bookpipeline - Tools to process books in a cloud based pipeline system

Age	Commit message (Collapse)	Author
2021-01-26	Make ListObjectsWithMeta generic again and create a specialised ↵	Nick White
	ListObjectWithMeta for single file listing, so we can still be as fast, but do not have a misleading api
2021-01-26	Improve lspipeline concurrency by removing WaitGroup stuff	Nick White

2021-01-26	Speed up lspipeline by making s3 requests concurrently and only processing ↵	Nick White
	single results from ListObjects requests
2020-04-14	Briefly document each of the commands in a godoc friendly way, and improve ↵	Nick White
	the cloudsettings documentation slightly
2020-04-07	Remove unused OCR queue (was superceded by the ocrpage queue some time ago)	Nick White

2020-04-07	gofmt	Nick White

2020-03-09	Add nobooks flag to lspipeline so it has a faster mode	Nick White

2020-02-27	Add documentation, license notices, and license	Nick White

2019-12-16	Fix usage message for getpipelinebook, and trim final slashes in lspipeline ↵	Nick White
	output
2019-12-03	Rewrite lspipeline book listing part to be much faster by taking advantage ↵	Nick White
	of the aws CommonPrefixes output
2019-11-19	Add ocrpage queue for processing individual pages	Nick White
	This should be a good way to get around the ongoing heartbeat issue, as individual page jobs will never come close to a the 12 hour mark that can cause the bug. The OCR page processing is done and working now, still to do is to populate the queue (rather than the ocr queue) after preprocessing / wiping.
2019-10-16	Sort book list in lspipeline by modified date	Nick White

2019-10-08	Separate out bookpipeline from catch-all go.git repo, and rename to ↵	Nick White
	rescribe.xyz/bookpipeline The dependencies from the go.git repo will follow in due course.