bookpipeline - Tools to process books in a cloud based pipeline system

Age	Commit message (Collapse)	Author
2021-07-12	Add test for upAndQueue function	Nick White
	This involved adding a test queue, so it can be run safely without intefering with the pipeline.
2021-02-01	Ensure DeleteObjects can handle over 1000 files to delete; fixes rmbook for ↵	Nick White
	large books
2021-01-26	Make ListObjectsWithMeta generic again and create a specialised ↵	Nick White
	ListObjectWithMeta for single file listing, so we can still be as fast, but do not have a misleading api
2021-01-26	Speed up lspipeline by making s3 requests concurrently and only processing ↵	Nick White
	single results from ListObjects requests
2021-01-26	Stop limiting keys returned from listobjectprefixes' api usage; this speeds ↵	Nick White
	up the request markedly
2020-12-14	Add rmbook tool	Nick White

2020-11-17	Add trimqueue and logwholequeue utilities which can help deal with weird ↵	Nick White
	queue states
2020-10-20	Improve logging by using Println, which ensures there is a space between ↵	Nick White
	arguments, even if all are strings
2020-05-19	Add getandpurgequeue debugging tool	Nick White

2020-04-14	Remove unused PreprocPattern, allow sensible defaults with aws setup, and ↵	Nick White
	add a little more documentation
2020-04-07	Improve documentation	Nick White

2020-04-07	Remove unused OCR queue (was superceded by the ocrpage queue some time ago)	Nick White

2020-04-07	gofmt	Nick White

2020-04-07	Separate out cloud settings into a separate file; cloudsettings.go	Nick White

2020-03-31	Disable autoshutdown by default for bookpipeline, and update to ami 0.11 ↵	Nick White
	(which reenables it for spot instances)
2020-03-31	[bookpipeline] Improve logging output	Nick White

2020-03-23	Add Log() function to Pipeliner interface	Nick White
	This simplifies things nicely from using conn.GetLogger().Println() to conn.Log()
2020-03-10	Update image used for spots	Nick White

2020-02-27	Add documentation, license notices, and license	Nick White

2019-12-13	Update StartInstance to point to the newest image	Nick White

2019-12-11	Use aws.go with mkpipeline too, plus fix one log.Fatal call in aws.go which ↵	Nick White
	should have been handled by caller
2019-12-05	Remove (the generally empty) files in the case of a failed download	Nick White

2019-12-03	Rewrite lspipeline book listing part to be much faster by taking advantage ↵	Nick White
	of the aws CommonPrefixes output
2019-11-19	Add ocrpage queue for processing individual pages	Nick White
	This should be a good way to get around the ongoing heartbeat issue, as individual page jobs will never come close to a the 12 hour mark that can cause the bug. The OCR page processing is done and working now, still to do is to populate the queue (rather than the ocr queue) after preprocessing / wiping.
2019-11-12	Add spotme command to start appropriate spot instances	Nick White

2019-10-29	Another attempt to fix the ongoing heartbeat issue	Nick White
	This time wait up to 1 second between attempts, reduce long polling time significantly, and attempt for longer before giving up.
2019-10-28	Try to fix heartbeat renew issue more fully	Nick White
	This approach first sets the remaining visibility timeout to zero. This should ensure that the message is available to re-find as soon as the process looks for it. Correspondingly the delay between checks is much shorter, as there shouldn't be a reason for much delay.
2019-10-17	Adjust the heartbeat searching function to hopefully have better luck at ↵	Nick White
	finding it and not letting another process steal it.
2019-10-16	Sort book list in lspipeline by modified date	Nick White

2019-10-08	Separate out bookpipeline from catch-all go.git repo, and rename to ↵	Nick White
	rescribe.xyz/bookpipeline The dependencies from the go.git repo will follow in due course.