summaryrefslogtreecommitdiff
path: root/aws.go
AgeCommit message (Collapse)Author
2021-07-12Add test for upAndQueue functionNick White
This involved adding a test queue, so it can be run safely without intefering with the pipeline.
2021-02-01Ensure DeleteObjects can handle over 1000 files to delete; fixes rmbook for ↵Nick White
large books
2021-01-26Make ListObjectsWithMeta generic again and create a specialised ↵Nick White
ListObjectWithMeta for single file listing, so we can still be as fast, but do not have a misleading api
2021-01-26Speed up lspipeline by making s3 requests concurrently and only processing ↵Nick White
single results from ListObjects requests
2021-01-26Stop limiting keys returned from listobjectprefixes' api usage; this speeds ↵Nick White
up the request markedly
2020-12-14Add rmbook toolNick White
2020-11-17Add trimqueue and logwholequeue utilities which can help deal with weird ↵Nick White
queue states
2020-10-20Improve logging by using Println, which ensures there is a space between ↵Nick White
arguments, even if all are strings
2020-05-19Add getandpurgequeue debugging toolNick White
2020-04-14Remove unused PreprocPattern, allow sensible defaults with aws setup, and ↵Nick White
add a little more documentation
2020-04-07Improve documentationNick White
2020-04-07Remove unused OCR queue (was superceded by the ocrpage queue some time ago)Nick White
2020-04-07gofmtNick White
2020-04-07Separate out cloud settings into a separate file; cloudsettings.goNick White
2020-03-31Disable autoshutdown by default for bookpipeline, and update to ami 0.11 ↵Nick White
(which reenables it for spot instances)
2020-03-31[bookpipeline] Improve logging outputNick White
2020-03-23Add Log() function to Pipeliner interfaceNick White
This simplifies things nicely from using conn.GetLogger().Println() to conn.Log()
2020-03-10Update image used for spotsNick White
2020-02-27Add documentation, license notices, and licenseNick White
2019-12-13Update StartInstance to point to the newest imageNick White
2019-12-11Use aws.go with mkpipeline too, plus fix one log.Fatal call in aws.go which ↵Nick White
should have been handled by caller
2019-12-05Remove (the generally empty) files in the case of a failed downloadNick White
2019-12-03Rewrite lspipeline book listing part to be much faster by taking advantage ↵Nick White
of the aws CommonPrefixes output
2019-11-19Add ocrpage queue for processing individual pagesNick White
This should be a good way to get around the ongoing heartbeat issue, as individual page jobs will never come close to a the 12 hour mark that can cause the bug. The OCR page processing is done and working now, still to do is to populate the queue (rather than the ocr queue) after preprocessing / wiping.
2019-11-12Add spotme command to start appropriate spot instancesNick White
2019-10-29Another attempt to fix the ongoing heartbeat issueNick White
This time wait up to 1 second between attempts, reduce long polling time significantly, and attempt for longer before giving up.
2019-10-28Try to fix heartbeat renew issue more fullyNick White
This approach first sets the remaining visibility timeout to zero. This should ensure that the message is available to re-find as soon as the process looks for it. Correspondingly the delay between checks is much shorter, as there shouldn't be a reason for much delay.
2019-10-17Adjust the heartbeat searching function to hopefully have better luck at ↵Nick White
finding it and not letting another process steal it.
2019-10-16Sort book list in lspipeline by modified dateNick White
2019-10-08Separate out bookpipeline from catch-all go.git repo, and rename to ↵Nick White
rescribe.xyz/bookpipeline The dependencies from the go.git repo will follow in due course.