summaryrefslogtreecommitdiff
path: root/bookpipeline/aws.go
AgeCommit message (Collapse)Author
2019-10-08Remove parts that have been moved elsewhere, and rename to rescribe.xyz/utilsNick White
bookpipeline is now at rescribe.xyz/bookpipeline preproc is now at rescribe.xyz/preproc integralimg is now at rescribe.xyz/preproc/integralimg
2019-10-02Add wipeonly queue and functionalityNick White
This is useful for prebinarised images, which don't need full preprocessing, but do require wiping, albeit with a more conservative threshold.
2019-09-24Rewrite GetInstanceDetails so page function is separateNick White
2019-09-24Move ec2 stuff out of lspipeline and into aws.goNick White
2019-09-23gofmtNick White
2019-09-23Move the sqs stuff out to aws.goNick White
2019-09-16Be more careful to try to grab the message after a heartbeat failure more ↵Nick White
quickly Rather than waiting for the whole length of a visibility timeout, in which time another process may grab the message, instead wait a short amount of time, each time the message is searched for. Also add a bit more logging.
2019-09-14Ensure enough time has elapsed before looking for the message to reget in ↵Nick White
the case of heartbeat running out
2019-09-11Work around the SQS limit of 12 hours of visibility timeoutNick White
This is done by checking for the error that is emitted in such a case, and if it's found trying several times to find the message back in the queue, and returning the message with an updated handle back to the caller to use in the future.
2019-09-05gofmtNick White
2019-09-04Rewrite heartbeat so errors during it will be reported, and the aws api ↵Nick White
doesn't rely on channels
2019-08-28Split out bookpipeline to cmd/Nick White
2019-08-23Fix gaping bugs by using correct queues and downloadsNick White
This has involved refactoring to make the interface simpler, and just use the URLs / IDs for the necessary queues and storage locations, rather than wrap these in functions.
2019-08-22gofmtNick White
2019-08-22Update usage string, and commentsNick White