summaryrefslogtreecommitdiff
path: root/aws.go
AgeCommit message (Collapse)Author
2021-08-17pipeline: use regular storage for tests, rather than a separate oneNick White
2021-08-02internal/pipeline: Add test (incomplete but working) for UploadImagesNick White
2021-07-13Fix up tests a bitNick White
2021-07-13gofmtNick White
2021-07-13aws: Only look up test queue id when asked for, as for most Init()s it won't ↵Nick White
be needed
2021-07-12Add test for upAndQueue functionNick White
This involved adding a test queue, so it can be run safely without intefering with the pipeline.
2021-02-01Ensure DeleteObjects can handle over 1000 files to delete; fixes rmbook for ↵Nick White
large books
2021-01-26Make ListObjectsWithMeta generic again and create a specialised ↵Nick White
ListObjectWithMeta for single file listing, so we can still be as fast, but do not have a misleading api
2021-01-26Speed up lspipeline by making s3 requests concurrently and only processing ↵Nick White
single results from ListObjects requests
2021-01-26Stop limiting keys returned from listobjectprefixes' api usage; this speeds ↵Nick White
up the request markedly
2020-12-14Add rmbook toolNick White
2020-11-17Add trimqueue and logwholequeue utilities which can help deal with weird ↵Nick White
queue states
2020-10-20Improve logging by using Println, which ensures there is a space between ↵Nick White
arguments, even if all are strings
2020-05-19Add getandpurgequeue debugging toolNick White
2020-04-14Remove unused PreprocPattern, allow sensible defaults with aws setup, and ↵Nick White
add a little more documentation
2020-04-07Improve documentationNick White
2020-04-07Remove unused OCR queue (was superceded by the ocrpage queue some time ago)Nick White
2020-04-07gofmtNick White
2020-04-07Separate out cloud settings into a separate file; cloudsettings.goNick White
2020-03-31Disable autoshutdown by default for bookpipeline, and update to ami 0.11 ↵Nick White
(which reenables it for spot instances)
2020-03-31[bookpipeline] Improve logging outputNick White
2020-03-23Add Log() function to Pipeliner interfaceNick White
This simplifies things nicely from using conn.GetLogger().Println() to conn.Log()
2020-03-10Update image used for spotsNick White
2020-02-27Add documentation, license notices, and licenseNick White
2019-12-13Update StartInstance to point to the newest imageNick White