Age | Commit message (Collapse) | Author | |
---|---|---|---|
2020-05-19 | Add getandpurgequeue debugging tool | Nick White | |
2020-04-14 | Remove unused PreprocPattern, allow sensible defaults with aws setup, and ↵ | Nick White | |
add a little more documentation | |||
2020-04-07 | Improve documentation | Nick White | |
2020-04-07 | Remove unused OCR queue (was superceded by the ocrpage queue some time ago) | Nick White | |
2020-04-07 | gofmt | Nick White | |
2020-04-07 | Separate out cloud settings into a separate file; cloudsettings.go | Nick White | |
2020-03-31 | Disable autoshutdown by default for bookpipeline, and update to ami 0.11 ↵ | Nick White | |
(which reenables it for spot instances) | |||
2020-03-31 | [bookpipeline] Improve logging output | Nick White | |
2020-03-23 | Add Log() function to Pipeliner interface | Nick White | |
This simplifies things nicely from using conn.GetLogger().Println() to conn.Log() | |||
2020-03-10 | Update image used for spots | Nick White | |
2020-02-27 | Add documentation, license notices, and license | Nick White | |
2019-12-13 | Update StartInstance to point to the newest image | Nick White | |
2019-12-11 | Use aws.go with mkpipeline too, plus fix one log.Fatal call in aws.go which ↵ | Nick White | |
should have been handled by caller | |||
2019-12-05 | Remove (the generally empty) files in the case of a failed download | Nick White | |
2019-12-03 | Rewrite lspipeline book listing part to be much faster by taking advantage ↵ | Nick White | |
of the aws CommonPrefixes output | |||
2019-11-19 | Add ocrpage queue for processing individual pages | Nick White | |
This should be a good way to get around the ongoing heartbeat issue, as individual page jobs will never come close to a the 12 hour mark that can cause the bug. The OCR page processing is done and working now, still to do is to populate the queue (rather than the ocr queue) after preprocessing / wiping. | |||
2019-11-12 | Add spotme command to start appropriate spot instances | Nick White | |
2019-10-29 | Another attempt to fix the ongoing heartbeat issue | Nick White | |
This time wait up to 1 second between attempts, reduce long polling time significantly, and attempt for longer before giving up. | |||
2019-10-28 | Try to fix heartbeat renew issue more fully | Nick White | |
This approach first sets the remaining visibility timeout to zero. This should ensure that the message is available to re-find as soon as the process looks for it. Correspondingly the delay between checks is much shorter, as there shouldn't be a reason for much delay. | |||
2019-10-17 | Adjust the heartbeat searching function to hopefully have better luck at ↵ | Nick White | |
finding it and not letting another process steal it. | |||
2019-10-16 | Sort book list in lspipeline by modified date | Nick White | |
2019-10-08 | Separate out bookpipeline from catch-all go.git repo, and rename to ↵ | Nick White | |
rescribe.xyz/bookpipeline The dependencies from the go.git repo will follow in due course. |