summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2019-10-29Another attempt to fix the ongoing heartbeat issueNick White
This time wait up to 1 second between attempts, reduce long polling time significantly, and attempt for longer before giving up.
2019-10-28Try to fix heartbeat renew issue more fullyNick White
This approach first sets the remaining visibility timeout to zero. This should ensure that the message is available to re-find as soon as the process looks for it. Correspondingly the delay between checks is much shorter, as there shouldn't be a reason for much delay.
2019-10-23getpipelinebook: default to downloading corresponding page images, and add ↵Nick White
option to download the original page images too
2019-10-23Manually calculate yticks, so they fall on reasonable numbersNick White
2019-10-23Add more annotations to graph; anything outside of the 80% "normal" band ↵Nick White
gets an annotation now, and that band is labelled
2019-10-17Adjust the heartbeat searching function to hopefully have better luck at ↵Nick White
finding it and not letting another process steal it.
2019-10-16Rewrite booktopipeline to use bookpipeline aws interfaceNick White
2019-10-16Sort book list in lspipeline by modified dateNick White
2019-10-16Ensure booktopipeline complains if given too many argumentsNick White
2019-10-16Another attempted fix to "too many open files" issueNick White
2019-10-16Ensure files are promptly closed by booktopipelineNick White
2019-10-11Ensure graph produces output by falling back on generic page numbers if none ↵Nick White
can be determined
2019-10-09Make confgraph and graph in general more resilient to bad inputNick White
2019-10-09Match prebinarised presegmented output from ocropus in wipepattern (named ↵Nick White
like "010001.bin.png")
2019-10-08Update paths of other rescribe importsNick White
2019-10-08Separate out bookpipeline from catch-all go.git repo, and rename to ↵Nick White
rescribe.xyz/bookpipeline The dependencies from the go.git repo will follow in due course.
2019-10-07Ensure wipe pipeline uses the expected png filesNick White
2019-10-02Improve usage notice for booktopipelineNick White
2019-10-02Add -prebinarised flag to booktopipelineNick White
2019-10-02gofmtNick White
2019-10-02Add wipeonly queue and functionalityNick White
This is useful for prebinarised images, which don't need full preprocessing, but do require wiping, albeit with a more conservative threshold.
2019-09-27Improve wiping procedure to work better with 2 column layoutsNick White
2019-09-27Fix crash bug when graph was used on source with less than 10 pagesNick White
2019-09-27One more update of graph.go to correspond to new go-chart, and improve usage ↵Nick White
for wipe
2019-09-27Hardcode to ignore "workhorse" from logsNick White
2019-09-27Update usage of go-chart to correspond to latest version of libraryNick White
2019-09-24gofmtNick White
2019-09-24Improve ssh logs; ensure only fully operational servers are tried, and ↵Nick White
ensure connections to new ips not in known_hosts still succeed
2019-09-24Do ssh log collection concurrentlyNick White
2019-09-24Get ssh logs from all running serversNick White
2019-09-24Add list of books done and in progress to lspipelineNick White
2019-09-24Rewrite GetInstanceDetails so page function is separateNick White
2019-09-24Move ec2 stuff out of lspipeline and into aws.goNick White
2019-09-23gofmtNick White
2019-09-23Move the sqs stuff out to aws.goNick White
2019-09-19Add queue listing to lspipelineNick White
2019-09-19Switch to using a goroutine for ec2 instance info, so can do all aws ↵Nick White
requests concurrently in due course
2019-09-18Add start of lspipelineNick White
2019-09-17gofmtNick White
2019-09-16Be more careful to try to grab the message after a heartbeat failure more ↵Nick White
quickly Rather than waiting for the whole length of a visibility timeout, in which time another process may grab the message, instead wait a short amount of time, each time the message is searched for. Also add a bit more logging.
2019-09-14Ensure enough time has elapsed before looking for the message to reget in ↵Nick White
the case of heartbeat running out
2019-09-12Don't prefix date/time to logs, as logger will store that anywayNick White
2019-09-11Increase size of graph to 4kNick White
2019-09-11Fix bug with graph that prevented the ticks from being correct, thus ruining ↵Nick White
the graph
2019-09-11Work around the SQS limit of 12 hours of visibility timeoutNick White
This is done by checking for the error that is emitted in such a case, and if it's found trying several times to find the message back in the queue, and returning the message with an updated handle back to the caller to use in the future.
2019-09-06Add flags to disable checking various queuesNick White
2019-09-05Handle no words found error in a better way so any page that is actually 0 ↵Nick White
confidence is recognised
2019-09-05Don't abort analysis if we encounter a hocr with no words, just skip itNick White
2019-09-05gofmtNick White
2019-09-05Update Pipeliner interface in getpipelinebook, and update some commentsNick White
generated by cgit v1.2.1-24-ge1ad (git 2.21.0) at 2024-05-18 16:05:37 +0000