summaryrefslogtreecommitdiff
path: root/cmd/lspipeline/main.go
AgeCommit message (Collapse)Author
2021-02-22lspipeline: Rename to lspipeline-ng, and restore pre concurrency version to ↵Nick White
lspipeline as there are some hard to debug issues in concurrency version
2021-01-26Make ListObjectsWithMeta generic again and create a specialised ↵Nick White
ListObjectWithMeta for single file listing, so we can still be as fast, but do not have a misleading api
2021-01-26Improve lspipeline concurrency by removing WaitGroup stuffNick White
2021-01-26Speed up lspipeline by making s3 requests concurrently and only processing ↵Nick White
single results from ListObjects requests
2020-04-14Briefly document each of the commands in a godoc friendly way, and improve ↵Nick White
the cloudsettings documentation slightly
2020-04-07Remove unused OCR queue (was superceded by the ocrpage queue some time ago)Nick White
2020-04-07gofmtNick White
2020-03-09Add nobooks flag to lspipeline so it has a faster modeNick White
2020-02-27Add documentation, license notices, and licenseNick White
2019-12-16Fix usage message for getpipelinebook, and trim final slashes in lspipeline ↵Nick White
output
2019-12-03Rewrite lspipeline book listing part to be much faster by taking advantage ↵Nick White
of the aws CommonPrefixes output
2019-11-19Add ocrpage queue for processing individual pagesNick White
This should be a good way to get around the ongoing heartbeat issue, as individual page jobs will never come close to a the 12 hour mark that can cause the bug. The OCR page processing is done and working now, still to do is to populate the queue (rather than the ocr queue) after preprocessing / wiping.
2019-10-16Sort book list in lspipeline by modified dateNick White
2019-10-08Separate out bookpipeline from catch-all go.git repo, and rename to ↵Nick White
rescribe.xyz/bookpipeline The dependencies from the go.git repo will follow in due course.