summaryrefslogtreecommitdiff
path: root/cmd
AgeCommit message (Collapse)Author
2021-02-22lspipeline: Rename to lspipeline-ng, and restore pre concurrency version to ↵Nick White
lspipeline as there are some hard to debug issues in concurrency version
2021-02-15getsamplepages: Add -prefix option, and use 'best' to get random page numbersNick White
The -prefix option is useful to us. Previously only a .jpg for page number 100 was retreived, which failed if the book had fewer (or unusually named) pages, and also didn't provide a corresponding .hocr at all (bug introduced with 48958d2). Using 'best', which is (effectively) randomly sorted, provides a guaranteed to exist page, and a random one at that.
2021-01-26Make ListObjectsWithMeta generic again and create a specialised ↵Nick White
ListObjectWithMeta for single file listing, so we can still be as fast, but do not have a misleading api
2021-01-26Improve lspipeline concurrency by removing WaitGroup stuffNick White
2021-01-26Speed up lspipeline by making s3 requests concurrently and only processing ↵Nick White
single results from ListObjects requests
2020-12-15[rmbook] Append / to end of bookname, to ensure e.g. "1" doesnt match all ↵Nick White
books starting with "1"
2020-12-15[rmbook] Add -dryrun flagNick White
2020-12-14Add rmbook toolNick White
2020-12-07[rescribe] Fix up *.hocr glob, which ensures that using a savedir that ↵v0.3.2Nick White
already has a hocr directory in it will work
2020-12-07[rescribe] Allow saving of results to somewhere other than a directory named ↵Nick White
after the book being processed
2020-12-03[rescribe] Fix portability issue where hocrs may not be correctly moved and ↵Nick White
txt-ified on windows
2020-11-30Merge branch 'master' of ssh://hammerhead/home/nick/rescribe/src/bookpipelineNick White
2020-11-30Add getstats toolNick White
2020-11-24[booktopipeline] Add a check to disallow adding a book that already existsNick White
This is important as if a book is added which has already been done, then an analyse job will be added every time a page is OCRed, which will clog up the pipeline with unnecessary work. Also if a book was added with the same name but differently named files, or a different number of pages, the results would almost certainly not be as intended. In the case of a book really wanting to be added with a particular name, either the original directory can be removed on S3, or "v2" or similar can be appended to the book name before calling booktopipeline.
2020-11-17Add trimqueue and logwholequeue utilities which can help deal with weird ↵Nick White
queue states
2020-11-17Remove _bin0.x from txt filenamesv0.3.0Nick White
2020-11-16[rescribe] Default to an appropriate tesscmd for WindowsNick White
2020-11-16[rescribe] Add txt output, only keep colour pdf, and reorganise files so ↵Nick White
they're more user-friendly
2020-11-16[rescribe] Mention in usage that things can be saved in a different directoryNick White
2020-11-10gofmtNick White
2020-11-10[rescribe] Enable custom paths to tesseract command to be set (also improve ↵Nick White
some error output)
2020-11-10[rescribe] Change -t to the path of the traineddata file, and set ↵Nick White
TESSDATA_PREFIX accordingly
2020-11-10[rescribe] Handle errors in processbook correctly, and improve console outputNick White
2020-11-10[getpipelinebook] Rewrite to use internal package functionsNick White
2020-11-10Switch booktopipeline to use internal pipeline functionsNick White
2020-11-09Add a couple of things that should not be forgottenNick White
2020-11-09Switch Preprocess() to take the thresholds to use, and have rescribe tool ↵separatelocalNick White
only use 0.1,0.2,0.3
2020-11-09[rescribe] Local only combo tool basically now working. Testing is still ↵Nick White
minimal.
2020-11-09[rescribe] work in progress at a self-contained local pipeline processor, ↵Nick White
called rescribe
2020-11-09[bookpipeline] Split most functionality out to package internal/pipelineNick White
No functionality changes, but this should make it easier to make custom builds using the pipeline in slightly different ways.
2020-11-09Add -autostop, so time to shutdown can be specified, and so the process can ↵Nick White
just be stopped after a period, rather than the whole computer shut down
2020-11-09[bookpipeline] Improve interface, particularly for local use, by disabling ↵Nick White
(failing) log saving, mail sending, and removing erroneous references to AWS
2020-11-09Set hocr config options directly rather than relying on 'hocr' config fileNick White
This ensures that bookpipeline will still work even if TESSDATA_PREFIX has been set to a directory without configs in it.
2020-10-20Improve logging by using Println, which ensures there is a space between ↵Nick White
arguments, even if all are strings
2020-10-20Add postprocess-bythresh cmdNick White
2020-09-22[booktopipeline] Check that all images are valid before adding to pipelineNick White
2020-09-15Abort and delete a failed wipeonly job, like we do with preprocessingNick White
There was no reason not to do this with wipeonly as well, and sure enough a single broken PNG image in a wipeonly task would cause the queue to exponentially fill as happened previously.
2020-09-01Fix confusing usage message for booktopipelineNick White
2020-08-24update getsamplepages to just get jpg pagesNick White
2020-08-19Add getsamplepagesNick White
2020-08-18Update preproc to v0.4.0 to enable vertical wipingNick White
2020-07-28Allow override of autodetected queues for booktopipelineNick White
2020-07-28Autodetect queue for booktopipeline based on file extensionAntonia Karaisl
2020-07-27Use os.Getenv() to find config dir, rather than rely on os.UserConfigDir(), ↵Nick White
as that isnt present on go1.11
2020-07-27Switch mail settings to an externally set fileNick White
2020-07-21[bookpipeline] If preprocessing fails, email us and remove the job from the ↵Nick White
queue This prevents the current situation where a failed preprocessing job is endlessly repeated, potentially spawning thousands of ocrpage jobs in its wake each time. Note that the email stuff works but requires putting secrets into .go files, so need to rewrite that to read from somewhere more sensible like a dotfile on the host.
2020-07-20Merge branch 'master' of https://git.rescribe.xyz/bookpipelineNick White
2020-07-20Update preproc to v0.1.4 to take advantage of vertical wiping parameters, ↵v0.2.5Nick White
and change WipeFile() to take advantage of them
2020-06-16[getallhocrs] Skip files which have already been downloadedNick White
2020-06-15Add getallhocrs toolNick White
generated by cgit v1.2.1-24-ge1ad (git 2.21.0) at 2024-11-09 06:08:09 +0000