summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2019-08-27Add annotations for pages with confidence below 70Nick White
2019-08-27Add basic graphing (still work to do, but basics are working)Nick White
2019-08-27Add basic analyse step, working but incompleteNick White
2019-08-23Expect source files to be .jpgNick White
2019-08-23Fix gaping bugs by using correct queues and downloadsNick White
This has involved refactoring to make the interface simpler, and just use the URLs / IDs for the necessary queues and storage locations, rather than wrap these in functions.
2019-08-22Generalise preprocessing and ocring to reuse common codeNick White
2019-08-22Switch to using flag to process command line, and allow different training ↵Nick White
to be passed
2019-08-22gofmtNick White
2019-08-22Update usage string, and commentsNick White
2019-08-22Improve timing of queue checksNick White
Now each queue is checked every 3 minutes, though the channel for each queue check request won't be rechecked until any previous job is completed.
2019-08-22Fix process finishing by closing dl channelNick White
2019-08-20Handle errors properly with goroutinesNick White
2019-08-20Handle errors correctly in main parts of programNick White
2019-08-20Substantially improve problematic object listing part of APINick White
Switch to regular non-concurrent stuff, concurrency is better handled by the main program anyway. Now we handle errors properly, and things are way simpler.
2019-08-20Add basic OCR support, and reorganise codeNick White
The previously committed thing didn't work, as listobjects was sending to a channel synchronously, so it was never being received. The current API isn't great, mixing synchronous and non-synchronous things, not handling errors consistently, and generally is over complicated. That will be fixed soon.
2019-08-20Split aws implementation from main.go in pipelinepreprocessNick White
2019-08-20Export qmsg typeNick White
2019-08-19Fix pipelinepreprocess segfaultsNick White
These were caused by using non-pointer methods, which meant that the values set in Init() were not saved.
2019-08-19Work in progress rearchitecture to use interfaces; currently pointers are ↵Nick White
screwy causing segfaults
2019-08-13Various improvements to pipelinepreprocessNick White
- Ensure temporary directory already being present isn't an issue - Remove temporary directory when done with it - Ensure any already preprocessed files aren't preprocessed themselves (this could happen in the case of a run stopping half way through)
2019-08-13Correct typo in bucket name for pipelinepreprocess; tested and seems to ↵Nick White
work, remarkably
2019-08-13Add bonus verbose log pointsNick White
2019-08-13Add booktopipeline tool (only lightly tested)Nick White
2019-08-13Reduce SQS WaitTime to something in-spec, and add bonus verbose log pointsNick White
2019-08-13Switch ksizes to use by preprocmultiNick White
2019-08-13Add basic verbose logging capabilities to pipelinepreprocessNick White
2019-07-25Add first draft of pipelinepreprocess - completely untested, will contain bugsNick White
2019-07-19rename setupawspipeline to mkpipelineNick White
2019-07-19rename pipelineaws to setupawspipelineNick White
2019-07-19Add aws pipeline setupNick White
2019-06-25Remove 0.6 binarisation threshold option from preprocmultiNick White
2019-06-25Experimentally adjust wipe threshold according to binarisation levelNick White
2019-06-11Name hocrs as pdfimages does, and preserve entities for hocrNick White
2019-06-11Add basic utility to turn an eebo xml into a set of hocr files (for hocr2pdf)Nick White
2019-06-03Add option to disable wiping for preproc and preprocmultiNick White
2019-06-03Add -m option to wipe to set minimum content area for wipe to proceedNick White
If content is very light or sparse it may be better to not wipe at all than wipe almost all of the content leaving a small strip. This is done now by aborting the wipe if the detected content takes up less than the minimum % of the page (default is 30%).
2019-05-15Return an error if page average calculation cant be done with hocrNick White
2019-05-14Rewrite pgconf to be more accurate by measuring average word confidence ↵Nick White
rather than average line confidence
2019-05-14pgconf: Don't print NaN if a page has no lines, and show the percentage, ↵Nick White
rather than float, for easier comparison
2019-05-14Add pgconf tool that prints the overall confidence for a whole page of hocrNick White
2019-05-14Basic cleanup of preprocmultiNick White
2019-05-14gofmtNick White
2019-05-14Add preprocmulti tool, that outputs multiple binarisation options quicklyNick White
2019-05-13Add preproc command, that binarises and preprocesses togetherNick White
Surprisingly opening an image takes a significant amount of the total processing time, so this actually saves quite a bit of time in the grand scheme of things.
2019-05-13Define flags in each test, so they arent erroneously picked up and used by ↵Nick White
cmds as they were defined in global package space
2019-05-13Use general integralimg functions for wipe functionsNick White
2019-05-13Add -slow flag to test to skip slow tests by defaultNick White
2019-05-13Reorganise image manipulation to separate integral image partsNick White
Also unify everything else under preproc/ Note that the UsefulImg interface should be used by the main functions, to simplify things, but this hasn't been done yet.
2019-05-13Start switching preproc to use interfaces moreNick White
2019-05-13Rename cleanup to wipe, and only export main functionNick White
generated by cgit v1.2.1-24-ge1ad (git 2.21.0) at 2024-11-23 04:05:46 +0000