bookpipeline/cmd/booktopipeline, branch v1.0.2 Tools to process books in a cloud based pipeline system Add PreNoWipe queue, that just does binarisation but no wiping 2022-02-28T16:17:35+00:00 Nick White git@njw.name 2022-02-28T16:17:35+00:00 21d49b546a27de6c53d8fe7d1a68d5a3b5506c93 






Make pipeline context-aware, so the rescribe tool can cancel jobs
2022-01-31T14:11:21+00:00

Nick White
git@njw.name

2022-01-31T14:11:21+00:00

550752fa2ab493fb6d10aa9d963fc45996c0d100












pipeline: use regular storage for tests, rather than a separate one
2021-08-17T12:18:04+00:00

Nick White
git@njw.name

2021-08-17T12:18:04+00:00

48c68cfa7f18992b26765c7b67c52c11687ad74a












internal/pipeline: Add test (incomplete but working) for UploadImages
2021-08-02T09:45:53+00:00

Nick White
git@njw.name

2021-08-02T09:45:53+00:00

4b7a4bd1b87d1dbba283e577aa03ce7e390d85d8












[booktopipeline] Add a check to disallow adding a book that already exists
2020-11-24T12:40:54+00:00

Nick White
git@njw.name

2020-11-24T12:40:54+00:00

0d914a5de3f8169d41df4fcff1ee4aea6d01afbe

This is important as if a book is added which has already been done,
then an analyse job will be added every time a page is OCRed, which
will clog up the pipeline with unnecessary work. Also if a book was
added with the same name but differently named files, or a different
number of pages, the results would almost certainly not be as
intended.

In the case of a book really wanting to be added with a particular
name, either the original directory can be removed on S3, or "v2"
or similar can be appended to the book name before calling
booktopipeline.





This is important as if a book is added which has already been done,
then an analyse job will be added every time a page is OCRed, which
will clog up the pipeline with unnecessary work. Also if a book was
added with the same name but differently named files, or a different
number of pages, the results would almost certainly not be as
intended.

In the case of a book really wanting to be added with a particular
name, either the original directory can be removed on S3, or "v2"
or similar can be appended to the book name before calling
booktopipeline.







Switch booktopipeline to use internal pipeline functions
2020-11-10T10:05:16+00:00

Nick White
git@njw.name

2020-11-10T10:05:16+00:00

c48630f590fe2c877e899948f1bf88458d3fd813












Add a couple of things that should not be forgotten
2020-11-09T18:57:21+00:00

Nick White
git@njw.name

2020-11-09T18:57:21+00:00

2c040f73ce7bbba480c441a0433fc8b4d6449254












Improve logging by using Println, which ensures there is a space between arguments, even if all are strings
2020-10-20T17:12:36+00:00

Nick White
git@njw.name

2020-10-20T17:12:36+00:00

644abdb177331107696b2caf3f0de8ac76a74702












[booktopipeline] Check that all images are valid before adding to pipeline
2020-09-22T15:45:28+00:00

Nick White
git@njw.name

2020-09-22T15:45:28+00:00

217a766647da7fa56289c0146705fec6f8cfea05












Fix confusing usage message for booktopipeline
2020-09-01T16:39:21+00:00

Nick White
git@njw.name

2020-09-01T16:39:21+00:00

14d2fcf4bd293b45d50bf34c568040fe9e7082f7