bookpipeline/internal/pipeline, branch v1.0.2 Tools to process books in a cloud based pipeline system Only generate full-size PDF if requested 2022-03-21T13:51:51+00:00 Nick White git@njw.name 2022-03-21T13:51:51+00:00 76d91ea8f65c6ad52efb24ac2c94b22c2908bc5c This avoids the issue that large PDFs require a lot of RAM, so there are chances of running out of memory. Plus it's a waste of space and time. 
This avoids the issue that large PDFs require a lot of RAM, so there
are chances of running out of memory. Plus it's a waste of space and
time.
Separate out fullsize pdf creation from colour pdf creation, so less memory is needed 2022-03-11T17:34:48+00:00 Nick White git@njw.name 2022-03-11T17:34:48+00:00 af8650c074bc111200b132b0918d44cacd423b6e 






Add initial support for full-size PDF generation
2022-03-11T13:36:59+00:00

Nick White
git@njw.name

2022-03-11T13:36:59+00:00

9d1382b69700129a66541d786ba3b784eda56e36

Some issues:
1) The PDF generation stores every page in memory while it constructs it. That means that
there's a higher chance of failure due to running out of memory with these. There's no
getting around this except by improving the PDF generation library, which is not easy.

2) Currently I've just changed the pipeline to always generate these full size PDFs, and
then the rescribe tool will just delete them if they weren't requested. This is bad in
particular because of point 1, and would probably cause issues of failures in the server
pipeline as a result

Therefore the plan is to add a tag to queue messages so that full size generation can be
selectively enabled.

Also, it should be split from the loop with colour pdf generation, as holding them both in RAM at
the same time is unnecessary.





Some issues:
1) The PDF generation stores every page in memory while it constructs it. That means that
there's a higher chance of failure due to running out of memory with these. There's no
getting around this except by improving the PDF generation library, which is not easy.

2) Currently I've just changed the pipeline to always generate these full size PDFs, and
then the rescribe tool will just delete them if they weren't requested. This is bad in
particular because of point 1, and would probably cause issues of failures in the server
pipeline as a result

Therefore the plan is to add a tag to queue messages so that full size generation can be
selectively enabled.

Also, it should be split from the loop with colour pdf generation, as holding them both in RAM at
the same time is unnecessary.







adjusted file renaming to make suffixes of png and jpg files lowercase and change jpeg to jpg
2022-02-28T16:41:30+00:00

Antonia Rescribe
antonia@rescribe.xyz

2022-02-28T15:41:17+00:00

787074fda5c9e4c7c959e20dd8ffe2d39f248a14












Add PreNoWipe queue, that just does binarisation but no wiping
2022-02-28T16:17:35+00:00

Nick White
git@njw.name

2022-02-28T16:17:35+00:00

21d49b546a27de6c53d8fe7d1a68d5a3b5506c93












Ensure that no new console windows are opened on Windows when executing Tesseract
2022-02-21T12:33:48+00:00

Nick White
git@njw.name

2022-02-21T12:33:48+00:00

69ac99da02988ad2ed675570ccaa5ff7777f0279












pipeline: Fail if no images are present
2022-01-31T17:09:16+00:00

Nick White
git@njw.name

2022-01-31T17:09:16+00:00

4fae7d93e0c07d5f6fc4c33db389c60d95276d01












Make pipeline context-aware, so the rescribe tool can cancel jobs
2022-01-31T14:11:21+00:00

Nick White
git@njw.name

2022-01-31T14:11:21+00:00

550752fa2ab493fb6d10aa9d963fc45996c0d100












internal/pipeline: if a graph cannot be created, don't leave an empty graph.png file, and allow failure to download that as it won't be created in the case of a 1 page book, which is fine
2022-01-17T13:10:33+00:00

Nick White
git@njw.name

2022-01-17T13:10:33+00:00

cbd4adcdf3b27fc157df526fcdcbac8d6b74bb81












internal/pipeline: Have DownloadPdfs() try to download all PDFs, but only return an error if none downloaded, as there are times when the colour PDF will not exist, which is fine
2022-01-10T16:15:52+00:00

Nick White
git@njw.name

2022-01-10T16:15:52+00:00

436551c5e6f9d96f82fdf31e01e422b2a937b6ee