bookpipeline/internal/pipeline, branch v0.3.3 Tools to process books in a cloud based pipeline system [rescribe] Allow saving of results to somewhere other than a directory named after the book being processed 2020-12-07T16:53:58+00:00 Nick White git@njw.name 2020-12-07T16:53:58+00:00 068ad0b666705a49ab22d7b48cd6a7d67b37f234 






Don't upload binarised pdf twice needlessly
2020-12-03T15:16:14+00:00

Nick White
git@njw.name

2020-12-03T15:13:22+00:00

38dbdd0b21fb363e3f63fd3ea50272975e98eb77

This can also result in the file being uploaded twice simultaneously,
as up() is running in a separate goroutine. This can cause failures
on Windows as the file is attempted to be removed by one upload
process while being open to upload by the other process. Probably it
could also fail if the process completed by one (so the file was
deleted) before being started by the other.





This can also result in the file being uploaded twice simultaneously,
as up() is running in a separate goroutine. This can cause failures
on Windows as the file is attempted to be removed by one upload
process while being open to upload by the other process. Probably it
could also fail if the process completed by one (so the file was
deleted) before being started by the other.







Some changes to ensure the pipeline works correctly on Windows
2020-11-16T17:43:27+00:00

Nick White
git@njw.name

2020-11-16T17:43:27+00:00

cfbb3481368714adcd734906d8a460b873551c90

There were a couple of places where a file was uploaded while still open,
which resulted in an attempt to remove it, which causes an error from
Windows.

The allOCRed function also included an assumption that the path separator
would be a /, which is always correct for AWS, and correct for local on
Linux and OSX, but not for local Windows. Fixed by leaving the separator
well alone.

Also, the local connection was not stripping leading \, like it did /,
which caused an issue with Windows local.

Windows local is now tested and working, at least through wine.





There were a couple of places where a file was uploaded while still open,
which resulted in an attempt to remove it, which causes an error from
Windows.

The allOCRed function also included an assumption that the path separator
would be a /, which is always correct for AWS, and correct for local on
Linux and OSX, but not for local Windows. Fixed by leaving the separator
well alone.

Also, the local connection was not stripping leading \, like it did /,
which caused an issue with Windows local.

Windows local is now tested and working, at least through wine.







gofmt
2020-11-10T12:30:15+00:00

Nick White
git@njw.name

2020-11-10T12:30:15+00:00

33f1726a4c9f8013dcde39e644281059d9766bc4












[rescribe] Enable custom paths to tesseract command to be set (also improve some error output)
2020-11-10T12:28:50+00:00

Nick White
git@njw.name

2020-11-10T12:28:50+00:00

ad7aaf490e78e969bb5495dfda06a33d2a176aec












[getpipelinebook] Rewrite to use internal package functions
2020-11-10T10:41:15+00:00

Nick White
git@njw.name

2020-11-10T10:41:15+00:00

198f8215f8dd0460608abcd03fa49451462c9d11












Switch Preprocess() to take the thresholds to use, and have rescribe tool only use 0.1,0.2,0.3
2020-11-09T18:55:36+00:00

Nick White
git@njw.name

2020-11-09T18:55:36+00:00

f19df9e8c1213a49c426caefd2fadc711f5faf11












[rescribe] Local only combo tool basically now working. Testing is still minimal.
2020-11-09T18:54:50+00:00

Nick White
git@njw.name

2020-11-09T18:29:56+00:00

a1de8862a091f9584220db40671a0d43346c4519












[rescribe] work in progress at a self-contained local pipeline processor, called rescribe
2020-11-09T17:33:52+00:00

Nick White
git@njw.name

2020-11-09T17:33:52+00:00

fc6becf5ed98e9c0815532fd76639c15eb481ed1












[bookpipeline] Split most functionality out to package internal/pipeline
2020-11-09T16:46:43+00:00

Nick White
git@njw.name

2020-11-09T16:46:43+00:00

4c7cdeb5646e84af3f15d4a7cd48f64d8086a6b9

No functionality changes, but this should make it easier to make custom
builds using the pipeline in slightly different ways.





No functionality changes, but this should make it easier to make custom
builds using the pipeline in slightly different ways.