bookpipeline, branch v0.3.3 Tools to process books in a cloud based pipeline system rescribe: change default training directory to trainings/ 2021-03-16T12:26:18+00:00 Nick White git@njw.name 2021-03-16T12:26:18+00:00 8ab178eaa23215c8f5ac41ca36f7863d56d06cf3 






lspipeline: Rename to lspipeline-ng, and restore pre concurrency version to lspipeline as there are some hard to debug issues in concurrency version
2021-02-22T16:02:55+00:00

Nick White
git@njw.name

2021-02-22T16:02:55+00:00

866aa409f8844ab2cb8d578672a703b4ddead30c












getsamplepages: Add -prefix option, and use 'best' to get random page numbers
2021-02-15T17:09:20+00:00

Nick White
git@njw.name

2021-02-15T17:09:20+00:00

44a027984044a55a8a483268ddf0b841e9f33e83

The -prefix option is useful to us.

Previously only a .jpg for page number 100 was retreived, which
failed if the book had fewer (or unusually named) pages, and also
didn't provide a corresponding .hocr at all (bug introduced with
48958d2). Using 'best', which is (effectively) randomly sorted,
provides a guaranteed to exist page, and a random one at that.





The -prefix option is useful to us.

Previously only a .jpg for page number 100 was retreived, which
failed if the book had fewer (or unusually named) pages, and also
didn't provide a corresponding .hocr at all (bug introduced with
48958d2). Using 'best', which is (effectively) randomly sorted,
provides a guaranteed to exist page, and a random one at that.







Merge branch 'master' of ssh://ssh.phx.nearlyfreespeech.net/home/public/bookpipeline
2021-02-05T17:15:51+00:00

Nick White
git@njw.name

2021-02-05T17:15:51+00:00

11470933e4fd379b4aefa4e2bab33662a72791c2












Update go-chart dependency
2021-02-05T17:09:34+00:00

Nick White
git@njw.name

2021-02-05T17:09:34+00:00

3e7da751b3ca917adb79674eac4ef2a3267e3984












Update AWS dependency to 1.37.1
2021-02-01T11:46:21+00:00

Nick White
git@njw.name

2021-02-01T11:46:21+00:00

a8c7481f0dc02bbda3b3a07091e9d61f6eb728b2












Ensure DeleteObjects can handle over 1000 files to delete; fixes rmbook for large books
2021-02-01T11:45:27+00:00

Nick White
git@njw.name

2021-02-01T11:45:27+00:00

16ea8034794ef030c969d586a7fc945bf4a2873a












Make ListObjectsWithMeta generic again and create a specialised ListObjectWithMeta for single file listing, so we can still be as fast, but do not have a misleading api
2021-01-26T14:56:10+00:00

Nick White
git@njw.name

2021-01-26T14:56:10+00:00

5c3cee66a90ce6ef87e125b3bf011a6903d38083












Improve lspipeline concurrency by removing WaitGroup stuff
2021-01-26T14:17:19+00:00

Nick White
git@njw.name

2021-01-26T14:17:19+00:00

54150b54cd06e3deba44e73b151070b74a4d8e76












Speed up lspipeline by making s3 requests concurrently and only processing single results from ListObjects requests
2021-01-26T13:50:02+00:00

Nick White
git@njw.name

2021-01-26T13:50:02+00:00

86cc5d6c921ac05e0d08f66b205b51e1f5adb938