MobileRead Forums - View Single Post - Smartphone Book Scanning: What's Still Missing for PDF and EPUB?

notimp · Yesterday, 05:36 PM

On size reduction:

epub is a zipped html, so no.

Image pdf, yes, many ways - but you lose information every step of the way. (There is one approach where you ID text vs images and use compressions that are best suited for either.

And there is the approach you saw in the linked ccc.de video above where you ID similar tokens and then remove all but one on most pages and just copy the remaining one into position several times, which leads to all sorts of issues (as also seen in the ccc.de video) - especially for people trying to get OCR able images out of the pdf in the future.

In short, yes - PDF has several varieties of compression just to meet different needs. Especially Adobes lossy compressions can save a bunch.

Most archivists stay away from them though - because again you lose information every step of the way.

Also NEVER (not even while whitening out shadows), reduce the image quality to just B/W. greyscale is almost all of the edge formation information of any letter. We want color or at least many levels of greyscale for OCR. This is more important than dpi. (300 is fine for most OCR purposes (images might need more) epub is largely authored with 150ppi in mind).

Also, all I've written is from a western languages (German, English) viewpoint.

Asian languages need better OCR and at least epub 3.0 - which is also why their VLM LLM models usually aim for better OCR quality and as a result are the better ones at least in the open weights field.

edit: On headings and chapter structure OCR LLMs on smartphone images are not good enough yet. But its easily fixed in a manual pass - because they most certainly will have different formating to the base text and are easily catchable. Sigil has all the tools to do that quickly already. (And page/chapter splitting and TOC creation in the ePub.) On a smartphone that toolset doesnt exist, and would be harder to create in the "touch first" paradigm. But then -- you'd mostly aim at folks tackling one chapter at a time to gain a TTS read along experience anyhow (epub as a byproduct you give them) -- and also should do this for legal reasons.

(Format switching on an entire book for personal use! Its so easy, buy my app - is something to talk about with your lawyer first.)

Yesterday, 05:36 PM	#3
notimp Addict Posts: 256 Karma: 892441 Join Date: Jul 2010 Device: K2i	On size reduction: epub is a zipped html, so no. Image pdf, yes, many ways - but you lose information every step of the way. (There is one approach where you ID text vs images and use compressions that are best suited for either. And there is the approach you saw in the linked ccc.de video above where you ID similar tokens and then remove all but one on most pages and just copy the remaining one into position several times, which leads to all sorts of issues (as also seen in the ccc.de video) - especially for people trying to get OCR able images out of the pdf in the future. In short, yes - PDF has several varieties of compression just to meet different needs. Especially Adobes lossy compressions can save a bunch. Most archivists stay away from them though - because again you lose information every step of the way. Also NEVER (not even while whitening out shadows), reduce the image quality to just B/W. greyscale is almost all of the edge formation information of any letter. We want color or at least many levels of greyscale for OCR. This is more important than dpi. (300 is fine for most OCR purposes (images might need more) epub is largely authored with 150ppi in mind). Also, all I've written is from a western languages (German, English) viewpoint. Asian languages need better OCR and at least epub 3.0 - which is also why their VLM LLM models usually aim for better OCR quality and as a result are the better ones at least in the open weights field. edit: On headings and chapter structure OCR LLMs on smartphone images are not good enough yet. But its easily fixed in a manual pass - because they most certainly will have different formating to the base text and are easily catchable. Sigil has all the tools to do that quickly already. (And page/chapter splitting and TOC creation in the ePub.) On a smartphone that toolset doesnt exist, and would be harder to create in the "touch first" paradigm. But then -- you'd mostly aim at folks tackling one chapter at a time to gain a TTS read along experience anyhow (epub as a byproduct you give them) -- and also should do this for legal reasons. (Format switching on an entire book for personal use! Its so easy, buy my app - is something to talk about with your lawyer first.) Last edited by notimp; Today at 03:56 AM.