Smartphone Book Scanning: What's Still Missing for PDF and EPUB?

[NurViewer]

Hello everyone!

I've been reading some interesting discussions here on MobileRead about digitizing printed books using smartphones.

In particular, I found the thread "New best practices tutorials" very informative:

https://www.mobileread.com/forums/sh...d.php?t=375385

The workflow described there — photographing book pages with a smartphone, processing the images with CamScanner, performing OCR using MinerU or Mistral OCR, and creating EPUB files — is impressive.

It made me think about an interesting question:

Smartphone scanning technology has become remarkably capable. But what are the biggest problems that remain when turning an entire printed book into a high-quality digital book?

Today, applications such as vFlat, CamScanner, and Adobe Scan already offer various combinations of automatic cropping, page flattening, dual-page scanning, image enhancement, and OCR.

However, producing a genuinely useful electronic book can still require several additional processing steps.

I'm particularly interested in the following areas.

1. Image correction after smartphone capture

Modern apps can correct perspective and curved pages, but how reliably do they handle difficult cases?

For example:

- Deep page curvature near the spine
- Fingers holding the book open
- Uneven lighting and shadows
- Thin paper with text showing through from the other side
- Illustrations extending across two facing pages

Which applications currently handle these problems best?

2. OCR accuracy versus document structure

Recognizing individual words is only part of the problem.

A useful digital book also needs to preserve:

- Paragraph boundaries and reading order
- Headings and chapter structure
- Italics and other text formatting
- Footnotes and references
- Tables, formulas, and illustrations
- Vertical Chinese and Japanese text

I've noticed that modern OCR and vision-language models can produce impressive text recognition results, but preserving the original document structure remains an interesting challenge.

3. Searchable PDF versus reflowable EPUB

A searchable PDF preserves the original appearance of the printed pages.

A reflowable EPUB can be much more comfortable on smaller e-readers, but converting complex layouts often introduces errors.

Would you prefer a workflow that produces both formats from the same scanned source?

4. Processing an entire book efficiently

Scanning a few pages is relatively easy.

But processing a book with 300–500 pages raises additional questions:

- Can page order and numbering be maintained automatically?
- Can missing or duplicated pages be detected?
- Can the final file size be reduced without damaging readability?
- Can OCR results be checked against the original images efficiently?
- How much manual correction is still necessary?

5. An integrated workflow versus multiple applications

This is perhaps the question I'm most interested in.

Many existing workflows involve several separate tools for image processing, OCR, PDF creation, EPUB conversion, and proofreading.

Would it be more useful to have these capabilities integrated into a single application, or do you prefer specialized tools for each stage?

I'd love to hear from members who regularly digitize printed books.

In your experience:

What is the single most frustrating or time-consuming part of turning smartphone photographs into a high-quality, searchable, readable e-book?

And have you found any software that handles most of the process particularly well?

Thanks for sharing your experiences!

[notimp]

An integrated workflow is possible and should even be possible on mobile within the next couple of years -

whats needed is:

1. Descewing, whitening out shadows, making text lines parallel and saving as a pdf for "easier one file handling" on mobile - in a way that is not tethered to a subscription, like the two solutions out there that people go for (Cam Scanner and vFlat scan).

2. An integrated workflow on mobile that ends at the "epub fully created stage". But that no one will use aside from nerds, so the integrated workflow on mobile would ideally stop at:

3. TTS in many languages done on at least a chapter of text of a book, with the user being presented with either a reading experience that shows text and does voice over creation on the fly - highlighting the words as they are spoken, and doing auto pageturn. But that needs the smartphone to be active on TTS all the time - or TTS that outputs .wav (you can even end at creating a m4b file, but users will likely only digitize one chapter at a time - so it would be many m4b files), then does .srt creation on the .wav (which was created by a TTS LLM) - the smallest optimized whisper models are fine/enough - and then dynamically can attach auto pageturns to time readings derived from the .srt. That would ultimately be the experience that more users would be interested in - especially in the smartphone space.

And that would be the end of a "fully integrated solution". (Give them epub (which has text reflow) as a byproduct.) Although with legal footnotes, because "format switching" while certainly legal for research and university work - all for personal use - might not be legal everywhere. So your app description would emphasize, that this is an app aimed at document digitzation and helping people to study/learn better - for legal reasons. Talk to your lawyer... If you are creating an integrated solution till the very end.

When you dont stop with epub, but go the kokoro route and offer audiobooks and read along experiences with automatic page turns -- issue: Some of the most widely used languages worldwide (German ) are not supported by stock Kokoro - and the projects to fold them in are quite scattered, and only support some - but maybe not all the best voices (Kokoro Martin stil remaining the best german one as of today).

Why still an issue - because of mobile device capabilities. The best open weights TTS model out there (Fish Audio Drama 3 as of now) needs the same hardware as Qwen3.8-Flash-Next on the professional side (= gpu server) to run at 1 hour per book speeds, and thats just not mass market.

So you are stuck with Kokoro until after the AI bubble pops - and if someone wants to invest time -- maybe looking at Ming-3.0-tts makes sense as well, but test voices they use on their hugginface page in german often have worse artefacting than Kokoro voices handcrafted by enthusiasts for that language.

Lets start with the main differenciation needed:

pdf and or epub.

They are two different audiences.

Pdf is office workers that dont know anything - but print all day. And desktop publishing houses.

ePub is basically .md (markdown) but prettier.

To address the needs of the pdf audience you need a serious amount of work put into pdf "cosmetics" and authoring. To create .md/.epub you do not.

(In our Smartphone photos to epub workflow pdf is mostly used as a carrier medium to not have to deal with 50-300 pages (image files), but with one to three files instead. This is different from serious .pdf creation.)

One small third group (out of contest) archivists are better off going for a format that captures everything. So pdfs or modern equivalents. Because many of them operate in a "we can throw out the original after" mode - and for that you want to capture as much as possible as accurately as possible (necessary lecture for that: https://media.ccc.de/v/31c3_-_6558_-..._david_kriesel watch this.)

So you first seperate per audience imho.

Or you spend a long time learning the state of the art processes of the DIY scanner movement. I'm approaching this from the "mostly interested in good epub quality" angle and thereby have blindspots in workflows on the pdf creation side of the scene. (Responsible disclosure. )

The stumbling block at the descewing and whitening and making lines of text parallel stage - is probably "good shadowcorrection" without cuting away the edges of fonts - which reduces OCR quality. So the "whitening out" part. The rest is math thats pretty well known by now - but that part still remains somewhat of an art (Cam Scanner is very good at it).

Also, concept wise - it is probably better if you retain double pages, and not cut them up at the ealiest possible stage - because at least humans understand the perspective artefacts more intuitively when they look at double pages (you could write logic that keeps track of those on single pages as well (as in, expect every other page to have similar artefacts)) - but think about it before you do.

Doublepages are a must for fast scanning with smartphones. Single pages are double the work.

Font shining through the back of a page causing artefacting issue - depends on how the LLM fails - but probably is easily detectible with a short "sanity" check using the 100 most common words in a language as a reference, or checking for "doubling" within a certain text range (as in AAAAAAAAAAAAAAAAAAAAAA should be quite easy to catch using an algo.. ). Because that should only appear on the back of pages that are otherwise empty. Non issue on pages that have text on it.

Aside from those "catastrophic fails" MinerU and MistralOCR have been very good with almost no additional work needed, if lighting was OK, and the image wasnt blurred through movement. (Using Cam Scanner as the image taking app so far.) There is a third visual language model based OCR engine worth mentioning... I think it is a bit more compute intensive, ... And I think its this one... https://huggingface.co/ATH-MaaS/OvisOCR3 worth the research.

Also there is a stumbling block here in terms of unified implementation, which might be the actual licensing those models can run under. As in - some of them dont just allow you to then create a commercial product around them and just sell it. Also this step (visual language model based OCR) needs GPU acceleration. (Otherwise it just takes too long, even with small LLMs that only need 5 GB of unified memory/VRAM) Which might be hard(er) to attain on mobile platforms. (The LLM world runs on python. Python GPU acceleration runs on pytorch -- and there is not an easy hook from there into the "I've created a unified app" world. This is work, but for solutions like MinerU processor wise, smarthphones are already there. (Current iPhones have everything thats needed. Current Android phones also.)) MinerU is a great fit, because it is fast and only needs 5GB of VRAM and is good enough for our purpose --

which is "predictive text paragraph interpretation" at the formerly scewed upper inner/outer parts of a (double) book page --

but afair, can not be freely resold if you integrate it into another app and then sell that because of licensing limitations (you might have to pay them). Ask your lawyer.

"Predictive text paragraph interpretation" is where you absolutely need them (/ a VLM based LLM model). Any wavey - slightly blurred line will throw off classic OCR like a mofo. The entire DIY OCR scene was made to create contraptions to have the text "ready for pdf" qualitywise -- the smartphone scans we can use for epub generation are not. Even after the best postprocessing.

epub focused folks need parallel lines (so OCR can draw rectangle targets), good de shadowing (whitening), without cutting into fonts, and "somewhat good descewing".

pdf focused folks need near perfection at this stage (scew yes, but controlled scew with predictable angles, so descewing can be perfect, and so on.)

From the epub creators perspective. Post processing needed after automation is minimal. Almost none. When using LLMs.

At this stage we can look at document structure. Its dictated by LLM model capability and the truth is, that italics are out. Structure recognition is better with better models. (Text, table, images, footnotes, ... good LLMs can deal with them well.)

Structure recognition was best in the past, when you let Finereader (not Visial language model based) recognize 10-20 different "font classes" per book, even when a book (scan at 300 dpi, using a flatbed scanner, so high quality, but no descewing (also none needed, because you press book against glass)) only had 5.

Then you moved that into Atlantis Word processor, and reduced the needed classes (it had "mark all text of a class" controls) manually. That way you moved almost all layout correctly out of the text, into the digital format and kept italics, and so on...

Not in the AI age anymore.

AI bros didnt optimize for it. They optimized for .md (markdown) compaitbility - which is - or which you can think about as "a certain complexity range". As in there is one main text class, and maybe 5 others, and of course tables, and yes subcaptions and so on --- but not 5 different main text classes, because they were needed for the layout of the book.

So they pressed the LLMs into that "form" during training - and this is what they will give you. So italics mostly is out (at least never reliably in). Some similar layouts get merged, but tables and images, and order recognition can be very good, depending on the LLM. Because they needed that to be perfect for high quality .md basically.

A point on "order recognition". It fails horribly if the LLM doesnt get what it expects. But is super fine, when the LLM gets what it expects. As in - the LLMs we are using are usually TINY models (fit in 5GB of VRAM with their entire context, which is huge for full books). So they are dumb as a rock if they find themselves in situations they werent trained on.

As in "double pages perfectly fine", but print 4 book pages onto on A4 (letter) sized sheet of pdf paper -- and they are having a drunken party evey 10th page or so - flipping order all over. (They dont catch and correct that. Which is interesting, because their predictive text recognition of blured out text on the edges often is better than human vision... So there they understand the probability of words in a paragraph derived from previous words on the book (Eliás with that accent on top gets recognized more often than Elias - even when the a is barely visible on one edge spot).

Can you automate failure correction? For the most part yes. (Finding a UI an a concept for a smartphone user to be able to do manual control post facto that is fast is some planing, and focus group testing, but it should be totally doable as well.)

Can you automate page checking. (Order, duplicates, missing pages.) Also yes. Although this one is harder - because telling user what you need in addition to what they gave you is more than most will be able to comprehend or be willing to provide... If not asked very simply and in a way that seems like no work to them..

In a workflow with an educated user keeping that step manual for the time being is not that much of an issue - as it flows together with checking for text errors on the beginning and the end of pages, where the scew artefacts are - so something you'd have to do manually anyhow, and page number removal (if one or two slipped the crop), and merging paragraphs at pagebreaks. So all of this can be one step in the process anyhow. The thing is - the text correction needed already is so "rare" after a model like MinerU, that you could almost automate all of this away. But it is also what you would want the user to manually be able to at least doublecheck in some form or fashion - if you aim for "responsible" and not "hey guys - my app is most easy for you". (You better aim at the latter if you want to sell your integrated solution.) So yes - you can almost skip this step entirely/automate it away. (Using current workflows already.)

And you'd have to do a "is a page missing" check almost entirely based on page numbers. Again, context continuity checking with LLMs, not with the small size we are using.

What about fingers and pagenumbers, do we need to cut them?

In theory no, in practice yes.

No - because most LLMs have been trained to ignore fingers no problem -- but when we get number into the .md that "could be page numbers or not", this becomes a huge issue later on. So cutting them off before the MAIN OCR step (again you'd need them for automatic recognition if LLM has all pages, so first OCR step at a certain sector of a page before main OCR step?) is almost necessary.

Also because folding the pages together later on depends in a decently not circumventable fashion on pagenumbers not having become part of the "rolling text". If you are doing it at the epub stage, you are basically folding together paragraphs (removing unremoved hyphens maybe also), with close to no idea if that number should be part of the text or not.

Some more advanced algo that looks for "number+1" and continuity not withstanding. (Which I never used.)

Also - very important for "paragraph integrity" -- in some book layouts you cant tell, if a "scene break" (so "time has passed" or the "scene changed") occured exactly from one page to another. So at the point of the page break.

It often either is not marked at all and only infered.

Especially in layouts where new paragraphs have no indents, but there is spacing between paragraphs.

More complex, but stay with me. There are book layouts that after such a break will have indent 0 on first paragraph lines after such a break, and 0.5 on every other paragraph. On pagebreak the first paragraph is always 0 indent - but you cant tell if it should be folded in with your 0.5 indent paragraphs or not - if you dont have the context. Because if a scene break is between pages -- there is no "larger empty space" that would indicate a scene break normally. Also because LLMs dont care so much about "larger empty space" and usually just give you one newline anyhow. (That layout getting lost in the process potentially.) But. I had workflows in the past (Finereader, not LLM based) that would account for that -- but it lead to the following.

The spacing between paragraphs is mostly lost to an LLM (so double spacing doesnt always result in double newlines).

And even if you get it perfectly (Finereader was very good at it) -- I worked with Finereader workflows in the past, that would Identify the first paragraph on a new page DIFFERENTLY. Reliably (so they had differnt structure, and you would apply a regex to make sure to makr those paragraphs as "first paragraphs" essentially). You then can set that to "indent = 0" and normal paragraphs to "indent = 0.5" just to notice that most of your indent = 0 paragraphs in the text were actual normal 0.5 even though they were special, because they were the first paragraph on a new page -- but the layout of the book just didnt differenciate them. So you can then correct them manually (by looking at their meaning within the paragraph and the paragraph before) - but it will take you ages, and you'll just hide them by making all of the paragraphs look the same in the end (see: https://github.com/hexgrad/kokoro/is...ent-6657235117 ), because its so much less work.

If you are a great chap, and can differenciate first paragraphs by magic (because the LLM prints two newlines differently, or leaves in an indication that allows you to capture them) maybe think about giving them different paragraph tags, but format them identically to normal text paragraphs (hide their specialness but keep the tags different).

So someone can later make the eBook perfect in an ePub editor, but no one else will know or care, or invest the time in that.

(For them it is easier to remove a special "first paragraph" than to add one in.)

The process actually goes:

1. Eliminate page numbers. (Now all text sections are just an endless list of paragraphs)

2. Do your "automatically merge paragraphs that dont end with a sentence end mark, with the next paragraph and put a normal SPACE between the last and the first word. Step.

3. Then see if you can still identify "first paragraphs on a page" paragraphs that remained, and make them a special html class.

4. Then do a manual check for hyphens "- " pass to see if auto dehyphening missed some. (The last word on a page can also be a hypenated one.. Fun.) This pass HAS to be made manually, because there is no way to identify words with hyphens that need to be pulled together from words with hyphens that are word-art-creations, automagically. Not even close.

Or ignore all of that, because the normal smartphone user will not care the least bit about any of that. An educated epub creator might care. If you are a consumer reading along in a TTS assisted ebook, you usually dont care either.

In practice in our use case - for most users this wont matter much. The .md to .epub script I linked in the new best practice tutorial also does de-textwurst processing. (Textwurst being my invented term for a part of book where auto dehypenation failed for the book text). Good one, that leaves in hyphens when its not sure.

And the general dehyphenation of the LLMs is very good at the curent stage - they just sometimes miss a paragraph or two -- and the .md to .epub script linked makes sure they dont look that out of the ordenary, when the LLM misses a paragraph.

LLMs folding together paragraphs in difficult layouts that should stay seperate prima facie, is something you wont solve through clever programming tricks. This is LLM training that they dont have to the extent to be perfect, not even the best ones. But in practice -- maybe 1 out of 100 users would notice, so this is even more of a non issues, depending on your target demographic. (We are talking about them not catching every new paragraph if a book is formated with no line indents and the same interline space throughout the book, so special hard to capture layout cases. With all other cases, OCR LLMs usually dont have issues.)

Double page images, should probably stay split in the epub paradigm, since most ereaders are single page anyway. If you want them to be merged, Manual step. Nothing that you can automate easily I think.

Why? Because image too small on eReader otherwise. And yes - Kindle tackles this with longpress on image, and you can get a rotated view that makes it bigger, and then click again and --- in practice no one uses it. And your grandmother wouldnt know that it exists.

If you are more on the publishing end of the spectrum and this is a requirement, then this is a manual step.

[notimp]

On size reduction:

epub is a zipped html, so no.

Image pdf, yes, many ways - but you lose information every step of the way. (There is one approach where you ID text vs images and use compressions that are best suited for either.

And there is the approach you saw in the linked ccc.de video above where you ID similar tokens and then remove all but one on most pages and just copy the remaining one into position several times, which leads to all sorts of issues (as also seen in the ccc.de video) - especially for people trying to get OCR able images out of the pdf in the future.

In short, yes - PDF has several varieties of compression just to meet different needs. Especially Adobes lossy compressions can save a bunch.

Most archivists stay away from them though - because again you lose information every step of the way.

Also NEVER (not even while whitening out shadows), reduce the image quality to just B/W. greyscale is almost all of the edge formation information of any letter. We want color or at least many levels of greyscale for OCR. This is more important than dpi. (300 is fine for most OCR purposes (images might need more) epub is largely authored with 150ppi in mind).

Also, all I've written is from a western languages (German, English) viewpoint.

Asian languages need better OCR and at least epub 3.0 - which is also why their VLM LLM models usually aim for better OCR quality and as a result are the better ones at least in the open weights field.

edit: On headings and chapter structure OCR LLMs on smartphone images are not good enough yet. But its easily fixed in a manual pass - because they most certainly will have different formating to the base text and are easily catchable. Sigil has all the tools to do that quickly already. (And page/chapter splitting and TOC creation in the ePub.) On a smartphone that toolset doesnt exist, and would be harder to create in the "touch first" paradigm. But then -- you'd mostly aim at folks tackling one chapter at a time to gain a TTS read along experience anyhow (epub as a byproduct you give them) -- and also should do this for legal reasons.

(Format switching on an entire book for personal use! Its so easy, buy my app - is something to talk about with your lawyer first.)

[NurViewer]

Thanks, notimp. I read both of your replies carefully. There's a lot of useful practical experience here.

Your points about shadow removal without damaging character edges, keeping double-page scans, and especially the problems with page numbers, hyphenation, paragraph boundaries and scene breaks are very helpful. These are exactly the kinds of details that can be missed when testing OCR on just a few pages.

We've recently started adding book scanning to our Android app, so these are useful areas for us to focus on. I also agree that keeping the original images and making it easy to compare them with OCR results would be important.

Your suggestion about TTS and synchronized reading is interesting too, especially for language learning.

One question: when converting a complete book, which part of the process still takes you the most manual work? And if you know of any particularly difficult public-domain scans, I'd be interested in testing them.

Thanks again for sharing such a detailed workflow!

[notimp]

4/5 of the manual work at the checking the formerly scewed parts of the pages for OCR errors (which is the same process in which I check for page numbers that stayed in after crop, and for correct pagebreak removals (paragraph merging) -- but its all mostly done because of double checking for OCR errors, pagebreak removal can be automated via Regex - 2, 3 left over pagenumbers that become part of the main text arent the world) - this is all one step though - where I'm comparing every page of the pdf against the epub text and making changes in the epub manually.

1/5 of the time: Pulling headline formatings onto the same level ("you all are h2 headlines"), merging pages that have unneeded chapter breaks (md to epub script looks at headline tags in the .md), adding additional chapter breaks and headline formating for the headlines that were missed by OCR (they usually never are shown as plain text and always have at least special formating, but might miss the headline tag), automatically (Sigil has a feature for it) recreating the TOC (epubs table of content), making sure the epub has the right title and author name and language tag in the metadata.

The second part maybe takes 5-10 minutes at most.

Also - If I just want to have an epub read to me with TTS I can skip all of them, aside from headings alignment and toc creation. (Because I create .wav files per chapter (based on the TOC). and then srts based on the .wavs).

Because the issues fixed in the 4/5 step are usually so minute - that for the unconcerning reader they dont matter. For a publisher they do. As in they almost never are text recognition errors anymore. The OCR almost always is right. Its just a split paragraph or a missed pagenumber out of 300 that I'm manually removing. And I'm doing this mostly because I have to check for huge OCR mistakes if I want to act responsibly. For personal use - I could almost skip that 4/5 step.

On perfect test books:

Children/Young adults literature. With Inserts in the text where a third person like a narrator speaks, ideally with an Image indicating those sections.

I dont have public domain books, that fit, but I can give you a recommendation.

You arent looking for books that are "very hard" to OCR (if you need that - look for scientific texts with many different nested layout levels, footnotes, and image subtitling) to test, you are looking for "short texts" with several challenging forms of "usual, normal" layouting that are a third of the length of a normal book when transforming them into a perfectly formed epub.

To learn.

My go to is always Die drei ??? und das Gespensterschloss by Robert Arthur, German edition - Where the earlier releases always had the Narrator (Hitchcock) pitch in at certain points of the Text.

Because it will teach you the shortcomings of conventional OCR (tesseract) on conventional book formating quite quickly.

Should you know the sources the AI bros frequented for training material for ChatGPT or Fable/Opus, you'll find pdfs of that book there as well. Those sources even have wiki pages, but I'll still not link them here. Or why not buy a used german book from an online retailer? Has to be a release from 2004 or before, because afterwards Kosmos lost the rights to use the Hitchcock branding.

The parts I usually look at in those books look like these - (Three pages for reference):

https://imgur.com/a/RjcFYJO

As for smartphone scans - make them yourself, test different smartphone cameras, childeren literature prefered, because "one entire book" is about a third of the size/volume.

edit:

Oh, and Shadowrun - 2XS by Nigel Findley (english or german edition) for the other type of challenging formating I was mentioning.

So first paragraph of a page with no indents, others with. Like this -

https://imgur.com/a/tugXYYQ

But imagine that on page 1 a sentence ended and on page 2 a new sentence started. What is it? A new Paragraph? A new scene after a scene break? A continuation of the previous paragraph?

Because the previous sentence in this case (on page 1) ended in a character that is not a sentence end mark ([^(.|?|!|<|>|«|‹|"|:|“)]), we would fold the actual starting paragraph on page 2 together with the previous one (possible via regex). But what if at the beginning of page 2 a new sentence had started?