Scanning Paperbacks with Yellow Background

[kbaerwald]

I want to scan 250-300 old paperbacks from the sixties (all paper with partly strong yellowish cast, characters mostly 8-9 point large, cheap print quality). Most books have 150-400 pages.

The process will be destructive. Cutting off the spine and sending the individual sheets through a Canon P-150 (double-side scanning). The resulting muli-page tifs will be processes further by Fineprint 11 OCR: targets are tagged and searchable pdf files.

I could do three different pre-processing steps

1. pre-processing the tif files with photographic means to filter the yellowish (really bad dark yellow) out and feeding the tifs into a black-and-white workflow
2. scanning the grayscale 600 dpi tif directly into Fineprint 11 and have Fineprint doing all the optimization
3. scanning the grayscale 600 dpi tif and move the files into Scan tailor first reducing to b&w and feeding into Fineprint 11

Given the number of books to process would you recommend the one or other process (or a completely different one)?
It should be done highly automated with a fairly good recogniten rate for OCR at the same time . File space is not really a problem.

In which steps should I do the optimization of contrast and brightness or should I give Fineprint control over it?

I have done some testing already with mixed results. Any recommendation is appreciated.

Klaus

[frostschutz]

If your scanner supports black&white scan (no gray, just those two colors, i.e. 1 bit), try that first. It should let you select a threshold (the point where a color is considered white, or black resp.) which you should change until only the text remains black and none of the background. If you're getting blotchy or thin characters regardless which threshold you choose, yank up the resolution a bit.

Note that this only works for text, not any kind of images (except maybe unshaded lineart).

Alternatively do a grayscale scan and threshold it yourself - unless the OCR software is really smart and able to choose the best threshold for itself.

If it's a really bad background color (such as dark red) you might have to do a color scan, or adjust how colors are translated to grayscale by the scanner (if there is such an option at all), since some colors end up being simply too dark in grayscale.

[DSpider]

FinePrint (http://fineprint.com/fp/) is at v6.25. Did you mean FineREADER ?

The yellow background doesn't matter, as long as the text is still legible. I came across quite a few scans of books highlighted in green, yellow and pink markers. Everything went better than expected, even with underlined text as long as it was in graphite pencil and not ballpoint pen or something else.

Quality trumps quantity every time. So my suggestion is to focus on quality and properly digitize them. Have you checked with Project Gutenberg yet? Google Books? Because chances are that many (if not all of them) belong to the public domain since their copyright probably expired decades ago. Some authors may not be alive today. Downloading them could save you a lot of work.

[kbaerwald]

@frostschutz (alzheimer)

Most of the books are SCIFI from that decade and I want to have them readable on my iRiver HD . Within certain limits I can adjust the image brightness but not the threshhold (exposure) directly. To determine the threshhold on a per book scale would mean some extra work which I would like to avoid. On the other hand: if I do grayscale I have more information in my image to compensate for exposure errors at a later point of the work flow.

I would be really interested in an opinion from an FineReader(!!!) user how well the integrated optimization works if I go into workflow no. 2

Like you wrote - my tendency is to avoid too much extra work and I would favour the pure b&w workflow from the beginning. Obviously I have to die one death or the other

@DSpider

Most of the German translated books are from American authors who still live or some German publishers still own the rights. As I own the paperback I can do private copies for my own usage. I have my doubts that there are titles in the Google world which can be downloaded legally.

Most of the prints are on cheap paper and carry a lot "wooden" artifacts in addition to the tint and I obviously have to do some despeckling too (which will be done by this marvellous sw "Scan Tailor").

I just will start and see what experience I will gather during the process itself.

Thanks for your answers - I will keep you informed here.

[DDHarriman]

Hello kbaerwald

Before beginning to do anything, I would advise you to do some testing first (before choosing) - specially do not cut the books yet!

If you can, get a normal plain scanner (not the cut sheet you mention), and opening one of the books (press the spine down when scanning, you where cutting the pages anyway) scan 10 to 30 pages of it (2 pages each scan) using the internal Finereader Pro 11 scanning option. Settings - 300 dpi, grey.

Finereader will (if chosen to do so) split the two pages into individual ones (the ones it cannot do so, one can split manually afterwards) and deskew them (this one can do also afterwards).

Crop the pages of all the surrounding extra space (you can even get rid of page numbers, header and footers if you want) - try to have the crop window for every page to be the same size.

Export the file as images, one for each page, as a tiff black and white page..
The program will do an outstanding job into making clear back and white pages out of grey pages (or color ones), also it will number them sequentially.
If in the back and white pages you find some with black stuff (or marks), you can go back to Finereader, delete (clean/erase) those marks/stuff and save the page with the same name.

Get your PDF creating tool and build a PDF file from the sequential numbered TIFF files.
Open the result PDF in your eBook reader and check if the result suits you.

I can say, paperback books like you told and with the old/sun burned paper you described, let one make PDFs (the way described) that can be read easily in 6” eBook readers (bigger screens, even better!). And one does not have to care about OCR even! (faster that way). Also you are not destroying the pBooks in the process!

Best regards,

[kbaerwald]

@DDHarriman

Thanks for your recommendations - it looks like many arguments speak for the direct usage of FineReader Pro.

PDF is my target so I spent some time with that. Unfortunately your method of choice (flatbed with double-page scan) gives bad results without OCRing the images. I have skewing etc. and the image does not look good on the eReader.
This is why I cut the paperbacks and scan it with the Canon P-150.

Currently I use the the mixed text-image output that FineReader delivers: the reading quality on my iRiver Story HD (1024x768) is acceptable and I still have the Finereader files for going further into epub production at a later point in time (if required). O.K. the file size is 4-9 MB average but the handling on the eReader is smooth.

Klaus

[Iain]

MY experience with discoloured old sci fi paper backs and finereader, is that it doesn't work too well.

I'm scanning my books descructively into multipage tiff files (using home-made software) then OCR ing with finereader. My more discoloured paperbacks have almost unreadably large levels of OCR errors. Books in the same series, same font and all which aren't discoloured convert fine.

An earlier respondants recommondation of PDF should work in that you will be able to read the text even if the OCR hasn't worked well (because the original image s retained in some fashion). However, you will loose reflow of the pages, which would be a no-no for me.

One trick which may be interesting is to use adaptive thresholding. Rather than picking a level, this elevates a pixel to black (or white) if it is darker (or lighter) than the immediately surrounding pixels. I've not done this for a whole book as yet, but my tests with single pages take a page which produces rubbish in Omnireader and generates near perfect text.

Sadly, the only way I know to get this facility is with the OpenCV image processing library. There may be commercial software which incorporates this, but I don't know it.

[steven522]

I have scanned several old paperbacks and converted various ways. These were very old, yellowed, and in some cases, water stained. The best results for the process that I have found, are the following steps:

Scan at a resolution of 300 DPI or better. The scan image seems to work best as grayscale, but color will work, too. Process images as jpeg files.

Run the scanned page images for the entire book through ScanTailor. ScanTailor will allow you to split pages, rotate slightly askew page images, and clean up the results to a nice clear black and white TIFF series. I can not stress how much ScanTailor will help here. If you have never used it, at least give it a try. You will love it.

The TIFF files from this point can be run through a PDF composer as-is for viewing. As long as your book consists of text with black and white line art images at best, you will have a very nice PDF. The TIFF files can also be processed through Finereader with a minimal number of OCR errors.

[DDHarriman]

Hello kbaerwald

You are right, one does get what you say, and that was why I mentioned “press the spine down when scanning”, and… if even so you still get the same - and this is after splitting the images automatically in Finereader Pro (my assumptions where, scanning “inside” Finereader) or manually is the program can not do it itself (and then desckew each page in Finereader) -, than you have the option you are using now or… get a book scanner.
(note my advice was for testing, and it looks you have already done that)

For the book scanner I advise (and still do to all my costumers) the Plustek OpticBook (the 3600 version is enough if you can still find it, or any other one, if not) - and once more you will not need to destroy to original pBook (sorry for me, it’s always the last option).

Finally, the decision to keep the Finereader files is a wise one - even if it will translate into hundreds/thousands of megabytes per book - as you will be able to build the real thing - a reflowing text eBook in any format you want - in the future.
Just take care it’s a hog of time and work… but for some (projects) well worth it.

Best regards,

[kbaerwald]

Taking into account all your good advices I processed one of my paperbacks with a less yellow cast: a fantasy novel with 526 pages (175.000 words) first published in 1946. For those who hate to destroy books I have to say that I own a more precious hardcover version which I would NOT handle that way.

The cutting is done in a simple way: cover is carefully removed, the book divided into 100 page chunks. The chunks are fixed on a workbench with screw clamps alongside of two wooden bars. With a sharp knife I cut off approx. 5mm of the spine. Done.

Looks like these two workflows are usable for me:

Target pdf

- Cutting the paperback and feeding into the Canon P-150: 25 pages at a time into the feeder, specs are roughly 10 double-pages per minute with 600 dpi grayscale which is very fast. A 500 page book is done in 30 min with handling, recans etc.
- Scanning grayscale with variable brightness; on a scale between 0-254 it is something between 160-200 depending on the yellowness of the book. The contrast is 10-20% more than average. This gives the OCR software enough room for interpretation and does not cut off too much information. Fortunately the scanner also allows for deskewing and other adjustments. It is always better to do these things at the very beginning of the workflow. Results are 600 dpi grayscales tifs, roughly adjusted but still with "jumping" layout.
- Tifs are then processed into Scan Tailor (excellent programme, already donated for it): in the end the text ist a centred b&w tif which can be directly handled by Adobe Acrobat or FineReader.
- Reading quality is acceptable to good on my eReader.

The whole process is realtively fast and yields a readable 700 page pdf ebook within a few hours.

Target epub/mobi

- The first two steps are the same but the tifs are then directly processed by FineReader 11
- The OCR quality of FineReader is really good: I compared that with Omnipage and tesseract (my favourite for German Fraktur text). Output is rtf with basic formatting (headings, body text with italics)
- I prefer MS Word for further processing: removing hidden and wrong hyphenations, correcting false OCR characters and CRs. Most of the time FineReader delivers a correct chaptering. Other word processors would be suitable too but I am experienced with WfW for some years.
- The finishing will be done with Jutoh (which is described elsewhere in this forum) with an epub/mobi output.

This workflow takes somewhat longer because of OCR: the 700 page example took me 4 hours of finalizing after the scanning. This pretty much depends on the quality and age of the book template. What comes out of it is a structured and easily readable eBook without much designing. Just enough for my fiction books.

I hope to get some more insights to share with you during the processing of next 300 paperbacks.

Klaus

[Beryll Snyder]

No problems with the automatic document feeding?
I used a Canon Office All-in-one, certainly not the best of solutions to start with(because it does not do duplex). The feeding mechanism seems to have problems with small pages und the rough surface of older paperbacks.
So there were omitted pages, and to put everything together again was fun.
So the P-150 apparently has a pretty good feeding mechanism, or do you have to watch it during the operation?
So I reverted to using my Plustek 3600. If the margins allow it, which they normally do, 6 pages a minute are realistic(300 dpi grayscale). So 700 pages would be scanned in 2 to 3(tops) hours.
The rest of the workflow ist more or less as described by you.

[kbaerwald]

Originally Posted by Beryll Snyder

No problems with the automatic document feeding?

The scanner is pretty new - currently I do not have problems but let us see after 10.000 pages

The process runs unattended: the feeder grabs 20-25 sheets of paper without problems. After that I reload the next chunk of paper and push a knob - no further interaction with the PC until all sheets are done. It even has a function to "detect" paper-feed failures by measuring the average length of a page. You can configure front and back of the sheet individually.

Do not forget this is a low-cost mobile scanner but for that price it does a good job. The twain driver can be configured with pretty detail and looks robust. What attracted me was the duplex scan and the high scan rates.

I can give you more information after 200 scanned books

Klaus

P.S. I scanned with a flatbed before: it is o.k. for one book or two - not for 200.

[DSpider]

Regardless the scanning method (document feeder, flatbed scanner, compact camera, mobile phone, etc), after you scan a book you should take a few minutes to look through the images and see if it skipped any pages. Especially with automatic document feeders. You never know...

Anyway, I think the scanning part is the easiest. Processing the scans takes much much longer: OCR-ing with FineReader, proofreading in FineReader, tracking down the fonts (or at least their closest match), vectorizing the cover and/or any other images in the book, doing the layout in Word, saving as PDF, optimizing the PDF, proofreading the final product... Add to that converting the Word file to HTML, loading it into Sigil, manually cleaning it up, stylizing it, saving as ePub, converting it to Mobi... Yeah. The scanning part is definitely the easiest. But I guess it depends how far you wanna go.

[kbaerwald]

Originally Posted by DSpider

Regardless the scanning method (document feeder, flatbed scanner, compact camera, mobile phone, etc), after you scan a book you should take a few minutes to look through the images and see if it skipped any pages. Especially with automatic document feeders. You never know...

Yes - the system is easy: when scanning I synchronize the "real" page numbers with the system's scan counter. After each feeder chunk of scans I see that the counter and the real page number is the same. Works perfectly! Moreover the Canon scanner has an "internal" warining system (see my previous input).

[Beryll Snyder]

Originally Posted by DSpider

The scanning part is definitely the easiest. But I guess it depends how far you wanna go.

I agree - and I never go far. I can live with the (mostly few)errors and imperfect formatting: simply too much work.