Getting Text content of book

[sgmoore]

What is the proper way of extracting the text from a book from a plugin?

I know I can do

os.system('ebook-convert' , ...

to create a temporary txt file and then read the contents of the temporary file into a string, but there is probably a better and faster way.

[kovidgoyal]

That way works fine.

[jackie_w]

I don't know whether it's better or faster but the calibre plugin 'Count Pages' contains some code for extracting book text into a big string. It uses it when calculating a wordcount for the book.

[isarl]

Instead of using Calibre's objects I find it simplest to use the Python library ebooklib. Calibre's container types work with exact MIME types whereas ebooklib simply lets me ask for all ITEM_DOCUMENTs in an ebook. Here is some sample code I have written which demonstrates using it to read ebook contents:

import ebooklib
import lxml

book = ebooklib.epub.read_epub("path/to/book.epub")
docs = list(book.get_items_of_type(ebooklib.ITEM_DOCUMENT)
# beware non-UTF8 content! E.g. you might need to .decode("latin1"), or some other encoding, instead.
doctree = lxml.etree.fromstring(docs[0].get_body_content().decode())

If you are interested in counting words then I recommend Calibre's calibre.spell.break_iterator.count_words function which reuses logic from the International Consortium for Unicode to get it “right” (± locale and quality of input text).

Good luck with your project.

[sgmoore]

Originally Posted by jackie_w

I don't know whether it's better or faster but the calibre plugin 'Count Pages' contains some code for extracting book text into a big string. It uses it when calculating a wordcount for the book.

It's definitely faster (a quick one off test shows that spawning ebook-convert is about five times slower).

Unfortunately it is not better and indeed not good enough. I have some files which look like they have been generated as epub files by Microsoft Word, and the count_pages algorithm produces text which is about four times larger than ebook-convert. (A quick glance shows thousands of font-family entries which have not been removed by count_pages).

[sgmoore]

Originally Posted by isarl

Instead of using Calibre's objects I find it simplest to use the Python library ebooklib.

Looks like that only works with epub files.

[isarl]

Originally Posted by sgmoore

Looks like that only works with epub files.

Yes, file format is going to affect how you solve this problem. The more disparate types you need to handle, the more work you need to do to ensure that each one is being handled correctly. TANSTAAFL

[BetterRed]

Originally Posted by sgmoore

It's definitely faster (a quick one off test shows that spawning ebook-convert is about five times slower).

Unfortunately it is not better and indeed not good enough. I have some files which look like they have been generated as epub files by Microsoft Word, and the count_pages algorithm produces text which is about four times larger than ebook-convert. (A quick glance shows thousands of font-family entries which have not been removed by count_pages).

AFAIK Microsoft Word itself is incapable of creating EPUB files directly.

The ePUBTools Word addin can create EPUBs from within Word, and there are a number of tools, including calibre, that will convert MS Word's native format DOCX files to EPUB.

Added: InDesign is a more likely candidate as the source of poorly formed EPUBs.

BR

[kovidgoyal]

If you care about speed use the extract_text() function from calibre.db.fts.text

[sgmoore]

Originally Posted by kovidgoyal

If you care about speed use the extract_text() function from calibre.db.fts.text

Thank you. That looks to do what I want. It does not produce the exact same results as ebook-convert but ignoring formatting and white-space issues its is extremely close. I tried it on about 1000 books and the worse case was still 99% similar and the vast majority of them were over 99.9% similar.

Also only takes about 1/20 of the time to call ebook-convert.

Thanks again.

[Quoth]

Calibre converting docx to epub works better than the plugins I've tried on Word & LO Writer. It seems to work better than Indesign and other commercial tools judging by the commercial ebooks from big publishers.
Indesign should only be used for fancy colour coffee table books and glossy magasines.

Calibre is jjust about perfect from properly formatted docx (made by Word or extra Save As in LO Writer) for novels. Also for ordinary novels direct PDF export from a differently formated copy of the Wp file beats Indesign too. Also you now can only rent Indesign.