Conversion: EPUB to TXT

Starson17 · 05-25-2010, 10:55 AM

I've got an EPUB that is filled with paragraphs like this:

Code:

<p>Epiphany Sunday of the new year of 1801 had been on the fourth of
January, and the next Term for King's Bench trials had, therefore,
waited to open on the seventh, with all the theatre, majesty, and
circumstance of which England was capable.</p>

Notice that the "ofJanuary" has no space between "of" and "January," but it does have a new line (0x0A). The same thing happens after "therefore," and "and". As I understand it, the new line character is seen as white space, and that's a word separator, so the EPUB displays correctly, putting a space between those words, not a line break. However, the conversion to .txt format (and rtf) does not seem to do that. All the words at the line ends have the new line characters stripped, but they aren't replaced by spaces, so I see lots of two-words combined into one-word form.

Is there a setting to fix this? A bug?

Thanks.

kovidgoyal · 05-25-2010, 11:23 AM

Sounds like a bug, opena ticket and assign it to either TXT output or rtf output.

Starson17 · 05-25-2010, 11:29 AM

Quote:

Originally Posted by kovidgoyal

Sounds like a bug

Thanks. I'll investigate a bit more, and perhaps be able to post some code with the ticket, or at least some pointers to where I think a change should be made. I haven't looked at the conversion code much. I always prefer studying the code when it will help me fix or improve something that's affecting me.

kovidgoyal · 05-25-2010, 11:34 AM

Quote:

Originally Posted by Starson17

Thanks. I'll investigate a bit more, and perhaps be able to post some code with the ticket, or at least some pointers to where I think a change should be made. I haven't looked at the conversion code much. I always prefer studying the code when it will help me fix or improve something that's affecting me.

Tickets that help me find and fix the problem are my favourite kind

user_none · 05-25-2010, 07:05 PM

newlines are replaced with space when generating TXT output... I will look into why this is happening. And I will need a sample because I cannot recreate this behavior.

The basic idea behind text output is take the XHTML, and replace all newlines with spaces. Then start reading though the XHTML string that is all one line pulling out text and adding newlines based on tags that represent paragraph breaks.

Starson17 · 05-25-2010, 09:01 PM

Quote:

Originally Posted by user_none

newlines are replaced with space when generating TXT output... I will look into why this is happening. And I will need a sample because I cannot recreate this behavior.

The basic idea behind text output is take the XHTML, and replace all newlines with spaces. Then start reading though the XHTML string that is all one line pulling out text and adding newlines based on tags that represent paragraph breaks.

You're going to deprive me of all the fun of figuring this out.

I'm currently trying to recover from some sort of constant reboot problem - will probably restore yesterday's backup. I assume the sample I posted from the EPUB wasn't enough to reproduce it? Oops - got to go, it just threw the pre - reboot error - hope this gets posted .....

user_none · 05-29-2010, 07:29 AM

Quote:

Originally Posted by Starson17

You're going to deprive me of all the fun of figuring this out.

At this point you might have to because I can't reproduce it.

Quote:

Originally Posted by Starson17

I assume the sample I posted from the EPUB wasn't enough to reproduce it?

Nope. Converted just fine for me.

Quote:

Originally Posted by Starson17

I'm currently trying to recover from some sort of constant reboot problem - will probably restore yesterday's backup. ...
Oops - got to go, it just threw the pre - reboot error - hope this gets posted .....

Could this be related?

Starson17 · 05-29-2010, 11:19 AM

Quote:

Originally Posted by user_none

At this point you might have to because I can't reproduce it.

Could you try again, but convert to rtf, not txt? I can't reproduce the problem in txt, but can in rtf when converting from a test zip file and from the original epub. The test zip is just a basic html file added to Calibre having a single paragraph <p> with <html>, <title> and <body> tags. Make sure the breaks in the para are just 0x0A.

Debug during conversion shows no problem at any of the 4 intermediate output stages, so I assume it would be in the rtf output stage.

Quote:

Could this be related?

No. I spent last night doing an image restore of my entire computer to last week's status and this morning putting back some settings that had changed. The computer's stable again. (Something bad happened when upgrading AVG antivirus - it's now gone.)

user_none · 05-29-2010, 12:04 PM

Quote:

Originally Posted by Starson17

Could you try again, but convert to rtf, not txt?

Verified and fix committed to driver-dev.

Starson17 · 05-29-2010, 12:08 PM

Quote:

Originally Posted by user_none

Verified and fix committed to driver-dev.

Do you mind telling me what change you made? I'd just like to get familiar with how the conversion code works.

user_none · 05-29-2010, 12:18 PM

Quote:

Originally Posted by Starson17

Do you mind telling me what change you made? I'd just like to get familiar with how the conversion code works.

Sure.

I've attached my changes. Basically all I changed was to read the XHTML into a string and replace all newline characters (windows, unix, and old mac) with a space.

The conversion pipeline has some basic requirements for passing data between the various parts. But once you get it into a stage it allows you to do pretty much whatever you want for actually processing the data. You'll notice a lot of similarities between some of the formats I've created and vast differences between other ones. Feel free to email me if you want/need a more in depth explanation (john AT nachtimwald.com).

Starson17 · 05-29-2010, 12:31 PM

Quote:

Originally Posted by user_none

Sure.

Thanks (and sorry for misleading you on the txt format - I could have sworn I saw it there too ...)

05-25-2010, 10:55 AM	#1
Starson17 Wizard Posts: 4,004 Karma: 177841 Join Date: Dec 2009 Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T	Conversion: EPUB to TXT I've got an EPUB that is filled with paragraphs like this: Code: <p>Epiphany Sunday of the new year of 1801 had been on the fourth of January, and the next Term for King's Bench trials had, therefore, waited to open on the seventh, with all the theatre, majesty, and circumstance of which England was capable.</p> Notice that the "ofJanuary" has no space between "of" and "January," but it does have a new line (0x0A). The same thing happens after "therefore," and "and". As I understand it, the new line character is seen as white space, and that's a word separator, so the EPUB displays correctly, putting a space between those words, not a line break. However, the conversion to .txt format (and rtf) does not seem to do that. All the words at the line ends have the new line characters stripped, but they aren't replaced by spaces, so I see lots of two-words combined into one-word form. Is there a setting to fix this? A bug? Thanks. Last edited by Starson17; 05-25-2010 at 11:30 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Preserving <br /> on epub -> txt conversion	billingd	Calibre	1	08-11-2010 06:24 AM
TXT conversion to ePub or LRF - paragraph formatting	Zapped	Calibre	6	10-23-2009 05:06 PM
HTML to TXT conversion	alkr	Calibre	3	10-02-2009 09:54 AM
Batch conversion of txt	BlackVoid	Sony Reader	8	11-17-2007 09:53 PM
conversion - pdf to txt?	fishcube	Sony Reader	1	10-24-2007 02:02 PM

05-25-2010, 11:23 AM	#2
kovidgoyal creator of calibre Posts: 45,168 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Sounds like a bug, opena ticket and assign it to either TXT output or rtf output.

05-25-2010, 07:05 PM	#5
user_none Sigil & calibre developer Posts: 2,487 Karma: 1063785 Join Date: Jan 2009 Location: Florida, USA Device: Nook STR	newlines are replaced with space when generating TXT output... I will look into why this is happening. And I will need a sample because I cannot recreate this behavior. The basic idea behind text output is take the XHTML, and replace all newlines with spaces. Then start reading though the XHTML string that is all one line pulling out text and adding newlines based on tags that represent paragraph breaks.

Advert

Advert