line formatting formatting question

[daesdaemar]

We are all aware of those pesky text documents that have a hard return/line break at the end of each line. Fortunately, there are usually two line breaks at the end of a paragraph so it is relatively easy to format out the line breaks at the end of each line and yet keep the paragraph structure.

Here's my problem:

I have a book in lit format that I converted to lrf in Calibre and the line breaks were scattered all over the place and it looked awful. I then converted the lit file with ConvertLit to html and then to text/rtf. There is a hard line break at the end of every single line, and there are not two breaks at the end of paragraphs. If I remove the line breaks I have a 400 page document with no structure at all.

I have also tried to convert in BookDesigner with the same problem -- no paragraph structure.

Any ideas?

[kovidgoyal]

The algorithm to use in such a case is based on average line length. First calculate the average line length and when there are lines significantly shorter than that, dont remove the breaks. That will take care of most of the breaks correctly

[daesdaemar]

Originally Posted by kovidgoyal

The algorithm to use in such a case is based on average line length. First calculate the average line length and when there are lines significantly shorter than that, dont remove the breaks. That will take care of most of the breaks correctly

I will try working on that... thanks.

[kacir]

For such books I use Vim script
(www.vim.org - a very powerful text editor)

You can write a command in Vim saying
"find every line NOT ending with dot, question mark, exclamation point or closing quote, optionally followed by a space character and join it with the next line"
:vglobal/[.!?"']\s*$/join
I often abbreviate the above command this way:
:v/[.!?"']\s*$/j
That is it.

You can also say:
"find every line ending with .!?" and enter an empty line after it"
"find every line shorter than (let's say) 50 characters and enter an empty line after it"
"find two empty lines and replace it with one empty line"
"Join paragraphs"
"delete empty lines"
That should take care about formatting 99 percent of excessive newline characters.

You have to tweak the above steps for a particular book, because every single misformated book is unique.

You can also try to have a look at the html file and try to distinguish between wanted and unwanted line breaks. Most often, unfortunately, the html file is generated by MSWord. MSWord is THE most horrible tool for producing html format.
You can also try to process html file with a program html_tidy http://www.w3.org/People/Raggett/tidy/

[JSWolf]

This does not sound like it is a legal LIT file. Where did this LIT come from?

[daesdaemar]

@kacir... very interesting. I will need to look into this program.

@JSWolf... this is actually a lit book that I have legally purchased several years ago. I am trying to format it for my Sony 505. I have actually run across several lit books that do not convert easily.

[kovidgoyal]

these are LIT books that embed TXT files instead of HTML files

[daesdaemar]

Originally Posted by kovidgoyal

these are LIT books that embed TXT files instead of HTML files

Based on my limited knowledge, that would explain some of the problems I've had with lit books.

[kacir]

Originally Posted by daesdaemar

@kacir... very interesting. I will need to look into this program.

If you need any help to get started with Vim, do not hesitate to ask.

[JSWolf]

With the wonky LIT, use lit2oeb to convert the LIT into it's component parts. Then you can fix it up so it converts the way you want.