lit2oeb -- calibre LIT extraction/conversion without ConvertLIT

[bkilian]

Originally Posted by kovidgoyal

As of version 0.4.83, lit2oeb powers lit2lrf

And also as of 0.4.83, it has become very difficult for me to use calibre Is there a way to tell it to use ConvertLit if lit2oeb fails dismally on a number of your books in irritating to spot ways that you don't notice until you try read the book?

Examples of failure:
Ticket #938 (Which makes my entirely automated book coversion process tricky at best)
Ticket #939 (In which a book that I spent ages scanning maps and cleaning them up suddenly has no maps)

And in general, it becomes _impossible_ to edit the resulting OPF and HTML files, which I tend to do a lot, when they're all in one line. ConvertLit appears to have no problems making an easy to read and edit file. (I had a hell of a time trying to add a <dc:Language> tag to a bunch of books because of this)

If I find anything else, I'll create corresponding bugs, but for the moment, I'm essentially stuck since I did a directory clean up and deleted all my LRF files to do a clean reconvert, and now I'll have to wait until these bugs are fixed.

[kovidgoyal]

It's the price of progress. The new code will be much easier to maintain in the long run.

You can run the output of lit2oeb through a pretty printer like html tidy if you want to edit the files by hand.

[llasram]

Originally Posted by bkilian

Examples of failure:
Ticket #938 (Which makes my entirely automated book coversion process tricky at best)
Ticket #939 (In which a book that I spent ages scanning maps and cleaning them up suddenly has no maps)

I've pushed fixes for these bugs up to my bug-fix branch -- Kovid should pick them up for the next release unless he's unhappy with them for some reason.

Originally Posted by bkilian

And in general, it becomes _impossible_ to edit the resulting OPF and HTML files, which I tend to do a lot, when they're all in one line. ConvertLit appears to have no problems making an easy to read and edit file. (I had a hell of a time trying to add a <dc:Language> tag to a bunch of books because of this)

IceHand has asked for this too, and it's certainly on the todo list. The problem is that the HTML contained in the LIT files is stored as lit2oeb produces them. ConvertLIT pretty-prints the HTML as it extracts it, but gets it wrong quite frequently, inserting whitespace where it doesn't belong and producing output with e.g. "S mall C aps" messed up. I've looked at pretty-printing with the HTML & XML parser/generators already in calibre, but they either have the same flaws as the ConvertLIT pretty-printer (BeautifulSoup) or don't always succeed in actualy pretty-printing "document-style" XML (lxml).

Originally Posted by bkilian

If I find anything else, I'll create corresponding bugs, but for the moment, I'm essentially stuck since I did a directory clean up and deleted all my LRF files to do a clean reconvert, and now I'll have to wait until these bugs are fixed.

I'm sorry you've run into some bugs, but please do submit any more issues you find and I'll get them fixed as quickly as I can.

[kovidgoyal]

I'll certainly merge the bug fixes before the next release. But it's going to be a little while until the next release as calibre is getting its job control system overhauled and is not in a usable state at the moment.

[kovidgoyal]

@llasram it may be woth implementing a simple custom pretty printer that just inserts new lines at the end of a bunch of pre defined tags (like <p>, <br>, <div>, <tr>, the various OPF tags etc) If we dont care about indentation, this should be easy enough to do.

[llasram]

Originally Posted by kovidgoyal

@llasram it may be woth implementing a simple custom pretty printer that just inserts new lines at the end of a bunch of pre defined tags (like <p>, <br>, <div>, <tr>, the various OPF tags etc) If we dont care about indentation, this should be easy enough to do.

I tried again running the markup through lxml, this time using a parser with the 'remove_blank_text' option enabled. This fixed the issue with pure-whitespace "tails" causing pretty-printing to fail entirely, at the cost of possibly removing relevant whitespace. That said, I checked several books and didn't see any rendering differences between enabling pretty-printing and not. That said, it seems safest to leave this as an option, so I've got the lxml-based option '--pretty-print' implemented and pushed up in my branch.

[kovidgoyal]

OK having it as an option is acceptable as I dont want to break things for the majority of users that go the direct lit->lrf route

[bkilian]

Originally Posted by kovidgoyal

I'll certainly merge the bug fixes before the next release. But it's going to be a little while until the next release as calibre is getting its job control system overhauled and is not in a usable state at the moment.

I don't need calibre, I just need a new lit2oeb.exe
Luckily I have an older version on my work PC, I think I can get all my LRF files recreated.

Thanks for all the work you guys do. I may sound ungrateful every so often, but calibre is a large part of the reason why I'm considering buying a PRS505 as my next device and not a Kindle.