Learning More About Cleaning Up Documents

[user_none]

Originally Posted by Agama

Can you recommend any resources, (books/websites), for learning Python...

Dive into Python! It's a book and a web site. The entire thing is available online free.

[Agama]

@ Manichean, user_none : Thanks for these links, both resources look really good. I have downloaded the book and am immediately impressed by the scope of Python.

@Idolse : The scripts are geared towards tidy up of markdown to ePub conversions and use the exploded ePub from calibre's Tweak ePub:
1) Renames the html split files which result from markdown -> ePub and updates the opf/ncx files to match.
2) Strips all class="calibre[0-9]*" attributes from an ePub and links in a custom stylesheet.
3) Tidies the OPF file by stripping out blank lines and splitting lines with multiple XML tags. These are not produced by calibre but some publishers seem fond of unreadable OPF files, (e.g. Feedbooks use lots of blank lines).
4) ToC editor which presents the ePub ncf file as a simple text file, (1 line per ToC entry), for insertions/deletions/ammendments/hierarchies then rebuilds it including playOrder.
5) (In progress) Applies a predefined set of regexes to a plain text file prior to conversion.

[Archon]

Idolse,

You wrote:
<<You should also check out the 'Heuristics' section of Calibre's conversion settings. A lot of the things you want to do may be covered there already.>>

From Calibre's Manual it says:
<<Enable heuristic processing
This option activates calibre‘s Heuristic Processing stage of the conversion pipeline. This must be enabled in order for various sub-functions to be applied>>
Now the manual has this choice right between "Page Setup" and "Structure Detection". If you look at the page in the manual there is no selection for these in the picture they show.
http://calibre-ebook.com/user_manual...tic-processing

I have tried bringing up the conversion screen and under structure detection I see options for Chapters, Headers and footers.

For new users of the program it is not clear exactly where to find these options.

Archon

[Archon]

dwanthny,

<<Right where you would expect it while in code view under the find and replace window, check Regular expression under search mode.>>

I apparently wasn't in "code view" and didn't realize I needed to be in that to search and replace with regex with Sigil. I may try that next time I test drive it. BBedit and Text wrangler have a "use grep" check box in their find and replace dialog.

I may have to re-evaluate Sigil for my purposes.

Thanks for the tip.
Archon

[user_none]

@Archon, Thanks for pointing out that error in the manual. The heuristics were just reorganized and added as a separate panel widget last week. It looks like that screen shot was over looked when the manual was updated.

[theducks]

Originally Posted by Archon

dwanthny,

<<Right where you would expect it while in code view under the find and replace window, check Regular expression under search mode.>>

I apparently wasn't in "code view" and didn't realize I needed to be in that to search and replace with regex with Sigil. I may try that next time I test drive it. BBedit and Text wrangler have a "use grep" check box in their find and replace dialog.

I may have to re-evaluate Sigil for my purposes.

Thanks for the tip.
Archon

You can't do multi-file S+R (any flavor) in Sigil unless you are in CV

Beware doing REGEX on a single file in BV, It will ignore the stuff you can't see and turn a simple (looking) replace into a

[Manichean]

Originally Posted by Archon

I have tried bringing up the conversion screen and under structure detection I see options for Chapters, Headers and footers.

For new users of the program it is not clear exactly where to find these options.

Ah, are you sure that you've got the latest version (0.7.42) installed? The header/footer part in structure detection should also be gone by that version, to be replaced by the new search & replace option.

[Archon]

Ahh yes I see that in 0.7.42.

Unfortunately, I am still on 0.7.38 because 0.7.40 didn't handle conversion of old rtf's properly and now 0.7.42 fixes that problem but now there is a bug with the window draw routine in the multi-file metadata edit screen since they added the Date reset functionality. It is usable but I have to keep resetting the window size every time I want to do a multi-file edit since it won't remember the window size from the last time.

So, since I use that function more than the search and replace (I haven't used that at all yet) I am sticking with 0.7.38 for now.

But, I will fire up 0.7.42 and test drive the S&R routine on a few documents when I get gong here. By then 0.7.43 will probably fix the window drawing bug.

Kovid and you guys are making this into a great document conversion utility along with its great cataloging features.

Happy Monday
Archon

[Agama]

Originally Posted by Archon

Kovid and you guys are making this into a great document conversion utility along with its great cataloging features.

Agreed! Sounds like time to get 0.7.42 installed and have a play with the new features.

[ldolse]

Originally Posted by Agama

@ Manichean, user_none : Thanks for these links, both resources look really good. I have downloaded the book and am immediately impressed by the scope of Python.

@Idolse : The scripts are geared towards tidy up of markdown to ePub conversions and use the exploded ePub from calibre's Tweak ePub:
1) Renames the html split files which result from markdown -> ePub and updates the opf/ncx files to match.
2) Strips all class="calibre[0-9]*" attributes from an ePub and links in a custom stylesheet.
3) Tidies the OPF file by stripping out blank lines and splitting lines with multiple XML tags. These are not produced by calibre but some publishers seem fond of unreadable OPF files, (e.g. Feedbooks use lots of blank lines).
4) ToC editor which presents the ePub ncf file as a simple text file, (1 line per ToC entry), for insertions/deletions/ammendments/hierarchies then rebuilds it including playOrder.
5) (In progress) Applies a predefined set of regexes to a plain text file prior to conversion.

Those mostly sound like things that probably would be best tied directly into the tweak epub feature rather than elsewhere in the pipeline. The regexes you're looking to apply to text conversion are something that could be done directly in the text input plugin, either as a new option or modifying an existing one, depending on what they are.

[Agama]

Originally Posted by ldolse

Those The regexes you're looking to apply to text conversion are something that could be done directly in the text input plugin, either as a new option or modifying an existing one, depending on what they are.

Thanks, I'll look into this - it's always better not to re-invent the wheel. There's just so much in calibre that it takes some time to work through all the options - and the developers keep on giving us more!