Sample Plugin for the Editor

[DiapDealer]

Originally Posted by kovidgoyal

Those messages are harmless. Uninstalling does not remove the entries for the toolbar so that if you re-install they are still there, in the chosen location. The editor simply ignores missing actions.

Good enough for me.

---------

In the spirit of getting some more Editor Plugin action happening, I'm attaching a plugin that has three different tools. All three can be set to run on the currently opened file or to run book-wide.

1) Span Div Editor: conveniently remove, change the attributes of (and/or convert to different html elements) those pesky, nestable spans and divs (as well as a few others).

2) Smarten Punctuation (the sequel): gives slightly more granular control over the the smartening process. It preserves pre-existing entities and lets you choose what you want to smarten. So if you prefer three periods to the ellipse character/entity, you can keep them if you like. Also lets you supply a text file that defines apostrophe exception words like 'cept and 'tis in order to get the apostrophe right (instead of an opening single quote). I'm attaching a sample file for that, but basically it's one word per line WITHOUT any apostrophe characters. You can also choose to go with SmartyPants' default of using numeric entities instead of unicode characters if you like.

3) Convert CM to EM: this one was one I did for roger64. It simply parses CSS files looking for attributes that have cm dimensions and converts them to em (based on a preset factor). Figured I'd toss that in.

Please scour the differences and make judicious use of the checkpoint system before trusting these tools on your babies!!

Not sure this is ready for a Prime-time plugin listing, so I thought I'd put it here.

[Rev. Bob]

NOTE TO ALL: Please forgive the slightly stream-of-consciousness writing below. I had relatively minor surgery yesterday and am still recovering. I'm just trying to throw some ideas out there; use whatever you like.

Originally Posted by DiapDealer

1) Span Div Editor: conveniently remove, change the attributes of (and/or convert to different html elements) those pesky, nestable spans and divs (as well as a few others).

Now that sounds neat. So if I've got a bunch of, say, <span class="italic"> or <div class="reallyaparagraph">, I could use this to switch those over to proper <i> and <p class="reallyaparagraph"> throughout the book without having to worry about nesting issues? And to hearken back to my own recent project, is it able to remove attributeless spans or (hope, hope) join adjacent identical spans?

Example of that last:

<span class="foo">blah blah </span><span class="foo">blah</span>

...would become, naturally:

<span class="foo">blah blah blah</span>

(And should such spans still count as adjacent when separated only by whitespace? I'd say yes, but there's certainly room for debate.)

Then again, what'd really be neat is the ability to correctly convert un-nested spans to nested ones. I see stuff like this all the time, particularly in ill-coded books that define a class for the paragraph, only to turn right around and wrap all of its text in a span that redefines everything...except it doesn't do a real wrap, because it doesn't nest spans:

<span class="foo">blah </span><span="ital">blah</span><span class="foo"> blah</span>

...which, of course, can usually be safely converted to:

<span class="foo">blah <span class="ital">blah</span> blah</span>

...although that feature may merit a note about unintended side-effects due to cascading, come to think of it.

Speaking of which - it'd really be handy to be able to say "all text files that call spreadsheet X" in search/replace. It's not an extremely common case, but if you're dealing with omnibus collections, it can be quite an issue. Just imagine one ebook containing three novels, each with a couple of dozen chapters and its own stylesheet...not fun. (Especially when the class names overlap but the classes' styles differ between sheets...)

Then again, I wish calibre's editor was generally more...is "CSS-aware" right? For instance, imagine being able to pick a class somewhere (in a sheet or on a page) and easily operate on it across the board, in the sheet which defines the class and in all the pages that use it. Rename it, replace it with another (existing or new) class, see if a class name is in use, convert a tag's style="definitions" into a class and make that change wherever it appears...that sort of thing. I mean, the editor's fine if you've got a basic ebook with one stylesheet, or multiple sheets with mutually exclusive class names or domains, but once there's any overlap, things get weird fast.

Originally Posted by DiapDealer

3) Convert CM to EM: this one was one I did for roger64. It simply parses CSS files looking for attributes that have cm dimensions and converts them to em (based on a preset factor). Figured I'd toss that in.

Any chance of expanding that to handle IN as well? And are you using the canonical "2.54 cm = 1 in = 6pc = 72 pt" conversion, or something else?

ADDENDUM: Also, 96px = 1in.

[DiapDealer]

Originally Posted by Rev. Bob

Now that sounds neat. So if I've got a bunch of, say, <span class="italic"> or <div class="reallyaparagraph">, I could use this to switch those over to proper <i> and <p class="reallyaparagraph"> throughout the book without having to worry about nesting issues?

Exactly. That's the plan anyway.

And to hearken back to my own recent project, is it able to remove attributeless spans or (hope, hope) join adjacent identical spans?

Yes to the first (one of my main uses, in fact), but unfortunately, no to the last. I can definitely see the merit in that, though. Would have to see if I can wrap my head around it.

Any chance of expanding that to handle IN as well? And are you using the canonical "2.54 cm = 1 in = 6em = 72 pt" conversion, or something else?

That's definitely in the ball park of what I'm doing, but I'll be honest ... someone gave me what they wanted (cm x 2.37106301584 = em) and I rolled with it. IN wouldn't be hard to add, though. I'd probably just need to add a gui to allow choosing the input dimension type. Right now, it only parses css files (long or short hand notation). It doesn't do style sections in the xhtml files yet. Simple css and @page rules, it should be able to handle. It won't be able to delve into @media rules, though.

[davidfor]

Originally Posted by DiapDealer

1) Span Div Editor: conveniently remove, change the attributes of (and/or convert to different html elements) those pesky, nestable spans and divs (as well as a few others).

Very nice and will save a lot of regex anguish. One thing is that I think <i> and <b> should be valid "Change tag to" values for a span. This is what the calibre editor uses when you press the "i" and "b" buttons.

2) Smarten Punctuation (the sequel): gives slightly more granular control over the the smartening process. It preserves pre-existing entities and lets you choose what you want to smarten. So if you prefer three periods to the ellipse character/entity, you can keep them if you like. Also lets you supply a text file that defines apostrophe exception words like 'cept and 'tis in order to get the apostrophe right (instead of an opening single quote). I'm attaching a sample file for that, but basically it's one word per line WITHOUT any apostrophe characters. You can also choose to go with SmartyPants' default of using numeric entities instead of unicode characters if you like.

I have been using a saved search for this. This is much better.

3) Convert CM to EM: this one was one I did for roger64. It simply parses CSS files looking for attributes that have cm dimensions and converts them to em (based on a preset factor). Figured I'd toss that in.

I can see the use, but I'm not sure if this is one I will use. As Rev. Bob said, handling other units would be good. But, there might be a problem with "pt". For pt, the font size might need to be treated differently to margins and other layout dimensions.

A very useful plugin. And the most impressive part, it even comes with "Help"

[DiapDealer]

Originally Posted by davidfor

Very nice and will save a lot of regex anguish. One thing is that I think <i> and <b> should be valid "Change tag to" values for a span. This is what the calibre editor uses when you press the "i" and "b" buttons.

If you're careful, you can edit the json settings file and add elements to the available "change to" options for the various elements. There's two json files in the 'plugin' folder of calibre's config directory for SpanDivEdit. The one you want ends in "settings". It won't exist until you've run the tool at least once, though. Add an "i" and a "b" to the "span_changes" list and you should be ready to rock. Throw in a "small" if you like.

A very useful plugin. And the most impressive part, it even comes with "Help"

Thanks. And I'll have you know a lot of thought went into those help pages.

[Rev. Bob]

Originally Posted by davidfor

I can see the use, but I'm not sure if this is one I will use. As Rev. Bob said, handling other units would be good. But, there might be a problem with "pt". For pt, the font size might need to be treated differently to margins and other layout dimensions.

Shouldn't be necessary, but I did make one mistake in my equation: "em" should be "pc" up there (and will be, shortly). The official list even defines 1in = 96px to make everything tidy.

[BobC]

Originally Posted by DiapDealer

In the spirit of getting some more Editor Plugin action happening, I'm attaching a plugin that has three different tools. All three can be set to run on the currently opened file or to run book-wide.

.....

Not sure this is ready for a Prime-time plugin listing, so I thought I'd put it here.

Nice. The cm to em will save me a lot of work when I build books using LibreOffice and export to EPUB - it will insist on using absolute measurements based on paper pages.

I need to play with the span/div one as it looks like a solution to another regular problem I have with badly formatted source EPUBS.

A big

BobC

[Terisa de morgan]

Originally Posted by DiapDealer

Good enough for me.

---------

In the spirit of getting some more Editor Plugin action happening, I'm attaching a plugin that has three different tools. All three can be set to run on the currently opened file or to run book-wide.

1) Span Div Editor: conveniently remove, change the attributes of (and/or convert to different html elements) those pesky, nestable spans and divs (as well as a few others).

2) Smarten Punctuation (the sequel): gives slightly more granular control over the the smartening process. It preserves pre-existing entities and lets you choose what you want to smarten. So if you prefer three periods to the ellipse character/entity, you can keep them if you like. Also lets you supply a text file that defines apostrophe exception words like 'cept and 'tis in order to get the apostrophe right (instead of an opening single quote). I'm attaching a sample file for that, but basically it's one word per line WITHOUT any apostrophe characters. You can also choose to go with SmartyPants' default of using numeric entities instead of unicode characters if you like.

3) Convert CM to EM: this one was one I did for roger64. It simply parses CSS files looking for attributes that have cm dimensions and converts them to em (based on a preset factor). Figured I'd toss that in.

Please scour the differences and make judicious use of the checkpoint system before trusting these tools on your babies!!

Not sure this is ready for a Prime-time plugin listing, so I thought I'd put it here.

I have played with the span/div option, and I've got a couple of "Index out of range". Is there any log I can get?

[DiapDealer]

Originally Posted by Terisa de morgan

I have played with the span/div option, and I've got a couple of "Index out of range". Is there any log I can get?

Launch the editor from the command-line with:

calibre-debug --edit-book

Hopefully it will spit out something useful (though I admit I didn't add as much debugging output as I probably should have).

I probably should have mentioned that while it shouldn't have much trouble with "ugly" markup, it's entirely likely that markup that's not well-formed could blow it up.

It's either that, or your ebook markup contains elements/attributes that require special handling that I haven't encountered in my own testing yet--which is entirely possible.

[phossler]

@DiapDealer --

In the spirit of getting some more Editor Plugin action happening, I'm attaching a plugin that has three different tools. All three can be set to run on the currently opened file or to run book-wide.

Sounds good. I spend way too much time just cleaning (mostly PD) epubs of unwanted and overly complex formatting just to make it readable. Usually due to being Calibre converted multiple times and to/from various formats (I think the record was 5 TOC XHTML files in one book)

Q1 - can you point me to a link as to how to install the PI in Win7 64 bit please?

When I'm trying to clean up the formatting in a book, I usually try to consolidate a lot of 'embedded in the text' formatting into a style sheet, as well as simplify. I don't really care (and who cares what the Kindle thinks) if some paragraphs have 1.25 line after and others have 1.26 line after.

This will leave me with a lot

of <span>blah blah</span> more blah blah <span>blah blah</span>

Q2 - sounds like your PI will strip the non-value add tags?

Q3 - could there be an option to remove some tag pairs separated by one (number could be settable?) character.

<p><b>I</b>n the beginning ..... </p>

Things like that might look ok in print, but IMHO it clutters the book

Q4 - Maybe and option to remove unused ID=" ..." ?

[DiapDealer]

Originally Posted by phossler

Q1 - can you point me to a link as to how to install the PI in Win7 64 bit please?

Editor plugins install just like any other manually installed calibre plugin. It's the same for all platforms: Preferences->Plugins->Load plugin from file. Then choose the zip file you downloaded.

Q2 - sounds like your PI will strip the non-value add tags?

It will strip the tags you tell it to strip--regardless of whether they add value or not. If you're talking about attributeless span tags, then yes. If you tell it to strip spans that have no attributes, it will do so. Nested or otherwise.

Q3 - could there be an option to remove some tag pairs separated by one (number could be settable?) character.

<p><b>I</b>n the beginning ..... </p>

Like Rev. Bob's idea of removing consecutive spans of the same class, this certainly has merit. I'm just not certain the logic required would lend itself well to the current structure and internal logic of the Span Div Editor's html parsing routine, though.

Q4 - Maybe and option to remove unused ID=" ..." ?

Same as above. It's certainly possible, but it would likely need to be part of a separate tool. Span Div Edit parses as it goes--one time through and it's all done. Removing "unused" IDs would require checking against the current file's css (inline or otherwise) and internal links, all the other (x)html files' links, as well as the opf and ncx files. It's a horse of a different color.

[Sabardeyn]

DiapDealer,
You have probably already considered this, but some potential issues that popped into my head.

What about conversational exchanges between characters like:
<p>"Yada, yada yada yada," he said.</p>
<p>"Blah! Blah, blah blah, blah blah blah</p>
<p>"Yada yada."</p>
While I've used paragraphs in this example they could be spans or divs. But wouldn't the non-value add stripping run the risk of merging this conversation into a single run-on comment by the first speaker?

Also the less frequently seen monologue of multiple paragraphs, which lacks a closing quotation mark until the final paragraph.

(I haven't downloaded the PI as yet - I've been busy at work - so perhaps these points are already accounted for.)

[eschwartz]

Possibly -- if the maker of the book formatted paragraphs using span tags. If so, the whole book would be shot to pieces and we'd need to kill the person who made it. And then revert it and fix every paragraph in the book.

No real solution for a situation like that -- we just have to assume the person using the plugin knows to check before overwriting the only copy.

[arspr]

Extremelly good plugin really, really useful.

Nevertheless some possible issues or suggestions.

Test the following text about smarten punctuation mark II:

  <p>''Tis is a dialogue' John said.</p>

  <p>''Whatever'' is not a dialogue.</p>

With an exception file which includes Tis.

If I tick that option I get:

  <p>‘’Tis is a dialogue’ John said.</p>

  <p>‘’Whatever’’ is not a dialogue.</p>

Why has the apostrophe before Whatever been converted to ’ ? (I can assure that Whatever is not in the exception file, in fact I've uploaded it).
And why has not the double apostrophe after Whatever been converted to a curly ” ?

But if I untick that option I get:

  <p>‘‘Tis is a dialogue’ John said.</p>

  <p>‘‘Whatever’’ is not a dialogue.</p>

Why duplicated single apostrophes are not collapsed into double ones?


And now a suggestion:

'90s is a decade contraction and it needs a right curly quote.
'90 is a number in a dialogue', John said, 'so it needs a left curly one.'

As coding an automate substitution rule for ' before numbers can be really hard, I propose asking the user each time an occurrence happens.



OTOH about the tag substitution feature:

• It only works on the current file, not in all text files. Why?
• I also vote for adding more options in addition to em & strong. Could you leave it open so any tag could be written?

[DiapDealer]

Definitely something up with that first example. I'll take a look. But as far as:

Originally Posted by arspr

Why duplicated single apostrophes are not collapsed into double ones?

That's never been a feature of the SmartyPants smartening algorithm. Unless you're talking about back-ticks. If so, I'm afraid I've purposely left the smartening of back-ticks (double or single) out of this entirely. I never run into back-ticks and it just confuses the hell out of things, frankly.

And now a suggestion:

'90s is a decade contraction and it needs a right curly quote.
'90 is a number in a dialogue', John said, 'so it needs a left curly one.'

That's why no automated smartening algorithm is going to be perfect.

As coding an automate substitution rule for ' before numbers can be really hard, I propose asking the user each time an occurrence happens.

But then that's not "automated." I'm just not that interested in pausing the processing for user interaction. That's something that S&R (regex or otherwise) is perfectly capable of doing. Basically, this is just a wrapper around SmartyPants (just like calibre's smartening implementation) that allows a little more granular control of SmartyPants and extends it's capabilities with the apostrophe exception file. Others are more than welcome to use what I've started and make it sing and dance to their own tune.

OTOH about the tag substitution feature:
It only works on the current file, not in all text files. Why?

Because you need to uncheck the box in the menu that limits it to the current file only.

I also vote for adding more options in addition to em & strong. Could you leave it open so any tag could be written?

As I mentioned above, other tags can be added to the json settings file to allow you to change the "replacement" tags to almost anything you want. There's just no nice gui dialog to do it for you.