[Editor Plugin] Otter's Editing Toolbag

[Lukusaukko]

There is now a new version with various fixes, some additions and UI clarifications. See the first two posts in this thread for the plugin, changes from last public release, and feature list.

Normalization now recognizes non-Unicode whitespace, and can itself use either narrow non-breaking space or non-breaking space to attach ellipses or dashes to preceding word to prevent line wrapping at the whitespace when using open style. Also supports adding non-braking whitespace between nested quote characters. I experimented with thin space and hair space in that role as well, but those are breaking whitespace and could result in in awkward line breaks in some circumstances.

I'm still on the fence if I should add support for word joiners to attach closed em dashes to preceding words - on one hand, em dashes often result in ugly line breaks with em dash starting a new line; on the other hand, this should really be something for the renderer to handle, and invisible control characters are rarely a good idea in text, plus it might result in even more awkward word spacing because em dash is quite wide.

[Lukusaukko]

New hotfix version 0.5.6: Previous 0.5.5 ellipse normalization inserted a thin space instead of a normal space after ellipses when the ellipse was at a DOM boundary ("word… <em>word</em>) and the attachment marker dropdown had narrow style selected - even if it was inactive and the attachment style was without markers.

This is now fixed, and hopefully didn't introduce any other regressions.

[JSWolf]

Why is this not in the list of plugins in calibre?

[Lukusaukko]

Originally Posted by JSWolf

Why is this not in the list of plugins in calibre?

I was thinking that until I'm fairly sure the basic functionality is guaranteed to work as intended, it would not be a good idea to include it there because people who do not read Mobileread would not know about possible issues. I'm assuming people who install it from the forums would be interested enough in providing feedback and help ironing out the bugs, so that once it's installable from Calibre's plugin browser, it would not have any showstoppers.

[JSWolf]

Originally Posted by Lukusaukko

I was thinking that until I'm fairly sure the basic functionality is guaranteed to work as intended, it would not be a good idea to include it there because people who do not read Mobileread would not know about possible issues. I'm assuming people who install it from the forums would be interested enough in providing feedback and help ironing out the bugs, so that once it's installable from Calibre's plugin browser, it would not have any showstoppers.

I've been using it and it's working well.

[JSWolf]

After testing the smarten punctuation, it could use some work.

The ellipses support and em dash support could use need fixing. You should have options to remove spaces around em dashes and ellipses. Also, an option to go from en dash to em dash.

Punctuation support could use an option to go from UK style quotes to US style quotes and US style to UK style.

After these fixes are done, it's ready to go.

[Lukusaukko]

Originally Posted by JSWolf

After testing the smarten punctuation, it could use some work.

The ellipses support and em dash support could use need fixing. You should have options to remove spaces around em dashes and ellipses. Also, an option to go from en dash to em dash.

Punctuation support could use an option to go from UK style quotes to US style quotes and US style to UK style.

After these fixes are done, it's ready to go.

There is an option to remove spaces around em dashes in Cleanup (it's intentional - smartening does the inital sweep and is close to what Calibre's internal smartening or Diaps original smartening does; it basically passes the text to Smartypants with some tweaks), but further fixing is delegated to Cleanup. But you are right, the wording is not obvious: if you select em dash, closed style then Remove markers and use ordinary spaces, it removes whitespace from em dashes. It doesn't support removing spaces from around en dashes, because that could introduce ambiguity in the text; is "word<en dash>word" a parenthetical dash or a range (e.g. "London–Glasgow route"). Maybe I could support that and then leave it to the user. Em dash <-> en dash could be a new option, with the same caveat; you might end up accidentally converting something like "10–12" to "10 — 12" or to "London–Glasgow route" to "London — Glasgow route".

Supporting quote conversion could be trickier, because the UK single primary quote style is easily confused with apostrophes when you are running an automatic conversion - there are a lot of edge cases where you need to figure out whether something is a closing quote or an apostrophe. Not impossible, I think, but something that merits a deeper look than it appears on the surface.

[JSWolf]

Originally Posted by Lukusaukko

There is an option to remove spaces around em dashes in Cleanup (it's intentional - smartening does the inital sweep and is close to what Calibre's internal smartening or Diaps original smartening does; it basically passes the text to Smartypants with some tweaks), but further fixing is delegated to Cleanup. But you are right, the wording is not obvious: if you select em dash, closed style then Remove markers and use ordinary spaces, it removes whitespace from em dashes. It doesn't support removing spaces from around en dashes, because that could introduce ambiguity in the text; is "word<en dash>word" a parenthetical dash or a range (e.g. "London–Glasgow route"). Maybe I could support that and then leave it to the user. Em dash <-> en dash could be a new option, with the same caveat; you might end up accidentally converting something like "10–12" to "10 — 12" or to "London–Glasgow route" to "London — Glasgow route".

What I've tested of Cleanup is working well. I'll test it some more.

There is one annoyance with both Smarten and Cleanup. I find it annoying that the option window opens when I press the icon. I'd rather pressing the icon just do it and the arrow have the option to change the settings. Like Edit Spans & Divs does.

Oh and there should be a separate option for en-dash and em-dash intead of having to keep changing the setting.

Supporting quote conversion could be trickier, because the UK single primary quote style is easily confused with apostrophes when you are running an automatic conversion - there are a lot of edge cases where you need to figure out whether something is a closing quote or an apostrophe. Not impossible, I think, but something that merits a deeper look than it appears on the surface.

I was hoping it would be not too difficult to do this. I just wish the UK would standardize on the US was and be done with it.

[Lukusaukko]

Discovered a few new and exciting bugs, but it will take a bit of time to fix and test. Maybe towards the end of the week.

• Books authored with HTML named character references (&mdash;, &nbsp;, &hellip; ) were not actually processed and could lose text (ampersands) in the worst case. This requires a book authored with HTML named entities such as &mdash; or &nbsp;. Files using only real Unicode characters, numeric references, or XML's five predefined entities (&amp;, &lt;, &gt;, &apos;, &quot; ) are unaffected. Losing ampersands additionally requires an undefined entity, which in EPUB 3 is an epubcheck error (operations on valid EPUB3 files should be safe). Relatively rare, but possible.
• If a whole book operation fails in the middle, reverting completely to previous state might not work. Shouldn't be an issue if the operation fully completes.
• With Use unicode characters instead of entities unticked, punctuation inside `style`, `script`, `pre`, `code`, `kbd`, `samp` and `var` was rewritten as numeric references, not left as is as would've been the intent. Rare-ish.

These are something that occur in relatively rare circumstances, but still worth fixing. Once these potentially destructive bugs are squashed, I will probably put this up into the index so it can be installed via the plugin manager interface.