Sample Plugin for the Editor

[DiapDealer]

Originally Posted by Sabardeyn

DiapDealer,
You have probably already considered this, but some potential issues that popped into my head.

What about conversational exchanges between characters like:
<p>"Yada, yada yada yada," he said.</p>
<p>"Blah! Blah, blah blah, blah blah blah</p>
<p>"Yada yada."</p>
While I've used paragraphs in this example they could be spans or divs. But wouldn't the non-value add stripping run the risk of merging this conversation into a single run-on comment by the first speaker?

My parsing method makes no "value" judgement. IF you had something like:

<span>"Yada, yada yada yada," he said.</span>
<span>"Blah! Blah, blah blah, blah blah blah</span>
<span>"Yada yada."</span>

and you tell the tool to delete any spans that have no attributes at all ... it will dutifully do so. The assumption here is that you KNOW what it is you want to delete/modify. But if you do happen to fudge something up, the differences dialog will show you what change were made and give you the option to cancel/revert those changes. It also sets a check point so you can return to the state before you ran the tool.

Also the less frequently seen monologue of multiple paragraphs, which lacks a closing quotation mark until the final paragraph.

that's the smartening tool. And it handles that situation pretty well.

[mrmikel]

The Jones boys (see Last Crusade) would be glad to know the quest for the holy grail of brainless epub conversion still is alive!

[arspr]

Originally Posted by DiapDealer

That's never been a feature of the SmartyPants smartening algorithm. Unless you're talking about back-ticks. If so, I'm afraid I've purposely left the smartening of back-ticks (double or single) out of this entirely. I never run into back-ticks and it just confuses the hell out of things, frankly.

Err, I asked this because Calibre does actually change them and I thought you were improving Calibre "native" process...

Originally Posted by DiapDealer

Because you need to uncheck the box in the menu that limits it to the current file only.

Originally Posted by DiapDealer

As I mentioned above, other tags can be added to the json settings file to allow you to change the "replacement" tags to almost anything you want. There's just no nice gui dialog to do it for you.

Ooops, I missed it...

Thanks for the hint. I've found it.

Nevertheless, a nice gui dialog perhaps?

[phossler]

Originally Posted by mrmikel

The Jones boys (see Last Crusade) would be glad to know the quest for the holy grail of brainless epub conversion still is alive!

More like 'The Temple of Doom'

[DiapDealer]

Originally Posted by arspr

Err, I asked this because Calibre does actually change them and I thought you were improving Calibre "native" process...

Are you certain we're talking about the same thing? If you mean that calibre's stock smartener will take two consecutive strait single quotes ('') and always collapse those into one curly double quote--well that's never been my experience. If you mean something else--like backticks; then yes, I made a conscious decision to not include support for ``stuff like this'' being converted into opening and closing curly double quotes.

I'll certainly look into making it possible for someone to turn that feature on if they like, but it's just not something I've ever needed. And in my testing, it usually introduced more anomalies than it was worth.

Nevertheless, a nice gui dialog perhaps?

I'll think about it.
But I've sort of made that "feature" a little difficult to implement on purpose. The potential for trouble is already there with the ability to change div to p. But if at possible, I'd like to avoid having to answer questions about why someone's ebook broke after they replaced a perfectly valid nested structure with html elements that cannot be nested.

Those with the knowledge to edit a json file will more likely know what they're doing with regard to markup.

[arspr]

Originally Posted by DiapDealer

That's never been a feature of the SmartyPants smartening algorithm. Unless you're talking about back-ticks. If so, I'm afraid I've purposely left the smartening of back-ticks (double or single) out of this entirely. I never run into back-ticks and it just confuses the hell out of things, frankly.

Well, I've just run Calibre's Smarten punctuaction (the curly double quotation mark button in the Editor toolbar) on that same text and I've gotten:

  <p>”Tis is a dialogue’ John said.</p>

  <p>”Whatever” is not a dialogue.</p>

which is quite odd in any case... It seems that a pair of ' is always converted into a curly right double quotation mark.

Is everything fine? Or is there some kind of bug here?

[Terisa de morgan]

I have a question and a request (if it's posiblie):

• When you select id in the options to clean a tag, the id must be a fixed string or it can be a regex?
• Request: Could you add <a> to the tags you clean (although without replacement tags, of course). It would be a nice feature for me.

Thank you very much

[DiapDealer]

Originally Posted by arspr

Well, I've just run Calibre's Smarten punctuaction (the curly double quotation mark button in the Editor toolbar) on that same text and I've gotten:

  <p>”Tis is a dialogue’ John said.</p>

  <p>”Whatever” is not a dialogue.</p>

which is quite odd in any case... It seems that a pair of ' is always converted into a curly right double quotation mark.

Is everything fine? Or is there some kind of bug here?

I don't understand what characters you're using for single-quotes (pre-smartening). When I try your examples, and I start with single, straight quotes ('):

<p>''Tis is a dialogue' John said.

<p>''Whatever'' is not a dialogue.</p>

my smarten tool produces exactly what I expect it to. None of my consecutive straight, single quotes are being converted into curly right double quotes.

Let's try this: I'm attaching a sample epub that contains your examples (and some of my own). I'm also attaching the same epub AFTER my smartening routine has processed it (using my original apostrophe exception file, all boxes checked, and the dashes combobox set to the middle of the three options). Please make sure you're getting the same results.

EDIT: ooh ... I see now what you mean about calibre's regular smartening routine. It DOES turn those consecutive straight single quotes into right curlies. I don't think it always did that.

EDIT 2: After studying SmartyPants again, the attempt to smarten backticks is exactly why all consecutive single quotes get converted to a right (closing) curly double quote -- and exactly why I chose to turn that feature off.

[DiapDealer]

Originally Posted by Terisa de morgan

I have a question and a request (if it's posiblie):

When you select id in the options to clean a tag, the id must be a fixed string or it can be a regex?

It's currently fixed only. I've been thinking about adding regex support there, but I'm hesitant to open that can of worms (hesitant meaning nothing else will get done in my life until I get it working ).

Request: Could you add <a> to the tags you clean (although without replacement tags, of course). It would be a nice feature for me.


Thank you very much

I've been tempted to do that myself (<a> tags). I just need to figure out how to handle the situation of having elements that can ONLY be deleted and not modified.

EDIT: did you ever get any more details on the "index out of range" error you were getting? I should really post a version that has more in the way of debug output.

[crutledge]

Would it be possible to publish Smarten Punctuation (the sequel) as a stand alone??

[Terisa de morgan]

Originally Posted by DiapDealer

It's currently fixed only. I've been thinking about adding regex support there, but I'm hesitant to open that can of worms (hesitant meaning nothing else will get done in my life until I get it working ).

I totally understand

Originally Posted by DiapDealer

I've been tempted to do that myself (<a> tags). I just need to figure out how to handle the situation of having elements that can ONLY be deleted and not modified.

Well, it will be a nice addition if you do it.

Originally Posted by DiapDealer

EDIT: did you ever get any more details on the "index out of range" error you were getting? I should really post a version that has more in the way of debug output.

I haven't tested it more. I am on vacation and preparing a travel (I begin my travel tomorrow at 7:20AM), so I'll test it after it.

[DiapDealer]

Originally Posted by crutledge

Would it be possible to publish Smarten Punctuation (the sequel) as a stand alone??

I guess that all depend on your idea of "stand alone." If you mean a stand alone with all of the functionality and features it currently enjoys as being a calibre plugin, then that's not very likely. A big part of the motivation for creating it was the fact that calibre was going to be doing a lot of the heavy lifting: maintaining/saving/opening the ebook container object; parsing all of its files into usable objects with descriptive attributes; side-by-side, visual comparison of the results (with the ability to roll them back). I just had to build an options widget and then hand the files off to a slightly modified SmartyPants script, and save what it gave back.

A stand alone that allowed a user to choose their options, and then feed an html file (or files) to my modified SmartyPants script which then outputs new file(s) isn't out of the question. But that's probably about it. It'd take me a decade of Sundays to adapt all of the functionality that calibre contributes to my little project into a stand alone tool.

[arspr]

Originally Posted by DiapDealer

Let's try this: I'm attaching a sample epub that contains your examples (and some of my own). I'm also attaching the same epub AFTER my smartening routine has processed it (using my original apostrophe exception file, all boxes checked, and the dashes combobox set to the middle of the three options). Please make sure you're getting the same results.

EDIT: ooh ... I see now what you mean about calibre's regular smartening routine. It DOES turn those consecutive straight single quotes into right curlies. I don't think it always did that.

EDIT 2: After studying SmartyPants again, the attempt to smarten backticks is exactly why all consecutive single quotes get converted to a right (closing) curly double quote -- and exactly why I chose to turn that feature off.

Good to know you've found the reason. Do you think it should be considered a bug and notified to Kovid in order to also disable it?

In addition, do you still need me to test your examples? (I've been too lazy to do it yet... )

Originally Posted by DiapDealer

Definitely something up with that first example. I'll take a look.

Have you found what it is happening here?

(Remember that Whatever is not in the exception list but it is also modified as if it were...)

[DiapDealer]

Originally Posted by arspr

Have you found what it is happening here?

(Remember that Whatever is not in the exception list but it is also modified as if it were...)

I can't get my smartening routine to duplicate what you claim it does with your sample. My "Whatever" is not treated like it's in the exceptions file. The straight single quote in front of "Whatever" gets converted to a left curly single quote like it should. See the sample epubs I uploaded.

[arspr]

Originally Posted by DiapDealer

I can't get my smartening routine to duplicate what you claim it does with your sample. My "Whatever" is not treated like it's in the exceptions file. The straight single quote in front of "Whatever" gets converted to a left curly single quote like it should. See the sample epubs I uploaded.

I'll have to... (in the end...) I'll do it ASAP.

Nevertheless one hint: I mounted my attached exception file from your example (downloaded from the initial post). And when I opened it (Windows Notepad), all the words were in the very same line with some kind of non-printable character between them. So I immediately thought about the UNIX-Windows CR/LF (or whatever) conflict and retyped those "return key" symbols in their place. So my file should be a pure genuine windows friendly one. Could it be the source of the trouble?

EDIT:
I've downloaded your books and ALSO my exceptions file (just in case). If I run the routine I get it wrong:

  <p>Bob said; “This is an example of quoted speech that ’specially goes on longer than one paragraph.</p>

  <p>“Then it resumes in the next paragraph where the I said; ’give ’em hell, Harry!’ and then it concludes.”</p>

  <p>“Then it resumes in the ’80s and the ’90s—”</p>

  <p>&nbsp;</p>

  <p>---’I hope it works to … everyone’s satisfaction.’</p>

  <p>‘’Tis is a dialogue’ John said.</p>

  <p>‘’Whatever’’ is not a dialogue.</p>

Look at "give", "I" and "Whatever"...

(Using Editor comparison tool, those are the only differences with your _after.epub book).