A situation that breaks "smarten punctuation"

[frostschutz]

Well, to be fair, there simply shouldn't be any spaces. Same way there shouldn't be spaces before ? ! . ,

Cause it looks like crap when the line happens to break there and the punctuation just goes into the next line
, because there was a space in front of it.

The smartypants code only looks at chars before/after the ", so it relies on the formatting alone. It does not have a logic that makes the first " in a paragraph opening, the next one closing, etc.

[HarryT]

Originally Posted by frostschutz

Well, to be fair, there simply shouldn't be any spaces. Same way there shouldn't be spaces before ? ! . ,

Cause it looks like crap when the line happens to break there and the punctuation just goes into the next line
, because there was a space in front of it.

The smartypants code only looks at chars before/after the ", so it relies on the formatting alone. It does not have a logic that makes the first " in a paragraph opening, the next one closing, etc.

When you have single quotes nested inside double quotes (or vice versa), though, it is conventional to separate them with a space. Without the space, it's very difficult to visually separate them.

[frostschutz]

Originally Posted by HarryT

When you have single quotes nested inside double quotes (or vice versa), though, it is conventional to separate them with a space.

I've looked at some of my paper books. For example in Harry Potter 5 there is no space whatsoever, it looks like this:

'I got a "P",' said Ron

'"Troll",' said George promptly

The only reason there is a space between " and ' is due to the comma that's pulled inside the quote. Otherwise there is no space whatsoever so the opening really looks like '''.


In Gregor and the Curse of the Warmbloods it looks like this:

"'Turn and turn and turn again,'" she said.

Now in that book there is a slight space between " and ', ' and ". But it's only hinted at, a regular space (between two words) is about three times as wide. So I believe this mini space to be courtesy of the font used by that book. Either that or there is a typographic extra slim space character.

I haven't found a book where it's actually a space. If it's actually necessary, a nonbreakable space should be used instead, as it otherwise can turn out ugly depending on screen / font size.

[HarryT]

It's clearly a space in my printed "Sherlock Holmes" books, and I think personally it looks a lot better like that - it's much easier to read. I guess it's an editorial choice at the end of the day.

[Perkin]

I've got a complete collection of Sherlock Holmes, reprints of the Strand magazines.
As well as what HarryT says there's a space following any open quote, also has spaces between words an question/exclamation marks and also another space if there's a following quote.
From A Study In Scarlet

" There is no clue ? " said Gregson.
" None at all," chimed in Lestrade.

Just also checked one of the Agatha Christie's I've got, and that is the same as well.
From The Murder at the Vicarage.

" So you're back at last," she observed.
" Is Colonel Protheroe here ? " I asked.

Quote-marks are actually curly type.

[frostschutz]

That's horrible.

I couldn't read an ebook like that. I'd have to fix this with search&replace first.

[dwig]

Originally Posted by frostschutz

Well, to be fair, there simply shouldn't be any spaces. Same way there shouldn't be spaces before ? ! . ,

Agreed, they shouldn't be there but it does happen.

The smartypants code only looks at chars before/after the ",

... but in looking at the characters after the quote it fails to recognize that it's at the end of a paragraph and should handle the quote differently. It acts as if there were printable characters after the quote. Quote + End of paragraph should result in Right Quote regardless of what occurs before the quote

[frostschutz]

Yes, sure. One could also argue that it should never close a quote when there it has no opening quote to partner it. But that's simply not how the code works at the moment. It's called smart punctuation but it's really just a simple heuristic which may or may not work.

It actually works quite well all things considered, but with strange formatting such as these space characters it fails.

Also I'm not sure if the original code has a concept of paragraphs at all, looks more like it's originally designed for plain text, such as what people write in a textarea form field.

[JSWolf]

Originally Posted by dwig

While you're at it, take a look at the inverse of HarryT's issue.

I've seen a reliable error occur when a quoted sentence ends with a space before the closing quote.

This (in the source document's HTML):

<p>He said, "Go away — "</p>

becomes (in the MOBI or ePub conversion):

He said, “Go away — “

instead of:

He said, “Go away — ”

If the trailing space is omitted the smart quotes function works correctly.

Where the em dash is are grammatically incorrect spaces. The spaces should not be there. So I'm not blaming the conversion but the source.

[dwig]

Originally Posted by JSWolf

Where the em dash is are grammatically incorrect spaces. The spaces should not be there.

It is not at all "grammatically" incorrect. It is, instead, typographically incorrect to use a full width space before and after the em dash. I listed them only because then can exist and do trip up smartypants' logic.

Em dashes should be wrapped with a thin space if the glyph isn't designed to provide the space. Leaving out the space is a modern electronic font habit since may fonts and layout programs kern (letterspace) the character properly with a space. Traditionally (read: hand set metal type) a thin space should be added before and after an em dash. Since most fonts in ereaders seem to not provide any kerning/letterspacing and since most ereaders won't break a line at an em dash when there is no space, ebooks should have a space before and after an em dash when it occurs in its proper position between two words.

The examples I listed occur when my S&R sweeps miss this inappropriate use of trailing em dashes. Using such em dashes seems to have been common with some writers and publishers. In truth, they should be ellipses. I some times change them and sometimes retain the original typography. When my S&R sweeps work correctly, the trailing em dashes are spaced with a non-breaking space to avoid the evil "only the punctuation wraps" error, omitting the trailing space before the end punctuation.

[Perkin]

Fixes for following should be in next release.

John said, “The man said ‘aaaaa’
” ‘bbbb’
” ‘cccc’
” ‘dddd’
” and then ended the story.”

He said, "Go away — "</p>

becomes (in the MOBI or ePub conversion):

He said, “Go away — “

instead of:

He said, “Go away — ”

Also fix for incorrect quotes when as following

A double quote—"within dashes"—would be nice.
A single quote—'within dashes'—would be nice.

@HarryT, (and others) can you test after next release and let me know if everything is now correct for you. (Works for me)

[Perkin]

@HarryT, and others, can you test and let me know if everything is now correct for you.

[HarryT]

Sorry, I've been away for the last week. Do I just download the latest version of Calibre and test with that?

[Perkin]

Yes, the changes went into the latest version. Can you test on file/s which was/were showing incorrect smartened quotes.

Thanks.

[dwig]

Originally Posted by Perkin

Yes, the changes went into the latest version. Can you test on file/s which was/were showing incorrect smartened quotes.

Thanks.

In 0.8.51 it still fails with a closing italics (</ i> or </ em>) tag just before the close quote. See attached examples (original tick-mark quotes in ePub and conversion to MOBI with smart quotes)