A situation that breaks "smarten punctuation"

HarryT · 03-12-2012, 01:17 PM

If you have a series of paragraphs in a book which are a continuing quote (ie paragraphs which don't have a close quote), and then within this, have quoted speech, the "internal" quotes always end up wrong if you use Calibre's "smarten punctuation" option.

Eg:

John said, "The man said 'aaaaa'
" 'bbbb'
" 'cccc'
" 'dddd'
" and then ended the story."

In the second and subsequent paragraphs, the "internal" quotes end up the wrong way around. Eg, 'bbbb' ends up starting with a close quote, and ending with an opening quote.

Unfortunately this is a quote style that's used extensively in the "Sherlock Holmes Omnibus" that I maintain here on MR.

Is it worth reporting this as a bug? I believe it's not actually Calibre code that does this, but an external library, right?

Billi · 03-12-2012, 02:33 PM

Does this site help you, especially the part beneath "Algorithmic Shortcomings"?

http://daringfireball.net/projects/smartypants/

HarryT · 03-12-2012, 02:48 PM

Thanks, Billi. Yes, that "shortcoming" is exactly the issue that's affecting me. Unfortunately in some of the Sherlock Holmes stories, this quoting style goes on for page after page.

HarryT · 03-12-2012, 04:28 PM

Actually reading it more carefully, I'm not sure it is the same thing. They are talking about the first quote in a sentence being converted to a left quote when it should be a right. I'm seeing the opposite.

Billi · 03-12-2012, 05:08 PM

Quote:

Originally Posted by HarryT

In the second and subsequent paragraphs, the "internal" quotes end up the wrong way around. Eg, 'bbbb' ends up starting with a close quote, and ending with an opening quote.

Does it end up this way: „‘bbb’ instead of „’bbb‘ (or the other way round, don't know how it is exactly in English)?
Then perhaps you can change the opening part with a simple search/replace for „‘ and the second with a regex search.

frostschutz · 03-12-2012, 05:15 PM

Quote:

Originally Posted by HarryT

John said, "The man said 'aaaaa'
" 'bbbb'
" 'cccc'
" 'dddd'
" and then ended the story."

For me the result is

Quote:

John said, “The man said ‘aaaaa’
” ‘bbbb’
” ‘cccc’
” ‘dddd’
” and then ended the story.”

So it turns around the ", not the '... which is not much better.

HarryT · 03-12-2012, 05:16 PM

I'm confusing myself now as to how it looks; I just remember that it doesn't get it right. I'll have to try it again and see in what exact way it breaks.

EDIT: I see that frostschutz has tried it. That's right - it's the opening double quotes it gets wrong. I imagine it's probably because it can't cope with the quotes in the preceding paragraph not being closed. Interestingly enough, though, if you remove the "inner" single quotes, then it DOESN'T mess up - it gets it right. ie, it copes OK with

"The start of a quote.
"Which goes on
"and on
"and then finishes."

but if you add the single quotes within the double quotes, it messes it up.

Billi · 03-12-2012, 06:09 PM

But again, can't you use a search/replace function?

- You could search for the "false closing quotes", they have a blank or nothing left to them.
- Or you could search for the "right closing quotes" and mask them, they always have a sign (letter, point, question mark...) left to them.
- Or you can search directly for this combination ”‘, it shouldn't occur on other places.

At least in theory this could work...

frostschutz · 03-12-2012, 06:16 PM

It considers those " as closing because a space is following them.

Without the space, it works.

HarryT · 03-12-2012, 06:21 PM

Ah, that would explain it. Thanks!

Yes, probably pretty easy to fix with search and replace.

Perkin · 03-13-2012, 06:30 AM

HarryT, what formats are you converting to/from?

HarryT · 03-13-2012, 06:54 AM

This is a Mobi to Mobi conversion. I don't think that matters, though; "Smarten Punctuation" is format-neutral.

Perkin · 03-13-2012, 07:32 AM

HarryT, I've got a fix that seems to work. I'll send it to user_none for his approval and it should be in future release.

HarryT · 03-13-2012, 07:33 AM

Great! Thanks - that's really good to hear.

dwig · 03-13-2012, 09:54 AM

Quote:

Originally Posted by Perkin

HarryT, I've got a fix that seems to work. I'll send it to user_none for his approval and it should be in future release.

While you're at it, take a look at the inverse of HarryT's issue.

I've seen a reliable error occur when a quoted sentence ends with a space before the closing quote.

This (in the source document's HTML):

<p>He said, "Go away — "</p>

becomes (in the MOBI or ePub conversion):

He said, “Go away — “

instead of:

He said, “Go away — ”

If the trailing space is omitted the smart quotes function works correctly.

03-12-2012, 01:17 PM	#1
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383043 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	A situation that breaks "smarten punctuation" If you have a series of paragraphs in a book which are a continuing quote (ie paragraphs which don't have a close quote), and then within this, have quoted speech, the "internal" quotes always end up wrong if you use Calibre's "smarten punctuation" option. Eg: John said, "The man said 'aaaaa' " 'bbbb' " 'cccc' " 'dddd' " and then ended the story." In the second and subsequent paragraphs, the "internal" quotes end up the wrong way around. Eg, 'bbbb' ends up starting with a close quote, and ending with an opening quote. Unfortunately this is a quote style that's used extensively in the "Sherlock Holmes Omnibus" that I maintain here on MR. Is it worth reporting this as a bug? I believe it's not actually Calibre code that does this, but an external library, right?

03-12-2012, 05:16 PM	#7
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383043 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	I'm confusing myself now as to how it looks; I just remember that it doesn't get it right. I'll have to try it again and see in what exact way it breaks. EDIT: I see that frostschutz has tried it. That's right - it's the opening double quotes it gets wrong. I imagine it's probably because it can't cope with the quotes in the preceding paragraph not being closed. Interestingly enough, though, if you remove the "inner" single quotes, then it DOESN'T mess up - it gets it right. ie, it copes OK with "The start of a quote. "Which goes on "and on "and then finishes." but if you add the single quotes within the double quotes, it messes it up. Last edited by HarryT; 03-12-2012 at 05:21 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Smarten punctuation slows conversion	kakkalla	Calibre	22	01-20-2011 07:22 AM
Configuration of "Smarten Punctuation"	Korben Dallas	Calibre	2	12-31-2010 07:12 AM
Simpler Way to Smarten Punctuation	Rand Brittain	Calibre	3	10-10-2010 08:16 PM
Thanks for new 'Smarten Punctuation' feature	jackie_w	Calibre	1	09-21-2010 02:53 PM
"The Situation" by Jeff VanderMeer - free download	Kingston	Deals and Resources (No Self-Promotion or Affiliate Links)	10	03-17-2008 11:39 PM

03-12-2012, 02:33 PM	#2
Billi Wizard Posts: 3,388 Karma: 14190103 Join Date: Jun 2009 Location: Berlin Device: Cybook, iRex, PB, Onyx	Does this site help you, especially the part beneath "Algorithmic Shortcomings"? http://daringfireball.net/projects/smartypants/

03-12-2012, 02:48 PM	#3
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383043 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	Thanks, Billi. Yes, that "shortcoming" is exactly the issue that's affecting me. Unfortunately in some of the Sherlock Holmes stories, this quoting style goes on for page after page.

03-12-2012, 04:28 PM	#4
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383043 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	Actually reading it more carefully, I'm not sure it is the same thing. They are talking about the first quote in a sentence being converted to a left quote when it should be a right. I'm seeing the opposite.

03-12-2012, 06:09 PM	#8
Billi Wizard Posts: 3,388 Karma: 14190103 Join Date: Jun 2009 Location: Berlin Device: Cybook, iRex, PB, Onyx	But again, can't you use a search/replace function? - You could search for the "false closing quotes", they have a blank or nothing left to them. - Or you could search for the "right closing quotes" and mask them, they always have a sign (letter, point, question mark...) left to them. - Or you can search directly for this combination ”‘, it shouldn't occur on other places. At least in theory this could work...

03-12-2012, 06:16 PM	#9
frostschutz Linux User Posts: 2,279 Karma: 6123806 Join Date: Sep 2010 Location: Heidelberg, Germany Device: none	It considers those " as closing because a space is following them. Without the space, it works.

03-12-2012, 06:21 PM	#10
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383043 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	Ah, that would explain it. Thanks! Yes, probably pretty easy to fix with search and replace.

03-13-2012, 06:30 AM	#11
Perkin Guru Posts: 655 Karma: 64171 Join Date: Sep 2010 Location: Kent, England, Sol 3, ZZ9 plural Z Alpha Device: Sony PRS-300, Kobo Aura HD, iPad (Marvin)	HarryT, what formats are you converting to/from?

03-13-2012, 06:54 AM	#12
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383043 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	This is a Mobi to Mobi conversion. I don't think that matters, though; "Smarten Punctuation" is format-neutral.

03-13-2012, 07:32 AM	#13
Perkin Guru Posts: 655 Karma: 64171 Join Date: Sep 2010 Location: Kent, England, Sol 3, ZZ9 plural Z Alpha Device: Sony PRS-300, Kobo Aura HD, iPad (Marvin)	HarryT, I've got a fix that seems to work. I'll send it to user_none for his approval and it should be in future release.

03-13-2012, 07:33 AM	#14
HarryT eBook Enthusiast Posts: 85,544 Karma: 93383043 Join Date: Nov 2006 Location: UK Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6	Great! Thanks - that's really good to hear.

Advert

Advert