Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 01-17-2018, 09:21 PM   #1
skipster10
Member
skipster10 began at the beginning.
 
Posts: 23
Karma: 10
Join Date: Jan 2013
Device: iPad Pro 10.5
iOS 11 Smart Punctuation Fouls Content Server Searches

Huge fan of Calibre (3.15 Win x64), but I'm getting regular complaints about this...

iOS 11 (e.g., iPad/iPhone) introduced "Smart Punctuation" that auto-replaces straight quotes with "Smart Quotes" and made this the default setting. This is actually useful in many cases/applications, but the Calibre Content Server search function requires straight quotes as delimiters (e.g., title:"this and that").

iOS doesn't have the ability to insert by character code or a Character Map applet, so these aren't viable work-arounds. Further, since the Content Server interface is designed to be accessed by regular users (not necessarily technically sophisticated), suggesting regular expressions or other non-intuitive work-arounds that the user must remember isn't viable. I'd also prefer not to instruct all users to change a default setting on their iDevice just to access the Calibre content server.

It'd be VERY useful if there was a tweak or Search preference that allowed the Search capability to treat smart quote delimiters as straight quote delimiters for users that desired. If it exists, I haven't found it yet.

Any suggestions?

Last edited by skipster10; 01-17-2018 at 09:25 PM.
skipster10 is offline   Reply With Quote
Old 01-17-2018, 11:16 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Sigh. Trust Apple to implement something like this with no way to turn it off on a per input field basis in Safari.

Two possible solutions:

1) Just blindly replace smart dquotes in the search expression before sending it to the server. Simple to implement, but it breaks searching for terms that contain smart dquotes.

2) Actually modify the search expression parser to handle smart dquotes as quotation delimiters. Since, IIRC, the parser was written by chaley, in its current form, he should comment on how feasible this is.
kovidgoyal is offline   Reply With Quote
Advert
Old 01-18-2018, 02:03 AM   #3
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
@chaley: I just pushed an implementation of supporting smart quotes as quote delimiters. I'd appreciate it if you could review it.
kovidgoyal is offline   Reply With Quote
Old 01-18-2018, 05:53 AM   #4
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 11,691
Karma: 6240117
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
@kovid: I am looking at it now and I am confident that it does what you intend, but the approach makes me nervous for several reasons:
  • From what I see via Google, the "smart quote" stuff only happens when you are typing, but doesn't happen when you paste. As such there is no guarantee that the quotes are balanced. Example: if I paste comments:" then type foo" then I will get comments:"foo”. The parser will fail.
  • Because the change is in the parser it affects all searches even if the source doesn't use smart quotes, such as the calibre app search bar and OPDS clients. Similar things might happen with machine-generated searches.
  • It changes the meaning of some searches. For example, today I can search for comments:“foo” and find “foo”. With this change I must search for comments:"“foo”".
  • iOS apparently also messes around with dashes, automatically converting double hyphens (--) to dashes (—). Some people say that just for fun iOS tosses in 0x00 bytes after them.
  • Apparently Apple is doing the same thing with other locales, for example changing "foo" to « foo » in a French locale or „foo“ in Germany.
Conclusion: changing the search/query parser introduces unexpected behavior and doesn't solve the problem for non-English locales. Instead it creates a never-ending headache as Apple continues to mess with it.

I think it would be better to deal with the problem directly in the content server, replacing "smart" quotes if the user agent is an iOS (or perhaps any Apple) device. That way you can:
  • limit the damage to one input source (content server searching)
  • limit the damage to one user agent if it can be trusted.
  • use a brute force replacement algorithm instead of trying to balance the things. By this I mean replace all “ and ” with " and so on.
  • deal with converting m-dashes back to -- (or not)
  • take locale into consideration (or not).
  • do the dirty in the browser or in the content server itself, whichever is better.
BTW: According to this thread, it is possible to turn off smart quotes on a field-by-field basis. I don't know if that level of control is made visible somehow to javascript in Safari.
chaley is offline   Reply With Quote
Old 01-18-2018, 06:41 AM   #5
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 11,691
Karma: 6240117
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
FWIW: The ICU tools include a transliterator called "Any-Publishing" that converts publishing punctuation to typewriter punctuation. For example, it converts ‘“foo”’ to '"foo"'. It might make sense to use it in a variant of "primary_contains" so searching ignores the accent types. I don't understand enough about how you did the ICU python bindings to know a) if this idea is feasible or b) how to do it if it is feasible.

It also might make sense to use something like the transliterator in the content server to process the search strings.
chaley is offline   Reply With Quote
Advert
Old 01-18-2018, 06:44 AM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by chaley View Post
BTW: According to this thread, it is possible to turn off smart quotes on a field-by-field basis. I don't know if that level of control is made visible somehow to javascript in Safari.
Yeah, I saw that, but I dont know of any way to do it in Safari via JS. If Apple are really using locale dependent quotes, then I agree, it's too much disruption, just to work-around Apple's idiocy. Probably best to use option 1 (dumb replacement in the browser when the user agent indicates iOS).

Last edited by kovidgoyal; 01-18-2018 at 06:46 AM.
kovidgoyal is offline   Reply With Quote
Old 01-18-2018, 06:52 AM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I tried this on an iPad and you can type normal quotes by long tapping the quote button. So I dont think we need to do anything.
kovidgoyal is offline   Reply With Quote
Old 01-18-2018, 09:25 AM   #8
skipster10
Member
skipster10 began at the beginning.
 
Posts: 23
Karma: 10
Join Date: Jan 2013
Device: iPad Pro 10.5
Kovid:

Good find on the long-press. Since this work-around requires the end-user to be aware of smart-quote problem and how to workaround it, Content Server end-user education is required. I’m not a novice, but with my eyesight and the size of an iDevice screen, I couldn’t see the smart quotes and it took me a while to figure out why my searches weren’t finding known entries in the database when accessing from iPad. If the user doesn’t have prior knowledge of the database contents, they may not be aware that their search results aren’t reflecting as intended.

My preference would be to solve the issue for the end user so no end-user knowledge is required, but if this workaround is the final answer, can the content server at least throw a warning and instructions to the user if the user agent is iOS and a search is entered with smart quotes?

Thanks much for a fantastic tool that we use daily!

Last edited by skipster10; 01-18-2018 at 09:31 AM.
skipster10 is offline   Reply With Quote
Old 01-18-2018, 11:50 AM   #9
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Sure, adding a warning is worthwhile. THough really, the fix is for apple to allow a website to specify that it does not want smart anything in an input field.
kovidgoyal is offline   Reply With Quote
Old 01-19-2018, 11:47 PM   #10
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
https://github.com/kovidgoyal/calibr...37d6d84aa1f66c
kovidgoyal is offline   Reply With Quote
Old 10-08-2019, 09:17 AM   #11
skipster10
Member
skipster10 began at the beginning.
 
Posts: 23
Karma: 10
Join Date: Jan 2013
Device: iPad Pro 10.5
It seems that version 4.0 lost the ability to handle iOS smart quotes in searches. The prior solution of asking to replace smart quotes worked well.

Last edited by skipster10; 10-08-2019 at 09:20 AM.
skipster10 is offline   Reply With Quote
Old 10-08-2019, 09:38 AM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,744
Karma: 22446736
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
There weren't any changes to that code in 4, what exactly do you mean by lost the ability?
kovidgoyal is offline   Reply With Quote
Old 10-08-2019, 09:42 AM   #13
skipster10
Member
skipster10 began at the beginning.
 
Posts: 23
Karma: 10
Join Date: Jan 2013
Device: iPad Pro 10.5
Prior to 4, if a search on the content server contained smart quotes (i.e., iOS default), the content server detected the smart quotes and queried the user if smart quotes should be treated as regular quotes. With 4.0, no query is thrown and the search result just finds nothing.
skipster10 is offline   Reply With Quote
Old 10-08-2019, 10:05 AM   #14
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 73,510
Karma: 126422064
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3
Quote:
Originally Posted by skipster10 View Post
Prior to 4, if a search on the content server contained smart quotes (i.e., iOS default), the content server detected the smart quotes and queried the user if smart quotes should be treated as regular quotes. With 4.0, no query is thrown and the search result just finds nothing.
The solution is so very simple. Just turn off smart punctuation.

https://www.howtogeek.com/344310/how...hone-and-ipad/
JSWolf is offline   Reply With Quote
Old 10-08-2019, 10:17 AM   #15
skipster10
Member
skipster10 began at the beginning.
 
Posts: 23
Karma: 10
Join Date: Jan 2013
Device: iPad Pro 10.5
Understood that it can be turned off. It is, however, the default setting for iOS. Thus, most users will have smart punctuation turned on and will not know how to turn it off. Furthermore, a search response with nothing returned is misleading when, in-fact, the issue is the smart punctuation. The former solution of prompting the user was an excellent solution that cautioned the end-user who didn't know any better. (Note: When upgrading to Version 4, even my first response was "Where'd all my books go?" when my searches started returning the null set. I'd gotten so used to the prompt...)

Last edited by skipster10; 10-08-2019 at 10:20 AM.
skipster10 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
3.0.0 Content Server and iOS version TomCombs Devices 15 04-18-2018 11:16 AM
Content Server View Books In Safari Issue With IOS 11 theengine Devices 1 09-28-2017 05:50 AM
Calibre Content Server Searches Page r_sauve Recipes 7 08-08-2016 01:27 PM
Accessing Calibre content server from iOS safari with username & password in address Richs Calibre 0 01-18-2014 02:02 PM
Trying to get to Content Server on OS X 10.7.2 using iPad 2 iOS 5.0.1 cdg Apple Devices 0 11-15-2011 03:25 PM


All times are GMT -4. The time now is 03:02 AM.


MobileRead.com is a privately owned, operated and funded community.