[GUI Plugin] Search the Internet

[Puciek]

Helo
With update of plugin i have this error:

calibre, version 0.8.15
ERROR: Nieznany wyjątek: <b>UnicodeEncodeError</b>:'latin-1' codec can't encode character u'\u0142' in position 65: ordinal not in range(256)

Traceback (most recent call last):
  File "calibre_plugins.search_the_internet.action", line 110, in search_web_link
  File "calibre_plugins.search_the_internet.action", line 116, in search_web_for_book
  File "calibre_plugins.search_the_internet.action", line 143, in open_tokenised_url
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u0142' in position 65: ordinal not in range(256)

I'm from Poland and i use letters: ą, ż, ź, ć, ł, ę, ó, ś, ń. Title or author without this letters are ok.
Sorry for my english...

[kiwidude]

@Puciek - welcome to MobileRead.

As for your error - I guess it must be a side-effect of the changes made to the safe_format() functions that I had to switch between since Calibre code got ripped out I used to use.

There is quite possibly a simple fix I can make in my code to workaround this but hopefully chaley or Kovid can tell me what it is. If they happen to stumble across this thread, the code is effectively doing this:

url = template_formatter.safe_format(tokenised_url, fixed_vals, 'STI template error', mi)
open_url(QUrl.fromEncoded(url))

It is that second line where it tries to encode the result from safe_format that is blowing up. The value of url at that point would likely be something like:

http://www.foo.com/myquery?search=ęń

[kovidgoyal]

The changes to safe_formatter were for thread safety, I don't see how they could possibly have any bearing on this. Why are you using QUrl.fromEncoded

As per the Qt documentation: Parses input and returns the corresponding QUrl. input is assumed to be in encoded form, containing only ASCII characters.

[kiwidude]

I only mentioned safe_formatter because the user said the problem occurred only when they upgraded to the latest version. The latest version only had a one line change of the import statement from the old safe_formatter that got deleted to the new one. Perhaps the user was wrong and this was always broken for this particular scenario, or perhaps there has been some other change to Qt. Or perhaps they were on a REALLY old version of this plugin.

I should have posted extra lines to give more info. The reason I use QUrl.fromEncoded is because the Url is not exactly like how I posted. Instead there are two further complications:

(1) The url may contain quotes as part of it's template specification, encoded as %22

http://www.google.com/#sclient=psy&q=%22{author}%22+%22{title}%22

(2) The text being injected into the template by safe_format gets "encoded" usually with utf-8 but for some websites like Amazon it is latin-1:

text = quote_plus(text.encode(encoding, 'ignore'))

Now the original way I had of opening urls (using webbrowser.open(url) rather than using Calibre's open_url function) coped with all of this perfectly fine. Then many months ago when I changed to use Calibre's open_url due to someone reporting a problem on some flavour of linux, I found that I could not just do open_url(QUrl(url)) because the %22 stuff does not translate.

So instead I was using QUrl.fromEncoding(url) - which solves the %22 stuff but apparently will barf on the other "encoded" foreign characters...

[kovidgoyal]

Use the unquote function from urllib and then QUrl(unquoteed_url)

[kiwidude]

Thx Kovid. Unfortunately the problem with doing that is it "undoes" the other stuff that the quoting is trying to achieve. Take for example this book author/title:

E. M. Cioran De l'inconvénient d'être né

To search Amazon.com.fr for that book, what I need supplied is for the URL to be in this format:

http://www.amazon.fr/s/ref=nb_sb_noss?url=search-alias%3Dstripbooks&field-keywords=E.%20M.%20Cioran+De%20l%27inconv%E9nient%20d%27%EAtre%20n%E9

My code I listed above using open_url(QUrl.fromEncoded(url)) does exactly that.

However if instead I use open_url(QUrl(unquote(url))) then what gets generated is this:

http://www.amazon.fr/s/ref=nb_sb_noss?url=search-alias=stripbooks&field-keywords=E.%20M.%20Cioran+De%20l%27inconv%C3%A9nient%20d%27%C3%AAtre%20n%C3%A9

which corresponds to something like:
"E. M. Cioran De l'inconvénient d'être né"

My head hurts

[Puciek]

Originally Posted by kiwidude

...Perhaps the user was wrong and this was always broken for this particular scenario...

No, using the previous version I added to about 500 titles covers and metadata using google and others portals

Originally Posted by kiwidude

...Or perhaps they were on a REALLY old version of this plugin...

Last working version of the plugin was 1.6.5

[kovidgoyal]

Try

QUrl.fromEncoded(quote(unquote(url)))

The proper fix is to not have partially quoted URLs in the first place. Start with keeping your URLs pure strings with no quoting, and only quote them just before calling open_url.

[kiwidude]

Thanks but that doesn't work either I'm afraid. All very frustrating for something that should be so freaking simple!

I have a template url which could be something like:

http://www.google.de/#sclient=psy&q=%22{author}%22+%22{title}%22

And I have data which I want to be substituted into those {xxx} tags. My original implementation just substituted in the "quoted" values (after any unicode encoding), so straight away I had the final string ready to just pass to a web-browser.

I think it is the open_url function requiring a QUrl which is making this all go pear shaped, unlike webbrowser.open() call which took the string directly.

To keep as a string and delay quoting, that would imply my starting point is something like this:

http://www.google.de/#sclient=psy&q="{author}"+"{title}"

However applying the quote function to the entire string results in a string that has all sorts of characters encoded - like the : after http etc etc. So that isn't going to work.

[kovidgoyal]

quoting is only supposed to apply to the path component of a URL. So you'd do it like this:

1) Start unencoded

2) Use urlparse.urlsplit to split the url into components.

3) Quote the correct components (path and query) encoded as UTF-8

quote(component.encode('utf-8'))

4) Use urlparse.urlunsplit to reconstitute the url

5) Use QUrl.fromEncoded

[kovidgoyal]

And IIRC for amazon rather than using utf-8 you want latin1

[kiwidude]

Thx for your patience Kovid. I'm trying what you suggest, but still no joy. The problem is that I have the "query" part from urlsplit, which will look like this:

'q="Abraham, Daniel"+"An Autumn War"'

Now if I do as you suggest and use quote(parts[3].encode(encoding, 'ignore')) that will turn into something like this:
'q%3D%22Abraham%2C%20Daniel%22%2B%22An%20Autumn%20 War%22'

So it is encoding things like the = and + signs.

[kovidgoyal]

Then do the quoting in the format() function.

final_url=template.format(title=quote(titlevar), author=quote(authorvar))

[Doctor Vapor]

Excellent, thanks

[kiwidude]

@Kovid - but there's a hole in my bucket...

How is that any different to what I began with? Other than what you show does not include the .encode()?

I'm close to the point of saying "sod the linux users" and putting it back to webbrowser.open().