[Metadata Source Plugin] Kobo Books

[davidfor]

I have just updated the plugin to fix the recent problems with connecting to the Kobo site. It appears to be some bot protection has been added.

The change is:

• Fix: Bot protection seems to have been added. Need to use a different user agent to get around it.

I don't know if this will last.

Calibre should announce the update in the next hour or so. If there are any problems, please report them here.

[soilheart]

Originally Posted by davidfor

Um, I think you want to allow localising the search to specific nations and languages. I have done this, and my testing didn't give any real advantage. And, if I remember rightly, caused as many issues as the current way of doing it.

Sorry for bringing up an older discussion, but I have a usecase for this. As the plugin works right now I get results from the Swedish Kobo store, which for some reason has replaced the # in the series name/number with "Nummer", so when getting data from the Swedish localization all my series end with "Nummer" which is quite annoying. For now I've downloaded the plugin and edited SEARCH_PATH by hand, which seems to have solved the issue for now. But adding a setting for selecting language (country doesn't seem as important, as it's the localization/translation that seems bad from what I can tell), would be appreciated so we with badly translated kobo stores don't have to manually update the plugin

[davidfor]

Originally Posted by soilheart

Sorry for bringing up an older discussion, but I have a usecase for this. As the plugin works right now I get results from the Swedish Kobo store, which for some reason has replaced the # in the series name/number with "Nummer", so when getting data from the Swedish localization all my series end with "Nummer" which is quite annoying. For now I've downloaded the plugin and edited SEARCH_PATH by hand, which seems to have solved the issue for now. But adding a setting for selecting language (country doesn't seem as important, as it's the localization/translation that seems bad from what I can tell), would be appreciated so we with badly translated kobo stores don't have to manually update the plugin

But that change would suggest you are not getting Swedish language for the synopsis and other parts. Is that what you really want? Adding that check is easy. The attached beta appears to work.

[soilheart]

Originally Posted by davidfor

But that change would suggest you are not getting Swedish language for the synopsis and other parts. Is that what you really want? Adding that check is easy. The attached beta appears to work.

Yes, as I read 99.9% of my books in English, I usually want to have the synopsis and other information in English anyway. From experience the synopsis isn't translated on the Swedish store either, so English books have English synopsis etc. (but I'm not 100% sure here, as I just started using this plugin for scraping data, and have only used the actual kobo store for purchases once or twice).

Thanks for the fix (I haven't tried it out yet though, as I'm still on my "custom" version right now).

[Shpounz]

Hello,

Looks like the plugin has stopped working (v1.8.3, calibre 5.37.0). You will find below a log file example (same with other books) :

calibre, version 5.37.0
ERREUR : Aucune correspondance trouvée: <p>Impossible de trouver un livre correspondant à votre recherche. Essayer de faire une recherche <b>moins spécifique</b>. Par exemple, utiliser seulement le nom de famille de l'auteur et un mot distinctif du titre.<p> Pour voir le journal en entier, cliquer "Afficher les détails.

Running identify query with parameters:
{'title': 'OLIVIER TWIST', 'authors': ['Charles Dickens'], 'identifiers': {}, 'timeout': 30}
Using plugins: Kobo Books (1, 8, 3)
The log from individual plugins is below

****************************** Kobo Books (1, 8, 3) ******************************
Found 0 results
Downloading from Kobo Books took 30.16534996032715
identify - title: "OLIVIER TWIST" authors= "['Charles Dickens']"
create_query - title: "OLIVIER TWIST"
create_query - after decode title: "OLIVIER TWIST"
create_query - title_tokens: "['OLIVIER', 'TWIST']"
Querying: https://www.kobo.com/search?Query=OL...clanguages=all
Failed to make identify query: 'https://www.kobo.com/search?Query=OLIVIER+TWIST+Charles+Dickens&fcmedia =Book&fclanguages=all'
Traceback (most recent call last):
File "mechanize\_urllib2_fork.py", line 1238, in do_open
File "http\client.py", line 1347, in getresponse
File "http\client.py", line 307, in begin
File "http\client.py", line 268, in _read_status
File "socket.py", line 669, in readinto
File "ssl.py", line 1241, in recv_into
File "ssl.py", line 1099, in read
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.kobobooks.__init__", line 173, in identify
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 287, in _mech_open
File "mechanize\_opener.py", line 193, in open
File "mechanize\_urllib2_fork.py", line 425, in _open
File "mechanize\_urllib2_fork.py", line 414, in _call_chain
File "calibre\utils\browser.py", line 28, in https_open
File "mechanize\_urllib2_fork.py", line 1240, in do_open
urllib.error.URLError: <urlopen error The read operation timed out>

************************************************** ******************************
The identify phase took 30.35 seconds
The longest time (30.165350) was taken by: Kobo Books
Merging results from different sources
We have 0 merged results, merging took: 0.00 seconds

[davidfor]

It isn't working here either. From a very quick look I did yesterday, it looks like Kobo have changed something in the site. I suspect they have changed how it is hosted or put something in front of it for protection. They have been using Akamai for hosting or protection, so they might have turned on more protection and this is blocking it.

I will have a look when I can, but, it will probably be a few days. And I won't be to surprised if it cannot be fixed.

[Shpounz]

Thanks for the feedback. Hopefully it can still work

[Uncle Robin]

Another "can't find" from Kobo. I bought this one a couple of days ago, and even tried manually adding the ISBN listed on its Kobo page:

Spoiler:

calibre, version 5.37.0
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters:
{'title': 'Death of a Delegate', 'authors': ['Cecilia Peartree'], 'identifiers': {'isbn': '9781005091491'}, 'timeout': 30}
Using plugins: Kobo Books (1, 8, 3), Google (1, 0, 3), Amazon.com (1, 2, 23)
The log from individual plugins is below

****************************** Kobo Books (1, 8, 3) ******************************
Found 0 results
Downloading from Kobo Books took 30.199804544448853
identify - title: "Death of a Delegate" authors= "['Cecilia Peartree']"
Querying: https://www.kobo.com/search?Query=97...clanguages=all
Failed to make identify query: 'https://www.kobo.com/search?Query=9781005091491&fcmedia=Book&fclanguage s=all'
Traceback (most recent call last):
File "mechanize\_urllib2_fork.py", line 1238, in do_open
File "http\client.py", line 1347, in getresponse
File "http\client.py", line 307, in begin
File "http\client.py", line 268, in _read_status
File "socket.py", line 669, in readinto
File "ssl.py", line 1241, in recv_into
File "ssl.py", line 1099, in read
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.kobobooks.__init__", line 173, in identify
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 287, in _mech_open
File "mechanize\_opener.py", line 193, in open
File "mechanize\_urllib2_fork.py", line 425, in _open
File "mechanize\_urllib2_fork.py", line 414, in _call_chain
File "calibre\utils\browser.py", line 28, in https_open
File "mechanize\_urllib2_fork.py", line 1240, in do_open
urllib.error.URLError: <urlopen error The read operation timed out>

************************************************** ******************************

****************************** Google (1, 0, 3) ******************************
Found 0 results
Downloading from Google took 3.2016921043395996
Making query: https://books.google.com/books/feeds...ewability=none
No results found, retrying with capitalized ISBN
Making query: https://books.google.com/books/feeds...ewability=none
No results found, retrying without identifiers
Making query: https://books.google.com/books/feeds...ewability=none

************************************************** ******************************

****************************** Amazon.com (1, 2, 23) ******************************
Found 0 results
Downloading from Amazon.com took 3.27168869972229
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
Server: auto
Making google query: https://www.google.com/search?q=%289...www.amazon.com
Failed to find any results on results page, with title: (9781005091491 or 978-1005091491) site:www.amazon.com - Google Search
No search engine results for terms: (9781005091491 OR 978-1005091491)
Trying the bing search engine instead
Making bing query: https://www.bing.com/search?q=%28978...www.amazon.com
Failed to find any results on results page, with title: (9781005091491 or 978-1005091491) site:www.amazon.com - Search
No search engine results for terms: (9781005091491 OR 978-1005091491)
No matches found with identifiers, retrying using only title and authors. Query: ['(9781005091491', 'OR', '978-1005091491)']
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
Server: auto
Making google query: https://www.google.com/search?q=Deat...www.amazon.com
Failed to find any results on results page, with title: Death of Delegate Cecilia Peartree site:www.amazon.com - Google Search
No search engine results for terms: Death of Delegate Cecilia Peartree
Trying the bing search engine instead
Making bing query: https://www.bing.com/search?q=Death+...www.amazon.com
Failed to find any results on results page, with title: Death of Delegate Cecilia Peartree site:www.amazon.com - Search
No search engine results for terms: Death of Delegate Cecilia Peartree
No matches found with query: ['Death', 'of', 'Delegate', 'Cecilia', 'Peartree']

************************************************** ******************************
The identify phase took 30.39 seconds
The longest time (30.199805) was taken by: Kobo Books
Merging results from different sources
We have 0 merged results, merging took: 0.00 seconds

I then realized I hadn't added book 2 in the series, so did that one, and Calibre did find metadata - from Amazon. A pox on Kobo for breaking this very useful plugin!

[ZodWallop]

Glad to see it's not just me that is no longer getting results from Kobo.

[davidfor]

As I said above, it is a change in the site. Or how it is hosted. I have not had much time for it, but, it doesn't return anything when I use curl to get the page. That will make it very hard to fix and might mean it cannot be.

[Uncle Robin]

Originally Posted by davidfor

As I said above, it is a change in the site. Or how it is hosted. I have not had much time for it, but, it doesn't return anything when I use curl to get the page. That will make it very hard to fix and might mean it cannot be.

Hence my imprecations against Kobo. Your plugin was invaluable, especially since I buy maybe 3 books a year from any site other than Kobo.

[13xforever]

interestingly, my proposed changes still work fine (I had to remove your user agent override from 1.8.3 though)

in __init__.py:

    def set_extra_headers(self, br):
        br.set_current_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/*,*/*;q=0.8')
        br.set_current_header('Accept-Language', 'en-US,en-UK,en;q=0.9,ja;q=0.8')
        br.set_current_header('Accept-Encoding', 'gzip')

and then later in identity: and download_cover: functions change
br = self.browser to br = self.set_extra_headers(self.browser)

also from testing, `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15` user agent seems to be the most reliable at the moment, but yes, it's a game of cat and mouse when there's no documentation and protection can change any moment

[JSWolf]

Originally Posted by 13xforever

interestingly, my proposed changes still work fine (I had to remove your user agent override from 1.8.3 though)

in __init__.py:

    def set_extra_headers(self, br):
        br.set_current_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/*,*/*;q=0.8')
        br.set_current_header('Accept-Language', 'en-US,en-UK,en;q=0.9,ja;q=0.8')
        br.set_current_header('Accept-Encoding', 'gzip')

and then later in identity: and download_cover: functions change
br = self.browser to br = self.set_extra_headers(self.browser)

also from testing, `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15` user agent seems to be the most reliable at the moment, but yes, it's a game of cat and mouse when there's no documentation and protection can change any moment

Where does the code in the code block go?

[13xforever]

It's a new function similar to others, but I wouldn't advise you to modify the code if you do not understand what you're doing as you can break it completely instead

[JSWolf]

Originally Posted by 13xforever

It's a new function similar to others, but I wouldn't advise you to modify the code if you do not understand what you're doing as you can break it completely instead

I can modify the code as long as I know where to put the new function. Where does it go?