Metadata search fails on many books

[lbutlr]

I’ve been having a lot of books fail to find any metadata, and some are books that are easily found on amazon, amazon uk, good reads, wikipedia, etc. The latest example of a relatively popular book is The Winter King by Cornwell, not an obscure book if not a NYT best seller, but it has ~50K ratings on Goodroeads, at least eight editions, and it is definitely on amazon. When trying to search for metadata it doesn’t even seem to try to search, just pops up a failure notice immediately.

I recently completely rebuilt my library, and I have a lot of books now that are not showing any metadata that I am certain DID contain valid meta data before, including all of the Cornwell Arthur series and Last Kingdom series, only some go which hav metadata now. Are there plugins that I should be adding to 6.1 that maybe I had in 5.x and have forgotten about?

[kovidgoyal]

Look at the log in the metadata download dialog.

[digitalw00t]

I blanked out all information, and put isbn:<number>, and it still shows zero search results, yet I can pull the information up via manual amazon or google search. An example is isnb: 9781568817118 for 3d Engine Design for Virtual Globes. (purchased through humblebundle for those curious)
I can pull up the information manually and copy/paste, but thought I'd chime in on this issue.

---------------------------------------------------------
calibre, version 6.2.1
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters:
{'title': 'tmpnXC8HQ', 'authors': None, 'identifiers': {'isbn': '9781568817118'}, 'timeout': 30}
Using plugins: Google (1, 0, 6), Amazon.com (1, 2, 28)
The log from individual plugins is below

****************************** Google (1, 0, 6) ******************************
Found 0 results
Downloading from Google took 1.9009332656860352
Making query: https://www.google.com/search?q=9781568817118&tbm=bks
Plugin Google failed
Traceback (most recent call last):
File "calibre\ebooks\metadata\sources\identify.py", line 47, in run
File "calibre\ebooks\metadata\sources\google.py", line 472, in identify
File "calibre\ebooks\metadata\sources\google.py", line 411, in identify_via_web_search
File "calibre\ebooks\metadata\sources\search_engines.py ", line 88, in query
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 429: Too Many Requests

************************************************** ******************************

****************************** Amazon.com (1, 2, 28) ******************************
Found 0 results
Downloading from Amazon.com took 0.47011470794677734
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36
Server: auto
Making google query: https://www.google.com/search?q=%289...www.amazon.com
Plugin Amazon.com failed
Traceback (most recent call last):
File "calibre\ebooks\metadata\sources\identify.py", line 47, in run
File "calibre\ebooks\metadata\sources\amazon.py", line 1536, in identify
File "calibre\ebooks\metadata\sources\amazon.py", line 1469, in search_search_engine
File "calibre\ebooks\metadata\sources\search_engines.py ", line 331, in google_search
File "calibre\ebooks\metadata\sources\search_engines.py ", line 88, in query
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 429: Too Many Requests

************************************************** ******************************
The identify phase took 2.00 seconds
The longest time (1.900933) was taken by: Google
Merging results from different sources
We have 0 merged results, merging took: 0.00 seconds

[kovidgoyal]

See the line

mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 429: Too Many Requests


you have made too many requests so google is blocking you. Just wait for a few hours and you will be unblocked.

[lbutlr]

I assume something similar was happening to me since when I checked today it found the metadata right away. It seems odd though, I checked on maybe 50 books yesterday which seems like an absurdly low amount to be hitting a limit on google. Also seems odd that both google and Amazon would fail at the same time (unless Calire aborts when google fails?).

A better error message in the UI would be most welcome.

[kovidgoyal]

Both amazon and google are searched via google. You can change the search engine used for amazon in Preferences->Metadata download but google generally works best.

As for a better error message, the complexity involved in doing that in this case is much higher than you might think.

[CookiePuss]

The same thing is happening to me too. I don't know what changed as I too used to be able get metadata for many books. Now after I get metadata for about 4 books in a row I get blocked by google from downloading any more. It's super annoying and only started a few days ago.

[theducks]

This may not be a Calibre (caused) issue:

Questions for those with recent fails:

1)Have you adjusted smaller, the tweak that affects time between requests. (NOT recommended making it lower: could get Calibre banned)

2)How many books WERE you doing in the group at a time?

3)How many groups THAT PERIOD before a fail?

[lbutlr]

All I can say is that this did not happen with version 5.x even when searching for metadata on hundreds of books. Coincidence, perhaps, but I find that very unlikely. Also, I did not have any issue with Google in the browser when this was not working in Calibre. 🤷🏻*♀️

[theducks]

Glad you have it figured.
For a bit, I thought the site had changed in a way that affected how Calibre scraped data.

[datostar]

I discovered today that several attempted metadata searches, not just on Amazon and Google failed with no results available. I had left on my internet VPN. Turning it off solved the problem. Possibly some tracking via IP address or location didn't work for them.

[catsknit]

I'm having the same issue, for the first time since I began using Calibre years ago. I usually process my books (import, then find metadata, primarily comments and covers, put in my own tags) in batches about once a month or so. Usually there are 10-15 books, sometimes more and I've never had it time out like this before. This time it stopped working after 5 books. And although I do now see that line about too many requests, the error message which pops up tells you to try searching with fewer parameters, nothing about too many requests. So of course I kept trying over and over, which has probably upped my number of requests even higher!
I'll try again later.

[CookiePuss]

Same here catsknit. I had no idea what the problem was until I checked the more info button to find the too many requests error. At first I had no idea what was going on, just that every single book came back with no metadata results.

Even when I use a VPN I can only grab metadata for about 4 books, then I get blocked by Google and no metadata gets returned for anything. If I close Calibre and switch IPs I can get metadata for 4 more books before being blocked again.

This isn't even large batch downloads, just 4 separate books individually. I don't know if Google changed something or if it's Calibre but it never happened before, even when grabbing metadata for dozens of books.