No longer able to download metadata from Google, Amazon, etc

[Joseph The Grave]

For some reason, I can no longer download metadata for any new (or old) ebooks-- it says "ERROR: NO Matches Found". However, I know there is metadata available because it works normally for a backup of my Calibre installation, but not my primary Calibre installation. It happened right after I imported a bunch of books. What causes something like this and is there a fix or a way to reset it?

[kovidgoyal]

look at the log in the metadata download window.

[Joseph The Grave]

Originally Posted by kovidgoyal

look at the log in the metadata download window.

Here's what it says:

Spoiler:

calibre, version 5.44.0
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters:
{'title': '1632862441 (N)', 'authors': ['Dylan Jones'], 'identifiers': {'isbn': '9781408860571'}, 'timeout': 30}
Using plugins: Google (1, 0, 4), Amazon.com (1, 2, 28), Edelweiss (2, 0, 1)
The log from individual plugins is below

****************************** Google (1, 0, 4) ******************************
Found 0 results
Downloading from Google took 0.6545839309692383
Making query: https://www.google.com/search?q=9781408860571&tbm=bks
Plugin Google failed
Traceback (most recent call last):
File "calibre\ebooks\metadata\sources\identify.py", line 47, in run
File "<string>", line 469, in identify
File "<string>", line 409, in identify_via_web_search
File "<string>", line 77, in query
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 429: Too Many Requests

************************************************** ******************************

****************************** Amazon.com (1, 2, 28) ******************************
Found 0 results
Downloading from Amazon.com took 0.5835056304931641
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15
Server: auto
Making google query: https://www.google.com/search?q=%289...www.amazon.com
Plugin Amazon.com failed
Traceback (most recent call last):
File "calibre\ebooks\metadata\sources\identify.py", line 47, in run
File "calibre\ebooks\metadata\sources\amazon.py", line 1536, in identify
File "calibre\ebooks\metadata\sources\amazon.py", line 1469, in search_search_engine
File "<string>", line 323, in google_search
File "<string>", line 77, in query
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals >.httperror_seek_wrapper: HTTP Error 429: Too Many Requests

************************************************** ******************************

****************************** Edelweiss (2, 0, 1) ******************************
Found 0 results
Downloading from Edelweiss took 0.024933815002441406
Currently Edelweiss returns random books for search queries

************************************************** ******************************
The identify phase took 0.83 seconds
The longest time (0.654584) was taken by: Google
Merging results from different sources
We have 0 merged results, merging took: 0.00 seconds

Did something maybe get corrupted in a plugin when I imported those ebooks? I can always go back to a working backup but I've since imported about 20 new books and would prefer not to have to import them again and do the metadata thing. Plus, I'd like to know what happened so I can prevent/fix it in the future. The download metadata feature is very convenient and useful and I use it all the time..

[Joseph The Grave]

Okay, I just fixed it on my own. Though it's suppose to be portable (the version I'm using), I noticed it will sometimes create 3 folders into my "C:\Users\username\AppData\Local" directory. All I had to do was delete those folders and now I can download metadata normally. This is good to know for whenever it happens again. An easy fix.

[theducks]

The last line
Too many requests

That means someone overloaded the server and they shut down the service for a while.
If it was you (we have no way to know WHO / why triggered that), remember to space out requests in the future to avoid this.

[Joseph The Grave]

Originally Posted by theducks

The last line
Too many requests

That means someone overloaded the server and they shut down the service for a while.
If it was you (we have no way to know WHO / why triggered that), remember to space out requests in the future to avoid this.

That seems logical but then when I tried retreiving metadata with a backup it worked normally, then when I went back to my primary installation, I got the error again But deleting the 3 folders mentioned in my post above seemed to completely fix it. However, I will be careful not to overload it in the future just in case it was me.

[kovidgoyal]

google's blocking (and google is nowadays used to search both amazon and google books) is based on IP address. If there are too many requests from a aprticular IP address it will temporarily block it. Just wait a few hours and it will get unblocked again.

[Ben Gruagach]

I'm not convinced this is an overuse issue unless Google has instituted a brand new policy which has just kicked in.

I've been experiencing the same problem since upgrading to Calibre 6.1 this week. This was never a problem in previous versions. I haven't changed my routine with Calibre - I'm not suddenly requesting many more metadata requests than what I have done in the past yet now I'm finding they just don't work after a mere handful of requests.

This makes calibre far less useful as many of us are managing (and trying to tidy up) large collections of ebooks accumulated over the years.

[kovidgoyal]

The error message is literaaly saying too many requests from the google servers. If you arent convinced by that, I dont know what to tell you.

[ownedbycats]

Originally Posted by kovidgoyal

google's blocking (and google is nowadays used to search both amazon and google books) is based on IP address. If there are too many requests from a aprticular IP address it will temporarily block it. Just wait a few hours and it will get unblocked again.

Also—this has happened to me in the past—it's possible that your internet provider gave you an IP address that had previously been used by a bad actor and got flagged as dodgy, causing it to be a lot more quick to block you. Depending on your provider, disconnecting your modem for a bit and then re-connecting can get you a new one.

[datostar]

Same might occur if using a VPN. Happened to me earlier today. No problem with the VPN off.

[Ben Gruagach]

Question: how many books can you request metadata for before you run into these automatic fails? For me it seems to allow me to get the metadata for five books and then any subsequent attempts automatically fail. Eventually (not sure if it's hours, or a day later) it lets me get more, but again only for a maximum of five books before it starts failing again.

Is anyone else able to get metadata for more than five or so books successfully at a go?

[Joseph The Grave]

Update: yeah, my "workaround" of deleting the username app data calibre folder doesn't actually work. It's back to its bad behavior. Using my ISP-issued IP, I can sometimes only download 2 or 3 metadatas before I get the error. However, using a VPN I can get a few more. Another workaround. I wonder if this is only a temporary thing with google since I only noticed it the last few days. It's annoying whatever it is.

[theducks]

Originally Posted by Joseph The Grave

Update: yeah, my "workaround" of deleting the username app data calibre folder doesn't actually work. It's back to its bad behavior. Using my ISP-issued IP, I can sometimes only download 2 or 3 metadatas before I get the error. However, using a VPN I can get a few more. Another workaround. I wonder if this is only a temporary thing with google since I only noticed it the last few days. It's annoying whatever it is.

Where are you (country)?
I ask, because I am not seeing any issue on queries for 4-6 books (I am going back and cleaning 'reviews' for just a good synopsis on some of my early entries.

[Joseph The Grave]

Originally Posted by theducks

Where are you (country)?
I ask, because I am not seeing any issue on queries for 4-6 books (I am going back and cleaning 'reviews' for just a good synopsis on some of my early entries.

I'm from the U.S. The only thing slightly odd about my configuration for my ISP's IP is that I'm using Google's public DNS servers (ie 8.8.8.8, 8.8.4.4) instead of the ones assigned by my ISP.