Download metadata and covers issues

[tomsem]

I'm trying to go through my library and add descriptions and other metadata using this feature. It's great, when it works.

However, something like 10-20% fail without any explanation as to the nature of the failure. I've tried tweaking the download behavior, but it hasn't really helped. Usually I'm usually only requesting metadata, not covers (I like to add covers in the metadata editor).

Also it seems very week at some things, like detecting and adding Series information.

So I wind up going to the Amazon book page (most of my purchases originate there) copying HTML and pasting it, adding some tags, etc., manually.

Is there some logging I can turn on?

Where is the source code for this? I might be able to at least figure out where it is getting tripped up, if not work on a PR.

[kovidgoyal]

Click the view log button to see the logs and if you want to read the source look in the metadata/sources directory of the calibre source code.

[tomsem]

I've been mostly using bulk update (command D) so was just seeing failure dialog with no log button while other jobs progressed (haven't been doing this for very long).

Sometimes the original ASIN is no longer in use, so failure may be expected. But this one is still good, so it should work.

#################### Log for The Book of Secrets ####################
Running identify query with parameters:
{'title': 'The Book of Secrets', 'authors': ['Fiona Kidman'], 'identifiers': {'mobi-asin': 'B00ADXVATE', 'isbn': '9781775533559'}, 'timeout': 30}
Using plugins: Google (1, 1, 1), Amazon.com (1, 3, 2)
The log from individual plugins is below

****************************** Google (1, 1, 1) ******************************
Found 0 results
Downloading from Google took 1.4956276416778564
Making query: https://books.google.com/books/feeds...ewability=none
No results found, doing a web search instead
Making query: https://www.google.com/search?q=9781775533559&tbm=bks
Plugin Google failed
Traceback (most recent call last):
File "calibre/ebooks/metadata/sources/identify.py", line 47, in run
File "calibre/ebooks/metadata/sources/google.py", line 534, in identify
File "calibre/ebooks/metadata/sources/google.py", line 439, in identify_via_web_search
File "calibre/ebooks/metadata/sources/search_engines.py", line 93, in query
File "mechanize/_mechanize.py", line 241, in open_novisit
File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class..httper ror_seek_wrapper: HTTP Error 429: Too Many Requests

************************************************** ******************************

****************************** Amazon.com (1, 3, 2) ******************************
Found 0 results
Downloading from Amazon.com took 98.48389458656311
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36
Server: ddg
Making ddg query: https://duckduckgo.com/html/?q=B00AD...azon.com&kp=-1
Failed to find cached page for: https://www.amazon.com/Book-Secrets-.../dp/B00ADXVATE
Failed to find cached page for: https://www.amazon.com/-/es/Fiona-Ki.../dp/B00ADXVATE
No search engine results for terms: B00ADXVATE
No matches found with identifiers, retrying using only title and authors. Query: ['B00ADXVATE']
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36
Server: ddg
Making ddg query: https://duckduckgo.com/html/?q=Book+...azon.com&kp=-1
Failed to find cached page for: https://www.amazon.com/Book-Secrets-.../dp/B00ADXVATE
Failed to find cached page for: https://www.amazon.com/Book-Secrets-.../dp/1459671988
Failed to find cached page for: https://www.amazon.com/-/es/Fiona-Kidman/dp/1459671988
Failed to find cached page for: https://www.amazon.com/livre-secrets.../dp/2757848933
Failed to find cached page for: https://www.amazon.com/-/es/Fiona-Ki.../dp/B00ADXVATE
Skipping non-book result: Result(url='https://www.amazon.com/Book-Secrets-Fiona-Kidman-2013-12-05/product-reviews/B01K94UAOW', title='Amazon.com: Customer reviews: The Book Of Secrets by Fiona Kidman (2013 ...', cached_url=None)
Skipping non-book result: Result(url='https://www.amazon.com/Books-Fiona-Kidman/s?rh=n%3A283155%2Cp_27%3AFiona+Kidman', title='Amazon.com: Fiona Kidman: Books', cached_url=None)
Skipping non-book result: Result(url='https://www.amazon.com/Fiona-Kidman/e/B001HPXQDA%3Fref=dbs_a_mng_rwt_scns_share', title='Amazon.com: Fiona Kidman: Books, Biography, Blog, Audiobooks, Kindle', cached_url=None)
Skipping non-book result: Result(url='https://www.amazon.com/Kindle-Store-Fiona-Kidman/s?rh=n%3A133140011%2Cp_27%3AFiona+Kidman', title='Amazon.com: Fiona Kidman: Kindle Store', cached_url=None)
Failed to find cached page for: https://www.amazon.com/Captive-Wife-.../dp/B00AM7FK7Y
Skipping non-book result: Result(url='https://www.amazon.com/Book-Secrets-Fiona-Kidman-ebook/product-reviews/B00ADXVATE', title='Amazon.com: Customer reviews: The Book of Secrets', cached_url=None)
Skipping non-book result: Result(url='https://www.amazon.com/Books-Fiona-Kidman/s?rh=n%3A283155%2Cp_27%3AFiona+Kidman&page=2', title='Amazon.com: Fiona Kidman: Books', cached_url=None)
Failed to find cached page for: https://www.amazon.com/-/es/Fiona-Kidman/dp/1775532860
Failed to find cached page for: https://www.amazon.com/Breed-Women-F.../dp/B008P627TS
Failed to find cached page for: https://www.amazon.com/At-End-Darwin.../dp/B00AM7FHEA
Failed to find cached page for: https://www.amazon.com/This-Mortal-B.../dp/B07D275M4W
Failed to find cached page for: https://www.amazon.com/All-Day-Movie.../dp/1489409319
Failed to find cached page for: https://www.amazon.com/Comme-au-cin-.../dp/2757881167
Failed to find cached page for: https://www.amazon.com/All-Day-Movie.../dp/B01FSSCF88
Failed to find cached page for: https://www.amazon.com/This-Mortal-B.../dp/B07VZNGK7F
No search engine results for terms: Book of Secrets Fiona Kidman
No matches found with query: ['Book', 'of', 'Secrets', 'Fiona', 'Kidman']

************************************************** ******************************
The identify phase took 98.49 seconds
The longest time (98.483895) was taken by: Amazon.com
Merging results from different sources
We have 0 merged results, merging took: 0.00 seconds
Failed to download metadata for The Book of Secrets


************************************************** ******************************

[kovidgoyal]

Your problems are caused by google blocking your IP for too many requests. JUst wait a few hours for it to be unblocked. And change the amazon server back to auto instead of using ddg.

[tomsem]

Amazon complains about too many requests also. I had hoped to complete this metadata project within my lifetime , but I'm lucky to get more than 20-25 done before it starts complaining.

[kovidgoyal]

Yes, sadly no one provides access to their databases for free without rate limits.