[Metadata Source plugins] Calibre internal plugins fail

[igorius]

Several internal calibre metadata source plugins fail in some ways since 6.17...
Search is extremely long, was some short seconds before:

****************************** Google Images Covers ******************************
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36')]
Downloaded cover: 915x1280
Took 108.63511800765991 seconds
Search URL: https://www.google.com/search?as_st=...=iar:t,ift:jpg
Downloading 10 covers

and

****************************** Amazon.com Covers ****************************** 
Request extra headers: [('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8'), ('Upgrade-insecure-requests', '1'), ('Referer', 'https://www.google.com/search?q=Waldwissen+Peter+Ibisch+Pierre+Wohlleben+site%3Awww.amazon.de'), ('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36')] 
Downloaded cover: 404x500 
Took 112.82284808158875 seconds 
No cached cover found, running identify
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:87.0) Gecko/20100101 Firefox/87.0
Server: auto
Making google query: https://www.google.com/search?q=%289783641294229+or+978-3641294229%29+site%3Awww.amazon.de
Failed to find any results on results page, with title: (9783641294229 or 978-3641294229) site:www.amazon.de - Google Suche
No search engine results for terms: (9783641294229 OR 978-3641294229)
Trying the bing search engine instead
Making bing query: https://www.bing.com/search?q=%289783641294229+or+978-3641294229%29+site%3Awww.amazon.de
Failed to find any results on results page, with title: (9783641294229 or 978-3641294229) site:www.amazon.de - Suchen
No search engine results for terms: (9783641294229 OR 978-3641294229)
No matches found with identifiers, retrying using only title and authors. Query: ['(9783641294229', 'OR', '978-3641294229)']
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36
Server: auto
Making google query: https://www.google.com/search?q=Waldwissen+Peter+Ibisch+Pierre+Wohlleben+site%3Awww.amazon.de
Downloading details from: https://webcache.googleusercontent.com/search?q=cache:lOil93li3e8J:https://www.amazon.de/Waldwissen-Erstaunliche-Erkenntnisse-umfassendste-Standardwerk-ebook/dp/B0BMGF5MQ7&cd=2&hl=de&ct=clnk&gl=de
Downloading details from: https://webcache.googleusercontent.com/search?q=cache:UJYcZhA9f1kJ:https://www.amazon.de/Waldwissen-Erstaunliche-Erkenntnisse-umfassendste-Standardwerk/dp/3453281497&cd=3&hl=de&ct=clnk&gl=de
Downloading details from: https://webcache.googleusercontent.com/search?q=cache:Jn82SU1tw5oJ:https://www.amazon.de/Waldwissen-verstehen-Erstaunliche-Erkenntnisse-Menschen/dp/B0C4B4689H&cd=4&hl=de&ct=clnk&gl=de
Error parsing title for url: 'https://webcache.googleusercontent.com/search?q=cache:Jn82SU1tw5oJ:https://www.amazon.de/Waldwissen-verstehen-Erstaunliche-Erkenntnisse-Menschen/dp/B0C4B4689H&cd=4&hl=de&ct=clnk&gl=de'
Traceback (most recent call last):
  File "calibre/ebooks/metadata/sources/amazon.py", line 439, in parse_details
  File "calibre/ebooks/metadata/sources/amazon.py", line 587, in parse_title
ValueError: No title block found

and

****************************** Big Book Search Covers ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko')] 
Failed to download valid cover 
Took 0.3754310607910156 seconds 
Failed to download cover from Big Book Search
Traceback (most recent call last):
  File "calibre/ebooks/metadata/sources/covers.py", line 42, in run
  File "calibre/ebooks/metadata/sources/big_book_search.py", line 43, in download_cover
  File "calibre/ebooks/metadata/sources/big_book_search.py", line 18, in get_urls
  File "mechanize/_mechanize.py", line 257, in open
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 404: Not Found

and

****************************** Edelweiss (2, 0, 1) ****************************** 
Found 0 results 
Downloading from Edelweiss took 0.18786096572875977 
Currently Edelweiss returns random books for search queries 

******************************************************************************** 
The identify phase took 97.76 seconds 
The longest time (0.187861) was taken by: Edelweiss 
Merging results from different sources 
We have 1 merged results, merging took: 0.00 seconds

and

****************************** Google Images Covers ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36')] 
Downloaded cover: 622x933 
Took 103.0151801109314 seconds 
Search URL: https://www.google.com/search?as_st=y&tbm=isch&as_q=Uhrmacher+Andy+Strau%C3%9F&as_epq=&as_oq=&as_eq=&cr=&as_sitesearch=&safe=images&tbs=iar:t,ift:jpg
Downloading 10 covers

and

****************************** Amazon.com Covers ****************************** 
Request extra headers: [('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8'), ('Upgrade-insecure-requests', '1'), ('Referer', 'https://www.google.com/search?q=Uhrmacher+Andy+Strau%C3%9F+site%3Awww.amazon.de'), ('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36')] 
Downloaded cover: 338x500 
Took 109.24892401695251 seconds 
No cached cover found, running identify
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36
Server: auto

and

****************************** Big Book Search Covers ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36')] 
Failed to download valid cover 
Took 0.7031939029693604 seconds 
Failed to download cover from Big Book Search
Traceback (most recent call last):
  File "calibre/ebooks/metadata/sources/covers.py", line 42, in run
  File "calibre/ebooks/metadata/sources/big_book_search.py", line 43, in download_cover
  File "calibre/ebooks/metadata/sources/big_book_search.py", line 18, in get_urls
  File "mechanize/_mechanize.py", line 257, in open
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 404: Not Found

and

****************************** Amazon.com (1, 3, 3) ****************************** 
Found 3 results 
Downloading from Amazon.com took 108.70860004425049

and

****************************** Amazon.com (1, 3, 3) ****************************** 
Found 1 results 
Downloading from Amazon.com took 8.86096477508545 
<snipped book result>
URL not found: 'https://webcache.googleusercontent.com/search?q=cache:nU0Ld69qPdYJ:https://www.amazon.de/Beta-Hearts-Roman-Marie-Gra%25C3%259Fhoff-ebook/dp/B07ZQ93NMS&cd=4&hl=de&ct=clnk&gl=de'
get_details failed for url: 'https://webcache.googleusercontent.com/search?q=cache:nU0Ld69qPdYJ:https://www.amazon.de/Beta-Hearts-Roman-Marie-Gra%25C3%259Fhoff-ebook/dp/B07ZQ93NMS&cd=4&hl=de&ct=clnk&gl=de'
Traceback (most recent call last):
  File "calibre/ebooks/metadata/sources/amazon.py", line 112, in parse_details_page
  File "mechanize/_mechanize.py", line 241, in open_novisit
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 404: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calibre/ebooks/metadata/sources/amazon.py", line 410, in run
  File "calibre/ebooks/metadata/sources/amazon.py", line 416, in get_details
  File "calibre/ebooks/metadata/sources/amazon.py", line 116, in parse_details_page
calibre.ebooks.metadata.sources.amazon.UrlNotFound: The URL https://webcache.googleusercontent.com/search?q=cache:nU0Ld69qPdYJ:https://www.amazon.de/Beta-Hearts-Roman-Marie-Gra%25C3%259Fhoff-ebook/dp/B07ZQ93NMS&cd=4&hl=de&ct=clnk&gl=de was not found (HTTP 404)

Hope that helps to get the errors...
Some rare times the search is short as before, but only in extremely rare cases.

[Wiggo]

I also noticed