Goodreads/Google Books Metadata Issues

[McStubb]

For the last week or so I have been unable to get metadata matches from either google books or Goodreads. This might be related to Goodreads rollout of AWS WAF tokens (in part)

Running identify query with parameters: 
{'title': "Clive Cussler's the Devil's Sea", 'authors': ['Dirk Cussler'], 'identifiers': {}, 'timeout': 30} 
Using plugins: Barnes & Noble (1, 5, 8), Goodreads (1, 8, 5), Goodreads More Tags (1, 2, 1, 2), Hardcover (0, 5, 0), Google (1, 1, 3), Amazon.com (1, 3, 18) 
The log from individual plugins is below 

****************************** Barnes & Noble (1, 5, 8) ****************************** 
Found 0 results 
Downloading from Barnes & Noble took 0.45362281799316406 
Querying: https://www.barnesandnoble.com/s/clive%20cussler%27s%20the%20devil%27s%20sea%20dirk%20cussler
No matches for identify query 

******************************************************************************** 

****************************** Goodreads (1, 8, 5) ****************************** 
Found 0 results 
Downloading from Goodreads took 0.3429744243621826 
[GoodreadsMoreTags] Integration active
identify - start. title=Clive Cussler's the Devil's Sea, authors=['Dirk Cussler'], identifiers={}
Querying: https://www.goodreads.com/search?search_type=books&search[query]=Clive+Cussler%27s+the+Devil%27s+Sea+Dirk+Cussler
No goodreads id via ISBN
Failed to get raw result for query: 'https://www.goodreads.com/search?search_type=books&search[query]=Clive+Cussler%27s+the+Devil%27s+Sea+Dirk+Cussler' 

******************************************************************************** 

****************************** Goodreads More Tags (1, 2, 1, 2) ****************************** 
Found 0 results 
Downloading from Goodreads More Tags took 10.00775694847107 
Timeout hit (10s) while waiting for results from the Goodreads plugin. Continuing with what we have, some results may not have tags.
Got no results from the Goodreads plugin, proceeding without integration
No goodreads identifier found, not grabbing extra tags 

******************************************************************************** 

****************************** Hardcover (0, 5, 0) ****************************** 
Found 1 results 
Downloading from Hardcover took 0.6016433238983154 


--- 
Title               : Clive Cussler's the Devil's Sea
Author(s)           : Dirk Cussler
Publisher           : Penguin Books Limited
Tags                : Adventure, Fiction
Series              : Dirk Pitt #26
Languages           : eng
Rating              : 3.9
Published           : 2021-01-01T05:00:00+00:00
Identifiers         : hardcover:clive-cusslers-the-devils-sea, hardcover-slug:clive-cusslers-the-devils-sea, hardcover-id:727999, hardcover-edition:31912074, isbn:9781405951593
Comments            : Fearless adventurer Dirk Pitt must unravel a historical mystery of epic importance in the latest novel in the beloved New York Times bestselling series created by the “grand master of adventure” Clive Cussler. When NUMA director Dirk Pitt discovers a forgotten plane crash in the Philippine Sea, he unearths clues pointing to an invaluable Buddhist artifact that has been lost for decades. But soon, he will learn that he’s not the only one making earthshattering discoveries. During a mission to recover a failed hypersonic missile from Luzon Strait, Pitt finds himself racing against a rogue Chinese military team…who have hijacked a ship strong enough to stir the ocean into a veritable Devil’s Sea. From the cold depths of the Pacific to the dizzying heights of the Himalayas, only Pitt and his children can unravel the mysteries that will preserve a religion, save a nation…and save the world from war. 
Searching for ids by Name Clive Cussler's the Devil's Sea Dirk Cussler
Found results=[727999] for query="Clive Cussler's the Devil's Sea Dirk Cussler"
Finding by book id [727999]
Comparing Clive Cussler's the Devil's Sea to Clive Cussler's The Devil's Sea (book:clive-cusslers-the-devils-sea)
Comparing Clive Cussler's the Devil's Sea to Clive Cussler's The Devil's Sea (edition:32739959)
Comparing Clive Cussler's the Devil's Sea to Clive Cussler's The Devil's Sea (edition:31912074)
Comparing Clive Cussler's the Devil's Sea to Clive Cussler's The Devil's Sea (edition:31947227)
Comparing Clive Cussler's the Devil's Sea to Clive Cussler's The Devil's Sea (edition:32663969)
Comparing Clive Cussler's the Devil's Sea to Clive Cussler's the Devil's Sea (edition:32660387)
filtering edition Edition(id=32660387, isbn_13='9780241552360', asin=None, title="Clive Cussler's the Devil's Sea", authors=[Author(name='Dirk Cussler', contribution='Author')], image='https://assets.hardcover.app/edition/32660387/77c9a3d6fb160336dc8c63428ccb6575e4a1724f.jpeg', language='eng', publisher='Michael Joseph', users_count=0, release_date=datetime.datetime(2021, 1, 1, 0, 0))
weighted similarity between ['Dirk Cussler'] and Author(name='Dirk Cussler', contribution='Author'): 2.0
overall similarity for Clive Cussler's the Devil's Sea (32660387): 2.0
filtering edition Edition(id=32739959, isbn_13='9780593419656', asin=None, title="Clive Cussler's The Devil's Sea", authors=[Author(name='Dirk Cussler', contribution='Author'), Author(name='Clive Cussler', contribution='Series Creator')], image='https://assets.hardcover.app/external_data/2623585/033b455803694ed6f86dce5124e0489e621b0bff.jpeg', language='eng', publisher=None, users_count=1, release_date=datetime.datetime(2021, 11, 11, 0, 0))
weighted similarity between ['Dirk Cussler'] and Author(name='Dirk Cussler', contribution='Author'): 2.0
weighted similarity between ['Dirk Cussler'] and Author(name='Clive Cussler', contribution='Series Creator'): 0.78
overall similarity for Clive Cussler's The Devil's Sea (32739959): 1.3900000000000001
filtering edition Edition(id=31912074, isbn_13='9781405951593', asin=None, title="Clive Cussler's The Devil's Sea", authors=[Author(name='Dirk Cussler', contribution='Author')], image='https://assets.hardcover.app/external_data/61622833/1b041913d49f5ddd422afa30d66d255df019f364.jpeg', language='eng', publisher='Penguin Books Limited', users_count=1, release_date=datetime.datetime(2021, 1, 1, 0, 0))
weighted similarity between ['Dirk Cussler'] and Author(name='Dirk Cussler', contribution='Author'): 2.0
overall similarity for Clive Cussler's The Devil's Sea (31912074): 2.0
filtering edition Edition(id=31947227, isbn_13='9780593419649', asin=None, title="Clive Cussler's The Devil's Sea", authors=[Author(name='Dirk Cussler', contribution='Author')], image='https://assets.hardcover.app/edition/31947227/a6f06c7ce981021cf22a42fda66681f4eb2c07bc.jpeg', language='eng', publisher='Penguin', users_count=1, release_date=datetime.datetime(2021, 11, 2, 0, 0))
weighted similarity between ['Dirk Cussler'] and Author(name='Dirk Cussler', contribution='Author'): 2.0
overall similarity for Clive Cussler's The Devil's Sea (31947227): 2.0
filtering edition Edition(id=32663969, isbn_13='9781405951586', asin=None, title="Clive Cussler's The Devil's Sea", authors=[Author(name='Dirk Cussler', contribution='Author')], image='https://assets.hardcover.app/edition/32663969/4b6a5bdd94ed1621c333a042f2bab72ca0a6cb91.jpeg', language='eng', publisher='Penguin Books', users_count=0, release_date=datetime.datetime(2021, 11, 11, 0, 0))
weighted similarity between ['Dirk Cussler'] and Author(name='Dirk Cussler', contribution='Author'): 2.0
overall similarity for Clive Cussler's The Devil's Sea (32663969): 2.0
Matched book.slug='clive-cusslers-the-devils-sea' to edition=Edition(id=31912074, isbn_13='9781405951593', asin=None, title="Clive Cussler's The Devil's Sea", authors=[Author(name='Dirk Cussler', contribution='Author')], image='https://assets.hardcover.app/external_data/61622833/1b041913d49f5ddd422afa30d66d255df019f364.jpeg', language='eng', publisher='Penguin Books Limited', users_count=1, release_date=datetime.datetime(2021, 1, 1, 0, 0))
Adding book slug 'clive-cusslers-the-devils-sea' to queue 

******************************************************************************** 

****************************** Google (1, 1, 3) ****************************** 
Found 0 results 
Downloading from Google took 0.606863260269165 
Making query: https://books.google.com/books/feeds/volumes?q=intitle%3AClive+intitle%3ACussler%27s+intitle%3ADevil%27s+intitle%3ASea%2Binauthor%3ADirk+inauthor%3ACussler&max-results=20&start-index=1&min-viewability=none
Failed to make identify query: 'https://books.google.com/books/feeds/volumes?q=intitle%3AClive+intitle%3ACussler%27s+intitle%3ADevil%27s+intitle%3ASea%2Binauthor%3ADirk+inauthor%3ACussler&max-results=20&start-index=1&min-viewability=none'
Traceback (most recent call last):
  File "calibre\ebooks\metadata\sources\google.py", line 518, in make_query
  File "mechanize\_mechanize.py", line 241, in open_novisit
  File "mechanize\_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 500: Internal Server Error 

******************************************************************************** 

****************************** Amazon.com (1, 3, 18) ****************************** 
Found 0 results 
Downloading from Amazon.com took 3.958552360534668 
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36
Server: auto
Making google query: https://www.google.com/search?q=Clive+Cussler%27s+Devil%27s+Sea+Dirk+Cussler+site%3Awww.amazon.com
Failed to find any results on results page, with title: Clive Cussler's Devil's Sea Dirk Cussler site:www.amazon.com - Google Search
No search engine results for terms: Clive Cussler's Devil's Sea Dirk Cussler
Trying the bing search engine instead
Making bing query: https://www.bing.com/search?q=Clive+Cussler%27s+Devil%27s+Sea+Dirk+Cussler+site%3Awww.amazon.com
Bing returned no results
No search engine results for terms: Clive Cussler's Devil's Sea Dirk Cussler
No matches found with query: ['Clive', "Cussler's", "Devil's", 'Sea', 'Dirk', 'Cussler'] 

******************************************************************************** 
The identify phase took 10.05 seconds 
The longest time (10.007757) was taken by: Goodreads More Tags 
Merging results from different sources 
We have 1 merged results, merging took: 0.00 seconds

[DNSB]

Google has been dead for metadata for quite a while. For Amazon, you need to use either Amazon servers (prone to blocking if too many requests) or the Wayback Machine (extremely slow). GoodReads has been up and down for a while and may now be officially dead. I've found Goodreads metadata to be a total mess which took more work to manually correct than I cared to spend.

[McStubb]

Not sure why you are being hostile to two of the largest sources of metadata. Google Books (Which I too have an API key for for another task) continues to provide valid reliable data for my app, just not for Calibre. Goodreads provide good enough data, may not be perfect but between GR and GB, its the best source I have been able to find. For GR I suspect its the AWS WAF tokens tokens that amazon is rolling out. They pushed this to imdb a while back already.

[DNSB]

As far as I am concerned, I am not hostile to Goodreads. I am simply aware that their metadata quality compared to other sources such as Amazon or Kobo is not up to contemporary community standards.

As for Google, Google no longer allows metadata download by calibre so using Amazon or other metadata supplier trying to obtain metadata from Google search does not work. And since we are in a forum for calibre users, that you are able to get metadata using an API key is not really relevant.

Due to the overenthusiastic strip mining by bots feeding AI's insatiable appetite for data, many metadata sources are no longer working and that is something that we are going to have to live with.

[kovidgoyal]

You are getting an internal server error from google -- likely an issue with google. I can tell you it continues ot work for me.

[JSWolf]

A lot of metadata has errors. That's why any metadata I want to edit, I do so manually. I see no point in downloading metadata, and overwriting all of my metadata. Then I have to check everything all over again to make sure it's correct. It's easier to just do it all manually.

[theducks]

I have resorted to using the Search the Internet plugin, which takes my browser to the page, where I copy the synopsis, then paste those into the comments section of the MDE (after filling in the easier, missing metadata by hand as Copy-paste is very tedious )