[Metadata Source Plugin] Goodreads

[Zhreca]

The plugin fails to retrieve the metadata from goodreads unless there is an entry in the id field. For example it fails to find books anymore from just title and author.
Here is an example error message:
calibre, version 9.8.0
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters:
{'title': 'Hue Tint & Shade', 'authors': ['Jordan Castillo Price'], 'identifiers': {}, 'timeout': 30}
Using plugins: Goodreads (1, 8, 5)
The log from individual plugins is below

****************************** Goodreads (1, 8, 5) ******************************
Found 0 results
Downloading from Goodreads took 0.22328877449035645
identify - start. title=Hue Tint & Shade, authors=['Jordan Castillo Price'], identifiers={}
Querying: https://www.goodreads.com/search?sea...e=books&search[query]=Hue+Tint+%26+Shade+Jordan+Castillo+Price
No goodreads id via ISBN
Failed to get raw result for query: 'https://www.goodreads.com/search?search_type=books&search[query]=Hue+Tint+%26+Shade+Jordan+Castillo+Price'

************************************************** ******************************
The identify phase took 0.40 seconds
The longest time (0.223289) was taken by: Goodreads
Merging results from different sources
We have 0 merged results, merging took: 0.00 seconds



The actual query in a web browser correctly finds the book.

[Drybonz]

Originally Posted by AnnieQuinn

For about two days now, have any of you experienced slow download times for book information?

Mine has been timing out, saying that metadata couldn't be retrieved after a certain length of time. It used to work normally.

[kiwidude]

Goodreads have gone the same path as Fantastic Fiction with AWS WAF protection. I have no solution for this at this time. That the plugin still allows direct scraping from detail pages directly (once you have a goodreads id) but will not let you do searches is the only mixed blessing but I wouldn't count on that lasting.

EDIT: Interestingly the Link to Goodreads feature still works in the Goodreads Sync plugin. This is hitting a slightly different endpoint (for the same purpose). It also happens to not use the mechanize browser and instead use the httplib2 library so there are two factors at play here to look at. I "might" be able to get something working with that alternate endpoint - at least until Goodreads shut the door on that one too...

EDIT 2: I have something working, but I'm not sure I want to release it. The only way I can get this alternate URL to work is to make use of my developer key that I use in the Goodreads Sync plugin, otherwise it gives a 401 unauthorized error. The problem with doing that is we are going to add a bunch more traffic onto the other endpoint with this id being highlighted. In 15 years of using this id Goodreads haven't suspended its use as yet, but this "might" be the straw that breaks the camel's back as far as they are concerned. That would end up breaking both plugins (both of which I personally use) with no recourse possible at all (Goodreads officially retired support for the APIs years ago, it isn't possible to get new developer keys). At which point I would have to just ditch Goodreads altogether and go with some other alternative.

Ugh.

[JBaby]

This is a major bummer. Amazon just has to ruin everything!

[kiwidude]

Indeed it is unfortunate. TBH I am surprised how long it has taken for sites like FF and Goodreads to reach this point of blocking scraping. I have no insight to their motives other than to suspect they allowed tools like this plugin to continue on the basis that users would still end up visiting their webpages in a browser anyway and it was free marketing. Certainly for Goodreads I like to think it is symbiotic - many people track their reading on Goodreads shelves, go looking there for recommendations/reviews of others etc from a one-click in Calibre of the scraped id. Likewise FF is unparalleled in the quality of it's series tracking (and better quality data generally).

Now though I suspect they are (understandably) concerned about AI/agent-based scraping. Where there is no chance that the data leads to a user visit, and instead they are incurring data traffic costs and the loss of their metadata to the exclusive gain of others. Plugins like this and FantasticFiction are collateral damage.