[Metadata Source Plugin] Goodreads

[Zhreca]

The plugin fails to retrieve the metadata from goodreads unless there is an entry in the id field. For example it fails to find books anymore from just title and author.
Here is an example error message:
calibre, version 9.8.0
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters:
{'title': 'Hue Tint & Shade', 'authors': ['Jordan Castillo Price'], 'identifiers': {}, 'timeout': 30}
Using plugins: Goodreads (1, 8, 5)
The log from individual plugins is below

****************************** Goodreads (1, 8, 5) ******************************
Found 0 results
Downloading from Goodreads took 0.22328877449035645
identify - start. title=Hue Tint & Shade, authors=['Jordan Castillo Price'], identifiers={}
Querying: https://www.goodreads.com/search?sea...e=books&search[query]=Hue+Tint+%26+Shade+Jordan+Castillo+Price
No goodreads id via ISBN
Failed to get raw result for query: 'https://www.goodreads.com/search?search_type=books&search[query]=Hue+Tint+%26+Shade+Jordan+Castillo+Price'

************************************************** ******************************
The identify phase took 0.40 seconds
The longest time (0.223289) was taken by: Goodreads
Merging results from different sources
We have 0 merged results, merging took: 0.00 seconds



The actual query in a web browser correctly finds the book.

[Drybonz]

Originally Posted by AnnieQuinn

For about two days now, have any of you experienced slow download times for book information?

Mine has been timing out, saying that metadata couldn't be retrieved after a certain length of time. It used to work normally.

[kiwidude]

Goodreads have gone the same path as Fantastic Fiction with AWS WAF protection. I have no solution for this at this time. That the plugin still allows direct scraping from detail pages directly (once you have a goodreads id) but will not let you do searches is the only mixed blessing but I wouldn't count on that lasting.

EDIT: Interestingly the Link to Goodreads feature still works in the Goodreads Sync plugin. This is hitting a slightly different endpoint (for the same purpose). It also happens to not use the mechanize browser and instead use the httplib2 library so there are two factors at play here to look at. I "might" be able to get something working with that alternate endpoint - at least until Goodreads shut the door on that one too...

EDIT 2: I have something working, but I'm not sure I want to release it. The only way I can get this alternate URL to work is to make use of my developer key that I use in the Goodreads Sync plugin, otherwise it gives a 401 unauthorized error. The problem with doing that is we are going to add a bunch more traffic onto the other endpoint with this id being highlighted. In 15 years of using this id Goodreads haven't suspended its use as yet, but this "might" be the straw that breaks the camel's back as far as they are concerned. That would end up breaking both plugins (both of which I personally use) with no recourse possible at all (Goodreads officially retired support for the APIs years ago, it isn't possible to get new developer keys). At which point I would have to just ditch Goodreads altogether and go with some other alternative.

Ugh.

[JBaby]

This is a major bummer. Amazon just has to ruin everything!

[kiwidude]

Indeed it is unfortunate. TBH I am surprised how long it has taken for sites like FF and Goodreads to reach this point of blocking scraping. I have no insight to their motives other than to suspect they allowed tools like this plugin to continue on the basis that users would still end up visiting their webpages in a browser anyway and it was free marketing. Certainly for Goodreads I like to think it is symbiotic - many people track their reading on Goodreads shelves, go looking there for recommendations/reviews of others etc from a one-click in Calibre of the scraped id. Likewise FF is unparalleled in the quality of it's series tracking (and better quality data generally).

Now though I suspect they are (understandably) concerned about AI/agent-based scraping. Where there is no chance that the data leads to a user visit, and instead they are incurring data traffic costs and the loss of their metadata to the exclusive gain of others. Plugins like this and FantasticFiction are collateral damage.

[Drybonz]

What are people using for metadata now that Goodreads isn't working?

[DNSB]

I use Amazon servers or Amazon with Duck-Duck-Go search, Kobo, Romance.io, Baen & Smashwords. For more cover images, I also use Apple Books covers and Kindle hi-res covers.

I tried Goodreads on and off and, for me, it was cold pancakes and no honey. The returned metadata was often total crap. The last couple of times, metadata was not returned.

I also tried Barnes and Noble but the metadata was low quality and the cover images were more thumbnails than full images.

[kiwidude]

I 100% agree that the metadata quality has become degenerated on Goodreads - they create far too many variations/editions of the book from disparate sources, so if the Goodreads ID you happen to link to is one of the "crap" versions there is nothing the plugin can pickup. Once upon a time there seemed to be more human curation and management of this but based on what exists there today they are either overwhelmed by volumes or not even bothering any more. Sometimes you can find another edition within the same set of data for the book, other times you have to search all over again for the title/author and find another whole new search result for it.

I should add that this plugin does still work provided you link your book to a Goodreads id first.

[JSWolf]

Originally Posted by DNSB

I use Amazon servers or Amazon with Duck-Duck-Go search, Kobo, Romance.io, Baen & Smashwords. For more cover images, I also use Apple Books covers and Kindle hi-res covers.

I tried Goodreads on and off and, for me, it was cold pancakes and no honey. The returned metadata was often total crap. The last couple of times, metadata was not returned.

I also tried Barnes and Noble but the metadata was low quality and the cover images were more thumbnails than full images.

I've never downloaded metadata that did not need human intervention to fix something. So it's not worth it since I'd have to double check what's downloaded anyway.

With Amazon no using some really stupidly long titles AI generated titles, that would have to be fixed for sure.

[JSWolf]

Originally Posted by kiwidude

I 100% agree that the metadata quality has become degenerated on Goodreads - they create far too many variations/editions of the book from disparate sources, so if the Goodreads ID you happen to link to is one of the "crap" versions there is nothing the plugin can pickup. Once upon a time there seemed to be more human curation and management of this but based on what exists there today they are either overwhelmed by volumes or not even bothering any more. Sometimes you can find another edition within the same set of data for the book, other times you have to search all over again for the title/author and find another whole new search result for it.

I should add that this plugin does still work provided you link your book to a Goodreads id first.

Goodreads should stop trying to auto-add all these different versions. Just add some main versions, double check, fix and wait until someone asks for another source with a link to a good enough source for the metadata.