[Metadata Source Plugin] Goodreads

[Drymarchon]

plugin is failing to find any books that match search consistently, this isn't the first book its done with with in the past couple days.

Running identify query with parameters: 
{'title': 'Two Witches and a Whiskey', 'authors': ['Annette Marie'], 'identifiers': {'isbn': '9781988153261', 'goodreads': '42087503'}, 'timeout': 30} 
Using plugins: Goodreads (1, 7, 3) 
The log from individual plugins is below 

****************************** Goodreads (1, 7, 3) ****************************** 
Found 0 results 
Downloading from Goodreads took 1.4515268802642822 
identify - start. title=Two Witches and a Whiskey, authors=['Annette Marie'], identifiers={'isbn': '9781988153261', 'goodreads': '42087503'}
Goodreads book url: 'https://www.goodreads.com/book/show/42087503'
Trying to parse book json for 2022 web page format
Json script node found, page in 2022+ html format
Could not parse all of title/authors/goodreads id from: 'https://www.goodreads.com/book/show/42087503'
Found Goodreads id: '42087503' Title: None Authors: None

[kiwidude]

Welcome to MobileRead @Drymarchon.

Yes this is a problem I was going to post here about that has only recently come up, as Goodreads have (once again) been fiddling with their web servers.

The problem is they are still giving different html responses to different web browser types (user agents). Plugins like Goodreads currently use some built-in calibre functionality to simulate a random user agent (web browser) hitting the Goodreads website in order to scrape the data. The randomness is "desired" in order that Goodreads doesn't see thousands of requests coming from one particular user string constantly and decides to just block it. The downside is their recent policy of publishing different website content means this plugin gets responses back that are not what it expects. Chrome works, Firefox doesn't etc.

I have a solution in mind (forcing Chrome user agents only), but can't make any plugin changes myself for a few more weeks yet. My computer and other possessions have passed through customs and available to be delivered to me but I keep delaying that as I am getting my house painted internally. I would rather get my items when I have somewhere I can unpack them more than temporarily. Also that assumes my computer does actually work when I try to turn it on, four months on and no doubt many times drop-kicked on it's journey around the world...

There is an open GitHub issue for this here, it will be one of the first items I tackle as this plugin is one I use all the time myself.
https://github.com/kiwidude68/calibre_plugins/issues/25

[Drymarchon]

Originally Posted by kiwidude

Welcome to MobileRead @Drymarchon.

[...]

There is an open GitHub issue for this here, it will be one of the first items I tackle as this plugin is one I use all the time myself.
https://github.com/kiwidude68/calibre_plugins/issues/25

Thanks! I don't know how I missed that issue, I checked the github repo before coming here but i guess I'm blind. I'll go ahead and subscribe to notifications for the aforementioned issue.

[C8H10N4O2]

I wanted to check in and see if any progress has been made on fixing what Amazon broke for the plugin. This is my primary data source for books and the other sources just don't hold a candle to the data quantity and quality that Goodreads has. I'm happy to help where I can; let me know if there is any testing or anything that you need help with.

Thanks!

[kiwidude]

@C8H10N4O2, as per my comment above I'm still not in a position to be able to make plugin code changes/releases as yet. The good news is overnight someone (changhuapeng) has contributed the necessary code change I mention above of using randomised chrome user agents. The bad news is I still have no ability to release it until I get my computer...

Allegedly my UK possessions finally get delivered tomorrow, but I have painters here until the end of next week (with a question mark on whether they will even manage to finish by then, with only 2 of the 4 painters supposed to be here today). I really "shouldn't" touch the dozens of boxes I shipped over that will just get piled into the garage until the painters are done. Realistically though I'm sure I will be tempted to open some of them to sort into piles this weekend for the highest priority items. Whether that will include my computer and all it's bits, time to assemble it somewhere temporarily, find out if it works let alone do a plugin code merge/release and box it back up again is difficult. But obviously I want to know what survived the trip ASAP more than anyone.

So we shall see what happens - hopefully (well unless the computer is broken which is the really "worst" case) it will be a couple more weeks at most.

[C8H10N4O2]

@kiwidude - I apologize for missing that part of your original update. I hope your move ends up being successful.

On that note, though, I patched my local copy with changhuapeng's change and it is working perfectly for me now. I appreciate all the work you have put into your published plugins. Your additions make my Calibre experience much better.

[kiwidude]

Release Notes:
https://github.com/kiwidude68/calibr...odreads-v1.7.4

Now using random chrome user agents, thanks to all for reporting this etc.

[kiwidude]

Release Notes
https://github.com/kiwidude68/calibr...odreads-v1.7.5

I've had to specify a minimum version for that random chrome user agent stuff, currently calibre v2.81.0 (thanks Kovid) but if you are having issues you really shouldn't be on a calibre that old anyways!

[kiwidude]

It was brought to my attention that the plugin is still intermittently failing, and I have verified this for myself. The problem as it turns out is not necessarily a user agent based thing. It just seems that some requests come back with some fairly minimal html intermittently. I had been thinking I could fallback if the json wasn't fully populated to just scraping the page, but in these circumstances the rest of the page html isn't containing the content either.

From a relatively quick look to figure all this out I frankly am unsure how I can proceed at this point. Goodreads have really messed with their servers with these updates they rolled out over the last year - perhaps this is some intentional anti-scraping feature, or perhaps it is just different builds of their site on different servers.

The crudest option to work around it may have to be just constantly retrying until the plugin gets a valid html response back. This could be pretty slow of course and no guarantees that you don't get 50 failed attempts in a row before one that succeeds.

The joys of web scraping unfortunately.

[ownedbycats]

I noticed it myself when adding metadata to a book earlier but thought it was a user error. Is there any indication that messing with the site design on different browser is a temporary measure, e.g. a very slow rollout?

[kiwidude]

Release Notes
https://github.com/kiwidude68/calibr...odreads-v1.7.6

This has some crude retry logic in it for up to 10 attempts. The most retries I have seen required in my brief testing is 3 so hopefully that should be sufficient to work around the issue.

I don't like it but I am at the mercy of what Goodreads supplies as a response at this point.

[kevn57]

I haven't been able to get any info from goodreads and thought I'd post this error report, not sure if my still using calibre 5.34 with the new plugin is the issue.

calibre, version 5.34.0
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters: 
{'title': 'Walden; Or, Life in the Woods', 'authors': ['Henry David Thoreau'], 'identifiers': {'isbn': '9780486284958', 'goodreads': '77667'}, 'timeout': 30} 
Using plugins: Goodreads (1, 7, 6) 
The log from individual plugins is below 

****************************** Goodreads (1, 7, 6) ****************************** 
Found 0 results 
Downloading from Goodreads took 0.0 
identify - start. title=Walden; Or, Life in the Woods, authors=['Henry David Thoreau'], identifiers={'isbn': '9780486284958', 'goodreads': '77667'}
Plugin Goodreads failed
Traceback (most recent call last):
  File "calibre_plugins.goodreads.__init__", line 51, in user_agent
ImportError: cannot import name 'random_common_chrome_user_agent' from 'calibre.utils.random_ua' (C:\Program Files\Calibre2\app\bin\python-lib.bypy.frozen\calibre\utils\random_ua.pyc)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calibre\ebooks\metadata\sources\identify.py", line 46, in run
  File "calibre_plugins.goodreads.__init__", line 211, in identify
  File "calibre\ebooks\metadata\sources\base.py", line 291, in browser
  File "calibre_plugins.goodreads.__init__", line 55, in user_agent
ImportError: cannot import name 'random_chrome_ua' from 'calibre.utils.random_ua' (C:\Program Files\Calibre2\app\bin\python-lib.bypy.frozen\calibre\utils\random_ua.pyc) 

******************************************************************************** 
The identify phase took 0.22 seconds 
The longest time (0.000000) was taken by: Goodreads 
Merging results from different sources 
We have 0 merged results, merging took: 0.00 seconds

[kiwidude]

Yes that is exactly what the issue is - upgrade your calibre. It seems the fallback code I have in there isn’t working to use a function that I thought existed in the older calibre.

[kevn57]

Originally Posted by kiwidude

Yes that is exactly what the issue is - upgrade your calibre. It seems the fallback code I have in there isn’t working to use a function that I thought existed in the older calibre.

Is there any way to d/l the previous version of the plugin? That still worked for me, but I just upgraded to stay current.

[ownedbycats]

Originally Posted by kevn57

Is there any way to d/l the previous version of the plugin? That still worked for me, but I just upgraded to stay current.

https://github.com/kiwidude68/calibre_plugins/releases