[Metadata Source Plugin] Goodreads

[theducks]

Originally Posted by Phil_C

Is there a way to keep the current columns for rating and votes, and enter numbers manually for newly added books? Manual entry does not seem to be allowed.

(Not a GR user but I read the manual linked in post #1)
Both those are actually saved as identifiers
Identifiers are a 2 part value. <Name> : <value>
So you can manually add those to {identifiers}
grvotes: <value>
grrating: <value>

Maybe someone will make a simple GRtool that you can manually fill in a form that mimics what was automatically done

[Terisa de morgan]

Originally Posted by theducks

(Not a GR user but I read the manual linked in post #1)
Both those are actually saved as identifiers
Identifiers are a 2 part value. <Name> : <value>
So you can manually add those to {identifiers}
grvotes: <value>
grrating: <value>

Maybe someone will make a simple GRtool that you can manually fill in a form that mimics what was automatically done

ActionChains would be enough (I'm using it for filling other data)

[Phil_C]

Changing the template changes the value for all books, so that doesn't solve this.

I have not used the Action Chain plugin. I fail to see how that could work with the Goodreads plugin.

It seems I need to find a way to copy all the current values from GR Rating and GR Votes to two new columns. Then I can do manual entry for newly added books.

I don't want to lose the current values for thousands of books.

[Comfy.n]

I have a minimal fix for the plugin, that fetches grvotes and grrating. Blurb is truncated because it is a server-side limit for the autocomplete API. Will post it shortly.

[Comfy.n]

This version can fetch grvotes and grrating fields, which I use a lot to sort my library. GR identifier is also available.

Published date and Publisher cannot be fetched. Comments (blurb) is limited because it's truncated to a certain characters limit server-side.

The autocomplete API also exposes numPages, so this could be used to fix the Count Pages feature.

[dunhill]

The plugin could not have an integrated browser that opens from Calibre to retrieve data, as if it were a store plugin.

[Phil_C]

Originally Posted by Comfy.n

This version can fetch grvotes and grrating fields, which I use a lot to sort my library. GR identifier is also available.

Published date and Publisher cannot be fetched. Comments (blurb) is limited because it's truncated to a certain characters limit server-side.

The autocomplete API also exposes numPages, so this could be used to fix the Count Pages feature.

This works! GR Identifier, grrating and grvotes are fetched. As you said, comments are cut short.

The grrating and grvotes are most important to me, as I also use them in other templates.

Wonderful!!!!

[dunhill]

Originally Posted by Phil_C

This works! GR Identifier, grrating and grvotes are fetched. As you said, comments are cut short.

The grrating and grvotes are most important to me, as I also use them in other templates.

Wonderful!!!!

You also saved my library, thank you.

[Waylander]

Originally Posted by Comfy.n

This version can fetch grvotes and grrating fields, which I use a lot to sort my library. GR identifier is also available.

Published date and Publisher cannot be fetched. Comments (blurb) is limited because it's truncated to a certain characters limit server-side.

The autocomplete API also exposes numPages, so this could be used to fix the Count Pages feature.

Does one install this in Calibre like any other plugin from an external source?

[dunhill]

Originally Posted by Waylander

Does one install this in Calibre like any other plugin from an external source?

From Preferences/Plugins, install from Load plugin from file after downloading the file posted in thread #830

[NeyNey]

Originally Posted by Comfy.n

This version can fetch grvotes and grrating fields, which I use a lot to sort my library. GR identifier is also available.

Published date and Publisher cannot be fetched. Comments (blurb) is limited because it's truncated to a certain characters limit server-side.

The autocomplete API also exposes numPages, so this could be used to fix the Count Pages feature.

Its overwriting existing goodreads id, is that working as intended?

[Matus]

Originally Posted by kiwidude

It was a good run for many years but Goodreads have finally closed the last loophole I know of for retrieving metadata.

To all of you asking "what is wrong" - I have already discussed this on previous pages. Goodreads are blocking web scraping use AWS WAF, for which there is not much that can be done to circumvent it. After all they sell it as a commercial service - if it was easy to workaround it wouldn't have any value.

The same issues apply to the web page scraping of page counts in the Count Pages plugin for multiple sources in there.

This Goodreads plugin along with the Fantastic Fiction ones are now dead for all intents and purposes. In the unlikely event someone out there comes up with a workaround in future of course they are welcome to submit a patch but I won't be spending any more time myself on it.

Hi, I am no Python developer, so I will just share what I did to solve the problem. I just call Selenium driver which runs Chrome browser to fetch page and return page content.

1. I have created a local proxy service that will connect to Goodreads, correctly handling WAF.
   • The script is attached as GoodReadsProxy.py
   • The full project for PyCharm can be downloaded from https://vonku.sk/GoodreadsProxy.zip (as I run on Windows, it contains win version of dependencies)
   • The proxy needs to be started as separate process before Calibre tries to use it ;-)
2. I have changed single line 81 in the plugin to call proxy instead of calling Goodreads directly. My updated plugin file is attached as Goodreads.zip
   

   raw = self.browser.open_novisit('http://localhost:8000/solve?url=' + self.url, timeout=self.timeout).read().strip()

With changes as above, I will just start the proxy and then I can download metadata from Goodreads with no issue. The only downside is that it is bit slow as it waits for 15 seconds to fetch it, but that is not a problem for me.

This is bit of hack, but I do not know much about how to create Calibre plugins and I did not want to invest too much time into it, just solve my problem any way possible. And for my needs, it works ;-).

Feel free to adapt it and move the implementation of connection to Goodreads from proxy directly to plugin or just use any other lib that can achive same, maybe something that is already available in Calibre core lits of libs.

I think of it as hack for me and POC for anyone willing to fix plugin...

[mokosk]

I also experimented a little bit with a possible way to work around the Goodreads WAF issues that have been affecting metadata downloads.

The idea is not to have the Calibre plugin access Goodreads directly. Instead, the plugin talks to a small local service running on the user's machine. That service controls the user's own Firefox browser through Selenium, navigates to the Goodreads book page, extracts the metadata (title, authors, series, tags, description, publisher, publication date, ISBN, rating, cover, etc.), and returns it to the plugin.

The architecture looks roughly like this:

Calibre plugin → Local metadata service → Native Firefox → Goodreads

Since the requests are performed through the user's actual browser session, this appears to avoid some of the issues caused by direct HTTP requests being blocked by the Goodreads WAF.

This is only an experiment at this stage, but it seems surprisingly viable. The main downside is the extra complexity of running a local background service and browser automation.

Do you think it would be worth discussing this approach with the original Goodreads metadata plugin developer as a possible optional backend?

[Comfy.n]

Originally Posted by Phil_C

This works! GR Identifier, grrating and grvotes are fetched. As you said, comments are cut short.

The grrating and grvotes are most important to me, as I also use them in other templates.

Wonderful!!!!

You're welcome.

Originally Posted by dunhill

You also saved my library, thank you.

Yep. I'm very used to the votes and ratings columns. And I have also this column for book details:

https://www.mobileread.com/forums/sh...d.php?t=363995

Originally Posted by Waylander

Does one install this in Calibre like any other plugin from an external source?

Correct.

Originally Posted by NeyNey

Its overwriting existing goodreads id, is that working as intended?

Yes, the post-AWF-apocalypse version should behave like the original plugin in this regard.