[Metadata Source Plugin] Goodreads

[BeckyEbook]

Google and Amazon seem to be different results, as there is only one author on Google and two on Amazon.

IMHO when Calibre finds optimal matching, it merges all information into one large entry.
Note the "See at" entry on the right side of your screenshots:

• When found it on Google, we see that it's Google.
• When found it on Amazon, we see that it's Amazon.
• When found in Goodreads, we can see that all information has been merged.

See hint for the main options when fetching metadata.

[davidfor]

Originally Posted by bc4me

Here's a couple of screenshots with Amazon and Google sources selected, then one with Goodreads, Amazon, and Google.

I believe that is working as designed. Though possibly a little unexpected with the interaction between the three

I think what is happening is that when you have just Google and Amazon active, there is not enough information to merge the results. The authors do not match. But, when Goodreads is added, the Goodreads and Google are merged because they have the same ISBN. Then, the Amazon is merged because there is an exact title and author match with the Goodreads result.

That's my guess. I have never looked at the merge algorithm to see what it does, but, the above makes sense with the results you are seeing. I have at times wanted a "no merge" option. But, I have also wanted a way to pick parts of the metadata from the different results. Especially the comment, as the comments in the merged aren't always the best.

I don't think the option @BeckyEbook will have a real effect on this. That is about what happens if the source returns multiple books. With that option on, you could see multiple entries if the source finds multiple good matches. But, if identifiers are used, in this case, Goodreads, Google and the ISBN, there should only be one match for each source.

[bc4me]

Originally Posted by BeckyEbook

Google and Amazon seem to be different results, as there is only one author on Google and two on Amazon.

IMHO when Calibre finds optimal matching, it merges all information into one large entry.
Note the "See at" entry on the right side of your screenshots:

• When found it on Google, we see that it's Google.
• When found it on Amazon, we see that it's Amazon.
• When found in Goodreads, we can see that all information has been merged.

See hint for the main options when fetching metadata.

Originally Posted by davidfor

I believe that is working as designed. Though possibly a little unexpected with the interaction between the three

I think what is happening is that when you have just Google and Amazon active, there is not enough information to merge the results. The authors do not match. But, when Goodreads is added, the Goodreads and Google are merged because they have the same ISBN. Then, the Amazon is merged because there is an exact title and author match with the Goodreads result.

That's my guess. I have never looked at the merge algorithm to see what it does, but, the above makes sense with the results you are seeing. I have at times wanted a "no merge" option. But, I have also wanted a way to pick parts of the metadata from the different results. Especially the comment, as the comments in the merged aren't always the best.

I don't think the option @BeckyEbook will have a real effect on this. That is about what happens if the source returns multiple books. With that option on, you could see multiple entries if the source finds multiple good matches. But, if identifiers are used, in this case, Goodreads, Google and the ISBN, there should only be one match for each source.

Thanks to both BeckyEbook and davidfor!
This is fascinating stuff the deeper you dig in to it. I'll keep playing with this as I now have a better understanding. And apologies for the poor formatting on the last posts, I'm figuring that out too now. Cheers!

[rglrglrgl]

I was just getting back into tending to my library after some time away from Calibre and noticed that the option to use the published date as specified per edition on Goodreads appears to have disappeared from the Goodreads Metadata Source plugin.

It seemed to have worked this way in 2018 and I can't see anything in this thread to say it changed.

I am married to having the publication date in the Calibre pubdate field and don't trust combining metadata sources to pull the correct edition date, I've treated Goodreads as my authoritative source for this stuff forever. I have large numbers of translations/critical editions of single works in my library so losing this ability would fundamentally break how I've structured a huge Calibre library.

Am I out of luck? Please tell me the option has moved somewhere I've not thought to look.

[davidfor]

Originally Posted by rglrglrgl

I was just getting back into tending to my library after some time away from Calibre and noticed that the option to use the published date as specified per edition on Goodreads appears to have disappeared from the Goodreads Metadata Source plugin.

It seemed to have worked this way in 2018 and I can't see anything in this thread to say it changed.

I am married to having the publication date in the Calibre pubdate field and don't trust combining metadata sources to pull the correct edition date, I've treated Goodreads as my authoritative source for this stuff forever. I have large numbers of translations/critical editions of single works in my library so losing this ability would fundamentally break how I've structured a huge Calibre library.

Am I out of luck? Please tell me the option has moved somewhere I've not thought to look.

I don't remember changing anything in the plugin related to the published date any time recently. And comparing the current code to the oldest version I have (1.1.9 from 2015). And that does not have any options for the publishing date. From the plugin's point of view, this is working as it always has.

It could be something has changed on GR that didn't break anything obviously in the plugin. To have a look at it, I need some examples that are not working for you and exactly what the problem is.

[rglrglrgl]

Originally Posted by davidfor

I don't remember changing anything in the plugin related to the published date any time recently. And comparing the current code to the oldest version I have (1.1.9 from 2015). And that does not have any options for the publishing date. From the plugin's point of view, this is working as it always has.

It could be something has changed on GR that didn't break anything obviously in the plugin. To have a look at it, I need some examples that are not working for you and exactly what the problem is.

I've been mistaken, it was on the general metadata prefs page as shown here



I'd assumed that with Goodreads carrying both dates that it was a control for Goodreads Metadata Source. My Error.

I found thisin the changelog where that option has been removed from Goodreads in general. I'd previously always gotten the edition-specific date from Goodreads into the pubdate column and had used a separate column for the year of first publication of the work. I have to assume this change was the culprit.

For an example, when I'm looking up John Berger's Ways of Seeing via the ISBN '9780141035796' Goodreads Metadata Source pulls the date 1972, where the corresponding Goodreads page lists the first published date as 1972 while this specific ISBN was published in 2008.

If the ability to prefer either the first published date or the edition-specific date could be added to the plugin I would be appreciative but as it hasn't came up once in the time since I also appreciate the limits on the time and energy of people who maintain these projects.

[davidfor]

Originally Posted by rglrglrgl

I've been mistaken, it was on the general metadata prefs page as shown here

Attachment 184692

I'd assumed that with Goodreads carrying both dates that it was a control for Goodreads Metadata Source. My Error.

I found thisin the changelog where that option has been removed from Goodreads in general. I'd previously always gotten the edition-specific date from Goodreads into the pubdate column and had used a separate column for the year of first publication of the work. I have to assume this change was the culprit.

For an example, when I'm looking up John Berger's Ways of Seeing via the ISBN '9780141035796' Goodreads Metadata Source pulls the date 1972, where the corresponding Goodreads page lists the first published date as 1972 while this specific ISBN was published in 2008.

If the ability to prefer either the first published date or the edition-specific date could be added to the plugin I would be appreciative but as it hasn't came up once in the time since I also appreciate the limits on the time and energy of people who maintain these projects.

That explains it. From memory, Worldcat was removed as the free version of the API was either removed, or restricted in ways that made it impractical for calibre (though I might be mixing that up with another site).

I suppose this hasn't come up as most people are more interested in the first published date. It shouldn't be hard to do. I'll put it on the list of things to look at the next time I need to touch the plugin.

[rglrglrgl]

Originally Posted by davidfor

That explains it. From memory, Worldcat was removed as the free version of the API was either removed, or restricted in ways that made it impractical for calibre (though I might be mixing that up with another site).

I suppose this hasn't come up as most people are more interested in the first published date. It shouldn't be hard to do. I'll put it on the list of things to look at the next time I need to touch the plugin.

That's what happened with Worldcat based on the digging I did.

Thanks for taking a look. The control for it being in Calibre but the effect of that control manifesting in the results from Goodreads Metadata Source makes me hopeful that the functionality is basically already in there.

[davidfor]

Originally Posted by rglrglrgl

That's what happened with Worldcat based on the digging I did.

Thanks for taking a look. The control for it being in Calibre but the effect of that control manifesting in the results from Goodreads Metadata Source makes me hopeful that the functionality is basically already in there.

The way that calibre handled this was simply to ignore the publishing dates from the other sources if that option was selected and Worldcat supplied a date. Each of the plugins have to decide how to handle each metadata field themselves. For Goodreads, there is code to look for "first published" in the book details and using that if it exists. It won't be hard to change that. Adding the option to the configuration and handling that will be more work.

[lairdb]

Is there a way to run the plugin so as to ONLY add the Goodreads ID? I see that I can deselect some of the metadata fields:

...but I can't deselect title, author, etc.

("Why do you want that? That's weird."
"Because I have lots of books with good clean metadata and no Goodreads ID, and I want to improve my Goodreads Sync. Particularly, the Goodreads practices around titles are not my preference*.")


((*Series in the title field is a terrible hack and often mangled, and I also prefer not to have the extended title in Calibre. (E.g. for goodreads:45358322, I'd rather have "Sizing People Up", not "Sizing People Up: A Veteran FBI Agent's User Manual for Behavior Prediction".)))

[davidfor]

The title and author cannot be deselected at the plugin level for any metadata source plugin. They can be deselected in the main metadata download preferences. That might do what you want. I have never experimented with this.

What I do do with when I want to add just an identifier to a book is do the search and copy the link displayed then cancel the search. Then, click on the paste button next to the identifiers field. If the URL is recognised, it will be parsed and added as an identifier. Not all metadata source plugins will do this, but, all the ones I maintain do. This includes Goodreads.

But, for Goodreads, you might be better to use the Goodreads Sync plugin. It has function to get the identifier from Goodreads. You can choose the book from the matches it finds and it lets you choose the edition.

[lairdb]

Originally Posted by davidfor

What I do do with when I want to add just an identifier to a book is do the search and copy the link displayed then cancel the search. Then, click on the paste button next to the identifiers field. If the URL is recognised, it will be parsed and added as an identifier. Not all metadata source plugins will do this, but, all the ones I maintain do. This includes Goodreads.

But, for Goodreads, you might be better to use the Goodreads Sync plugin. It has function to get the identifier from Goodreads. You can choose the book from the matches it finds and it lets you choose the edition.

Thanks for your reply. Yes, both of those methods will work for semi-automated, but with hundreds of books that I'd like to update, that's not presently practical.

Originally Posted by davidfor

The title and author cannot be deselected at the plugin level for any metadata source plugin. They can be deselected in the main metadata download preferences. That might do what you want. I have never experimented with this.

Good tip -- so I experimented. In case it's useful to anyone:

I deselected all fields in the main metadata download preferences, deselected all sources except goodreads, and did a download metadata only. Although the "review downloaded metadata" screen suggests that new content was downloaded even to the deselected fields, it lies; what's shown on the left is an echo of the previously existing content (author, title), a disregarded value (publisher='undefined'), or values that will be merged with an existing multivalue field (identifiers).

Short version: this works. Deselect all fields in the main metadata download preferences, deselect all sources except Goodreads, download metadata only. Goodreads identifiers will be merged into the identifiers field.

1. Yay.

2. davidfor, thanks for the hint, and for the plugin.

3. Unfortunately, this will be less directly useful when the GoodreadsSync plugin stops working due to GRs decision to discontinue the API -- but the metadata is still some of the best. Again, thanks to davidfor.

[shawneedj]

Hello,
I just did a new install of Calbre 5.11.0 and installed the Goodreads metadata plugin but I can't get it to work, also having the same problem with the Fantastic Fiction plugin. I'm using Calibre on Windows 10x64 for the first time, I've used older versions of Calibre on Windows 7 for a long time with the Goodreads and FF plugin with no problems.
When I try to get metadata on a book I get this in the log for Goodreads and a similar message for Fantastic Fiction, something about a certificate failure. I've read about stuff about Goodreads and their API but not knowledgeable enough to understand it. Is that why I can't get metadata from Goodreads anymore and if so why is it doing it for Fantastic fiction too?
Or do I just not have something set up right?
Thanks for any help.

calibre, version 5.11.0
ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click "Show details".

Running identify query with parameters:
{'title': 'The Green Mile', 'authors': ['King, Stephen'], 'identifiers': {'isbn': '9780752821467'}, 'timeout': 30}
Using plugins: Goodreads (1, 5, 1)
The log from individual plugins is below

****************************** Goodreads (1, 5, 1) ******************************
Found 0 results
Downloading from Goodreads took 0.29161977767944336
identify - start. title=The Green Mile, authors=['King, Stephen'], identifiers={'isbn': '9780752821467'}
get_goodreads_id_from_identifiers - isbn=9780752821467
get_goodreads_id_using_api - identifiers=9780752821467
Querying using autocomplete API: https://www.goodreads.com/book/auto_...=9780752821467
Failed to make identify query: 'https://www.goodreads.com/book/auto_complete?format=json&q=9780752821467'
Traceback (most recent call last):
File "mechanize\_urllib2_fork.py", line 1229, in do_open
File "http\client.py", line 1255, in request
File "http\client.py", line 1301, in _send_request
File "http\client.py", line 1250, in endheaders
File "http\client.py", line 1010, in _send_output
File "http\client.py", line 950, in send
File "http\client.py", line 1424, in connect
File "ssl.py", line 500, in wrap_socket
File "ssl.py", line 1040, in _create
File "ssl.py", line 1309, in do_handshake
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.goodreads.__init__", line 136, in get_goodreads_id_using_api
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 287, in _mech_open
File "mechanize\_opener.py", line 193, in open
File "mechanize\_urllib2_fork.py", line 425, in _open
File "mechanize\_urllib2_fork.py", line 414, in _call_chain
File "calibre\utils\browser.py", line 29, in https_open
File "mechanize\_urllib2_fork.py", line 1233, in do_open
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)>

Failed to trying to get Goodreads id using auto_complete API
Traceback (most recent call last):
File "mechanize\_urllib2_fork.py", line 1229, in do_open
File "http\client.py", line 1255, in request
File "http\client.py", line 1301, in _send_request
File "http\client.py", line 1250, in endheaders
File "http\client.py", line 1010, in _send_output
File "http\client.py", line 950, in send
File "http\client.py", line 1424, in connect
File "ssl.py", line 500, in wrap_socket
File "ssl.py", line 1040, in _create
File "ssl.py", line 1309, in do_handshake
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.goodreads.__init__", line 199, in identify
File "calibre_plugins.goodreads.__init__", line 173, in get_goodreads_id_from_identifiers
File "calibre_plugins.goodreads.__init__", line 136, in get_goodreads_id_using_api
File "mechanize\_mechanize.py", line 241, in open_novisit
File "mechanize\_mechanize.py", line 287, in _mech_open
File "mechanize\_opener.py", line 193, in open
File "mechanize\_urllib2_fork.py", line 425, in _open
File "mechanize\_urllib2_fork.py", line 414, in _call_chain
File "calibre\utils\browser.py", line 29, in https_open
File "mechanize\_urllib2_fork.py", line 1233, in do_open
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)>

************************************************** ******************************
The identify phase took 0.41 seconds
The longest time (0.291620) was taken by: Goodreads
Merging results from different sources
We have 0 merged results, merging took: 0.00 seconds

[davidfor]

@shawneedj: That is a certificate error. The usual solution is to open the Goodreads page in the Microsoft browser and let it update the central store. After that, the plugin should work again.

[shawneedj]

Okay,

I just did a another brand new install of windows 10, new install of Calibre 5.11.0
and I went to both the Goodreads and Fantastic fiction websites to update the certificates.
Still get the same messages as in my previous post.

I thought it might be a bug in the latest Calibre since it just came out Feb 12, so I uninstalled that version and tried Calibre 5.9.

Still didn't work.

I'm using Firefox 85.0.2 for my browser, would that make any difference?

There is a setting in Firefox options
"Query OCSP responder servers to confirm the current validity of certificates"
which is currently "on"
I have no idea what that is or whether it has anything to do with Calibre but since it had something about checking certificates, I turned it off.

Plugins still didn't work

As I said, I used older versions of Calibre and the plugins with Windows 7 for years and they always worked great but this is the first I've used Windows 10, so there may be some setting or something I have wrong in Windows or Calibre.

Any other suggestions of what I could try?

Thanks