[Metadata Source Plugin] Goodreads - Page 6

jaredv · 02-21-2012, 04:57 PM

Hello, in the last few weeks it seems as if the Goodreads metadata plugin has not been retrieving any metadata. I am using the both the latest version of Calibre 0.8.40 and the latest version of the goodreads metadata plugin 1.0.9. Whenever I retrieve metadata, I receive back results from Barnes & Noble, Amazon, as well as the other sources I am using; however, I do not get any results from Goodreads. This is true, even if I had previously retieved metadata from Goodreads and the Goodreads identifier is already listed, or if I manually enter the Goodreads identifier.
I have also had a few books created by goodreads authors, that are not available from other places. I have listed the goodreads identifer manually and the metadata still is not retrieved. I only get a response that no metadata was found.
I have been logged into the Goodreads website when trying to download the metadata.
I know that Goodreads recently changed the source of their metadata and no longer gets a feed from Amazon. Has this effected the plugin? Has the way metadata is handled changed, and the plugin needs to be changed to accomidate it?

Thanks in advance for your assistance and prompt response.

Jared

kiwidude · 02-21-2012, 06:21 PM

@jaredv - welcome to MobileRead.

I haven't had any problems with it, and I just tried it right now and it was fine. And no the Amazon/Goodreads relationship hasn't directly affected this plugin - other than there are some goodreads ids that have disappeared for some books, and covers that were missing while they reloaded more.

Give some specific goodreads id examples, also post the contents of your log in a spoiler. Make double sure you do have the Goodreads plugin enabled of course

. In fact try just having the Goodreads plugin enabled to make sure all output is only related to that.

jaredv · 02-21-2012, 09:07 PM

I tested the goodreads plugin with a few books. I started by using a book I had previously downloaded metadata to using the goodreads plugin. I retrieved the metadata again, and even with the goodreads identifier still there no goodreads results were presented. I had disabled all the other sources except for goodreads.
The book is The Structure of Magic II

Here is the results from trying to download the metada from Goodreads, with the identifier already present:

Spoiler:

Clicking on the goodreads identifier link in the details page before attempting to redownload the metadata correctly took me to the goodreads page for the book. Which is:

Spoiler:

With all the other sources enabled: Barnes & Noble, Amazon, Doubian, etc. The metadata downloaded correctly from the other sources except goodreads.

I tried a 2nd Ebook. This was an Ebook I downloaded from my Barnes & Noble nook library, and imported directly into Calibre from the BN site.
The book is the Sisterhood of Dune by Brian Herbert, Kevin J. Anderson.
The only metadata that importated into Calibre was the title, author, and publisher.
Once the ebook was in Calibre, I disabled the plugins for all the metadata sources except Goodreads and downladed the metadata. I received the No Metadata found error. Details below. However copying the search URLS listed and pasting them in a web browser brings up the correct book. Renabling the rest of the sources, correctly downloaded metadata from those sources. Except for goodreads.

Downloading metadata with only Goodreads enabled:

Spoiler:

Finally after downloading the metadata for sisterhood of dune from all the other sources. I dow it a second time with all of the sources enabled including goodreads, knowing that more information about the book would be available then just the title and author. Again goodreads did not successfully return metadata but the log showed slightly different results:

Spoiler:

I hope this all helps out.

Thanks,
Jared

kiwidude · 02-22-2012, 05:53 AM

@Jared - perfect response thanks, exactly the sort of stuff I can make use of. Looking at the logs you displayed:

Code:

Could not find title/authors/goodreads id for u'http://www.goodreads.com/book/show/12904336-sisterhood-of-dune'
Goodreads: u'12904336' Title: 'Sisterhood of Dune' Authors: None

There is logic in the plugin that says if it can't find *all* of a goodreads id, title and authors as the result of scraping a web page then it aborts returning any results, displaying that first error line. The line underneath displays what it did manage to find - which as you can see is that while it got the goodreads id and title, it did not manage to detect the authors for that book.

Now it could well be that Goodreads have made a small change to their web page which is causing that issue. It is interesting that when I tried it last night I did not have that problem, though it could be like other websites have done in the past that the new website changes had not yet been globally rolled out and we were hitting web servers delivering different versions of the page.

Probably won't have time to look at this tonight but certainly will tomorrow night - thanks for the details, it greatly helps.

jaredv · 02-22-2012, 12:14 PM

@kiwidude I am glad I could help. Thanks for your quick response. Please let me know if I can be of any assitance to you for anything else.

kiwidude · 02-22-2012, 04:46 PM

@jaredv - I have just given that exact goodreads id for Sisterhood of Dune a go and it works perfectly for me (from the UK - where are you?). Perhaps there is some sort of regional web server issue going on. Or maybe like Kovid faced with Amazon there is some sort of other issue? I never did look at his code to see exactly what he changed on that.

No-one else has reported an issue - perhaps a few other users could try for that book. Just create an empty book, set the identifiers to be goodreads:12904336, disable the other metadata download plugins and then do a metadata download to see whether it works or not.

Just for the record jaredv, this is what I get back:

Spoiler:

tilia · 02-22-2012, 05:47 PM

I'm in Norway, and had no problem downloading metadata from Goodreads for that id.

ElizabethN · 02-23-2012, 12:16 AM

No problem from Nebraska (about 10pm MST) for that goodreads id.

GMFuller · 02-23-2012, 12:27 AM

It works from BC, Canada (9:30PM PST).

kiwidude · 02-23-2012, 03:01 AM

Thanks very much folks - jaredv, don't know what to tell you as clearly I am not alone in it working for me

.

When I get time I will send you a special build which will save the html that the plugin finds to a file, so I can compare it to what I am seeing.

jaredv · 02-23-2012, 06:28 PM

@kiwidude I will try what you described with the empty book and the goodreads ID. I am in the US in NY State.

jaredv · 02-23-2012, 07:34 PM

@kiwidude : I created an empty book, and did not put the author or title in the book, just the goodreads id you gave me. It still did not successfully retrieve the metadata. You can see the results below. I did notice that it successfully resolved the book title, but not the author.
All my other metadata sources were disabled. I tried to download just the cover as well, and nothing was returned.

Spoiler:

After not being successful not retrieving the metadata with the completly empty book and only the goodreads id; I looked at your data and put in the tite and author. Again i was unsucessful in downloading any metadata.

Spoiler:

Looking more closely at your results and mine; I did notice two differences.

Spoiler:

It seems as if in the author search, your search was looking for author by LastName, FirstName and my search was looking for FirstName LastName

I have calibre configured to set the Author as FirstName LastName and the Author Sort to LastName, FirstName.

Attached is a screen cap of the author and title section from my Edit Metadata dialog box.

The only other 'unique' thing may be that I am using Ubuntu 11.10 for my operating system.

I hope this all helps. Feel free to send me a test plugin to try and get more info.

Thanks again for all your help.

Jared

jaredv · 02-23-2012, 07:38 PM

@kiwidude My only other thought about the author sort, is when I had 'unknown' author and title, the title was successfully found using the goodreads id, but not the author.

Thanks again.

kiwidude · 02-24-2012, 03:05 AM

@jaredv - thanks but I'm afraid all that author stuff is a complete red herring.

In the situation where a goodreads id is specified the plugin doesn't look at any of title or author (hence why my instructions above for others to test it said only to create the empty book and specify the id). The author name actually on your book is only used in the situation where the plugin has no isbn or goodreads id, and hence has to do a "search" against the goodreads site and match by title/author. Even in that situation author sort order does not matter as the plugin takes care of the permutations.

I need to see the actual html you are receiving back from Goodreads. You are a Windows user, right? Use the attached version of the plugin, which will write out a file called at c:\goodreads.html when you try the metadata download. Run it once for that same Sisterhood of Dune book above, then immediately put that in a zip and post or PM a link to it back to me. I will take a look to compare with what I see.

jaredv · 02-25-2012, 09:15 PM

Quote:

Originally Posted by kiwidude

@jaredv -... You are a Windows user, right? Use the attached version of the plugin, which will write out a file called at c:\goodreads.html when you try the metadata download.

@kiwidude

I am not using Windows. I am using linux, specifically Ubuntu 11.10.

Do you have hard coded paths in your plugin? Since I don't have a 'c drive' I did notice that the html output file was written in my home directory as a file named "c:\goodreads.html"

Here is the html output using the debug plugin you sent me.

Thanks for the time you are taking to resolve this.

02-21-2012, 09:07 PM	#78
jaredv Junior Member Posts: 7 Karma: 10 Join Date: Feb 2012 Device: nook color	Test of goodreads plugin I tested the goodreads plugin with a few books. I started by using a book I had previously downloaded metadata to using the goodreads plugin. I retrieved the metadata again, and even with the goodreads identifier still there no goodreads results were presented. I had disabled all the other sources except for goodreads. The book is The Structure of Magic II Here is the results from trying to download the metada from Goodreads, with the identifier already present: Spoiler: calibre, version 0.8.40 ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click Show Details. Starting download Query: title:The Structure of Magic II; authors:John Grinder & Richard Bandler; douban:2539780, amazon:0831400498, isbn:9780831400491, goodreads:821993 Running identify query with parameters: {u'authors': [u'John Grinder', u'Richard Bandler'], u'identifiers': {u'douban': u'2539780', u'amazon': u'0831400498', u'isbn': u'9780831400491', u'goodreads': u'821993'}, u'timeout': 30, u'title': u'The Structure of Magic II'} Using plugins: Goodreads The log from individual plugins is below **************************** Goodreads ************************** Request extra headers: [('User-agent', 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)')] Found 0 results Downloading from Goodreads took 1.0803771019 Goodreads book url: u'http://www.goodreads.com/book/show/821993' Could not find title/authors/goodreads id for u'http://www.goodreads.com/book/show/821993' Goodreads: u'821993' Title: 'The Structure of Magic II: A Book About Communication and Change (Book 2)' Authors: None ********************************************** ************************** The identify phase took 1.24 seconds The longest time (1.080377) was taken by: Goodreads Merging results from different sources and finding earliest publication dates from the xisbn service We have 0 merged results, merging took: 0.00 seconds Clicking on the goodreads identifier link in the details page before attempting to redownload the metadata correctly took me to the goodreads page for the book. Which is: Spoiler: http://www.goodreads.com/book/show/821993 With all the other sources enabled: Barnes & Noble, Amazon, Doubian, etc. The metadata downloaded correctly from the other sources except goodreads. I tried a 2nd Ebook. This was an Ebook I downloaded from my Barnes & Noble nook library, and imported directly into Calibre from the BN site. The book is the Sisterhood of Dune by Brian Herbert, Kevin J. Anderson. The only metadata that importated into Calibre was the title, author, and publisher. Once the ebook was in Calibre, I disabled the plugins for all the metadata sources except Goodreads and downladed the metadata. I received the No Metadata found error. Details below. However copying the search URLS listed and pasting them in a web browser brings up the correct book. Renabling the rest of the sources, correctly downloaded metadata from those sources. Except for goodreads. Downloading metadata with only Goodreads enabled: Spoiler: calibre, version 0.8.40 ERROR: No matches found: <p>Failed to find any books that match your search. Try making the search <b>less specific</b>. For example, use only the author's last name and a single distinctive word from the title.<p>To see the full log, click Show Details. Starting download Query: title:Sisterhood of Dune; authors:Brian Herbert & Kevin J. Anderson Running identify query with parameters: {u'authors': [u'Brian Herbert', u'Kevin J. Anderson'], u'identifiers': {}, u'timeout': 30, u'title': u'Sisterhood of Dune'} Using plugins: Goodreads The log from individual plugins is below ************************** Goodreads ************************** Request extra headers: [('User-agent', 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)')] Found 0 results Downloading from Goodreads took 1.44699001312 Querying: http://www.goodreads.com/search?sear...e=books&search[query]=Sisterhood+of+Dune+Brian+Herbert Examining up to 7 editions: http://www.goodreads.com/work/editions/15313650 No matches found with query: u'http://www.goodreads.com/search?search_type=books&search[query]=Sisterhood+of+Dune+Brian+Herbert' ********************************************** ************************** The identify phase took 1.64 seconds The longest time (1.446990) was taken by: Goodreads Merging results from different sources and finding earliest publication dates from the xisbn service We have 0 merged results, merging took: 0.00 seconds Finally after downloading the metadata for sisterhood of dune from all the other sources. I dow it a second time with all of the sources enabled including goodreads, knowing that more information about the book would be available then just the title and author. Again goodreads did not successfully return metadata but the log showed slightly different results: Spoiler: ************************** Goodreads ************************** Request extra headers: [('User-agent', 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)')] Found 0 results Downloading from Goodreads took 2.90368914604 Querying: http://www.goodreads.com/search?sear...e=books&search[query]=9781429986571 ISBN match location: u'http://www.goodreads.com/book/show/12904336-sisterhood-of-dune' Goodreads book url: u'http://www.goodreads.com/book/show/12904336-sisterhood-of-dune' Could not find title/authors/goodreads id for u'http://www.goodreads.com/book/show/12904336-sisterhood-of-dune' Goodreads: u'12904336' Title: 'Sisterhood of Dune' Authors: None ********************************************** **************************** I hope this all helps out. Thanks, Jared

02-22-2012, 05:53 AM	#79
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	@Jared - perfect response thanks, exactly the sort of stuff I can make use of. Looking at the logs you displayed: Code: Could not find title/authors/goodreads id for u'http://www.goodreads.com/book/show/12904336-sisterhood-of-dune' Goodreads: u'12904336' Title: 'Sisterhood of Dune' Authors: None There is logic in the plugin that says if it can't find all of a goodreads id, title and authors as the result of scraping a web page then it aborts returning any results, displaying that first error line. The line underneath displays what it did manage to find - which as you can see is that while it got the goodreads id and title, it did not manage to detect the authors for that book. Now it could well be that Goodreads have made a small change to their web page which is causing that issue. It is interesting that when I tried it last night I did not have that problem, though it could be like other websites have done in the past that the new website changes had not yet been globally rolled out and we were hitting web servers delivering different versions of the page. Probably won't have time to look at this tonight but certainly will tomorrow night - thanks for the details, it greatly helps.

02-22-2012, 04:46 PM	#81
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	@jaredv - I have just given that exact goodreads id for Sisterhood of Dune a go and it works perfectly for me (from the UK - where are you?). Perhaps there is some sort of regional web server issue going on. Or maybe like Kovid faced with Amazon there is some sort of other issue? I never did look at his code to see exactly what he changed on that. No-one else has reported an issue - perhaps a few other users could try for that book. Just create an empty book, set the identifiers to be goodreads:12904336, disable the other metadata download plugins and then do a metadata download to see whether it works or not. Just for the record jaredv, this is what I get back: Spoiler: Starting download Query: title:Sisterhood of Dune; authors:Herbert, Brian; goodreads:12904336 Running identify query with parameters: {u'authors': [u'Herbert, Brian'], u'identifiers': {u'goodreads': u'12904336'}, u'timeout': 30, u'title': u'Sisterhood of Dune'} Using plugins: Goodreads The log from individual plugins is below **************************** Goodreads ************************** Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.78 Safari/532.5')] Found 1 results Downloading from Goodreads took 2.50899982452 --- Title : Sisterhood of Dune Author(s) : Brian Herbert & Kevin J. Anderson Publisher : Tom Doherty Associates Tags : Science Fiction, Fantasy Rating : 3.6 Published : 2012-01-03T00:00:00+00:00 Identifiers : isbn:9781429986571, goodreads:12904336 Comments : It is eighty-three years after the last of the thinking machines were destroyed in the Battle of Corrin, after Faykan Butler took the name of Corrino and established himself as the first Emperor of a new Imperium. Great changes are brewing that will shape and twist all of humankind. The war hero Vorian Atreides has turned his back on politics and Salusa Secundus. The descendants of Abulurd Harkonnen Griffen and Valya have sworn vengeance against Vor, blaming him for the downfall of their fortunes. Raquella Berto-Anirul has formed the Bene Gesserit School on the jungle planet Rossak as the first Reverend Mother. The descendants of Aurelius Venport and Norma Cenva have built Venport Holdings, using mutated, spice-saturated Navigators who fly precursors of Heighliners. Gilbertus Albans, the ward of the hated Erasmus, is teaching humans to become Mentats…and hiding an unbelievable secret. The Butlerian movement, rabidly opposed to all forms of “dangerous technology,” is led by Manford Torondo and his devoted Swordmaster, Anari Idaho. And it is this group, so many decades after the defeat of the thinking machines, which begins to sweep across the known universe in mobs, millions strong, destroying everything in its path. Every one of these characters, and all of these groups, will become enmeshed in the contest between Reason and Faith. All of them will be forced to choose sides in the inevitable crusade that could destroy humankind forever…. Goodreads book url: u'http://www.goodreads.com/book/show/12904336' ********************************************** **************************** The identify phase took 2.60 seconds The longest time (2.509000) was taken by: Goodreads Merging results from different sources and finding earliest publication dates from the xisbn service We have 1 merged results, merging took: 0.51 seconds

02-22-2012, 05:47 PM	#82
tilia Evangelist Posts: 432 Karma: 1720909 Join Date: Mar 2011 Device: Voyage, K3	Sisterhood of Dune I'm in Norway, and had no problem downloading metadata from Goodreads for that id.

02-24-2012, 03:05 AM	#89
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	@jaredv - thanks but I'm afraid all that author stuff is a complete red herring. In the situation where a goodreads id is specified the plugin doesn't look at any of title or author (hence why my instructions above for others to test it said only to create the empty book and specify the id). The author name actually on your book is only used in the situation where the plugin has no isbn or goodreads id, and hence has to do a "search" against the goodreads site and match by title/author. Even in that situation author sort order does not matter as the plugin takes care of the permutations. I need to see the actual html you are receiving back from Goodreads. You are a Windows user, right? Use the attached version of the plugin, which will write out a file called at c:\goodreads.html when you try the metadata download. Run it once for that same Sisterhood of Dune book above, then immediately put that in a zip and post or PM a link to it back to me. I will take a look to compare with what I see. Last edited by kiwidude; 02-26-2012 at 04:13 AM. Reason: Removing attachment as purpose completed

02-21-2012, 04:57 PM	#76
jaredv Junior Member Posts: 7 Karma: 10 Join Date: Feb 2012 Device: nook color	Hello, in the last few weeks it seems as if the Goodreads metadata plugin has not been retrieving any metadata. I am using the both the latest version of Calibre 0.8.40 and the latest version of the goodreads metadata plugin 1.0.9. Whenever I retrieve metadata, I receive back results from Barnes & Noble, Amazon, as well as the other sources I am using; however, I do not get any results from Goodreads. This is true, even if I had previously retieved metadata from Goodreads and the Goodreads identifier is already listed, or if I manually enter the Goodreads identifier. I have also had a few books created by goodreads authors, that are not available from other places. I have listed the goodreads identifer manually and the metadata still is not retrieved. I only get a response that no metadata was found. I have been logged into the Goodreads website when trying to download the metadata. I know that Goodreads recently changed the source of their metadata and no longer gets a feed from Amazon. Has this effected the plugin? Has the way metadata is handled changed, and the plugin needs to be changed to accomidate it? Thanks in advance for your assistance and prompt response. Jared

02-21-2012, 06:21 PM	#77
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	@jaredv - welcome to MobileRead. I haven't had any problems with it, and I just tried it right now and it was fine. And no the Amazon/Goodreads relationship hasn't directly affected this plugin - other than there are some goodreads ids that have disappeared for some books, and covers that were missing while they reloaded more. Give some specific goodreads id examples, also post the contents of your log in a spoiler. Make double sure you do have the Goodreads plugin enabled of course . In fact try just having the Goodreads plugin enabled to make sure all output is only related to that.

02-22-2012, 12:14 PM	#80
jaredv Junior Member Posts: 7 Karma: 10 Join Date: Feb 2012 Device: nook color	@kiwidude I am glad I could help. Thanks for your quick response. Please let me know if I can be of any assitance to you for anything else.

02-23-2012, 12:16 AM	#83
ElizabethN reader, ebook junkie Posts: 110 Karma: 436806 Join Date: Dec 2007 Location: western nebraska Device: in use kindle oasis 2nd and 3rd gen, kobo forma, boox poke5	No problem from Nebraska (about 10pm MST) for that goodreads id.

02-23-2012, 12:27 AM	#84
GMFuller Connoisseur Posts: 71 Karma: 2794644 Join Date: Sep 2010 Device: Kobo	It works from BC, Canada (9:30PM PST).

02-23-2012, 03:01 AM	#85
kiwidude Calibre Plugins Developer Posts: 4,636 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Thanks very much folks - jaredv, don't know what to tell you as clearly I am not alone in it working for me . When I get time I will send you a special build which will save the html that the plugin finds to a file, so I can compare it to what I am seeing.

02-23-2012, 06:28 PM	#86
jaredv Junior Member Posts: 7 Karma: 10 Join Date: Feb 2012 Device: nook color	@kiwidude I will try what you described with the empty book and the goodreads ID. I am in the US in NY State.

02-23-2012, 07:38 PM	#88
jaredv Junior Member Posts: 7 Karma: 10 Join Date: Feb 2012 Device: nook color	@kiwidude My only other thought about the author sort, is when I had 'unknown' author and title, the title was successfully found using the goodreads id, but not the author. Thanks again.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] Goodreads Sync	kiwidude	Plugins	1721	04-18-2024 10:22 AM
[Metadata Download Plugin] Goodreads Metadata Deprecated	kiwidude	Plugins	30	04-23-2011 02:10 PM
[Covers Plugin] Goodreads Covers Deprecated	kiwidude	Plugins	13	04-17-2011 05:09 PM
metadata plugin	redneck_momma	Plugins	1	05-21-2010 08:41 PM

Advert

Advert