[Metadata Download Plugin] Goodreads Metadata **Deprecated**

[kiwidude]

As of Calibre 0.8 this plugin is deprecated and will no longer work. Instead you should use the replacement Goodreads Metadata Source plugin which is compatible with Calibre 0.8 onwards.

This plugin allows Calibre to read book information from goodreads.com when you choose to download/fetch metadata. Calibre currently comes with plugins for a number of information sources such as Amazon and Googlebooks. Adding this plugin can potentially increase both the success rate and quality of information retrieved for some of your books.

It is the first of a Goodreads trinity of plugins I will publish in this forum. The other two are for Goodreads cover downloads and shelve syncing.

Main Features of v1.3.3

• Can retrieve title, author, series, isbn, comments, publisher, publication date and tags
• Does not require the ISBN of a book to retrieve metadata, provided both title and author are specified for your Calibre book. In that circumstance it will do a Goodreads title/author search matching with the first result found if any.
• Publication date is the "first published" date where available. This is brilliant for users like me who just want a "date the author wrote it" for sorting purposes rather than "date this particular ISBN edition was published" that other metadata sources provide. In a future version I will make this configurable.
• Tags are created by a mapping between the most popular shelves a book is on and my own hard-coded subset of tag names. You can see the popular shelves on the lower right of the book detail web page on Goodreads. In a future version this may be configurable. For now if you see a popular shelf name listed that is not below for a book you want a particular tag for let me know. Below is the current mappings - Goodreads shelf name on the left, one or more Calibre tags it will map to on the right.
  
  Spoiler:

  calibre_tag_lookup = {
                  'Anthologies': 'Anthologies',
                  'Adventure': 'Adventure',
                  'Adult Fiction': 'Adult',
                  'Adult': 'Adult',
                  'Art': 'Art',
                  'Biography': 'Biography',
                  'Biography Memoir': 'Biography',
                  'Business': 'Business',
                  'Chick-lit': 'Chick-lit',
                  'Childrens': 'Childrens',
                  'Classics': 'Classics',
                  'Comics': 'Comics',
                  'Graphic Novels Comics': 'Comics',
                  'Contemporary': 'Contemporary',
                  'Cookbooks': 'Cookbooks',
                  'Crime': 'Crime',
                  'Fantasy': 'Fantasy',
                  'Feminism': 'Feminism',
                  'Gardening': 'Gardening',
                  'Gay': 'Gay',
                  'Glbt': 'Gay',
                  'Health': 'Health',
                  'History': 'History',
                  'Historical Fiction': 'Historical',
                  'Horror': 'Horror',
                  'Comedy': 'Humour',
                  'Humor': 'Humour',
                  'Health': 'Health',
                  'Inspirational': 'Inspirational',
                  'Sequential Art > Manga': 'Manga',
                  'Modern': 'Modern',
                  'Music': 'Music',
                  'Mystery': 'Mystery',
                  'Non Fiction': 'Non-Fiction',
                  'Paranormal': 'Paranormal',
                  'Religion': 'Religion',
                  'Philosophy': 'Philosophy',
                  'Politics': 'Politics',
                  'Poetry': 'Poetry',
                  'Psychology': 'Psychology',
                  'Reference': 'Reference',
                  'Romance': 'Romance',
                  'Science': 'Science',
                  'Science Fiction': 'Science Fiction',
                  'Science Fiction Fantasy': 'Science Fiction,Fantasy',
                  'Self Help': 'Self Help',
                  'Sociology': 'Sociology',
                  'Spirituality': 'Spirituality',
                  'Suspense': 'Suspense',
                  'Thriller': 'Thriller',
                  'Travel': 'Travel',
                  'Paranormal > Vampires': 'Vampires',
                  'War': 'War',
                  'Western': 'Western',
                  'Language > Writing': 'Writing',
                  'Writing > Essays': 'Writing',
                  'Young Adult': 'Young Adult',
                  }

Special Notes:

• This plugin uses web page scraping to retrieve the data from Goodreads. I've tried to cover as many scenarios as I have found to date but there could be some I've missed. If the data is missing or doesn't match what you would expect from looking at the Goodreads book web page, post all the details including the URL and I will take a look.
• Series information from Goodreads can sometimes be a range for an omnibus/boxset version such as "Harry Potter #1-7". In this circumstance it will assign it the first number in the range specified of "Harry Potter [1]" in Calibre. In a future version this may be configurable, for those of you who put omnibus versions at 0 or 99 for instance.
• Kovid intends in future to make changes to the API for metadata download plugins to improve the functionality and performance. When he publishes that new version, this plugin may stop working. Check back here for updates over the next few months.

Installation Notes:

• Download the attached zip file and install the plugin as described in the Introduction to plugins thread.
• Note that this is not a GUI plugin so it is not intended/cannot be added to context menus/toolbars etc.
• You can enable/disable and configure both it and the other metadata download plugins like Amazon, Google Books etc in Preferences -> Plugins -> Metadata download plugins

Version History:

Spoiler:

Version 1.3.3 - 18 Apr 2011
Fix a typo in the Anthologies tag mapping, remove action-adventure legacy mapping.

Version 1.3.2 - 16 Apr 2011
And yet another change to the website by Goodreads that broke tags again.

Version 1.3.1 - 13 Apr 2011
Fix for yet another website update by Goodreads that broke tags again.

Version 1.3 - 13 Apr 2011
Change the URL searched against as Goodreads have changed their website.
Change all the tag mappings as Goodreads no long has Popular Shelves and has Genres instead.

Version 1.2.1 - 28 Feb 2011
Re-release of 1.1.3 due to incorrect version numbering of earlier release.

Version 1.1.3 - 22 Mar 2011
Return [] for results when no isbn match or no title match
Strip anything in brackets off the title before doing search

Version 1.1.2 - 07 Mar 2011
For compatibility with Calibre v0.7.48 return [] instead of None when no tags for a book.
Changed mapping for historical-fiction to Historical instead of History,Fiction

Version 1.1.1 - 01 Mar 2011
Additional tag mappings as per StevieRaveOn post #12

Version 1.1 - 28 Feb 2011
Fix for scraping rating value with international numeric machine settings
Addition of a horror->Horror and non-fiction->Non-Fiction tag mappings
Change fiction/general-fiction tag mappings from General to Fiction

Version 1.0 - 16 Feb 2011
Initial release of Goodreads Metadata plugin

Thanks due to stormcloude for testing and requesting me to put together a working version based on an earlier attempt by tmcone on this ticket.

[silasgreenback]

Installed this alongside the covers plugin and they both seem to be working nicely.

[stormcloude]

You are a rockstar and went so far above and beyond what I was hoping for when I asked about this. Thank you!

[jumpuphigh]

This is super-helpful. Thanks so much!

[theomeier]

I get an error message and no action on activating the Goodreads plugin:

Calibre Version 07.46

Traceback (most recent call last):
File "site-packages\calibre\gui2\metadata\bulk_download.py", line 245, in handle_result
File "site-packages\calibre\gui2\metadata\bulk_download.py", line 253, in set_metadata
File "site-packages\calibre\library\database2.py", line 1583, in set_metadata
File "site-packages\calibre\library\database2.py", line 1556, in doit
File "site-packages\calibre\library\database2.py", line 2359, in set_rating
File "site-packages\calibre\library\sqlite.py", line 250, in run
IntegrityError: constraint failed

[kiwidude]

Originally Posted by theomeier

I get an error message and no action on activating the Goodreads plugin:

Calibre Version 07.46

Traceback (most recent call last):
File "site-packages\calibre\gui2\metadata\bulk_download.py", line 245, in handle_result
File "site-packages\calibre\gui2\metadata\bulk_download.py", line 253, in set_metadata
File "site-packages\calibre\library\database2.py", line 1583, in set_metadata
File "site-packages\calibre\library\database2.py", line 1556, in doit
File "site-packages\calibre\library\database2.py", line 2359, in set_rating
File "site-packages\calibre\library\sqlite.py", line 250, in run
IntegrityError: constraint failed

Now that is a new one I haven't seen before. Can you please supply the details of the books you had selected? Can you narrow it down to a single book that causes the error and post the ISBN/title author of that book?

[theomeier]

The error happens with every book. See the debug protocol enclosed:

calibre Debug log
calibre 0.7.46
Windows-Vista-6.0.6002-SP2
Windows
('Windows', 'Vista', '6.0.6002')
Python 2.7.0
Windows: ('Vista', '6.0.6002', 'SP2', 'Multiprocessor Free')
Starting up...
Started up in 118.303000212
openlibrary.org covers: HTTP Error 404: Not Found
amazon.com covers: cover for 9780312576554 found
openlibrary.org covers: HTTP Error 404: Not Found
amazon.com covers: cover for 9780982952306 found
openlibrary.org covers: cover for 9780451229571 found
amazon.com covers: cover for 9780451229571 found



I have the pdf ISBN extraction sript from MEKK installed:

Originally Posted by bazbar

For anyone who cares this is how I finally got the many thousands of ISBNs I had in my unorganized ebooks into Calibre on Windows. This is a giant pain in the ass and I might have missed some steps that I went through but at least this might save you a little time. It is better than getting a repetitive stress injury by doing it all by hand (I know, I did about 2000 books by hand before I gave up on that nonsense).

Again, if anyone cares, I feel like ISBN recognition should be a core attribute of any software that is trying to organize ebooks - it is the critical piece of information that allows for proper organization and recognizing it should be as automated as possible. If you don't want to make it happen automatically at least make it an option or somehow build this Python script I'm running into the system so we can choose to run it on our libraries without all of the nonsense below? Anyway, my $.02. Goodluck to anyone else trying.

-------
Download pdftotext from foolabs.com:
ftp://ftp.foolabs.com/pub/xpdf/xpdf-3.02pl4-win32.zip
Extract the contents, or pdftotext.exe alone, to a directory. Add the directory to your path.


1) download and install Python 2.6. do not use a newer version since you will need binary distributions of packages that don't exist for newer versions.
http://www.python.org/ftp/python/2.6.6/python-2.6.6.msi


2) install setuptools that works with Python 2.6
http://pypi.python.org/packages/2.6/...f9589aafe053dc


3) install binary of lxml that works with Python 2.6
http://pypi.python.org/packages/2.6/...145e15a7b878fb


3a) if the binary above failed with an error about a missing dll then download that dll from the web and put it in the same directory as the exe (awesome, right? I know. I love downloading random dlls because they don't include the visual c distribution stuff)


4) set enviornment variables - make sure the following are all in your path:
C:\Program Files\Calibre2\;C:\Python26\Scripts;


5) open command prompt and try 'easy_install mekk.calibre'

5a) if step 5 fails with some kind of error then you probably need to install a compiler - I used MinGW:
http://sourceforge.net/projects/ming...9.exe/download
When that runs if you have options make sure you include gcc and MAKE (I think I selected them all but I don't think you need to)

5b) tell easy_install to use the MinGW compiler
This is done by creating a file called "distutils.cfg" at C:\Python26\Lib\distutils that contains following:

[build]
compiler=mingw32

add MinGW to the path: c:\MinGW\bin;

5c) close command prompt, reopen, retry easy_install mekk.calibre

6) edit .calibre-utils file (if it exists for you) - it should be in your C:\Documents and Settings\<user> folder or similar based on operating system. If you didn't install one of the commands listed then set it to nothing like so:
catdoc =
archmage =
djvutxt =
calibredb = calibredb
pdftotext = pdftotext

Make sure to keep the calibredb line (Calibre should already be in the path). I only have pdftotext working in mine. If it doesn't exist then run the program once, it will likely error out eventually when it finds a missing command, then track down the config file and edit it.

7) run calibre_guess_and_add_isbn (or any of the other options in mekk's script)
-----

automatic ISBN recognition win32 windows calibre

[kiwidude]

Originally Posted by theomeier

The error happens with every book. See the debug protocol enclosed:

calibre Debug log
calibre 0.7.46
Windows-Vista-6.0.6002-SP2
Windows
('Windows', 'Vista', '6.0.6002')
Python 2.7.0
Windows: ('Vista', '6.0.6002', 'SP2', 'Multiprocessor Free')
Starting up...
Started up in 118.303000212
openlibrary.org covers: HTTP Error 404: Not Found
amazon.com covers: cover for 9780312576554 found
openlibrary.org covers: HTTP Error 404: Not Found
amazon.com covers: cover for 9780982952306 found
openlibrary.org covers: cover for 9780451229571 found
amazon.com covers: cover for 9780451229571 found



I have the pdf ISBN extraction sript from MEKK installed:

I took one of those ISBN's above (9780312576554), created an empty book, disabled all the other metadata download plugins except for Goodreads and did Ctrl+D. There was no issue whatsoever for me, metadata was downloaded and populated. I don't have the pdf ISBN extraction script installed which you mention above, I am not sure how it could interfere but then I'm no python guru.

If you PM me with your email address, I will send you a special debug build which will generate a bunch of output so we can hopefully see a little better what may be going on.

[StevieRaveOn]

First of all, thank you so much for this plugin. I've always wanted my tags to just pretty much have genre type information and this looks perfect.

Can you please add "Horror" as a tag? That's top tag for most Stephen King novels.

Also, can you add "Non-Fiction" and maybe not have Fiction and General-Fiction switch to "Fiction" instead of "General"?

[kiwidude]

Originally Posted by StevieRaveOn

First of all, thank you so much for this plugin. I've always wanted my tags to just pretty much have genre type information and this looks perfect.

Can you please add "Horror" as a tag? That's top tag for most Stephen King novels.

Also, can you add "Non-Fiction" and maybe not have Fiction and General-Fiction switch to "Fiction" instead of "General"?

Nice timing, I was going to push a new release out later today so will take a look at the category changes you requested and include those. No Horror category was a bit of an oversight, clearly not a genre that came to mind

If anyone else has any other tag change/addition requests, now would be a good time to suggest them. You can see the list of current mappings on the first post in the spoiler.

[kiwidude]

This update has the following changes:

• Fix for scraping rating value with international numeric machine settings
• Addition of a horror->Horror and non-fiction->Non-Fiction tag mappings
• Change fiction/general-fiction tag mappings from General to Fiction

Download from the first post in this thread.

[StevieRaveOn]

Awesome, thank you so much. Like I said, you have no idea how much this helped me, I hated having tons of tags. I just wanted my tags to be simple genres and you saved me hours of time.

Here is a list of all the popular shelfs. http://www.goodreads.com/shelf Some ones from page 1 that I know would help my collection:


'nonfiction': 'Non-Fiction', (sometimes it's tagged as 1 word)
'biography': 'Biography',
'autobiography': 'Biography',
'memoir': 'Biography',
'reference': 'Reference',
'poetry': 'Poetry',
'historical': 'History',
'politics': 'Politics',
'philosophy': 'Philosophy',
'travel':'Travel',
'psychology':'Psychology',
'music':'Music',
'self-help': 'Self Help',
'historical-fiction': 'History, Fiction'

[kiwidude]

I've just pushed up a newer version which has the additional tag mappings suggested by StevieRaveOn above (thanks for those).

[DoctorOhh]

Originally Posted by StevieRaveOn

'historical-fiction': 'History, Fiction'

Just my 2 cents. All of the mapping have been good but Historical Fiction is a genre all its own and rarely should be lumped in with History.

Of course this will all be mute if you are kind enough to make this a customizable section.

[kiwidude]

Originally Posted by dwanthny

Just my 2 cents. All of the mapping have been good but Historical Fiction is a genre all its own and rarely should be lumped in with History.

Fair point. In my own library I use a tag called "Historical", so I'll change the mapping to that in the next release.

Of course this will all be mute if you are kind enough to make this a customizable section.

Haha, too true. I knew providing default mappings would potentially be contentious but everyone has played nicely in the sandpit so far which is great. I think some Calibre users are so used to overwriting tags to get exactly how they want that a few "dud" mappings now and then out of the list is worth the majority being close enough.

Once Kovid tells me his new API is completely nailed down I'll look to do the major changes to it including customisable shelf/tag mappings. I saw a comment from a feature ticket today that mentions a new identifiers table so will push a new version out once that is released making use of that. It will allow a closer integration between this plugin and my "Goodreads Sync" one - as by using Ctrl+D to get metadata I will be able to store the Goodreads ID from this plugin, so your book will automatically be "linked" for use by the Sync plugin. It would also make sense that if you link a book first using Sync and then do Ctrl+D, that I download metadata for that specific Goodreads book which will greatly increase the chances of a match. I can also scrape other id's that Goodreads has like ASIN and store those too.