Wanted: Goodreads plug-in

[kiwidude]

Originally Posted by nickdma

Glad to see some progress on this. I saw your post on GoodReads, I hope they response favorably.

I'm sure you saw their response, which for anyone reading this was that they have some degree of "leniency" before they warn about an application abusing their API. I've tried to explain that this particular usage (synchronising with a user's GoodReads shelves) will hopefully encourage users to the GoodReads site.

Just a note on shelves. The 3 default shelves are "exclusive", which means a book can only be on one at any given time. Adding a book to "read" takes it off "to-read/currently-reading" for you.

Nice to know. In fact I can't find an API function to remove a book from a shelf so that will be one less menu option in the plugin . As you say you will just add the book to a "different" shelf in Goodreads if you put it on the wrong one. I was already intending a "View shelf" menu option which will take you direct to the shelf contents on Goodreads so you could delete it completely from there if you messed up.

I actually did some playing around with the GoodReads api this weekend. I put together a perl script (cause that's what I'm comfortable with) to fetch a user's shelf content, and update a custom column in calibre. I created a "tag" column in calibre with a value for each person in the family. I now run my script daily to fetch their "read" and "to-read" lists, compare those with the calibre db, and update the "read" custom column adding their name if they've read the book.

This lets me maintain a single library (it gets rsynced to their computers) with read tracking.

Interesting ideas, and touches on one of the thoughts I had of the synchronisation possibly being two ways. i.e. the ability to somehow update books in Calibre to indicate something about the shelf they are on in Goodreads.

One "problem" I see is coming up with a list of options as to how people would want to flag those books in Calibre. Some people would use a tag, others would use a custom column like you have, there may be other ways I haven't thought of. It's almost like you want a mini rules engine for each shelf in terms of if book on this shelf, add "xxx" tag to book in Calibre, if book on this shelf, put "yyy" in the "zzz" custom column etc.

Also you have in passing mentioned another issue I hadn't considered at all - multiple Goodreads user accounts with a single Calibre usage. So if a Calibre user of this plugin has multiple accounts, choosing the Goodreads account must be part of the action. And as each user could need different "actions" if we are synchronising data to Calibre (e.g. different values in a custom column) there needs to be per account customisation of those.

The final missing piece is an easy way to put a calibre book on a GoodReads shelf.

I had a bugger of a time dealing with comparing books between calibre and goodreads. Variations in title and author spelling, etc. ISBN helps, but not everyone placed the book on their shelf using the same ISBN.

Yeah that was always going to be an issue and not an easy one to solve - even if we had a Goodreads metadata download plugin which isn't likely. I'm going to keep the plugin simple (initially at least) and focus only on ISBN. If there is a mismatch, I will find a way to give as much information to the user as possible about it and leave it to them to resolve the issue (such as updating the ISBN in Calibre then trying again).

There could be issues when adding books to shelves too - such as books that have no ISBN, or the ISBN is not known by Goodreads (dunno how likely that is given they seem fairly confident of their database sources but it could happen I guess). Again all I can think of at the moment is to present a dialog after the add summarising the information on the books that could not be added and let the user resolve it. They have the "Search the Internet" plugin to let them find the book easily on the Goodreads site, they can then paste that into the Calibre data and then add to shelf again.

This has been a big help. Now when I go to look for content to send to my reader, I can use a saved query to restrict calibre to all the books I haven't read, and then poke around by series/author etc. I can easily see which of those books someone else has read as well. I like to try to read things the kids are reading.

Very cool, love the idea of "reading with the family" and integrating that into how you synchronise with Calibre.

[kiwidude]

Originally Posted by stormcloude

That's pretty much how I thought it would be. Thanks for confirming my suspicions.

And I'm also guessing there's no way to make each Calibre installation look independent and unique, so they're not all lumped together under the same application umbrella.

The issues are identical I believe to the LibraryThing plugin. An "application" is identified by a "devkey". You could try to apply for multiple devkeys, but as stated in the LibraryThing thread it wouldn't take them long to figure out what was going on and likely just complicate the devkey process which currently is nicely simple.

You'd think the goodwill from users and Calibre bringing them more readers to check the site out would be a benefit, but I guess not.

From a Calibre perspective we like to think that However I can't think of a genuine reason why a Calibre user will feel the need to open a Goodreads account only because they now get some ratings, book covers or whatever when they download metadata for a book. How many people opened librarything.com accounts (until they were forced to!)? Or if you were forced to open an account, how many people have regularly if at all gone back to librarything since they signed up?

I think it would require a direct approach from Kovid to gain approval for a metadata download plugin and justify the benefits to them. However given we have a "working" alternative from Amazon that does most of what people need I don't think there is any real priority on doing that. IMHO it is a "nice to have" to fill gaps in Amazon's data - it would be great but not critical to using Calibre today.

OTOH creating features that interact with the social aspects of the Goodreads site do I believe create reasons for Calibre users to register and visit their website and is hopefully easier to justify. After all page visits are what such sites live and die on. Hopefully this "shelve" based plugin will survive in that space.

[nickdma]

Also you have in passing mentioned another issue I hadn't considered at all - multiple Goodreads user accounts with a single Calibre usage. So if a Calibre user of this plugin has multiple accounts, choosing the Goodreads account must be part of the action. And as each user could need different "actions" if we are synchronising data to Calibre (e.g. different values in a custom column) there needs to be per account customisation of those.

I had assumed that any preferences for the plugin, such as userid for GoodReads, would be stored locally, and not in the calibre database. At least in my case (though I can see other's sharing calibre on a single PC), each user has their own calibre install. We just happen to use the same database.

One "problem" I see is coming up with a list of options as to how people would want to flag those books in Calibre. Some people would use a tag, others would use a custom column like you have, there may be other ways I haven't thought of. It's almost like you want a mini rules engine for each shelf in terms of if book on this shelf, add "xxx" tag to book in Calibre, if book on this shelf, put "yyy" in the "zzz" custom column etc.

All points which have so far kept me from "releasing" my script. I'm not opposed to sharing, but it's hardly shrink wrap software. It's got many hardcoded settings, such as the calibre custom table name, and users and such.

My particular use case probably isn't all too common I would think. I would think the most common use case would be to simply sync a custom read column with GoodReads. But then, it's just as easy to mark it as read per your normal calibre workflow, and then use the plugin you're creating to "export" that to GoodReads. In my case, I had need to obtain "read" data for others. If it was just myself, a custom column would have been adequate.

Yeah that was always going to be an issue and not an easy one to solve - even if we had a Goodreads metadata download plugin which isn't likely. I'm going to keep the plugin simple (initially at least) and focus only on ISBN. If there is a mismatch, I will find a way to give as much information to the user as possible about it and leave it to them to resolve the issue (such as updating the ISBN in Calibre then trying again).

So far, I've found going from calibre to GoodReads pretty easy and reliable. I haven't done exhaustive testing, but I've yet to lookup an ISBN on GR and not find it. I would think if you ask GR for a book ID by ISBN (if available in calibre) you'll get a hit quite often. If no hit by ISBN, then a title/author should hit. The GR search engine seems to do a good job of sorting out quirky title/author mashups.

So I think in terms of the "add to shelf" plugin, it should be right quite often. As you said, when it's not, the search the internet plugin fills in the gap.

You remove a book from a shelf with:
http://www.goodreads.com/shelf/add_t...ove&book_id=id
I don't see much need for the plugin to remove a book, sending them to their shelves is probably sufficient. One could argue that not allowing removal of books is best, since it prevents mishaps with a users shelves.

[kiwidude]

Originally Posted by nickdma

I had assumed that any preferences for the plugin, such as userid for GoodReads, would be stored locally, and not in the calibre database. At least in my case (though I can see other's sharing calibre on a single PC), each user has their own calibre install. We just happen to use the same database.

In your particular case that will work fine. But there are plenty of users out there who do not have individual windows accounts but still share a PC. So I will implement it as if the latter is the case and that will allow for both cases. If you define multiple users in the plugin configuration dialog, you will get a second level menu below the action e.g. Goodreads -> Add to shelf->User: Grant->to-read whereas if you only have a single user defined that "User: xxx" level menu is not required.

All points which have so far kept me from "releasing" my script. I'm not opposed to sharing, but it's hardly shrink wrap software. It's got many hardcoded settings, such as the calibre custom table name, and users and such.

My particular use case probably isn't all too common I would think. I would think the most common use case would be to simply sync a custom read column with GoodReads. But then, it's just as easy to mark it as read per your normal calibre workflow, and then use the plugin you're creating to "export" that to GoodReads. In my case, I had need to obtain "read" data for others. If it was just myself, a custom column would have been adequate.

True, I happen to mark my books with a tag to indicate they have been read rather than a custom column. And I'm sure I'm not the only one, so again supporting both cases would be required.

The question of "syncing" of course raises the issue of "who is the master". If I select a bunch of books and add to a shelf in Goodreads, that implies Goodreads is the master (as I am not removing books from my Goodreads shelf that were not in my selection).

If I were to add an "Update from shelf" menu option at some point (allowing a book to be tagged or a custom column populated as discussed), what do we do about Calibre books that were tagged from a previous update but are no longer on that shelf? In the case of the "read" shelf this is hopefully less likely to happen. However if you chose to synchronise the "currently-reading" shelf, this could happen a lot. Should we leave it up to users to manually remove such tags in Calibre not now in the Goodreads list? Or do we instead call the option "Sync with shelf", choose Goodreads to be the master and automatically update other Calibre books no longer on the shelf by removing the tag value?

So far, I've found going from calibre to GoodReads pretty easy and reliable. I haven't done exhaustive testing, but I've yet to lookup an ISBN on GR and not find it. I would think if you ask GR for a book ID by ISBN (if available in calibre) you'll get a hit quite often. If no hit by ISBN, then a title/author should hit. The GR search engine seems to do a good job of sorting out quirky title/author mashups.

Good to know. I'm thinking about having a dialog displaying the results of an add operation - listing books in order of those not able to be synced (e.g. no ISBN or no match in Goodreads), those that were added and those that were on the list already (as I think I will add a call to get the list of books on a shelf before doing an add - no point in calling add 15 times if 14 were already on the shelf). From this dialog you will ultimately be able to do actions to fix the problem, such as with a right-click try to find an ISBN and then try again.

If I ever implement the "Update from shelf" or "Sync from shelf" option it could similarly be used to resolve mismatches in ISBN values to again retry as required without having to try to note down the offending books, manually fix up and start over.

You remove a book from a shelf with:
http://www.goodreads.com/shelf/add_t...ove&book_id=id
I don't see much need for the plugin to remove a book, sending them to their shelves is probably sufficient. One could argue that not allowing removal of books is best, since it prevents mishaps with a users shelves.

Ahh, nice to know if not exactly intuitive I had only scanned the top level API names at this point and didn't spot a remove option. I think I will still have the remove option in there, I'll just throw a warning dialog up before it to remind the user of the risks if they got it wrong.

[DoctorOhh]

Originally Posted by kiwidude

The issues are identical I believe to the LibraryThing plugin. An "application" is identified by a "devkey". You could try to apply for multiple devkeys, but as stated in the LibraryThing thread it wouldn't take them long to figure out what was going on and likely just complicate the devkey process which currently is nicely simple.

Actually the Librarything plugin doesn't ID itself with a Devkey. Many moons ago Kovid tried this and it quickly exceeded the 1000 requests per day and that key was banned. I believe he scrapes the website for the data currently.

Originally Posted by kiwidude

OTOH creating features that interact with the social aspects of the Goodreads site do I believe create reasons for Calibre users to register and visit their website and is hopefully easier to justify. After all page visits are what such sites live and die on. Hopefully this "shelve" based plugin will survive in that space.

This is very true. This type of plugin should be a win win scenario for the calibre user and Goodreads. If the user wasn't active on Goodreads why would they even use the plugin. I know it might make me bring my shelves current and go to the site more frequently then I do now.

[kiwidude]

Ok, I've been working on this the last few days and here is what it looks like so far...

The initial release will focus just on pushing information into Goodreads (i.e. updating your shelves in Goodreads with books from Calibre). So here is the functionality implemented to date in the plugin:


When you first start after installing the plugin, the only option you will have in the menu is "Customize plugin", which will take you to this dialog, showing the result of a user being setup:


As you can see I am supporting the possiiblity of multiple Goodreads accounts for users sharing the same machine logon. The very first time you create a "user profile" for the plugin you just give it a name (to appear in submenus if you have multiple of them), click the Authorize button and confirm in the launched Goodreads web page as I described in an earlier post. At that point it will display a list of the shelves you can work with. In a future version this dialog could be expanded to support "actions" associated with each shelf for an "Update from Goodreads" menu option I have mentioned previously, like "add tag xxx" or "put value yyy in custom column yyy".

The add to/remove from shelf actions will attempt to do as few API calls as possible, reading the contents of the shelf and then only updating for books in your Calibre selection that it needs to. However one of the big problems mentioned previously is missing ISBNs (or total lack thereof for some books). At the moment, after the action I display this summary dialog:

Here you can see information on the results of the add or remove action. You also have the opportunity to update the ISBN value to do a retry with various right-clicks to help get the data you need.

However this is only part of the longer term approach. I think what I need to do is add a custom column to Calibre containing the "Goodreads Id" and populate it as the plugin works with your selected books. That way any books which genuinely do not have an ISBN can still be "matched" and not have to ask the user repeatedly. It will also over time reduce the number of API calls being made as at the moment for every ISBN I have to lookup the Goodreads Id before making another call to add to the shelf.

With that in place I should be able to have some additional functionality on that summary dialog such as using the Goodreads API to search for books matching title/author, display in a popup and allow you to choose.

The "View shelf" option just launches a web browser with the selected shelf displayed in Goodreads.

It is fully functional at the moment, but as I am about to commit surgery on it to investigate the custom column approach I'm not making it available to play with as yet. Thoughts/suggestions welcomed.

[kovidgoyal]

See add_custom_book_data in database2 that's an API specifically for use by plugins.

[kiwidude]

Originally Posted by kovidgoyal

See add_custom_book_data in database2 that's an API specifically for use by plugins.

Awesome, thanks Kovid, that does sound just what I am after...

[kiwidude]

Here are a bunch of screenshots from the latest version. I've changed the flow a bit from my original ideas on it but the basic fundamentals are still the same of add/remove to shelves, syncing from shelves, and managing the links between a Calibre book and an equivalent in Goodreads.

The "Sync from shelf" functionality has made it into this first version, I decided it was too much fun to leave out. It allows you to specify a number of actions to be performed for each shelf, such as adding/removing tags, or adding/removing values from custom columns. So for instance you could set a #read Y/N custom column to Y for books on your 'read' shelf on Goodreads when you sync from it.

I'm pretty much done with the development other than fixing any bugs and addressing any suggestions for usability improvements to menus/dialogs etc. So if you fancy giving it a test, please drop me a PM with your e-mail. It's about 3,000 lines of code and certainly the most complex plugin I have written to date so there's bound to be something I have missed along the way...

[kiwidude]

Thanks to all of you who PM'd me - you should have the latest beta version that I sent just now. If you don't receive it please check your spam filters etc. Once I get your feedback with any changes I hope to release the plugin on this forum sometime this week.

[nickdma]

I think alot of people will be pleased with this plugin. Kiwidude did an amazing job with it.

It's making my life as "house librarian" alot easier already. It's very cool to be able to keep our GoodReads shelves and "read" status in sync with Calibre now.

Huge thanks to Kiwidude for all his hard work and late nights spent on this one!

[hedwig]

This looks like an amazing plugin. Thanks for your hard work.
One quick question - will it also work with Mac OS Snow Leopard?
Thanks.

[kiwidude]

Originally Posted by hedwig

One quick question - will it also work with Mac OS Snow Leopard?
Thanks.

I honestly have no idea, but my guess is that it shouldn't be an issue. It doesnt do anything with the file system which was the compatibility issue that my "Open With" plugin used to have, so I don't see why it shouldnt. If you want to try the beta version to see just PM me your email, it would be nice to find out since I can't test it myself.

Thanks for the support guys, I can't deny this plugin being many long days of work to develop. We seem to be on the home stretch with it now though fingers crossed.

[nickdma]

See, I told you it would be simple

[hedwig]

Tested it on my imac and it works perfectly.
Thanks very much.