[GUI Plugin] LibraryThing Match

[Bradles]

LibraryThing is a social cataloging web application for storing and sharing book catalogs and various types of book metadata. It is used by authors, individuals, libraries, and publishers.

LibraryThing Match is a calibre plug-in for comparing your calibre e-book list with your LibraryThing catalog.


Main Features:

• Filter your LibraryThing books by Collection, Tags, N most recently added.
• Sort you LibraryThing books by author, title, entry date, entry date reversed.
• Display a list of your LibraryThing books for side-by-side comparison with your calibre e-books
• Compare the two lists using a fuzzy matching algorithm to easily identify books which do/do not exist in both lists.
• Save matches in a calibre custom column.
• Save your LibraryThing ratings in a calibre custom column (half-star ratings).

Special Notes:

• Requires calibre 5.0.0 or later.
• The LibraryThing interface works with public LibraryThing catalogs only, which excludes books you have marked as "Private".

Installation Steps:

• Install the plugin from calibre preferences (Preferences - Plugins - Get new plugins).
• Alternatively, install it manually by downloading the attached zip file, and installing as described in the Introduction to plugins thread (look for "How do I install a plugin manually?").

Paypal Donations:

• If you find this plugin useful please feel free to show your appreciation.

Version History:

Spoiler:

Version 0.2.0 - 29 Mar 2021
LibraryThing Match en español. Spanish translation thanks to @dunhill.
Added option to save LT book id to a calibre identifier.
Added "About" box with link to this thread.

Version 0.1.1 - 21 Feb 2021
Fix an edge case in the matching algorithm.

Version 0.1.0 - 18 Feb 2021
Initial release of LibraryThing Match plugin

[Bradles]

This is a test version of the LibraryThing Match plugin for use with calibre 6 when it is released. It will also work with current versions of calibre from 5.0 upwards.

See the calibre 6 / Qt6 thread for details:

https://www.mobileread.com/forums/sh...d.php?t=344064

This version of the plugin can be installed manually for testing purposes. Report any issues at the bottom of this thread.

[Duke9700]

I am very excited about this plugin. LT deserves far more love than it seems to get!

For those of us who use Authors names as FN LN (I know - scandalous!) the plugin will try and match calibre first names to LF last names e.g. Adam-Troy Castro is paired up with John Joseph Adams.

Are there any tricks or possible future configuration for this? I'm assuming I would just go back and forth in the Preferences>>Advanced>>Tweaks and switch between fn ln and ln, fn for Author Sort. If anyone has a better or more elegant method I'd love some ideas!

[Bradles]

Originally Posted by Duke9700

For those of us who use Authors names as FN LN (I know - scandalous!) the plugin will try and match calibre first names to LF last names e.g. Adam-Troy Castro is paired up with John Joseph Adams.

Thanks Duke.

I would not expect a very high probability match in that situation. The algorithm splits each string into "tokens" (i.e. words), sorts them, then compares the shortest combination. It includes the book title, so only if the titles are very similar as well would you get a high probability match. You can expect quite a few low probability matches, e.g. 50% and under, just from these random word matches.

The LT API also returns author first last, so it would be fairly easy to add that as a LTMatch configuration option.

Let me know if you are seeing anything different and I'll look into it further.

[Bradles]

Responding to a post in the "LTMatch Beta" thread (which is now closed).

Originally Posted by jjmcgaffey

So I'm an LT and calibre user - theoretically they match perfectly, since most of my ebooks have been entered by importing a calibre-exported database. Theoretically. I'm delighted to see your plugin!

Once again, thanks for taking the time to test my plugin, and persisting despite frustrations.

First run - very frustrating. The documentation is findable online (in this thread), but it's not reachable directly from the plugin - so I clicked on the plugin and it complained about no librarything_id column, so I made a librarything_id column (of the wrong type - floating point), which meant I had to restart calibre, then I noticed the part about the librarything_rating column...didn't make it, I didn't feel like restarting... The documentation is really necessary, as it's perfectly possible to make a mess pushing random buttons on the plugin. Is it possible to make a button that's a link to the documentation (and somehow point it out, especially on first run)?

It should be easy enough to include a button/link to the first post above. It might end up being in an "About" dialog.

My full LT library is 11,000+ books; my ebook collection is 5,000+. I tried telling LTMatch to look for 10,000 books, and it timed out. So I reduced it to 1000. The loading was pretty fast - less than a minute, I think. The matching took longer, about 5-6 minutes; not unbearable. It's matching very nicely, and finding an awful lot of books I have listed in calibre that aren't showing up on LT (some of them may be matters of edited authors or titles, but determining that will require doing a full check).

One of the limitations of the LT API is not being able to search by collection. So if you use a LT Collection to identify your ebooks (as I do), the plugin has to download ALL your LT books, then filter for books in the "Ebooks" collection.

Many LT users use tags to identify ebooks, or newly added books, and searching by tags is fast. In fact, I also use tags to identify the source of the ebook, e.g. Kindle, Google, Apple Books. This makes for faster searching and the plugin only downloads what it needs.

The first lot matched perfectly - maybe 50 or 90 books, with just a few gaps. Then there were huge blank spaces on the LT side; not sure why, I'm pretty sure those books are in fact there. Also some complete mis-matches, with low probability; in at least one case, there was a much better match higher up, not sure why it was ignored. But it's not hard to force the match (except that with a list this long, just scrolling through it is hard).

Hopefully the "50 or 90" was not from 1000 books! The gaps are simply "spacers" to ensure that matched books are always next to each other in the lists.

Failing to identify the best match is a concern, but this sometimes happens with series, where the series name is part of the book title. If you share the names of the books concerned (by DM if you like) I can have a look.

I think I'll start over using only 100 at a time, now that (having found and read the documentation) I understand that once it's matched it doesn't need to be looked at again. Doing 1000 at a time, there's too much chance of a mis-match. So I closed the window without having it accept matches.

I think the key to successful matching, is to have a LT list that you expect will fully encompass the calibre list. This isn't really possible with small batches, as you'll always be getting the same LT list, and changing the sort order won't help much.

There are a number of ways to filter your calibre list to help though, e.g. Authors whose surname starts with A:

author_sort:~^A

Then B, then C, etc. LTMatch works with the currently displayed calibre list, so you can at least focus your efforts.

Is this useful info? Are there other things you want to know? I'll be trying the plugin again soon, and will check back here to update my experience with it.

Very useful info. I'd be interested to know how you identify ebooks in LT. I haven't actually seen too many users with an Ebooks collection like me.

I'm having some serious frustrations with the forum - I tried a quick reply and it logged me out, tried it again and it appeared as a blank post and then disappeared entirely. Hoping this one will work.

Sorry, I can't help with that.

You may be interested to know, I'm working on a LibraryThing Metadata plugin too. It's like the Amazon, Google, Goodreads, etc plugins, except it downloads your metadata and covers from LT. (If you have a matching book.) It builds on the LTMatch plugin, by using the librarything_id to go directly to your book.

It's not quite ready for testing yet, but I'll surely post something in this thread when it is.

[jjmcgaffey]

OK, I didn't understand about the matched books not being matched again... I'll try your suggestion of filtering calibre by author's initial.

I do use a collection to mark all my LT ebooks. I suppose I could Power Edit and tag them all with something, but that's a little annoying. Can LTMatch deal with wildcards in tags? I do have most of them marked with _import(date) - that is, the last batch I added are tagged _import210316. If I could tell LTMatch to look for _import*, that would work (for most of them, and I should tag the ones that don't have it anyway).

OK, I tried to set LTMatch going with all the calibre authors that begin with A, and max results 12000 (to exceed my LT library). I got

"Can't connect to LibraryThing due to an error.
This could be due to an incorrect username. See error details below."

and when I clicked Show Details it said

"timeout('The read operation timed out')"

So I can't set it at my whole library. Hmm.

OK, weird! I tried it with tag _import* and max results 6000 - I got the same error, but the details say

"URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)'))"

Sounds like that asterisk messed something up. I guess I'm stuck at the moment. I'll check back with you later - thanks!

[Bradles]

Originally Posted by jjmcgaffey

and when I clicked Show Details it said

"timeout('The read operation timed out')"

I did some investigating, and the default timeout is None, i.e. no limit. I'm not specifying any timeout in the API call either. I tested (using the "Test Connection" button) with a 15,000 book library (thankyou CovenantNetwork whoever you are) and successfully retrieved 14980 books in around 2 minutes.

Perhaps there is something going on at your end, i.e. proxy server, VPN or something? Could speed be an issue? My broadband is 30Mbits, so not particularly fast.

Also, per your suggestion, I've added an "About" box, which includes a link to this thread. Not that it will do you much good, but some future user may benefit. An update with that and a couple of minor changes to be released in the next couple of days.

Also also, you could try firing off a test query in your browser, using a URL of the form:

https://www.librarything.com/api_getdata.php?userid=sample&showstructure=0&max=10&showCollections=1&showRatings=1&showTags=1&booksort=author,title&responseType=json

[Bradles]

LibraryThing Match version 0.2.0 released. Updates:

• LibraryThing Match en español. Spanish translation thanks to @dunhill.
• Added option to save LT book id to a calibre identifier.
• Added "About" box with link to this thread.

Enjoy!

[dunhill]

Enjoy !!!

[aklau75]

calibre, version 5.14.0
ERROR: Install plugin failed: A problem occurred while installing this plugin. This plugin will now be uninstalled. Please post the error message in details below into the forum thread for this plugin and restart calibre.

Traceback (most recent call last):
File "calibre\gui2\dialogs\plugin_updater.py", line 709, in _install_clicked
File "calibre\customize\ui.py", line 472, in add_plugin
File "calibre\customize\ui.py", line 61, in load_plugin
File "calibre\customize\zipplugin.py", line 293, in load
File "importlib\__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "calibre\customize\zipplugin.py", line 192, in exec_module
File "calibre_plugins.fictiondb.__init__", line 11, in <module>
ImportError: cannot import name 'quote' from 'urllib' (C:\Program Files (x86)\Calibre2\app\bin\python-lib.bypy.frozen\urllib\__init__.pyc)

[Bradles]

This looks like a fictiondb issue, which has not been updated for python3 and therefore doesn't work with calibre 5+. See this thread for more:

https://www.mobileread.com/forums/sh...=171674&page=7

[Bradles]

Here are a couple of custom columns I added that link to the LT website. (See attached images for configuration.)

LT Find: Direct link to your book on librarything.com, or search if it doesn't exist.

Template

program:
    id = select(field('identifiers'), 'ltid');
    if id then
        strcat('<a href="https://www.librarything.com/work/book/', id , '">', field('title'), '</a>')
    else
        strcat('<a href="https://www.librarything.com/title/', field('title'), ' ', field('author'), '">', field('title'), '</a>')
    fi;

LT Add: Direct link to your book on librarything.com, or an "add books" search using Title + Author.

Template

program:
    id = select(field('identifiers'), 'ltid');
    if id then
        strcat('<a href="https://www.librarything.com/work/book/', id , '">', field('title'), '</a>')
    else
        strcat('<a href="https://www.librarything.com/addbooks?search=', field('title'), ' ', field('author'), '">', field('title'), '</a>')
    fi;

If you add tags when adding books you need to enter the tags and re-search.

[mihrimah]

Hi;

I'm trying to set up this plugin (calibre v5.33.2, plugin v 0.2.0), and I'm getting a "Can't connect to LibraryThing due to an error" message. The Details panel reads "JSONDecodeError('Expecting value: line 1 column 1 (char 0)')".

This only happens when I try to retrieve 20 or more books at a time; anything up to 19 works fine. Am I doing something wrong?

[Bradles]

Hi mihrimah,

That error can occur if you have not specified "LibraryThing user" or "Max results" in the Customize LibraryThing Match" screen, which you can access from the "Configure..." button.

Please ensure those values are filled in and try again.

If you are still having problems you can message me. Just in case you like reading user manuals, there's a short one attached to the first post.

Brad.

PS: Sorry your first post had to be about an error with my plugin...

[mihrimah]

I've got those filled in; it only happens when "Max results" is 20 or above. I'll take some screenshots and message them to you.