Wikidata GUI

[compurandom]

This plugin imports near arbitrary metadata from Wikidata for books that already have a wikidata ID. This includes a bulk search feature to try to do exact title/author searches in wikidata to find books and can try to match books by other identifiers as well. Use the wikidata metadata plugin to find and add IDs to books that are not found by these methods.

There's a todo.txt in the zip file with the complaints I personally have and possible future features. Discussion of these or others will prioritize their implementation.

Features supported in version 2.0.0:

• Bulk search for books in wikidata using exact match on title, author (faster and less likely to hit query limit than the metadata plugin version)
• Property, Tag, and Identifier editor to help mange wikidata before merging into books
• Import from books or manually enter arbitrary wikidata properties and entities and mark for filtering
• Import from books external identifiers in wikidata for editing
• Merge in series data with support for a second series in a custom column
• Identifier editor to manage importing and translating of identifiers back and forth to URIs
• Convert and search by external identifiers already in book metadata
• Partially merge selected properties when adding new properties.
• Mark books to indicate merge status
• Option to hide uninteresting imported properties and tags
• Import empty placeholder books for books by same author and books in series

Version History:

Spoiler:

Version 2.0 - 1 Jan 2021
This is mostly a bug fix version, fixing several long standing bugs that have irritated me.
This also now works in python3 for calibre 5.

* Upgrade minimum version to calibre 5 since there's no testing with calibre 4 (If you are interested in testing, let me know.)
* Changes for python3 syntax, unicode changes
* Upgrade SPARQLWrapper
* Tweak error messages in multiple places so the user actually sees them
* Try harder to differentiate between no books found and wikidata got an error and crashed
* Attempt to recognize wikidata throttling (if there is any)
* Change GET to POST to allow querying more books at once, now it times out if you ask for too many, and books with single quotes in their title now can be matched.
* Update the view so metadata changes show up immediately
* Update user agent string to fit wikidata standards

Version 1.3 - 24 Feb 2019

Bulk search for books in wikidata by exact match of title, author
- Use this instead of the Wikidata metadata plugin for faster but less complete results.
- Note: don't select too many books at once or it will fail. (The other plugin gets rate limited on bulk import.)
Add features to import other books in series or by same author
* match existing books
* select books and add empty placeholder books for them

User interface for the book import feature is not yet complete -- suggestions welcome!

Version 1.2.1 - 11 Feb 2019
Fix bug with missing mark_* options in defaults
strip spaces for tags and properties

Version 1.2.0 - 28 Jan 2019

Add support for importing series data
second series supported in a custom column

fixed bug involving merging tags

Version 1.0.1 - 22 Jan 2019
Fix a crash during merge caused by occasional unexpected data from wikidata. This is a quick fix to capture the crash so that the offending book can be marked as in error for manual review and continue with the merge.

A more extensive fix is needed to properly handle conversion to target datatypes and possibly do a better job of merging data based on that.

Version 1.0.0 - 20 Jan 2019
New features:
* Search for books in wikidata by identifier
* Merge button on customize tabs
* Merge button on properties/tags tab can merge just selected properties
* Optionally mark books with merge status
* Option to hide uninteresting imported properties and tags
* convert overdrive gutenberg ids to gutenberg ids
* convert urls to ids from id preferences
* option to disable some delete confirmations
* convert overdrive gutenberg ids to gutenberg ids
* convert and delete urls matching imported IDs
* option to disable some delete confirmations
* option to chain convert IDs
Check forum thread for upcoming features and to give feedback on feature priority.

Bugs fixed:
* sorting now preserves selection
* Fixed a typo in new identifier default tag generation
* Saving ID visibility works now
* Messed with column widths
* Updated missing tooltips
* Performance improvements when updating metadata
* "undefined" dates now update properly

Version 0.3.0 - 13 Jan 2019
Initial release of Wikidata GUI plugin

Majority of planned features are implemented
Most features seem to not be buggy anymore

Known bugs:

Spoiler:

• This version claims to only work in Calibre 5, because it hasn't been tested in Calibre 4.
• Import books dialog doesn't (yet) save anything from matched books (unmatched books can be marked to import)
• Edited author/title on imported books is not used (yet)
• Bulk query will probably still choke if you select an insane number of books, but if you pick merely too many, wikidata times out trying to find them all. But you can select more than before.
• Bulk query chokes on titles containing double quotes and control characters
• Handles multivalue fields poorly when inserting into a non-string field (luckily this isn't very common--needs documentation)
• Probably needs more documentation, please ask questions!
• Partially implemented series and author import -- implement metadata update
• Need to add advanced algorithms to do secondary search as well as sort and rank books in a series when wikidata has incomplete information

[compurandom]

Please note... development on this plugin is driven by feedback.
I haven't gotten any feedback, so I haven't felt like developing it.

If you feel like testing the new version on calibre 4, I'd be happy to apply patches for any errors that come up., but with no feedback, I'll just assume nobody that uses calibre 4 wants it.

Here's the last version tested on Calibre 4 (python2): version 1.3

[compurandom]

I'll keep this message up to date with status of unimplemented features.

I've added support for importing series information.
This now supports all the metadata the metadata plugin supports, except this can import *two* series into a book if you create a custom column for it.

The default settings are really intended as examples, but if anyone thinks I should include more, let me know.

The next features I'm considering are:

• Improve and clean up book import dialog. (Suggestions welcome!)
• Allow updating metadata of matching books found by author and series
• Book import future features: update books missing series info in matched books; import books not listed as in series, but having next/previous links (red links); generate indexes for books with no wikidata series index from next/previous book links
• Handle books with multiple editions: pull metadata and pubdate from parent editions and multivalue pubdates; search forward and back for gutenberg editions
• Advanced metadata merging for multivalue entries: replace overwrite column with a function: replace / if empty / min / max /average? / append? / other? (this would fix several bugs)
• Merge our custom identifier handler with the calibre ID handler in Preferences->Look and Feel->Book Details->Create rules to convert identifiers into links (this is probably mostly cosmetic to the user but would preserve identifiers independent from this plugin)
• add special handler for URL type properties (wrap in html, wrap in markdown, or insert as identifier uri/url/urn or try to xlate url to id)

If Any of these interest you or you want some other feature, discuss it below and help me prioritize implementation...

[tlc]

My first book attempt:

calibre 3.39.1 embedded-python: True is64bit: True
Darwin-17.7.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '17.7.0', 'Darwin Kernel Version 17.7.0: Fri Nov 2 20:43:16 PDT 2018; root:xnu-4570.71.17~1/RELEASE_X86_64')
Python 2.7.15
OSX: ('10.13.6', ('', '', ''), 'x86_64')
Interface language: None
Successfully initialized third party plugins: Quality Check (1, 9, 11) && Goodreads (1, 4, 0) && Clean Metadata (0, 0, 6) && Kindle Collections (1, 7, 23) && Wikidata (1, 3, 0) && Wikidata gui (1, 2, 0) && Embed Comic Metadata (1, 4, 0) && Find Duplicates (1, 6, 3)
Traceback (most recent call last):
File "calibre_plugins.wikidata_gui.ui", line 35, in merge_wikidata
File "calibre_plugins.wikidata_gui.ui", line 228, in merge_actual
KeyError: 'mark_updated'

[tlc]

Works when "Mark update books" is set in customize -> options.

[compurandom]

Thanks, I'll look into it, but probably not until late sunday.
Not sure I've run it with that off. :-> oops.
I'd prefer to release it with the next feature set, but not sure how long that will take, so I might patch it earlier.

[compurandom]

Originally Posted by tlc

Works when "Mark update books" is set in customize -> options.

Looking at the code, I bet if you set the option, close the dialog, and then go back and unset the option, it'll still work.

I think I have a fix; the bug is that I forgot to add a few new options to the defaults table, so if you don't set them, they don't have a value and it chokes.

Note: fixed in 1.2.1, just released.