[Metadata Source Plugin] wikidata

[BetterRed]

Originally Posted by compurandom

I'm actually kinda surprised calibre doesn't have an identifier registration system.

I haven't looked at what you are providing, but it sounds a bit like the "Preferences->Look and Feel->Book Details->Create rules to convert identifiers into links" settings.

[davidfor]

Originally Posted by compurandom

I'm actually kinda surprised calibre doesn't have an identifier registration system. I more or less wrote one. If I were to generalize this, I'd add a column to also replace get_book_url_name() and probably use **kwargs instead of a fixed array.

The metadata source plugins act as the identifier registration system. The rules BR pointed to were added a while ago for identifiers without the backing of a plugin. You should be able to populate that programmatically. But, I'll have a look your solution in the next few days.

[compurandom]

Originally Posted by BetterRed

I haven't looked at what you are providing, but it sounds a bit like the "Preferences->Look and Feel->Book Details->Create rules to convert identifiers into links" settings.

Ah so! But mine is missing name, and that one is missing regex to transform a pasted URL. And the button to get there is so wide I didn't see it as a button.

sigh. Neither is a complete solution. And I'd still need to manage linking identifiers to wikidata entities.

Maybe later I'll dig into that and see how it works and maybe synchronize the data in it with mine.

[davidfor]

Originally Posted by compurandom

Ah so! But mine is missing name

The name is what is displayed in the metadata panel. I'm not sure how you can not have that.

and that one is missing regex to transform a pasted URL.

I built the code into the metadata editor. Press the button next to the IDs, and it is matched against the rules and then the metadata source plugins (using "id_from_url") to find an identifier.

And the button to get there is so wide I didn't see it as a button.

I was caught by that when it was added. And I'd been involved in the discussion to add it.

sigh. Neither is a complete solution. And I'd still need to manage linking identifiers to wikidata entities.

Maybe later I'll dig into that and see how it works and maybe synchronize the data in it with mine.

To me you could have your plugin generate them from its config. At least for the common. But, if you are generating an identifier for anything that exists in WikiData, that probably isn't practical. And I'm more curious about what you have done.

[compurandom]

Originally Posted by davidfor

The name is what is displayed in the metadata panel. I'm not sure how you can not have that.

I didn't really need it, didn't want another column in my already crowded table, and didn't want to think about how to generate a default from the data already in wikidata, although I might have overlooked something. For now, I just set name to be the same thing as the identifier tag.

In fact, if I can leverage this interface, I might move more of my existing data into it, and just leave two or three columns in my interface to link to it. More to the point, I could make a model that saves data in both places, views all of it, and keeps only 2-3 columns (wdid, identifier, wikidata description) actually stored in my plugin's preferences.

I will have to play with that hidden editor. It would be nice if it had an API that was advertised in the plugin documentation pages as an alternative to the 3 (4?) functions.

Also, I was kind of looking for a way for the wikidata plugin to not step on other plugin's toes. That's why I have a checkbox to disable display of ID's, in case you have a plugin that already does it itself that you want to integrate with.

I intended on actually adding some the id's that other plugins already use as defaults with the display box unchecked, but I haven't gotten around to finding them (I know there are at least 3). IDs that are handled elsewhere can still be imported from wikidata, and eventually I will add something to search for books in wikidata based on linked identifiers to get their wikidata IDs. The wikidata metadata plugin does this already, but only for gutenberg IDs (and ISBNs, but I'm not sure that works).

The point is to have full bidirectional linking between wikidata and other sources; ultimately, linking on any source should cause a cascade of data to become available on all the sources.

(large button)

I was caught by that when it was added. And I'd been involved in the discussion to add it.

The button itself is fine, but can we shrink it to a normal size instead of full width?

[compurandom]

Originally Posted by Jay Dugger

I feel as if I ask for the sun, the moon, and the stars, but if I don't ask...

Ask and you shall receive. I think almost everything you asked for is now available either through this metadata plugin or the GUI plugin I just published.

The field series may map to the column Series;

This plugin will import that now along with series index. Series data is in the todo list for the GUI version of the plugin, both the simple version in this plugin and an extended version that can generate indexes from prevous/next columns. It'll be interesting to see how well that works if I get around to it.

genre to a custom column "Genre;" and so on.

publisher to Publisher

illustrator to a custom column of the same name

These would be easy to add by importing the properties. Illustrator and genre are already in the defaults as examples. Note that you have to create the custom column yourself.

language of work or name to Languages (though this should append, and not overwrite)

This one is messier; I haven't tried importing it. If the languages is a multivalue field, it might work as is; uncheck the overwrite column to merge data in multivalue columns.

publication date to Published

This is another messy one; Do you want the publication date of the current edition, or the oldest date for any edition? Maybe you want the inception date rather than the publication date? This plugin does a simplistic job at attempting to grab the oldest date out of several.

I'll be writing code for the GUI plugin eventually to handle some of these cases in a more sophisticated way and track down metadata shared by multiple editions but not propagated. For now, you can directly import edition publication date and inception date and give a priority between them. Eventually, I might add merge rules of some sort. (Average publication date anyone?)

narrative locations and characters also to a custom column of the same name, but these two wikidata properties might only apply to fiction...

In my set of test books, I found a surprising number of non-fiction books that included location and characters. Importing characters can quickly make a mess. One of the books put several hundred characters into a tags field.

The way I have it now, you can import these directly, unchanged, or run them through the entity filter and pick and choose how many you want to keep and if you want to respell them.

The Commons category and the topic's main category as tags

That should work, but I didn't test it. I suppose the commons category could come in as a link as well, but I didn't even look to see if that is possible from the calibre side.

All available identifiers as Ids.

That is there and somewhat well tested, although as per discussion above, I may change how it works internally.

Note that wikidata supports somewhere on the order of 3500 identifiers, so "all available" may become silly fast. What I support is importing identifiers in selected books into the preferences editor for filtering, adjusting, and manual labeling. Currently, only one identifier of each type is supported. I think this can be fixed, but I don't currently have plans for it.

[BetterRed]

Originally Posted by davidfor

I built the code into the metadata editor. Press the button next to the IDs, and it is matched against the rules and then the metadata source plugins (using "id_from_url") to find an identifier.

When I press the button to the left of the {ids} text box, in the Metadata Edit General tab, calibre interrogates the clipboard for an ISBN number, if there's nothing resembling an ISBN I'm prompted to enter one.

If I right click in the {ids} text box, there's an "Edit identifiers in a dedicated window" option, it pops a simple text box into which I can enter keyname:value pairs e.g. books_etc:9780571345809 or torygraph_review:mere-wife-review-beowulf-updated-modern-new-york/

Am I missing something?

I'd rather enter a name (maybe select from a drop down list of what's in Preferences->L&F) followed by a complete URI including the protocol name, the latter is usually waiting in the clipboard. As it is I have to paste it into the text box and carefully chop off the front bit e.g. "https://www.telegraph.co.uk/books/what-to-read/"

Re the Wide Button, my problem with it is that I forget where it is. Book Details seems an unlikely place - but I can't think of a less unlikely place

BR

[davidfor]

If the clipboard contains the URL for an identifier, when you click the paste button, the URL is parsed to get the identifier. For example, if you have the Goodreads Metadata source plugin installed, and have "https://www.goodreads.com/book/show/13487260" in the clipboard and press the button, you should get "goodreads:13487260" added as an identifier. Not all metadata source plugins have the necessary method. All the ones I maintain do, and I have been meaning to go through the built-in ones to add it. It also works if you have the URL that matches one of the rules.

If the contents of the clipboard don't match a metadata source or a rule, it falls back to looking for an ISBN.

[BetterRed]

Ah- most of my 'books' aren't commercial books as such, and what I have in the clipboard could be anything, a url to a newspaper review or a link to the download location at somewhere like Stratfor or Lowy or a Twitter tweet

Every time I think of using the ids column I go back to pasting them into a long text custom column.

BR