[Metadata Source Plugin] wikidata

[davidfor]

Originally Posted by compurandom

I'm looking into how to add data to custom columns. At first pass, it looks like it isn't hard, as long as the custom column name is fixed ahead of time. ( @kovidgoyal let me know if there's gotchas or flexibility in this.)

Are you sure of that? I agree that setting the value of a custom column isn't hard, and I do it in a lot of plugins. But, the metadata source plugin doesn't actually have access to the book. The updating is done after the metadata merging has been done and this doesn't allow updating of custom columns. This has been asked about plenty of times here with a strict "can't do it" as the response.

I'm taking Jay Dugger's feature list as a priority list and will implement them in that order, but delaying harder ones unless discussion here indicates otherwise.

In the next 3-5 days I'll see if I can add (some of) illustrator, narritive locations, and characters.

I have to admit I'm most curious about how you plan to handle all the identifiers. Adding them as URLs is easy enough, but to get a site specific identifier to work needs support from a metadata source plugin or an identifier rule. You could test the URL against each metadata source plugin and the identifier rules to decide which to use.

The latter statement reminds me to recommend adding the method "id_from_url". But, I just looked at the code and you have it commented out with a "What?" type comment. See https://www.mobileread.com/forums/sh...d.php?t=311978 for how it is used.

[compurandom]

Originally Posted by davidfor

Are you sure of that?

I'm sure I can make it work. I'm not sure I can make it work without modifying calibre.

At the very least, it appears that the metadata GUI supports adding fields, I tested that much. I didn't test if I can get the field into the book. I may have to dig for that and add a line of code or two somewhere.

This has been asked about plenty of times here with a strict "can't do it" as the response.

Let me know where, so that when I fix it, I can let them know. :-P

I have to admit I'm most curious about how you plan to handle all the identifiers.

This I have tested. It shouldn't be too hard. I just don't want to supoprt every single identifier in existence.

For translating from identifier to url, I had imagined making a dictionary of identifiers with a template to create the URL. This could even be fully automated, pulling the template itself from wikidata.

Going from URL to identifier is a two part job.

When pulling from wikidata, we already know which template it will match, so extracting the ID is easy. I might not be able to automate that, but manually making a regex for it is trivial.

The second half would be translating a generic URL to an identifier.
I hadn't planned on doing that at all, but...

The latter statement reminds me to recommend adding the method "id_from_url". But, I just looked at the code and you have it commented out with a "What?" type comment. See https://www.mobileread.com/forums/sh...d.php?t=311978 for how it is used.

Thanks a lot for that! I was wondering where that function was used. I'll test that as soon as I can.

If I understand correctly, this function is used to translate the URI when importing the book independently from downloading metadata with the plugin.

I'll certainly translate a few URI's automatically, but it might be tricky to get them all without just iterating over the list of all templates, which might or might not be a performance problem. If it's not, I should be able to do them all easily as long as I can make good regex's to match the URIs.

[davidfor]

Originally Posted by compurandom

I'm sure I can make it work. I'm not sure I can make it work without modifying calibre.

At the very least, it appears that the metadata GUI supports adding fields, I tested that much. I didn't test if I can get the field into the book. I may have to dig for that and add a line of code or two somewhere.

I'm not sure if I am following you there. Calibre supports custom columns. The metadata editor automatically builds entry fields for them. Or do you mean in the Metadata download configuration?

Let me know where, so that when I fix it, I can let them know. :-P

There are far to many places to name. I'd suggest at least once in every metadata source plugin thread. And plenty of times in other threads about downloading metadata.

The real issue would be that all metadata source plugins would need to be updated to use custom columns. And there would need to be a way to identify the columns for each possible piece of metadata from the source.

This I have tested. It shouldn't be too hard. I just don't want to supoprt every single identifier in existence.

For translating from identifier to url, I had imagined making a dictionary of identifiers with a template to create the URL. This could even be fully automated, pulling the template itself from wikidata.

Going from URL to identifier is a two part job.

When pulling from wikidata, we already know which template it will match, so extracting the ID is easy. I might not be able to automate that, but manually making a regex for it is trivial.

The second half would be translating a generic URL to an identifier.
I hadn't planned on doing that at all, but...

Your using an API to get the data, so that probably makes some of this easier. But, if you don't translate both ways, then adding the identifier is fairly pointless. And won't display anywhere.

Thanks a lot for that! I was wondering where that function was used. I'll test that as soon as I can.

If I understand correctly, this function is used to translate the URI when importing the book independently from downloading metadata with the plugin.

I'll certainly translate a few URI's automatically, but it might be tricky to get them all without just iterating over the list of all templates, which might or might not be a performance problem. If it's not, I should be able to do them all easily as long as I can make good regex's to match the URIs.

I did this because I frequently ended up with a URL in the clipboard for a metadata source. Especially when the download didn't merge the versions I wanted. The button in the metadata editor already supported pasting an ISBN. I just extended this. And Kovid extended it a little bit before I finished my change.

[BetterRed]

@compurandom - you probably recognise this, but... please don't used fixed custom column names (as implied/suggested by Jay Dugger), when it has been done previously it has caused a considerable friction.

Thanks BR

[compurandom]

Originally Posted by BetterRed

@compurandom - you probably recognise this, but... please don't used fixed custom column names (as implied/suggested by Jay Dugger), when it has been done previously it has caused a considerable friction.

Thanks for the feedback, that's one of the reasons I delayed implementing these features. I really would like it to be a preference -- although maybe not on the preferences pane, but in a tweaks section? Not sure how to do that either yet.

However, as the touched_fields array is suggested to be a frozen set, I don't know if modifying it will have an effect on what fields are accepted -- it may be too late to modify it by the time identify is called. This is one of the things I'm a bit unsure of and haven't tested yet.

[BetterRed]

Yeah - custom columns feature heavily in GUI plugins, but IIRC they don't feature at all in Metadata Source plugins. But they have configuration panels so that's where you'd allow the user to specify them - should be no need to delve into the arcane world of Plugin Tweaks.

But, but, but -- can a Metadata Source plugin even write to custom columns. The fact that none of them do makes me wonder if it is because they cannot. David and/or Kovid would know.

Might need two plugins [Metadata Source Plugin] Wikidata Basics and [GUI Plugin] Get Wikidata Advanced.

BR

[kovidgoyal]

No metadata plugins are not allowed to write to custom columns.

[compurandom]

Originally Posted by BetterRed

Might need two plugins [Metadata Source Plugin] Wikidata Basics and [GUI Plugin] Get Wikidata Advanced.

Are two plugins really needed, or can I mash all the functionality into one?

Some of the things I'd like to do would be really messy from a metadata plugin, so maybe it would be cleaner to put custom column functionality in a GUI plugin anyway.

I'm thinking, rather than one query that grabs a huge mass of data that will be mostly thrown away, start small and match the book to the database with the metadata plugin, and then select a group of previously identified books to import the rest.

[davidfor]

Originally Posted by BetterRed

Yeah - custom columns feature heavily in GUI plugins, but IIRC they don't feature at all in Metadata Source plugins. But they have configuration panels so that's where you'd allow the user to specify them - should be no need to delve into the arcane world of Plugin Tweaks.

But, but, but -- can a Metadata Source plugin even write to custom columns. The fact that none of them do makes me wonder if it is because they cannot. David and/or Kovid would know.

That's the thing. The metadata source plugin doesn't actually update the library. It extracts the details and passes them back to the metadata download process. That merges the results and presents them to you. After that, the library is updated. I do not believe this will handle custom columns.


Edit: Should have refreshed before posting.

[davidfor]

Originally Posted by compurandom

Are two plugins really needed, or can I mash all the functionality into one?

Some of the things I'd like to do would be really messy from a metadata plugin, so maybe it would be cleaner to put custom column functionality in a GUI plugin anyway.

I'm thinking, rather than one query that grabs a huge mass of data that will be mostly thrown away, start small and match the book to the database with the metadata plugin, and then select a group of previously identified books to import the rest.

Doing it that way does mean two plugins. One thing it does is allow people to use the metadata download to get the WikiData identifier while getting other metadata. The second plugin just downloads the metadata knowing that the identifier is correct and doesn't need to check. But, you probably still want a way to set the identifier from the second plugin.

The Goodreads sync does this. If the goodreads identifier exists, it can use it for the function, but you can also search and set the identifier. But, it doesn't get any metadata.

[compurandom]

As it looks like metadata plugins can't mess with custom columns, and implementing that as a GUI feature will take a while, I have decided to release version 1.2 now with series and ISBN support.

I chose to add only the id for ISBN for now, as additional IDs will require more research. IDs imported from wkidata should:
* only have one ID of that type per book preferably (if it has more, explain why all of them should be imported, or explain how to pick one of them)
* The label for the ID needs to match or at least not conflict with IDs used by other metadata plugins
* If it uses the same ID label as another plugin, the ID format must match that plugin exactly
* If we support saving the ID, it would be nice to also be able to search by that ID

Other possible future additions to this plugin would be:
* improved searching (especially by ISBN -- I need samples to test)
* better handling of books that have multiple editions in wikidata (i.e., check parent entity for more metadata, check sub-editions for earlier publication dates)
* importing more metadata as tags (or save this for the fancier GUI version of this plugin)

I'll start working on a GUI version during the next 3 weeks. The first version of that will be dedicated to importing metadata into custom columns.

[compurandom]

The good news is that I'm about 50% done with a GUI plugin that will let you import nearly any simple wikidata property relationship you want into nearly any column (custom or not) and any IDs you want without making any changes to the plugin -- just preferences changes.

The bad news is that I got so badly sidetracked in the QT MVC model that the plugin is only 50% done and I may not have a lot of time to work on it in the next week.

But when it is done, it'll be pretty and extremely flexible.

The end product will use this plugin to handle assigning wikidata IDs to books and formatting any IDs imported by either plugin and importing covers. The other plugin will handle importing all other possible metadata (with some overlap), but it would be preferable to import it with the GUI plugin because it can merge data where this metadata plugin can only overwrite it.

[compurandom]

I've posted version 1.3. This is a minor update for this plugin, but a huge update for metadata in general.

This adds a module that can handle arbitrary identifiers. (@davidfor: you might like how I did this...)

You won't see a lot of change (beyond possible bugs fixed) in this plugin without the accompanying Wikidata GUI plugin that allows you to manage new tags. (Will post that in a few minutes.)

[davidfor]

Originally Posted by compurandom

This adds a module that can handle arbitrary identifiers. (@davidfor: you might like how I did this...)

I don't have a need for WikiData, so I wasn't going to look at this plugin. But, now I'll have to

[compurandom]

Originally Posted by davidfor

I don't have a need for WikiData, so I wasn't going to look at this plugin. But, now I'll have to

I'm actually kinda surprised calibre doesn't have an identifier registration system. I more or less wrote one. If I were to generalize this, I'd add a column to also replace get_book_url_name() and probably use **kwargs instead of a fixed array.