Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

View Poll Results: What features would you like added to this plugin?
Release existing bug fixes now 2 66.67%
Add options to make search more flexible 1 33.33%
Offer fewer options 0 0%
Link all matching gutenberg IDs 0 0%
Only link gutenberg ID for exact edition match 0 0%
Import more wikidata fields 1 33.33%
Handle wikidata entries with multiple book editions correctly 2 66.67%
Multiple Choice Poll. Voters: 3. You may not vote on this poll

Reply
 
Thread Tools Search this Thread
Old 12-17-2018, 02:26 AM   #16
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 15,885
Karma: 25956868
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo,Aura H2O,Glo HD,Aura ONE,Clara HD,Forma;tolino epos
Quote:
Originally Posted by compurandom View Post
I'm looking into how to add data to custom columns. At first pass, it looks like it isn't hard, as long as the custom column name is fixed ahead of time. ( @kovidgoyal let me know if there's gotchas or flexibility in this.)
Are you sure of that? I agree that setting the value of a custom column isn't hard, and I do it in a lot of plugins. But, the metadata source plugin doesn't actually have access to the book. The updating is done after the metadata merging has been done and this doesn't allow updating of custom columns. This has been asked about plenty of times here with a strict "can't do it" as the response.
Quote:
I'm taking Jay Dugger's feature list as a priority list and will implement them in that order, but delaying harder ones unless discussion here indicates otherwise.

In the next 3-5 days I'll see if I can add (some of) illustrator, narritive locations, and characters.
I have to admit I'm most curious about how you plan to handle all the identifiers. Adding them as URLs is easy enough, but to get a site specific identifier to work needs support from a metadata source plugin or an identifier rule. You could test the URL against each metadata source plugin and the identifier rules to decide which to use.

The latter statement reminds me to recommend adding the method "id_from_url". But, I just looked at the code and you have it commented out with a "What?" type comment. See https://www.mobileread.com/forums/sh...d.php?t=311978 for how it is used.
davidfor is online now   Reply With Quote
Old 12-17-2018, 08:41 AM   #17
compurandom
Addict
compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.
 
Posts: 342
Karma: 172672
Join Date: Jun 2015
Device: kobo aura h2o
Quote:
Originally Posted by davidfor View Post
Are you sure of that?
I'm sure I can make it work. I'm not sure I can make it work without modifying calibre.

At the very least, it appears that the metadata GUI supports adding fields, I tested that much. I didn't test if I can get the field into the book. I may have to dig for that and add a line of code or two somewhere.

Quote:
This has been asked about plenty of times here with a strict "can't do it" as the response.
Let me know where, so that when I fix it, I can let them know. :-P

Quote:
I have to admit I'm most curious about how you plan to handle all the identifiers.
This I have tested. It shouldn't be too hard. I just don't want to supoprt every single identifier in existence.

For translating from identifier to url, I had imagined making a dictionary of identifiers with a template to create the URL. This could even be fully automated, pulling the template itself from wikidata.

Going from URL to identifier is a two part job.

When pulling from wikidata, we already know which template it will match, so extracting the ID is easy. I might not be able to automate that, but manually making a regex for it is trivial.

The second half would be translating a generic URL to an identifier.
I hadn't planned on doing that at all, but...

Quote:
The latter statement reminds me to recommend adding the method "id_from_url". But, I just looked at the code and you have it commented out with a "What?" type comment. See https://www.mobileread.com/forums/sh...d.php?t=311978 for how it is used.
Thanks a lot for that! I was wondering where that function was used. I'll test that as soon as I can.

If I understand correctly, this function is used to translate the URI when importing the book independently from downloading metadata with the plugin.

I'll certainly translate a few URI's automatically, but it might be tricky to get them all without just iterating over the list of all templates, which might or might not be a performance problem. If it's not, I should be able to do them all easily as long as I can make good regex's to match the URIs.
compurandom is offline   Reply With Quote
Old 12-17-2018, 09:36 AM   #18
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 15,885
Karma: 25956868
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo,Aura H2O,Glo HD,Aura ONE,Clara HD,Forma;tolino epos
Quote:
Originally Posted by compurandom View Post
I'm sure I can make it work. I'm not sure I can make it work without modifying calibre.

At the very least, it appears that the metadata GUI supports adding fields, I tested that much. I didn't test if I can get the field into the book. I may have to dig for that and add a line of code or two somewhere.
I'm not sure if I am following you there. Calibre supports custom columns. The metadata editor automatically builds entry fields for them. Or do you mean in the Metadata download configuration?
Quote:
Let me know where, so that when I fix it, I can let them know. :-P
There are far to many places to name. I'd suggest at least once in every metadata source plugin thread. And plenty of times in other threads about downloading metadata.

The real issue would be that all metadata source plugins would need to be updated to use custom columns. And there would need to be a way to identify the columns for each possible piece of metadata from the source.
Quote:
This I have tested. It shouldn't be too hard. I just don't want to supoprt every single identifier in existence.

For translating from identifier to url, I had imagined making a dictionary of identifiers with a template to create the URL. This could even be fully automated, pulling the template itself from wikidata.

Going from URL to identifier is a two part job.

When pulling from wikidata, we already know which template it will match, so extracting the ID is easy. I might not be able to automate that, but manually making a regex for it is trivial.

The second half would be translating a generic URL to an identifier.
I hadn't planned on doing that at all, but...
Your using an API to get the data, so that probably makes some of this easier. But, if you don't translate both ways, then adding the identifier is fairly pointless. And won't display anywhere.
Quote:


Thanks a lot for that! I was wondering where that function was used. I'll test that as soon as I can.

If I understand correctly, this function is used to translate the URI when importing the book independently from downloading metadata with the plugin.

I'll certainly translate a few URI's automatically, but it might be tricky to get them all without just iterating over the list of all templates, which might or might not be a performance problem. If it's not, I should be able to do them all easily as long as I can make good regex's to match the URIs.
I did this because I frequently ended up with a URL in the clipboard for a metadata source. Especially when the download didn't merge the versions I wanted. The button in the metadata editor already supported pasting an ISBN. I just extended this. And Kovid extended it a little bit before I finished my change.
davidfor is online now   Reply With Quote
Old 12-17-2018, 04:19 PM   #19
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 11,847
Karma: 10633600
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@compurandom - you probably recognise this, but... please don't used fixed custom column names (as implied/suggested by Jay Dugger), when it has been done previously it has caused a considerable friction.

Thanks BR
BetterRed is offline   Reply With Quote
Old 12-17-2018, 09:25 PM   #20
compurandom
Addict
compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.
 
Posts: 342
Karma: 172672
Join Date: Jun 2015
Device: kobo aura h2o
Quote:
Originally Posted by BetterRed View Post
@compurandom - you probably recognise this, but... please don't used fixed custom column names (as implied/suggested by Jay Dugger), when it has been done previously it has caused a considerable friction.
Thanks for the feedback, that's one of the reasons I delayed implementing these features. I really would like it to be a preference -- although maybe not on the preferences pane, but in a tweaks section? Not sure how to do that either yet.

However, as the touched_fields array is suggested to be a frozen set, I don't know if modifying it will have an effect on what fields are accepted -- it may be too late to modify it by the time identify is called. This is one of the things I'm a bit unsure of and haven't tested yet.
compurandom is offline   Reply With Quote
Old 12-17-2018, 11:51 PM   #21
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 11,847
Karma: 10633600
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Yeah - custom columns feature heavily in GUI plugins, but IIRC they don't feature at all in Metadata Source plugins. But they have configuration panels so that's where you'd allow the user to specify them - should be no need to delve into the arcane world of Plugin Tweaks.

But, but, but -- can a Metadata Source plugin even write to custom columns. The fact that none of them do makes me wonder if it is because they cannot. David and/or Kovid would know.

Might need two plugins [Metadata Source Plugin] Wikidata Basics and [GUI Plugin] Get Wikidata Advanced.

BR
BetterRed is offline   Reply With Quote
Old 12-18-2018, 01:19 AM   #22
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 33,913
Karma: 10254168
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No metadata plugins are not allowed to write to custom columns.
kovidgoyal is offline   Reply With Quote
Old 12-18-2018, 01:30 AM   #23
compurandom
Addict
compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.
 
Posts: 342
Karma: 172672
Join Date: Jun 2015
Device: kobo aura h2o
Quote:
Originally Posted by BetterRed View Post
Might need two plugins [Metadata Source Plugin] Wikidata Basics and [GUI Plugin] Get Wikidata Advanced.
Are two plugins really needed, or can I mash all the functionality into one?

Some of the things I'd like to do would be really messy from a metadata plugin, so maybe it would be cleaner to put custom column functionality in a GUI plugin anyway.

I'm thinking, rather than one query that grabs a huge mass of data that will be mostly thrown away, start small and match the book to the database with the metadata plugin, and then select a group of previously identified books to import the rest.

Last edited by compurandom; 12-18-2018 at 01:39 AM.
compurandom is offline   Reply With Quote
Old 12-18-2018, 01:58 AM   #24
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 15,885
Karma: 25956868
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo,Aura H2O,Glo HD,Aura ONE,Clara HD,Forma;tolino epos
Quote:
Originally Posted by BetterRed View Post
Yeah - custom columns feature heavily in GUI plugins, but IIRC they don't feature at all in Metadata Source plugins. But they have configuration panels so that's where you'd allow the user to specify them - should be no need to delve into the arcane world of Plugin Tweaks.

But, but, but -- can a Metadata Source plugin even write to custom columns. The fact that none of them do makes me wonder if it is because they cannot. David and/or Kovid would know.
That's the thing. The metadata source plugin doesn't actually update the library. It extracts the details and passes them back to the metadata download process. That merges the results and presents them to you. After that, the library is updated. I do not believe this will handle custom columns.


Edit: Should have refreshed before posting.
davidfor is online now   Reply With Quote
Old 12-18-2018, 02:04 AM   #25
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 15,885
Karma: 25956868
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo,Aura H2O,Glo HD,Aura ONE,Clara HD,Forma;tolino epos
Quote:
Originally Posted by compurandom View Post
Are two plugins really needed, or can I mash all the functionality into one?

Some of the things I'd like to do would be really messy from a metadata plugin, so maybe it would be cleaner to put custom column functionality in a GUI plugin anyway.

I'm thinking, rather than one query that grabs a huge mass of data that will be mostly thrown away, start small and match the book to the database with the metadata plugin, and then select a group of previously identified books to import the rest.
Doing it that way does mean two plugins. One thing it does is allow people to use the metadata download to get the WikiData identifier while getting other metadata. The second plugin just downloads the metadata knowing that the identifier is correct and doesn't need to check. But, you probably still want a way to set the identifier from the second plugin.

The Goodreads sync does this. If the goodreads identifier exists, it can use it for the function, but you can also search and set the identifier. But, it doesn't get any metadata.
davidfor is online now   Reply With Quote
Old 12-19-2018, 11:38 AM   #26
compurandom
Addict
compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.
 
Posts: 342
Karma: 172672
Join Date: Jun 2015
Device: kobo aura h2o
As it looks like metadata plugins can't mess with custom columns, and implementing that as a GUI feature will take a while, I have decided to release version 1.2 now with series and ISBN support.

I chose to add only the id for ISBN for now, as additional IDs will require more research. IDs imported from wkidata should:
* only have one ID of that type per book preferably (if it has more, explain why all of them should be imported, or explain how to pick one of them)
* The label for the ID needs to match or at least not conflict with IDs used by other metadata plugins
* If it uses the same ID label as another plugin, the ID format must match that plugin exactly
* If we support saving the ID, it would be nice to also be able to search by that ID

Other possible future additions to this plugin would be:
* improved searching (especially by ISBN -- I need samples to test)
* better handling of books that have multiple editions in wikidata (i.e., check parent entity for more metadata, check sub-editions for earlier publication dates)
* importing more metadata as tags (or save this for the fancier GUI version of this plugin)

I'll start working on a GUI version during the next 3 weeks. The first version of that will be dedicated to importing metadata into custom columns.
compurandom is offline   Reply With Quote
Old 01-01-2019, 08:18 PM   #27
compurandom
Addict
compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.
 
Posts: 342
Karma: 172672
Join Date: Jun 2015
Device: kobo aura h2o
The good news is that I'm about 50% done with a GUI plugin that will let you import nearly any simple wikidata property relationship you want into nearly any column (custom or not) and any IDs you want without making any changes to the plugin -- just preferences changes.

The bad news is that I got so badly sidetracked in the QT MVC model that the plugin is only 50% done and I may not have a lot of time to work on it in the next week.

But when it is done, it'll be pretty and extremely flexible.

The end product will use this plugin to handle assigning wikidata IDs to books and formatting any IDs imported by either plugin and importing covers. The other plugin will handle importing all other possible metadata (with some overlap), but it would be preferable to import it with the GUI plugin because it can merge data where this metadata plugin can only overwrite it.

Last edited by compurandom; 01-13-2019 at 08:58 PM. Reason: about to update
compurandom is offline   Reply With Quote
Old 01-13-2019, 09:06 PM   #28
compurandom
Addict
compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.
 
Posts: 342
Karma: 172672
Join Date: Jun 2015
Device: kobo aura h2o
I've posted version 1.3. This is a minor update for this plugin, but a huge update for metadata in general.

This adds a module that can handle arbitrary identifiers. (@davidfor: you might like how I did this...)

You won't see a lot of change (beyond possible bugs fixed) in this plugin without the accompanying Wikidata GUI plugin that allows you to manage new tags. (Will post that in a few minutes.)
compurandom is offline   Reply With Quote
Old 01-13-2019, 10:17 PM   #29
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 15,885
Karma: 25956868
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo,Aura H2O,Glo HD,Aura ONE,Clara HD,Forma;tolino epos
Quote:
Originally Posted by compurandom View Post
This adds a module that can handle arbitrary identifiers. (@davidfor: you might like how I did this...)
I don't have a need for WikiData, so I wasn't going to look at this plugin. But, now I'll have to
davidfor is online now   Reply With Quote
Old 01-13-2019, 10:46 PM   #30
compurandom
Addict
compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.compurandom can program the VCR without an owner's manual.
 
Posts: 342
Karma: 172672
Join Date: Jun 2015
Device: kobo aura h2o
Quote:
Originally Posted by davidfor View Post
I don't have a need for WikiData, so I wasn't going to look at this plugin. But, now I'll have to
I'm actually kinda surprised calibre doesn't have an identifier registration system. I more or less wrote one. If I were to generalize this, I'd add a column to also replace get_book_url_name() and probably use **kwargs instead of a fixed array.

Last edited by compurandom; 01-13-2019 at 10:50 PM.
compurandom is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Metadata Source Plugin] Comicvine chewt0y Plugins 60 06-29-2018 06:26 PM
[Metadata Source Plugin] BiblioNETka.pl fenuks Plugins 3 03-09-2017 09:26 PM
[Metadata Source Plugin] Biblionet.gr wrangly Plugins 7 12-30-2016 07:20 AM
[Metadata Source Plugin] Skoob rodrigoccurvo Plugins 10 02-07-2015 08:20 AM
[Metadata Source Plugin] DIZZIE_NL Pr.BarnArt Plugins 7 08-11-2014 02:48 PM


All times are GMT -4. The time now is 10:33 PM.


MobileRead.com is a privately owned, operated and funded community.