Custom columns and metadata sources

[chaley]

@JKenP: Although I agree with your point in many situations, this isn't one of them.

We are talking about 4 things:

- A conceptual entity w/some relationships dictionary for *book bibliographic data*. This dictionary would help standardize terms and their semantics. For example, the dictionary could contain the term "in print", noting that this term means that the book is still available from a "publisher", and may have a "last print run date". This dictionary is documentation, and has nothing to do with calibre code. I can't see any objection to the creation of this dictionary.

NB: I draw a distinction between book bibliographic data and, for example, research paper bibliographic data. The two domains seem similar, but are in detail quite different. Calibre should not try to become Endnote or Zotero.

- A calibre implementation of the dictionary. This dictionary is a subset of the above, providing agreed-upon names and abstract types, then mapping the abstract types onto one of calibre's column types. To be sure, this process will generate discomfort, because the mapping of abstract type to calibre type will often be approximate.

- A user-supplied mapping of dictionary terms to database columns. This mapping permits me to name my columns however I want, while specifying at the entity level the semantics of the data.

- Metadata plugins that use the above three things to automatically insert data into the database. The writer of the plugin determines the meaning of the information supplied by the metadata source, maps it to the closest equivalent in the calibre dictionary, and uses the map to store the data.

Calibre already does all of the above in one form or another. I consider this work almost to be a refactoring effort, moving to using the same term for the same thing, changing code that doesn't respect the mappings, and using common code where the same thing is done.

Responding to your last paragraph: note that we are not proposing many of the other things that come with SAP customization: active relationships, customizable user interfaces, data integrity rules, report generation, event triggers, and interconnection of wildly-differing domains such as CRM and JIT inventory management. What is being proposed here is codification of something that many of us already have in mind and is already implemented throughout calibre: the meaning of the data we store about books.

@kiwidude: what is it you would like from me? I would be happy to build the code that maps from external dictionary field names to columns (this is trivial), and to do the work so that edit metadata single updates the custom data. Building the data dictionary will be work over time, with the metadata plugin authors providing a push. A gatekeeper is probably necessary, but we have a pleasant sufficiency of these. Is there something else?

[kiwidude]

@dwathny/charles - thx for backing me up on this and your support, very well timed as I was starting to wonder why I bother.

@charles - if you have some time to help that would be brilliant. I hadn't asked as I know like most of us you always have a backlog of stuff to do and unless it is something that has your interest I don't want to be an unnecessary diversion.

As to how you or others could help - well, as much as you are willing . My comfort zone is making the changes to the metadata source plugins but they are as trivial to me as the custom column changes sound to you.

So - what else is there? These come to mind for now...

- defining exactly what the API will look like for the metadata plugins to define what additional fields they support. This must cater for both "Calibre known" fields, and plugin specific fields, and as you have suggested also allow defining the necessary custom column data types

- deciding what merge behaviour these fields will have if any, through the identify() phase of the sourcing. There are certainly fields which it makes no sense at all to try to merge or average. Can you think of/do we need to cater for any which should, and if so how do we define that? The easy answer may be that any such fields must be special cased as Calibre known fields in the identify() phase.

- what will the configuration screen look like, and where will it be accessed from? Is it a button placed on the metadata configuration dialog launching a separate dialog?

- what configurability will a user have for these additional fields to turn them on/off at both plugin and global level? For instance are they just appended to the existing list of checkboxes like title, author, series etc when you go to configure a plugin? What about the global settings - should these extra fields appear in the same set? Should for instance a field be listed only if a mapping exists to a custom column for it?

As examples of the sorts of additional metadata we could pull back, see the following. If you think of others on the pages that we should consider, feel free to add to this. I am sure you would agree that dealing with actual use cases will help discovering the issues we need to deal with.

Goodreads: example book
- Rating (floating point value)
- Number of ratings (int value)
- Pages (int value)
- Awards (tags or comments?)
- Other possibilities like characters, setting and original title?

B&N: example book
- Rating
- Number of ratings
- Pages
- Price (what to do about currencies, numeric or text?)

Fantastic Fiction: example book
Nothing obvious I see but others may spot something

Webscription: example book
- Price

Amazon: example book
- Rating
- Number of ratings
- Pages
- Price

Worldcat: example book
Not currently a metadata source as such, but I will add the link here for interest since you can see this page when you click on the ISBN in the book details pane. It has stuff like genres, other title, awards.

[kovidgoyal]

Since the Rating is already pulled into a standard field, why do you want it to be customized? And the only real use I see for number of ratings is to decide whether to trust the ratings or not. Which can become a configurable threshold. The user sets it and the plugins only download rating info if the number is higher than that.

I suggest adding a field extra_fields to the Source class that lists the extra fields the plugin can download. Add some config UI to create mappings, display the extra fields in the global and plugin specific lists only if the mappings have been configured. The config UI would most likely be a button on the global screen that pops up a dialog with a list of available extra fields.

Add knowledge of the following fields to the merge code, the config code (to check that the mapping is to a column of the correct type).

price (the problem here is how to deal with multiple currencies)
pages

whatever else you come up with that is fairly standard

[kiwidude]

Originally Posted by kovidgoyal

Since the Rating is already pulled into a standard field, why do you want it to be customized? And the only real use I see for number of ratings is to decide whether to trust the ratings or not. Which can become a configurable threshold. The user sets it and the plugins only download rating info if the number is higher than that.

A couple of reasons. The biggest as I mentioned above is that the way Calibre averages ratings currently means that just about every book I have comes out as 3 stars . You could have 1000 ratings from a site at 3.98 - and it is 3 stars. You could have a rating from one source of 4 and another at 3.98 - and it becomes 3 stars. Basically the rounding makes every book look the "same". Allowing storing the rating in a floating point field is one of the ways of circumventing this issue.

As for the number of ratings, yeah there is a few ways that could be used. What you suggest is one option. A weighted average would be another. A third is to allow a user to display it if they want to. There is no great surprise to see the first few ratings for a book all at 5 stars as the author/publisher tries to sway things. Seeing that a book is rated at x stars from 1500 reviews vs another from 3 reviews does tend to sway the credence you give to it.

Thanks for all the other comments too btw.

[jbienko1]

Hi,

Any news in this topic?

[DoctorOhh]

Originally Posted by jbienko1

Any news in this topic?

News on what topic?

This thread is complete as far as development issues between developers is concerned. If you do not have specific questions about a development issue and how to implement same issue in code then please start a new thread in another forum.

If you have a specific question and want to implement code to accomplish something please be more specific.

[jbienko1]

Originally Posted by dwanthny

News on what topic?

This thread is complete as far as development issues between developers is concerned. If you do not have specific questions about a development issue and how to implement same issue in code then please start a new thread in another forum.

If you have a specific question and want to implement code to accomplish something please be more specific.

Sorry. I agree I wasn't precise enough.
Actually I was interested, if it is possible (and how) to use custom columns in metadata plugins. So before asking, I started search forum for such information. I found this thread.
As I understood, there is no such possibility at this moment and here was discussion about possible implementation methods. I just was interested, if we can expect such functionality in near future. And this was the reason of my question.

[DoctorOhh]

Originally Posted by jbienko1

Sorry. I agree I wasn't precise enough.
Actually I was interested, if it is possible (and how) to use custom columns in metadata plugins. So before asking, I started search forum for such information. I found this thread.

I can see how you would have been attracted to this thread.

Moderator Notice
For future reference please read this sticky post on the purpose of this subforum.

[fenuks]

I would say genre column with set of permitted values. Even if page didn't contain information about genre it can be extracted from tags since we can compare them with list of genres.

[dbolack]

My initial thought is sort of neither.

My impulse is to alter the fields to store arrays and allow the user to select ( defaulted value and per field ) which value to go with. Store the value with the plugin name as the index and the corresponding value.

Not sure yet how much a pain that would be to deal with. I should think this would be simpler to implement and the user to use. Might not be as elegant, but it should be effective.

As to custom column meta data, two thoughts.

One, a check on install for custom columns "expected" by the plugin and prompt the user if they should be created if missing.

Two, go ahead and store metadata we don't know how to deal with.

Probably both.

[JSWolf]

Any word on allowing plugins to populate custom columns?

[dbolack]

Originally Posted by JSWolf

Any word on allowing plugins to populate custom columns?

I haven't touched this code in some time, I'm afriad.

[theducks]

Originally Posted by JSWolf

Any word on allowing plugins to populate custom columns?

The Page Count PI does this (2 user define Integer columns )

[Ramblurr]

Sorry to resurrect this old thread, but I'm still not clear on the answer to the question:

In 2014, can a metadata source plugin provide values for custom columns?

I'm working on a plugin and would love to be able to do this.

The "Download metadata fields" section of the "Configure metadata download" window seems to indicate that metadata source plugins can set these custom fields, as my custom fields are listed in that section. But setting the fields in my plugin doesn't seem to work:

mi = Metadata(title, authors)
# neither of these work
mi.set("#customfield", customvalue)
mi.set("customfield", customvalue)
mi.customfield = customvalue

[kovidgoyal]

No, it is not possible.