Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Development

Notices

Reply
 
Thread Tools Search this Thread
Old 05-21-2011, 06:44 PM   #1
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,220
Karma: 1333994
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Custom columns and metadata sources

I've kind of touched on this subject a couple of times without a potential solution as yet. Kovid has said he will consider a patch but there are a few issues which would need to be resolved that I would appreciate ideas and suggestions on.

The challenge is to find a way to pass values for custom columns from a metadata source plugin through the new 0.8 metadata api. Why? Because it would allow metadata sources to scrape additional data – like the number of pages in the paperback edition, price, number of users who rated the book, etc. Basically any piece of data a particular website could offer in its detail page would become a potential candidate for a custom column.

There are three challenges to this problem I see:
  1. Allowing the user to configure for a metadata source what additional fields they would like scraped into what columns. On the surface just a plugin configuration screen issue which at its simplest would just allow specifying a custom column name for a metadata source field. There would be limitations – metadata sources are not per library configurable so you would have to use the same column in all your databases for instance.
  2. Changing all Kovid’s merge and persistence stuff in identify.py so that values in a custom column would survive the merge process. Including I guess “merging” values in terms of handling two plugins both returning a page count (just pick one), or two plugins returning differing custom columns. Not a showstopper I think.
  3. The most challenging possibly is setting the custom column value on the result Metadata object in the metadata source plugin. The problem being there is no db object. So how can I get the column definition to be able to set a custom column value on the object?

Any thoughts appreciated.
kiwidude is offline   Reply With Quote
Old 05-22-2011, 08:04 AM   #2
JKenP
Member
JKenP will become famous soon enoughJKenP will become famous soon enoughJKenP will become famous soon enoughJKenP will become famous soon enoughJKenP will become famous soon enoughJKenP will become famous soon enough
 
Posts: 22
Karma: 510
Join Date: May 2011
Device: nook
I'm sure featureitis was the precursor for the Tower of Babel. Even the best design and implementation can melt down under that kind of pressure.

My suggestion would be for you to add a single field that could then be repeatedly filled from their spreadsheet containing your comma delimited data and the hundred or so additional column these type have a crying need for. Let them do it on their own dime. They can just keep reimporting that field until they get as confusing as the code changes they cry for.
JKenP is offline   Reply With Quote
 
Enthusiast
Old 05-22-2011, 08:58 AM   #3
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,220
Karma: 1333994
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
I didn't realise I had to qualify my "thoughts appreciated" with "useful thoughts appreciated".

Take your rants elsewhere, the main forum seems to be a good place lately given the tone of some of the recent posters in there. This is the development forum, for discussing technical implementation questions. That you have no need for additional metadata beyond that supplied by Calibre today is perfectly fine, and any changes in this regard won't affect you in the slightest.

However there are people out there who do want to store additional metadata with their books - data that is available from the same web pages we pull title, authors, comments etc from. Just because it isn't part of the standard Calibre set of database fields does not make it any less relevant to some users. Would I personally use all of the potential this feature would expose? No - though I would like to store an actual number of pages, and perhaps some of the rating stuff. Having the whole calibre database normalised to "3 stars" because of the basic way the ratings are stored and averaged now I do not find particularly useful. Also there are people who are storing articles via DOI etc in their library asking for a metadata source - care to bet there isn't some field relevant to those that may come up in future that doesn't fit in Calibre's default set? Opening the API a little further would allow more disparate metadata sources - heck even people who want to store a row for their audio books could have custom columns storing the running time or whatever with the right plugin.

I had started drafting this post in an email to Charles as "Mr Custom Column" but thought I should use the dev forums instead for wider opinions and suggestions.
kiwidude is offline   Reply With Quote
Old 05-22-2011, 11:44 AM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 24,817
Karma: 4369673
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
On 3: I dont see why you would need to do that in the source plugins. Simply create an attribute named extra_metadata = {field1: val1, field2 : val2, ...}

Do some kind of merging on extra_metadata for results from all sources. Return a result that contains extra_metadata from the identify function. Leave it up to the consumer of the result to map the data from fields to custom cols.
kovidgoyal is offline   Reply With Quote
Old 05-22-2011, 12:18 PM   #5
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,220
Karma: 1333994
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Thx Kovid. I guess the answer to that is a question of how "intrusive" this additional functionality should be on the Calibre API. You say the consumer of the identify() function - but that will need to be back on the GUI thread well out of the metadata sourcing function right?

I guess what I had in mind would have kept the metadata plugins self sufficient. So the result of identify() is a Metadata object that is ready to be persisted. The configuration of the mapping and the applying would be plugin specific - the good being that the rest of Calibre doesn't care, the bad being duplication of configuration screens etc per plugin (plus the technical issues of no db object to have the column definitions).

If I instead defer that mapping, how would I know what mappings to apply? Or does the "map this field if it exists to this custom column name" become a generic configuration of Calibre's metadata downloading, in a similar way to how you globally set things like max number of tags, swap author names etc? That would reduce the configuration duplication, but you would have the problem of knowing the list of possible "extra fields" the user could map (unless you extended the API to allow each plugin to define a set of additional fields and iterated across the plugins). And you would still need to apply that mapping presumably right at the last moment possible when you know you are back with a db available again...

Or I could be off base of course :-)
kiwidude is offline   Reply With Quote
Old 05-22-2011, 12:31 PM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 24,817
Karma: 4369673
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
IMO allowing the plugins to enumerate their extra fields and having a global config will be both easier to program and easier to use. The config for this could then simply have a plugboard type mapping of fields to columns. This also has the advantage of allowing per library settings.

Logically speaking, the job of mapping fields to columns cannot be done by the download infrastructure. The download infrastructure is designed to run in a context that has no db, for example, via the fetch-ebook-metadata cmdline tool.

The only downside is that plugin authors will have to be careful to use the same field names for the same information, but that shouldn't be too hard. We can add comments to the base class enumerating the more common "extra" fields.

EDIT: The only major problem I see is that the extra fields will have no semantics associated with them, so it will be hard to design code to pick the best one. I would suggest have a set of "standard" extra fields that the base class knows about and can merge intelligently.
kovidgoyal is offline   Reply With Quote
Old 05-22-2011, 02:17 PM   #7
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 4,705
Karma: 800144
Join Date: Jan 2010
Location: France
Device: Many android devices
I am starting from the point of ignorance: I don't know where downloaded metadata is written to the db. I assume that it all ends up at some point at library2.set_metadata.

There are two issues to solve, I think. The first is to have a common set of names for diverse and extra metadata. For example, various editions of books have different page counts that might be called 'pages' or 'length'. Authors in multi-authored books sometimes are contributors, sometimes are authors. Plugins should be able to query and add to this set of names.

The second problem is mapping the data to columns in the database. Here I am in agreement with Kovid -- this is best done as a mapping of metadata name to column. Set_metadata can implement this mapping. The only complexity I see is handling multiples: should values be merged or should they replace?

One way to proceed is to have a dialog in preferences that defines the names the user wishes to use. The set to choose from could come from the installed plugins (this is a push system), or the plugins could use these names (a pull system). Both have merit. The user would choose what column to use for storing information under the name, and if necessary any auxiliary information such as merging.

Both push and pull systems should permit specifying a data type for a name. The dialog would use this to prevent absurdities such as assigning a text comments field to an integer. Some special cases could be allowed, such as series-type fields carrying along the series index (somehow).

Bottom line: as far as I can see, the major problem is having an extensible set of names with known semantics. Once we have these, mapping the names to columns (custom or otherwise) is not hard.
chaley is offline   Reply With Quote
Old 05-22-2011, 03:50 PM   #8
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,220
Karma: 1333994
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
I like the sound of per library settings, that sounds rather more flexible. And standardising on some common field names like "pages" shouldn't be an issue. I think there would need to be the flexibility to allow a plugin to have some additional field(s) that are bespoke to it without them being centrally defined in the Calibre code, so defined as part of a plugin's API sounds a plan.

Charles you raised an interesting point about the specification including the data types a custom column must conform to for matching - I've been able to get away with hard-coding this when it has been self-contained in the plugin.

I also hadn't seriously considered series type fields which are probably going to be a pain to deal with. The only use I could think of such a field would be for an "alternate series", but then you need to find a metadata source that could populate it (none of the B&N, Goodreads, Fantastic Fiction, Webscription plugins could populate this). I guess you don't like shortcuts

As for where this mapping translation takes place, I wonder whether we can get away with waiting until set_metadata. The situation that comes to mind is the edit metadata dialog, which of course has the ability to display custom column values. Maybe it is a special case. I would have thought you would have to get the metadata object updated into custom columns by when the Download Metadata button has finished so it can refresh the dialog contents (which are not yet persisted)?
kiwidude is offline   Reply With Quote
Old 05-22-2011, 04:00 PM   #9
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 4,705
Karma: 800144
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by kiwidude View Post
As for where this mapping translation takes place, I wonder whether we can get away with waiting until set_metadata. The situation that comes to mind is the edit metadata dialog, which of course has the ability to display custom column values. Maybe it is a special case. I would have thought you would have to get the metadata object updated into custom columns by when the Download Metadata button has finished so it can refresh the dialog contents (which are not yet persisted)?
set_metadata already quite happily sets custom columns, so there would be no problem providing a method to merge the 'named' information into the custom columns in the mi structure.

We would also need to change update_from_mi in the single metadata dialog to update the custom widgets, which it currently does not do. This would be a pain because of the number of changes, but the intellectual difficulty is low.

EDIT: looking at the code, I think updating the custom widgets from an mi structure is straightforward. Add one method to a base class then add a loop over the mi structure that passed an mi structure instead of a book_id.

Last edited by chaley; 05-22-2011 at 04:08 PM.
chaley is offline   Reply With Quote
Old 05-25-2011, 07:24 PM   #10
JohnnyRocks
Zealot
JohnnyRocks has learned how to read e-booksJohnnyRocks has learned how to read e-booksJohnnyRocks has learned how to read e-booksJohnnyRocks has learned how to read e-booksJohnnyRocks has learned how to read e-booksJohnnyRocks has learned how to read e-booksJohnnyRocks has learned how to read e-booksJohnnyRocks has learned how to read e-books
 
Posts: 116
Karma: 960
Join Date: Feb 2011
Device: PocketBook 360
So is this something that is doable? It would be IMMENSELY helpful as right now I'm manually entering all the ratings for all the books in my library (a couple thousand), and it's hugely time consuming.
JohnnyRocks is offline   Reply With Quote
Old 05-27-2011, 07:24 AM   #11
JKenP
Member
JKenP will become famous soon enoughJKenP will become famous soon enoughJKenP will become famous soon enoughJKenP will become famous soon enoughJKenP will become famous soon enoughJKenP will become famous soon enough
 
Posts: 22
Karma: 510
Join Date: May 2011
Device: nook
Thanks for straightening me out, Kiwi or Mr. Dude. I appreciate your addition of the useful adjective. In watching this thread, useful seem to ebb and flow with rather esoteric demands. Everybody has their field de jour to throw into the useful category.

Just because you can do or specify something doesn't mean you should.

Words may or may not be rants but definitions are interesting.

–verb (used without object)
1. to speak or declaim extravagantly or violently; talk in a wild or vehement way; rave: The demagogue ranted for hours.

2. disagree mildly with Kiwidude.

Last edited by JKenP; 05-27-2011 at 07:39 AM.
JKenP is offline   Reply With Quote
Old 05-27-2011, 08:20 AM   #12
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,220
Karma: 1333994
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
@JKenP - well we can agree to disagree on your input being "useful" to this thread to date. As I said above this a technical forum for the Calibre developers to discuss the most appropriate way to solve a problem. I'm not interested in people posting here that they have no use for a particular feature when it is so obviously an optional addition. Every few days there come up other potential uses - even yesterday when someone suggested wanting a plugin to be able to find books in their library which had won awards. Guess what an extension to the Goodreads metadata plugin to populate an awards custom column could do for them if this feature were available?

@JohnnyRocks - obviously we have come to our own solution between us via PM outside this thread but to officially answer your question anyways for the benefit of others. The correct solution to this problem appears to involve a lot more Calibre code changes than I had hoped, which means I will need to vamp some input from Kovid/Charles. Which makes it all a lot more complicated to get done. I am still keen on it, but I'm not going to take it on myself right now without help.
kiwidude is offline   Reply With Quote
Old 05-27-2011, 06:49 PM   #13
JKenP
Member
JKenP will become famous soon enoughJKenP will become famous soon enoughJKenP will become famous soon enoughJKenP will become famous soon enoughJKenP will become famous soon enoughJKenP will become famous soon enough
 
Posts: 22
Karma: 510
Join Date: May 2011
Device: nook
Kiwidude, many demand kitchen-sink plugins for any number of product. End users always have varied needs that they're happily to express. If followed to its final form, the world would only need a single program that did everything.

Calibre is a solid product with an exceptional feature set. It has a defined database that others can access with their own applications

I have been through the design process with clients and have seen those change over time. Those were modest and well thought out changes. There are gotchas along the route but sometimes change is beyond an option.

There are departments that even had additional need for "their" fields. Those always seemed best served to me with linkage from a secondary database and apps to service those needs.

But, all of that was definable. Here, it is a Chinese menu with unlimited ordering from column a and column b. That makes for a maintenance and code nightmare. We always compromise code standards over time even with modest change requirements. It is the rare event when that is 100% avoided.

You want to provide all those nifty extra fields? Great. Link to the basic db and add to the secondary all those fields -- however many thing you see a need for. It lets the original product be and preform the services intended. You'll need to code for all that rococo but what the heck, you'll have fun doing it. You can throw in a gargoyle on one of the pediments to remind you of me.

Or you could become a SAP programmer or user and really have the kitchen sink but I wouldn't wish that on anybody.
JKenP is offline   Reply With Quote
Old 05-27-2011, 07:10 PM   #14
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,220
Karma: 1333994
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Jkenp - you continue to spout nothing of value to this thread and again miss the point. Data would be stored in custom columns - a feature calibre already has. It would merely allow users to automate the populating of those columns with data they choose to put in them, rather than manually copy pasting them. It is not changing the calibre database schema, and it won't affect users who choose to not gather the extra data from the metadata sources.
kiwidude is offline   Reply With Quote
Old 05-28-2011, 05:51 AM   #15
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,573
Karma: 12369681
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by JKenP View Post
Kiwidude, many demand kitchen-sink plugins for any number of product. End users always have varied needs that they're happily to express. If followed to its final form, the world would only need a single program that did everything.
Since being off topic is fashionable, here's my 2 cents.
@JKenP it has been mentioned before that this is a development forum. So far your posts have added little to this thread. Reading your posts tell me you aren't stupid....

@Kiwidude I applaud your queries into expanding the capabilities of the APIs available to GUI plugin designers. Often I don't figure I'll use the features you have helped expand in the past, often I'm wrong.

Hopefully you'll find the time to to structure your concept in such a way as to get the input from Kovid/Charles that you require. They both have been very responsive to your ideas in the past once y'all wrapped your minds around the concept. Keep up the

Last edited by DoctorOhh; 05-28-2011 at 05:54 AM.
DoctorOhh is online now   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Custom Columns - How are you using yours? nynaevelan Library Management 19 04-18-2011 12:42 AM
Custom Columns Question Greg Waddell Calibre 5 03-31-2011 12:34 AM
New Metadata class (empty books) and custom columns kiwidude Development 14 03-12-2011 06:47 AM
0.7.46 and custom columns meme Library Management 4 02-21-2011 04:21 AM
Custom Columns - the Future? Starson17 Calibre 2 07-13-2010 09:56 AM


All times are GMT -4. The time now is 07:27 AM.


MobileRead.com is a privately owned, operated and funded community.