Best strategy for metadata management for Kobo using Calibre?

[davidfor]

Originally Posted by ceridwen

At the moment, I'm using a very crude approach: I've created a custom column, and I have two scripts that literally invoke calibredb repeatedly. The AO3 script handles AO3 tags directly (bypassing the epub) and only calls calibredb to look up the calibre id and then to run calibredb set_metadata. The non-AO3 script looks up the Tags column and then runs calibredb set_metadata to add some of them to the custom column. This is slow and awkward, so I'd really like to write a plugin to do it instead. I spent some time looking through the API documentation for plugins, but I didn't see any discussion of how to write a plugin that does the kinds of things I want here. Does anyone know of a good example plugin that could get me started or some other resource that would help me?

Some of that might be possible with search and replace. It really depends on if the rules are simple enough.

Some more can be done with the tag mapping. Though, it doesn't do custom columns.

A plugin to set custom columns isn't that hard to write. For examples:

The Count Pages plugin is the one I normally point people to (or Modify ePub, but it doesn't fit with what you want). It can apply its function over multiple books and set multiple custom columns. It only does numeric columns, but it isn't much of a stretch to do tag-like colums. It is useful if you want to something in a batch job. Plus, it should be the one that does the updates in the best manner.

Both the Reading List and Goodreads Sync plugins will set the tags or tag-like columns. The former is probably simpler, but is setting a single value. The later is generating the list of tags from Goodreads and then setting the columns. Import List also does multiple updates of columns based on data from CSV files or web sites.

My Kobo Utilities plugin does various things were it updates columns. The collection fetch will update a tag-like custom column with the current collections the books are in. Not sure if I really want to suggest looking at it, as there is some very ugly code in the plugin. But, it does do just about anything you want to in a plugin except actually modify a book.

[ceridwen]

For the curious, everything I needed for my particular use-case (I'm happy running a command-line script periodically to update my tags, I don't need a GUI) I found in the database API. I call:

db.all_book_ids()

Fetch the metadata for each book with:

db.get_metadata()

Do all the tag processing, build my set of collections, and pass the result to:

db.set_field('#collections', book_id_to_val_map)

For filtering down tags to a subset small enough for collections, I excluded all tags that were found on only one work or on one author's works, normalized tags by singularizing the last word and normalizing upper/lower case, keeping only fandom and freeform tags from AO3, and then only tags with 5 or more occurrences in my database.

At this point, I think I have more to gain by improving the metadata I'm using as input than refining the collections processing. In particular, most fanfiction sites have additional contextual information about what tag represents like the Category, Relationship, Fandom, and Freeform classifications for AO3. For AO3 I wrote my own scraper because I also needed it to handle canonicalization which FanFicFare definitely doesn't do. I took a look at FanFicFare's capabilities for other sites. Internally, it calls a method self.story.setMetadata() that takes two arguments, a type of metadata and its actual value. However, even after I looked through the config settings, it's not immediately clear to me how to use that. What I really want is a way to just dump key-value pairs into a custom column in Calibre or a file, with FanFicFare's command tool. FanFicFare also doesn't do complete metadata processing for some sites, so I'll either have to extend it or write my own scraper, again.