Managing subgroups of books, for example "genre" (long post)

[chaley]

(Superseded by a section in the calibre manual: http://calibre-ebook.com/user_manual/sub_groups.html.)

Some people wish to organize the books in their library into subgroups, similar to subfolders. The most common wish is to create genre hierarchies, but there are many others. One user asked for a way to organize textbooks by subject and course number. Another wanted to keep track of gifts by subject and recipient. I will use the genre example for the rest of this post.

Before I go on, please note that I am not talking about folders on the hard disk. Subgroups are not file folders. Books will not be copied anywhere. Calibre's library file structure is not affected. Instead, I am talking about a way to display subgroups of books within a calibre library.

The commonly expressed requirements for subgroups such as genres are:

• A subgroup (e.g., a genre) must contain (point to) books, not categories of books. This is what distinguishes subgroups from user categories.
• A book can be in multiple subgroups (genres). This distinguishes subgroups from physical file folders.
• Subgroups (genres) must form a hierarchy; subgroups can contain subgroups.

Tags give you the first two. If you tag a book with the genre then you can use the tag browser (or search) for find the books with that genre, giving you the first. Many books can have the same tag, giving you the second. The problem is that tags don't satisfy the third requirement. They don't provide a hierarchy.

Calibre's new hierarchy feature gives you the third, the ability to see the genres in a 'tree' and the ability to easily search for books in genre or sub-genre. For example, assume that your genre structure is similar to the following:

Genre
. History
.. Japanese
.. Military
.. Roman
. Mysteries
.. English
.. Vampire
. Science Fiction
.. Alternate History
.. Military
.. Space Opera
. Thrillers
.. Crime
.. Horror
etc.

By using the hierarchy feature, you can see these genres in the tag browser in a tree form:



As you can see, in this example the outermost level (Genre) is a custom column. The genres themselves appear under that column. Genres containing sub-genres appear with a small triangle next to them. Clicking on that triangle will open the item and show the sub-genres, as you see with History and Science Fiction.

Clicking on a genre will search for all books with that genre or children of that genre. For example, clicking on Science Fiction will give all three of the child genres, Alternate History, Military, and Space Opera. Clicking on Alternate History will give books in that genre, ignoring those in Military and Space Opera. Of course, a book can have multiple genres. If a book has both Space Opera and Military genres, then you see that book if you click on either genre. Searching is discussed in more detail below.

Another thing you can see from the image is that the genre Military appears twice, once under History and once under Science Fiction. Because the genres are in a hierarchy, these are two separate genres. A book can be in one, the other, or (doubtfully in this case) both. For example, Winston Churchill's World War II books could be in "History.Military". David Weber's Honor Harrington books could be in "Science Fiction.Military", and in "Science Fiction.Space Opera" for that matter.

Once a genre exists, that is the genre has been applied to at least one book, you can easily apply it to other books by dragging a book from the library view onto the genre you want the book to have. You can also apply them in the metadata editors. More on this below.

SETUP
Your question by now might be "how did I set all of this up?". There are three steps: 1) create the custom column, 2) tell calibre that the new column is to be treated as a hierarchy, and 3) add genres.

I created the custom column in the usual way, using Preferences -> Add your own columns. I used "genre" as the lookup name and "Genre" as the column heading. The column type is "Comma-separated text, like tags, shown in the tag browser."



Then after restarting calibre, I told calibre that the column is to be treated as a hierarchy. I went to Preferences -> Look and Feel and entered the lookup name "#genre" into the "Categories with hierarchical items" box. I pressed Apply and was done with setting up.



At the point there are no genres. We are left with the last step: how to apply a genre to a book. A genre does not exist until it appears on at least one book. To apply a genre for the first time, we must go into some detail about what a genre looks like in the metadata for a book.

A hierarchy of 'things' is built by creating an item consisting of phrases separated by periods. Continuing the Genre example, these items would "History.Military", "Mysteries.Vampire", "Science Fiction.Space Opera", etc. Thus to create a new genre, you pick a book that should have that genre, edit its metadata, and enter the new genre into the column you created. Continuing my example, if I want to assign a new genre "Comics" with a sub-genre "Superheros" to a book, I would 'edit metadata' for that (comic) book, choose the Custom metadata tab, and then enter "Comics.Superheros" as shown in the following (ignore my other custom columns):



After I do the above, I see in the tag browser:



From here on, to apply this new genre to a book (a comic book, presumably), I can either drag the book onto the genre, or add it to the book using edit metadata in exactly the same way as I did above.

SEARCHING
The easiest way to search for genres is to use the tag browser, clicking on the genre you want to see. Clicking on a genre with children will show you books with that genre and all child genres. However, this might bring up a question. Just because a genre has children doesn't mean that it isn't a genre in its own right. For example, a book can have the genre "History" but not "History.Military". How do I search for books with only "History"?

The tag browser search mechanism knows if an item has children. If it does, clicking on the item cycles through 5 searches instead of the normal three. The first is the normal green plus, which shows you books with that genre only. The second is new: a doubled plus (shown below), which shows you books with that genre and all sub-genres. The third is the normal red minus, which shows you books without that exact genre. The fourth is new: a doubled minus, which shows you books without that genre or sub-genres. The fifth is back to the beginning, no mark, meaning no search.



RESTRICTIONS
If you search for a genre then create a saved search, you can use the 'restrict to' box to create a virtual library of books with that genre. This is most useful if you want to do other searches within the genre or to manage/update metadata. For this example I created a saved search named 'History.Japanese' by first clicking on the genre Japanese in the tag browser to get a search into the search box, entering History.Japanese into the saved search box, then pushing the "save search" button (the green box with the white plus, on the right-hand side).



Once I have done that, then I can use this search as a restriction.

[chaley]

When using a setup similar to the one described above, you might want to use the information in templates such as send-to-device and save-to-disk. Calibre 0.7.48 has a new template function to faciliate this. For example, you can slice off the first part of a hierarchy and add it to the save path. Using plugboards, you could add it to the title or what-have-you.

Taken from the calibre template language manual (http://calibre-ebook.com/user_manual/template_lang.html, single-function mode):

subitems(val, start_index, end_index) – This function is used to break apart lists of tag-like hierarchical items such as genres. It interprets the value as a comma-separated list of tag-like items, where each item is a period-separated list. Returns a new list made by first finding all the period-separated tag-like items, then for each such item extracting the start_index-th to the end_index-th components, then combining the results back together. The first component in a period-separated list has an index of zero. If an index is negative, then it counts from the end of the list. As a special case, an end_index of zero is assumed to be the length of the list.

Examples:
Assuming a #genre column containing "A.B.C":
{#genre:subitems(0,1)} returns "A"
{#genre:subitems(0,2)} returns "A.B"
{#genre:subitems(1,0)} returns "B.C"

Assuming a #genre column containing "A.B.C, D.E":
{#genre:subitems(0,1)} returns "A, D"
{#genre:subitems(0,2)} returns "A.B, D.E"

[nynaevelan]

Hi Chaley:

Is it possible to use one of your examples above with the following plugboard and custom send to device template:



send to device template: {tags:re((.*?)(\,|$).*,\1)}/{series} {series_index:0>2s} - {title}

Nyn

[chaley]

Originally Posted by nynaevelan

Hi Chaley:

Is it possible to use one of your examples above with the following plugboard and custom send to device template:

[snip]

send to device template: {tags:re((.*?)(\,|$).*,\1)}/{series} {series_index:0>2s} - {title}

Nyn

Yes, but without knowing what you want to do, I can't go further than that.

[nynaevelan]

Chaley:

Right now I have Calibre setup to send ebooks to my k2 with the above template so that they are divided into folders based on their tag. I have a plugboard setup to add the stripped series name, series index and title to the title field. I also use the hierarchy information in order to have some of my tags using the hierarchy functionality. And I use Meme's kindle collections plugin to organize my collections based on their tags. But now the tags have the Read.xx in them, I would like to possibly rename them so that they will only have the sub.tag name. While at the same time keeping my send to template, plugboard template and continuing to use the plugin. Is this possible or will I have to re-evaluate to determine which of these functions I have to live without?

[chaley]

Originally Posted by nynaevelan

Right now I have Calibre setup to send ebooks to my k2 with the above template so that they are divided into folders based on their tag. I have a plugboard setup to add the stripped series name, series index and title to the title field. I also use the hierarchy information in order to have some of my tags using the hierarchy functionality. And I use Meme's kindle collections plugin to organize my collections based on their tags. But now the tags have the Read.xx in them,

So one issue is that the tags column is being used for multiple things: the kindle annotations read.xx stuff, kindle collections, and something hierarchical (you didn't say what it is)?

I would like to possibly rename them so that they will only have the sub.tag name.

By 'them', do you mean the contents of the tags column? You want the tags column to contain only the sub.tag (the hierarchy) stuff? That isn't possible if you use the read.xx annotation function.

While at the same time keeping my send to template, plugboard template and continuing to use the plugin. Is this possible or will I have to re-evaluate to determine which of these functions I have to live without?

You can probably have it all, but not with the organization you have.

Looking at the send template, it splits off the first tag (no matter what it is) and uses it for a folder name. This will be a problem if the tag you want ever falls alphabetically after 'Read.xx' or (possibly) the kindle collection information.

My gut feeling is that you should create a different custom column for each different 'meaning'. Mixing information with different 'meanings' together can only lead to complexity and confusion. That said, it is very hard for me to give specific advice, because I don't have a kindle and therefore don't use either the meme's collections plugin or pchrist's amount-read stuff. It is clear that you should use the tags column for the amount-read, because the annotation is put into tags automatically. You should use another custom tags-like column for the sub.name, separating it from the other stuff (like read.xx). Finally, I would think that putting kindle collections into yet another column would simplify your life, but I don't know that.

The detailed answers to your questions will require assistance from someone who uses a kindle and understands the interplay between the various bits of information you are manipulating.

[nynaevelan]

Thanks, I think I will take your advice and split my setup to create another custom column for the tags that will be using the hierarchial information. That way I can get to use all the more advanced features that you all have introduced. Thanks for your time and assistance.

Nyn

[ViktorShchedrin]

Many thanks for the thread. Is it way to sync Calire genres with GoodReads.com ? and - bulk sync and build genre hierarchy - Calibre book's genre using ISBN ? AFAIK this goodreads metadata sync https://www.mobileread.com/forums/sho...d.php?t=121641 did not synced genre. Thanks.

[chaley]

Originally Posted by ViktorShchedrin

Many thanks for the thread. Is it way to sync Calire genres with GoodReads.com ? and - bulk sync and build genre hierarchy - Calibre book's genre using ISBN ? AFAIK this goodreads metadata sync https://www.mobileread.com/forums/sho...d.php?t=121641 did not synced genre. Thanks.

I know of no way to build genre hierarchies automatically from downloaded metadata. You might be able to do something with search/replace if you know which tags you wish to use as genre and how to form a hierarchy of them.

You might ask on that goodreads thread if the plugin author (kiwidude) is willing to do something. I have no idea if he would support the idea, how possible it is, or how much work it would be.

[DoctorOhh]

Originally Posted by ViktorShchedrin

Many thanks for the thread. Is it way to sync Calire genres with GoodReads.com ? and - bulk sync and build genre hierarchy - Calibre book's genre using ISBN ? AFAIK this goodreads metadata sync https://www.mobileread.com/forums/sho...d.php?t=121641 did not synced genre. Thanks.

That is the Goodreads metadata download plugin, you might want to look at the Goodreads Sync GUI plugin.

[kiwidude]

This discussion is perhaps better to be in the Goodreads Sync thread but I will make a few comments.

I would guess that Viktor has a number of shelves in Goodreads using them as a genre, like 'science-fiction' etc. With the initial release of Goodreads sync it was possible to set an action to apply when syncing from a shelf, such as setting a value in a custom genre column. However that feature it wasn't really intended to be used for genres, as if you have a large number of genre shelves it would be a pain to setup all the rules and one by one sync them all every time you make a change.

In the 1.1 version I added a "Download shelves as tags" feature, which would let you map shelf names to tags in Calibre. Similar functionality in some ways, except the idea is you select a bunch of books in Calibre that you have on Goodreads shelves, choose this menu option and have the "tags" column assigned values based on what shelves each book is on.

What chaley suggests above from my quick reading is an additional custom column to contain your genre tags. This is not currently possible with the Goodreads Sync plugin using the "Download shelves as tags" feature, though you could do it using the sync feature, with the same issues I described above.

What I could do is enhance the "Download shelves as tags" feature to allow you to specify an alternate custom tags column to put the values into, which sounds useful anyway. So you could get them placed into a #genre custom column for instance. You would still need to setup the tag mappings into how you want them, and it would still be dependent on how you organised your Goodreads shelves. For instance to have a "Science Fiction.Space Opera" genre tag set you would need to have some shelf in Goodreads that you could identify to map this to, like 'sci-fi-space-opera' or whatever. If you just have a 'sci-fi' shelf, there is no automagic to put it into a sub-genre.

[Huisie]

Chaley, thanks for your excellent explanation both here and in the manual.
Until I read this post I hadn't a clue how to populate a custom column.

My first attempt was for the example you used, to create genre tags.

Unable to populate a custom column, I tried another route:

• - I created a user category (Genre) and
• - set "Tags" as a filter.

As "genre"-type tags are to be found amongst the other tags (are these what's referred to as "Social Metadata"?) it was fairly easy to select those that represented genres.

My method seemed easier - I was merely manipulating info that was already there rather than having to create it.

I couldn't create a custom column from that list, though.
Does this matter, as long as I am able to sort by "Genre", one way or another?

Other questions I have are:

• What is the difference between these methods?
• What limitations does each one come with?

General confusion:
To this novice, it appears custom columns can also be user categories and/or groups.
I can't understand the difference between columns, user categories and groups.
I know you gave an explanation (quoted below) but I can't get my head around it:

A subgroup (e.g., a genre) must contain (point to) books, not categories of books. This is what distinguishes subgroups from user categories.

[chaley]

Originally Posted by Huisie

My method seemed easier - I was merely manipulating info that was already there rather than having to create it.

I couldn't create a custom column from that list, though.
Does this matter, as long as I am able to sort by "Genre", one way or another?

What matters is that what you do works for you.

General confusion:
To this novice, it appears custom columns can also be user categories and/or groups.
I can't understand the difference between columns, user categories and groups.
I know you gave an explanation (quoted below) but I can't get my head around it:

I am probably diving into the deep end, but here goes...

We are dealing with two basic concepts, 'thing' and the 'idea of thing'. For example, consider a stone. The stone exists, you can throw it, etc. It is a 'thing' or (what I like better) an 'instance'. Now consider telling a friend over the phone that you have the stone. Your friend is left with an 'idea of stone', not an 'instance of stone'. Suppose your friend gets interested and calls other friends to find out what stones they have. Your friend's 'idea of stone' now encompasses many instances of stone. In other words, for your friend, 'stone' is a category, not an instance. A third party might note that 'stone' has two very different meanings. It is ambiguous; its meaning depends on context.

Now assume that your friend gets a stone. Suddenly your friend must deal with 'instance' and 'idea' at the same time, and must work out a system for knowing which meaning applies, when. We have the same problem in calibre. A term can describe an instance, a category, or both . The questions then become "How do we disambiguate between the meanings?" and "How do we manipulate the meanings?" To do this in a computer program like calibre, we must have a way of concretely representing (modeling) the various meanings of information.

In calibre, we can say that a column (any column) is a category, representing the 'idea of something'. A row is a book, an instance of something. The cell (row/column) is therefore an instance of a category. For example, the Authors column contains names of authors (instances at an abstract level). The Titles column contains instances of titles. If you create a custom column, then one hopes that the column contains instances consistently fitting a category (idea).

The difficulty that arises is that an instance might fit into more than one category. For example, an author can be categorized by nationality, location, style, gender, genre, hair color, and a host of other notions, none of which are conveyed by the idea 'Author'. Tags are even worse, because the notion of tag is much more abstract than author. Calibre deals with this difficulty by providing 'user categories'. When you create a user category, you are working with the 'idea' of something. What you put into the category are the instances that somehow are connected to that idea. You might create a category called Spanish Authors, an 'idea' that would provide more information about authors. You might create a category called 'Bad Things', another idea which could contain instances of almost any kind (tag, author, publisher, etc).

Bottom line: categories represent an 'idea' about and are independent of a book. Categories can exist even if the library is empty. Columns are categories. Book metadata is a row of tags, authors, etc representing 'instances' of category defined by the columns. User categories are similar to columns in that they convey additional 'ideas' about instances.

Regarding hierarchical items: it could be argued that they are both a category and an instance. However, for convenience we must pick one definition. In calibre, a hierarchical item exists only if it appears on a book, so it is an instance, not a category.

[ViktorShchedrin]

Originally Posted by kiwidude

I would guess that Viktor has a number of shelves in Goodreads using them as a genre, like 'science-fiction' etc. With the initial release of Goodreads sync it was possible to set an action to apply when syncing from a shelf, such as setting a value in a custom genre column. However that feature it wasn't really intended to be used for genres, as if you have a large number of genre shelves it would be a pain to setup all the rules and one by one sync them all every time you make a change.

many thanks, you explaination of my gues is 100% correct. i said about hierarchy genre (based on goodreads genres) generated instead "linear" tags. Of course my wish if it possible and not difficult for developers.

[Huisie]

@Chaley

Thanks for the info and the detailed (and zen-like ) reply.

In another post I made, kiwidude also gave an excellent reply (post #5).

You both have given me lots to mull over, but that's the nature of the Calibre beast, I guess.

Being confused by options is better than being confounded by none.