Columns, Categories, Groups & Metadata

[Huisie]

Calibre cannot but be complex, owing to the source material.
Kovid and crew have done brilliantly in designing, improving and maintaining such a flexible and powerful system for managing it all.

No only that, but the manuals, help videos, etc. + the indefatigable experts in this forum are a tremendous help.
Even with all these sources to draw from, I find myself becoming mired in the options and unavoidable complexities.

I love the fact that there are options to sort so thoroughly (it's like bibliophile crack).
You can sort multi-series/sagas (such as the Riftwar saga) properly, you can create proper multi-genre/sub-genre classifications ... the list goes on.

The questions about how it all interrelates, and what each element does, grow too, however.

Some questions:
What are the differences between user categories, custom columns and groups?
Generally, how do they allow you to use/manipulate the metadata?
How can they affect metadata, such as when saving-to-disk?
It appears that some custom columns can be considered user categories too?
What's the difference between metadata fields like {author} and {author_sort} or {title} and {title_sort}?

Example 1:
With the Riftwar books (by Raymond, E. Feist), there is the overall saga with various chronological series and some of the series tie in together.
A possible order might be:

Riftwar (overall series/saga)
- 1. The Riftwar Saga
- 2. The Empire Trilogy
- 3.1 Krondor's Sons
- 3.2 Legends of the Riftwar
- 3.3 Riftwar Legacy
- 4. Serpentwar Saga
- 5. Conclave of Shadows, etc

I'd like to represent this in Calibre and be able to transfer this info to disk/device, but am unsure which combination of columns/categories/groups to use without tying myself into knots.

Example 2:
Chaley put together an excellent explanation of how to manage Genres, using custom columns.
Until I read that post I hadn't a clue how to populate a custom column.

I had tried another route by:
- creating a user category and
- using tags as a filter.
- This way, it was easy to select from the tags those that represented genres.
My method seemed easier - I was merely manipulating info that was already there rather than having to create it.
I couldn't create a custom column from that list, though.

What is the difference between these methods, I wonder?
What limitations does each one come with?

I know Chaley included an explanation of the differences between tags and user categories and custom columns/groups but I can't follow that though.
It is somewhat like listening to an explanation of advanced physics - the overall conclusion makes sense but I get lost when trying to folow the explanation of how the conclusion is derived.

My motive in asking all this is not to elicit hand-holding or get others to do the hard yards for me.
I find myself stalled in my understanding:
- in a "the big picture is just SO big" kind of way and
- in a "can't see the forest for the trees" kind of way

I suspect I'm not the only newbie who is struggling with these concepts and how to use them concretely.

I'm wanting to stimulate some discussion on these ideas that would help further my (and others) understanding,
Hopefully, this would lead to me being more of a help and less a hindrance in future - grow the pool of experienced and clued-up users by helping some newbies over the initial barriers.

[Huisie]

I suspected this post might be too long and rambling to elicit a response, but it does reflect my confusion at how several elements interrelate, so thought it worth a shot.

More than 60 views without response bears this out, I think.

Can one of the mods close his post, please?

I will reconsider my questions and ask them in a manner that can be answered.

Thanks in advance.

[theducks]

Originally Posted by Huisie

I suspected this post might be too long and rambling to elicit a response, but it does reflect my confusion at how several elements interrelate, so thought it worth a shot.

More than 60 views without response bears this out, I think.

Can one of the mods close his post, please?

I will reconsider my questions and ask them in a manner that can be answered.

Thanks in advance.

60 views without an answer are nothing arround here
You question was polite and thought out.

Let'r ride. (besides, I am interested in the answers that may appear. I don't use anywhere near 50% of the features Calibre has to offer, some, simply because it never occured to me to use it in that way. )

[Nyssa]

I agree with and ditto what theducks has said. 60 views means, we're interested but we don't know how to answer your question. I'm guessing some of those people have done what I've done and subscribed to your post so that we can also see the answer, should one come along. Don't give up hope.

[kiwidude]

It is just an awful lot of questions. Seriously, you could write an entire manual on all the topics you attempt to ask. It was a very polite ask, but it would take hours of posts to cover them all.

I think you just need to get a couple of basic concepts right in your mind:

Custom columns are a way of allowing you to store or display additional information about a particular book. Such as a genre, whether you have read it etc. Whatever you can think of. There are lots of ways of skinning a cat, some people use tags to do the same thing in some cases. In other cases a custom column is the only way to go, such as if you wanted a column recording the date you last read the book.

I think your confusion with user categories comes because you see custom columns appearing in the tag browser on the left. Think of the tag browser as a giant point and click way of accessing your database. It lets you point and click to filter your database by an attribute of a book, such as its author or series. For convenience it also lets you point and click filter for certain custom column types too, that is why if you add a "Genre" custom column it appears in the tag viewer. So a custom column is *not* a user category, it just gets displayed in the tag browser next to genuine user categories.

Custom columns are most frequently used just to allow you to view/store additional information about a book. There are scenarios where people use them for other purposes to do with plugboards and sending books to their devices but ignore that for now, it is a niche usage.

For a genuine user category, you must use the Manage user categories button to create them. There is a thread here discussing ideas for how people find it useful. Fundamentally a user category is just a way of building a group of books together as a clickable filter by some criteria be it authors, tags, series or publishers. In some ways it is very similar to a saved search, but just a lot more user friendly for some purposes. For instance a "Favourite Authors" user category containing 20 specific author names is easier to manage than a giant "or" based search.

However Calibre has such flexibility it can be difficult to decide which approach to take at times. For instance to manage favourite authors you could instead put a Favourite tag on each book, or a custom yes/no column called Favourite. However if you are thinking about that categorisation at a level other than book (this is a favourite *author*) then to avoid the ongoing maintenance every time you get a new book by that author you would be better off having a user category instead.

As for {author} vs {author_sort}. Say you store your authors displayed on screen as FN LN, like Tom Clancy. The author sort should have been set to "Clancy, Tom" so that when you sort on screen the books appear under C rather than under T. When you save books to disk or to a device, if you were to use the {author} field then "Tom Clancy" will get used, if you use {author_sort} then "Clancy, Tom" will get used. Similar with {title} and {title_sort} - Calibre has the ability to strip the "The" and "A" etc and stick them at the end for display or sorting purposes so depending on your settings you may want something different.

In my library I set the author to be LN, FN the same as the author sort, so it does not matter whether I use {author} or {author_sort}. Likewise I don't like the default title sorting/display of the trailing ", A" and ", The" etc, so after applying tweaks again it makes no difference whether I use {title} or {title_sort}. There are "tweaks" you have to set and hoops to jump through unfortunately, it is an area of Calibre that needs cleaning up with a wizard screen on startup imho.

So - whether to use a tag, a custom column, or a user category depends on whether you want to store book specific data or view across a higher level of abstraction.

[Huisie]

Firstly to "the ducks" and "Nyssa":

Thanks for the encouraging remarks.
Should I be worried that such experienced users are also curious?

Part of the reason I asked for termination was that I thought breaking up my "theme" into smaller and more digestible questions might be better.

For example, I asked some questions related to "Groups/sub-groups" in a thread by Chaley - he used "Genres" as an example.
(Well worth a read, by the way!)

I didn't want to find myself guilty of double-posting, but also (selfishly) didn't want to hang around forever waiting for answers that might never come.

So, my request to close this down seemed the sensible option, one that respected the forum and members.

I am happy for this to stay open AND (with permission) continue to seek smaller bits of info.
These can then be brought back in here and added to the whole.


Secondly, to "kiwidude"

Thank you for replying and that "Guru" tag is well-deserved!
I very much agree with you when you say:

Seriously, you could write an entire manual on all the topics you attempt to ask.

Ironically, you then provide such a detailed and concise explanation that, if I were asking how to put together IKEA flatpack, the manual would stay in its wrapper.


I still have to properly digest it all and think it through - just wanted to leave some words of thanks and appreciation to y'all.

[Manichean]

Originally Posted by Huisie

Can one of the mods close his post, please?

We don't close threads normally, other for reasons of extreme misbehaviour on the part of one or more participants.

[kiwidude]

Calibre is a massive program, it is easy to understand feeling overwhelmed if you look too far under the hood too quickly

Going back to another of your original questions of the Riftwar saga example. You shouldn't need any custom columns or user categories to both store that information in Calibre and output it to disk. The title, author and series columns will store all the information you need for that.

What to do next depends on where you want to export the books to - are you wanting to send them to an ereader (if so then what device), or are you wanting to save them to a folder on disk? If you give some specifics then I am sure people can suggest any changes if necessary to your save/send to device templates and plugboards.

[Huisie]

@Manichean:

Thanks for the clarification.
When you're still wet behind the ears it's best to tread softly.
It's nice to know that here you don't have to fear the big stick...


@kiwidude:

Thanks for addressing the Riftwar example, too.
It's a great feature in Calibre that it takes your input, builds its own copy and creates its database from that.
You don't have to fear going horribly wrong and borking your originals - they are still there.

In my case, I am currently wrangling thousands of books into some order.
The condition of everything from file names to metadata to formatting runs the gamut from execrable to excellent.

Once I have everything in an order that works for me, I would like to be able to save to disk and replace the originals.
I can save that as backup and browse via Calibre or as files on PC.
Also, I want to be able to send-to-device in an order that makes sense to me.

Using Feist's Riftwar as example, I envision a save-to-disk order of:

• E:\Library\Fiction\F\Feist, Raymond E.\Riftwar Arc\<series_arc_index>- <series>\<series_index> - <title>\<book>

In other words:

• E:\Library\Fiction\F\Feist, Raymond E.\Riftwar Arc\01 - Riftwar Saga\01 - Magician\Magician.epub

I can figure out how to write the regex and expressions to import and export using the default metadata, such as title/author/series/series index, etc.

What I'm wondering is how to manage multi-series or arcs/sagas.
Amongst Calibre's glittering array of options, how can I choose to:

• display individual series within a larger arc
• display what order they are in or index them
• export that info to disk or device?

Some questions re questions:

• Is this where groups, sub-groups and hierarchies come into play?
• To save-to-disk/device such added info, do I need to use plugboards to work with it, rather than using the standard methods?

[kiwidude]

Thanks for the specifics of what you are trying to do.

My first question would be to double check to make sure you do indeed need to do some of this at all. You say you want to "save to disk to replace the originals" and you want the file structure to "browse as files on PC". Do you have a particular reason in mind why you want to browse via the file structure, rather than just using Calibre? You can do everything you can do in the file system and an awful lot more from Calibre. Why is it that you feel you want to browse books using the file system?

If it is just a "nice to have" rather than an actual tangible need, I would suggest living without it for a while and see how you get on. Maintaining both a file system based directory copy of your books and a calibre library will double your effort to keep things up to date. I would suggest most Calibre users do not bother with a file system based duplication of the files. There is one scenario where it is required that I am aware of, and that is if you have a device that cannot be sent to directly using Calibre, however I believe that is the exception rather than the norm.

The next point you raise is multiple series. There are threads on this forum about how people handle these. Here are a couple for you:
https://www.mobileread.com/forums/sho...d.php?t=121412
https://www.mobileread.com/forums/sho...d.php?t=119522

Have a read through those for some ideas of how to approach it. If you want to use a subcategory type approach in the tag browser, I guess you would have to use a series name something like:
Riftwar Arc.01 Riftwar Saga.Magician [1]

If you don't care about the tag browser nesting, then you could use multiple series columns for the book.

When it comes to "exporting", again it depends on exactly what the destination is and the type of device you have. For instance if you have a Kindle the file path is unimportant, particularly if you only manage your Kindle from Calibre (storing them as by title per author is sufficient), as the Kindle uses the internal book metadata not the physical filename for MOBI files.

On other devices the file path does play a role. You can use the Calibre template language to come up with a file path how you want. Exactly how to do that depends on the decisions you make on how to store the series in the first place.

I would caution you against trying to get too complex/clever as every extra "layer" of columns and series depth levels etc just adds to your maintenance. It also significantly increases the complexity of your save templates and plugboards if you are trying to incorporate all that additional information. You could always add it as you need it at a later date. Get your books in Calibre, with a nice clean set of basic title/author/series/cover/description metadata and a genre. Then look to enhance it when you have been using it for a while. Not only may you find that your requirements change, but remember that Calibre is continually changing as well - the heirarchical categories feature only appeared in the last week or two alone. By the time you find you *really* need a particular feature, there could well be an even better or simpler way to do it.

With that said, if you absolutely must have that file path today I am sure someone can help you out with it, once you have specified exactly how and where you are storing all those elements that make up the series information.

[jackie_w]

Originally Posted by kiwidude

...Get your books in Calibre, with a nice clean set of basic title/author/series/cover/description metadata and a genre. Then look to enhance it when you have been using it for a while...

Excellent advice for anyone new to calibre. I couldn't agree more.