Best Practices for Management of Virtual Libraries and Tags

[Sunshine82]

Hi, everyone.

I am trying to better understand the Calibre design behind the virtual libraries and tags.

1) Is it possible to nest virtual libraries within each other, creating a hierarchy of tiered virtual libraries within a single, shared virtual library? For example, if I begin with one main root virtual library entitled Non-Fiction, could I then create sub-libraries within it: 1) History; 2) Science; 3) Philosophy; 4) etc?

2) Am I correct in my understanding that tags cannot be nested within each other? Do tags all exist on the same hierarchical level?

3) Are tags considered metadata? Is there a find-and-replace command within Calibre that might help me better manage the metadata -- including automated tags that I would like to reword (often resulting from Calibre's metadata searches)?

4) Does anyone have any overall comments or suggestions for best practices in setting up detailed virtual libraries and customized tags? I am especially interested in defining tags for genres in a way that can be organized into tiers, if that is possible.

Thanks for your help!

[theducks]

Tags ARE metadata
In the Tag browser: Right click on a tag and you can rename it (or remove All uses)
OR
Right click on a tag: Manage (tags) to work wit the entire list. All usage will be maintained at the end under the new rules.


A VL is a view (of the main) of whatever the conditions you set up.
Note: Searching respects the view so if your VL is 'To Read' and you look for the tag non-fiction. those are just what you will see.
VL's do not actual change the overall Library, although any change you make there is really on the main library.

[isarl]

1) Virtual libraries are built on search strings. If you create additional virtual libraries based on search strings which are strictly more restrictive than the original one, then you could think of them hierarchically, sure. But each is simply another Virtual Library, there is no hierarchy structure Calibre enforces upon them.

2) You are incorrect; you can have hierarchical tags. Preferences → Look & feel → Tag browser → “Hierarchy and searching” tab → “Select categories with hierarchical items:” → Ensure “Tags” is checked (or whichever column you wish to use – below, I suggest that you may wish to use a custom column, instead). See also the tutorial in the Calibre User Manual on managing hierarchical categories in the Tag Browser.

3) Again, see the Calibre User Manual (hotlinked directly to section on search/replace while editing metadata for multiple books at a time). However, see also Preferences → Metadata download (in the Sharing group) →“Create rules to filter/transform tags”. Many rule types are possible including Remove, Replace, Keep, Split, and others. If you find yourself making the same replacements many times on your library's existing metadata, then adding rules here can help prevent the need for making those same changes again in future when downloading metadata for new books.

4) You may wish to create a custom column for the genre, rather than storing it with the rest of your tags. As mentioned above, you should have no trouble making this custom column hierarchical. Organizing genres into VLs should be very simple, though do make sure to read – you guessed it: – the Calibre User Manual which will explain to you the particular syntax to use when searching values in a hierarchical field.

[ownedbycats]

For 1), you may also want to check Additional Restrictions in the VL menu. It lets you put another saved search on top of your VL.

[Sunshine82]

Thank you for the thorough feedback, everyone. Much appreciated!

After some time spent with the manual (thanks for those links, isarl), I am beginning to better understand Calibre's capabilities. But some questions remain.

1) Where do I find the Additional Restrictions in the Virtual Library? Is this setting accessed somewhere in Preferences?

2) Thanks. This issue is now resolved.

3) Is there a way to preemptively apply a search and replace command (or some equivalent command) unilaterally to a column's metadata? For example, if I wish to default the metadata in my Tags column to titlecase, can a command be set up to either modify ALL the existing metadata within a certain field (i.e. a Tags column) or to at least program the modification of ALL future imported metadata? If so, what might this particular command look? Is it possible to use a simple text command or would a regular expression be ideal? (The coding involved is still a little confusing...)

4) I have created a customized Genre column in addition to the default Tags column. Is there a way to correlate the metadata in the Tags column with the Genre column? For example, can the metadata in one field (Tags) be referenced in another field (Genre)? [If this is confusing, I have attached a screenshot to provide context.] I'm wondering if there is a more efficient process for creating the metadata infrastructure for my customized Genre column that can take its data from an existing source.

More to the point, is there a way to setup the metadata hierarchy within my customized Genre column independently of assigning the related metadata to existing books in my library?

[ownedbycats]

Additional Restrictions is here

[BetterRed]

Originally Posted by Sunshine82

. . .

1) Where do I find the Additional Restrictions in the Virtual Library? Is this setting accessed somewhere in Preferences?

Originally Posted by Sunshine82

. . .
3) Is there a way to preemptively apply a search and replace command (or some equivalent command) unilaterally to a column's metadata? For example, if I wish to default the metadata in my Tags column to titlecase, can a command be set up to either modify ALL the existing metadata within a certain field (i.e. a Tags column) or to at least program the modification of ALL future imported metadata? If so, what might this particular command look? Is it possible to use a simple text command or would a regular expression be ideal? (The coding involved is still a little confusing...)

Does this help:

Originally Posted by Sunshine82

. . .
4) I have created a customized Genre column in addition to the default Tags column. Is there a way to correlate the metadata in the Tags column with the Genre column? For example, can the metadata in one field (Tags) be referenced in another field (Genre)? [If this is confusing, I have attached a screenshot to provide context.] I'm wondering if there is a more efficient process for creating the metadata infrastructure for my customized Genre column that can take its data from an existing source.

More to the point, is there a way to setup the metadata hierarchy within my customized Genre column independently of assigning the related metadata to existing books in my library?

Probably need to use a virtual column: i.e. one with Column type: Column built from other columns, behaves like tags and a suitable template.

Have a play around with something simple ==>> The calibre template language.

BR

[Sunshine82]

Thank you very much for the help, BetterRed and ownedbycats.

I have resolved the issue with Additional Restrictions and Search and Replace commands. Making progress.

However, the Calibre template language is a bit daunting. I have done some coding before, but it's not my specialty. So, I'm not quite sure what I'm looking for. From what I can tell, I almost need an inverse of the subitems and sublist functions -- but I'm still not sure.

It might be easier to explain what I am looking to accomplish in order to determine whether it is possible or if there is another way of accomplishing my goal. I am wondering if it's possible to pre-program the customized column I created entitled Genre with a comprehensive list of possible entries in advance; if so, this would allow me to create the hierarchy tree ahead of time before then assigning the designated genre path to particular books. [I have attached a screenshot to provide context.]

The issue that I foresee is that if I build one column from another (i.e. basing my customized Genre column off of the Tags column), I can mine the content/vocab from Tags but wouldn't be able to specify the context/logic for the Genre column. In other words, is it possible to replicate the arbitrary user-defined hierarchy of a customized column (like what I created for Genre) in a way that is systematic and applies globally to an entire library independently of particular pre-existing book entries? Does this make sense? I am wondering if there is a more efficient way of creating and assigning genre hierarchies to a bulk range of books.

[ownedbycats]

If you want to see an example of the Calibre template language in action, here's one I use:

program:

## Splitting tags
	if 
		'^(Fiction|Nonfiction|Magazines & Periodicals)' in $#booktype 
	then 
		split_tags = re($tags, '\.', ',') 
	else
		split_tags = ''
	fi;

## Removing a few unwanteds and sorting
	cleaned_tags = list_sort(list_difference(
		split_tags,
		'[Cleanup], Fiction, Nonfiction, Magazines & Periodicals, Cultures & Regions, Social Issues',
		','), 0, ',');

## Adds 'Omnibus' if present in #admintags
	if 
		'omnibus' 
	in 
		$#admintags 
	then 
		cleaned_tags  = 'Omnibus' & if cleaned_tags then ', ' & cleaned_tags fi
	fi;

	cleaned_tags

This does several things:

1. If the custom column #booktype is set to Fiction, Nonfiction, or Magazines & Periodicals, it 'splits' the tag at the periods. Fiction.Science Fiction becomes Fiction, Science Fiction. If a book doesn't have one of the defined entries in #booktype, it returns an empty value (to prevent template errors).

2. It then removes a number of selected entries from the tags and sorts the remaining results.

3. Lastly, if 'omnibus' is present in custom column #admintags, it adds Omnibus to the beginning of the string.

Originally Posted by Sunshine82

It might be easier to explain what I am looking to accomplish in order to determine whether it is possible or if there is another way of accomplishing my goal. I am wondering if it's possible to pre-program the customized column I created entitled Genre with a comprehensive list of possible entries in advance; if so, this would allow me to create the hierarchy tree ahead of time before then assigning the designated genre path to particular books. [I have attached a screenshot to provide context.]

Take a look at the Tag Mapper.

[isarl]

Originally Posted by Sunshine82

The issue that I foresee is that if I build one column from another (i.e. basing my customized Genre column off of the Tags column), I can mine the content/vocab from Tags but wouldn't be able to specify the context/logic for the Genre column. In other words, is it possible to replicate the arbitrary user-defined hierarchy of a customized column (like what I created for Genre) in a way that is systematic and applies globally to an entire library independently of particular pre-existing book entries? Does this make sense? I am wondering if there is a more efficient way of creating and assigning genre hierarchies to a bulk range of books.

I'm not really clear on what exactly you are hoping to accomplish here. Perhaps some concrete examples would help to clarify. I see a long and detailed hierarchy in your attached screenshot but what might help more is a few books, the values you want to use for your Tags field for those books, and what you want your Genre field to look like for them (and how much of that you want to be automatic). If you've already tried something and you're getting errors, it would be helpful to see what you've tried and what the errors say, too.

Despite the confusion I expressed above, it also seems like you want to pre-define values for your Genre hierarchy without first having to apply these values to individual books. The Entities Manager plugin might be what you are seeking, which allows you to manage a value for a particular column independently of any book records, although I don't use it myself so can't speak from firsthand experience.

[BetterRed]

Originally Posted by isarl

. . .

it also seems like you want to pre-define values for your Genre hierarchy without first having to apply these values to individual books.

In other words a "Controlled Vocabulary (CV)" See Getty and Wikipedia

Calibre has no direct way of implementing CVs, if the column in question has a limited set of values you could create an empty book as a placeholder, and add all the possible values to it.

If you edit the column in question (e.g. Genre) from the book list then you can press Shift+F2 on a book's Genre cell and the Manage items: Genre dialogue will pop, this allows values to be assigned from the current list, and the addition of new values.

Originally Posted by isarl

The Entities Manager plugin might be what you are seeking, which allows you to manage a value for a particular column independently of any book records, although I don't use it myself so can't speak from firsthand experience.

IIRC Entities Manager sprang from my wish to have a place to store extra data about entities such as Publisher, Series, Tags etc. DaltonST went to considerable effort to satisfy my wishes as can be seen in the EM plugin thread. But the complexity of me using it outweighed the advantages it offered to me… so I abandoned my use of the plugin.

BR

[isarl]

Originally Posted by BetterRed

Calibre has no direct way of implementing CVs, if the column in question has a limited set of values you could create an empty book as a placeholder, and add all the possible values to it.

What about the custom column type for containing a limited set of values? Such a column could still be configured to be treated as hierarchical, couldn't it?

[ownedbycats]

Originally Posted by isarl

What about the custom column type for containing a limited set of values? Such a column could still be configured to be treated as hierarchical, couldn't it?

Yes, it can. Here's an entry called "Test.enum" (for enumerated) after the column was set to hierarchical in preferences > look & feel > tag browser.

[isarl]

Originally Posted by ownedbycats

Yes, it can. Here's an entry called "Test.enum" (for enumerated) after the column was set to hierarchical in preferences > look & feel > tag browser.

Attachment 199848

So then this might be an option for delineating a hierarchical controlled vocabulary. The only caveat is that values cannot contain a comma. (If this is important to you then you could use some other character in its place which looks similar but will not be treated by Calibre as a comma. For example (but you could use other choices), a turned comma: ⸲)

I've just tested and confirmed that columns with a fixed set of permitted values can have that set of permitted values changed after creation. (This would seem to be important for something as broad as a Genre category, which I assume would occasionally throw up new genres that were not initially considered.)