Sorting hierarchical tags

[kjdavies]

I love that the tag browser supports hierarchical tags. Tags make it much easier for me to organize my books, and hierarchical tags make it much easier for me to organize my tags.

I notice that the tags/categories appear to be sorted before adding to the tag browser. On the face of it I think this is a sound design decision... but I see a possible improvement.

Hierarchical tags use the period ('.', 0x2e) as the level delimiter. This character is almost immediately before the digit characters (0 is 0x30) and after many other characters such as space (' ', 0x20), hyphen ('-', 0x2d), and comma (',', 0x2c). This means when my library has 'RPG.D&D', 'RPG.Pathfinder', 'RPG unsorted', and 'RPG-new', they show up in the following order:

• RPG unsorted
• RPG-new
• RPG
  • D&D
  • Pathfinder

I suggest, before sorting the strings, replacing '.' (0x2e) with tab (0x09 -- which does not appear in tag text as best I can see). Then split on tab instead of period to populate the tag browser (applying the original strings -- periods rather than tabs -- so the existing tag filtering works). This would cause the tag browser to look like:

• RPG
  • D&D
  • Pathfinder
• RPG unsorted
• RPG-new

I think this is a more obvious and user-friendly layout, in this scenario. In my example I obviously could change the tag strings (replace 'RPG unsorted' and 'RPG-new' with other values), but that approach isn't always available.

[theducks]

Uh! Hierarchical tags sort just fine. The delimiter is a period
AND
You need to declare what fields are hierarchical in preferences:Look&Feel:Tag Browser:Hierarchy... (a tab): tick columns (that are valid for this mode)

[kjdavies]

Originally Posted by theducks

Uh! Hierarchical tags sort just fine. The delimiter is a period
AND
You need to declare what fields are hierarchical in preferences:Look&Feel:Tag Browser:Hierarchy... (a tab): tick columns (that are valid for this mode)

Yes, I understand how to get the hierarchies to appear. And the entries do sort in a consistent manner. I have observed that the consistent manner is not necessarily what people might expect, in that if you have 'RPG.Something' and 'RPG Group.Something Else', you might expect at the top level to see 'RPG' before 'RPG Group'... which is not what happens.

[chaley]

Originally Posted by kjdavies

I love that the tag browser supports hierarchical tags. Tags make it much easier for me to organize my books, and hierarchical tags make it much easier for me to organize my tags.

I notice that the tags/categories appear to be sorted before adding to the tag browser. On the face of it I think this is a sound design decision... but I see a possible improvement.

Hierarchical tags use the period ('.', 0x2e) as the level delimiter. This character is almost immediately before the digit characters (0 is 0x30) and after many other characters such as space (' ', 0x20), hyphen ('-', 0x2d), and comma (',', 0x2c). This means when my library has 'RPG.D&D', 'RPG.Pathfinder', 'RPG unsorted', and 'RPG-new', they show up in the following order:

• RPG unsorted
• RPG-new
• RPG
  • D&D
  • Pathfinder

I suggest, before sorting the strings, replacing '.' (0x2e) with tab (0x09 -- which does not appear in tag text as best I can see). Then split on tab instead of period to populate the tag browser (applying the original strings -- periods rather than tabs -- so the existing tag filtering works). This would cause the tag browser to look like:

• RPG
  • D&D
  • Pathfinder
• RPG unsorted
• RPG-new

I think this is a more obvious and user-friendly layout, in this scenario. In my example I obviously could change the tag strings (replace 'RPG unsorted' and 'RPG-new' with other values), but that approach isn't always available.

This isn't an unreasonable idea, but I need to reflect a bit on whether it is worth the inevitable bug reports about the change in behavior.

[Terisa de morgan]

Originally Posted by chaley

This isn't an unreasonable idea, but I need to reflect a bit on whether it is worth the inevitable bug reports about the change in behavior.

[theducks]

Originally Posted by Terisa de morgan

And I will raise you

[kjdavies]

Originally Posted by chaley

This isn't an unreasonable idea, but I need to reflect a bit on whether it is worth the inevitable bug reports about the change in behavior.

Indeed, which is why I posed it as a suggestion rather than a request

[chaley]

The change is in calibre source.

[MickiTee]

Originally Posted by chaley

This isn't an unreasonable idea, but I need to reflect a bit on whether it is worth the inevitable bug reports about the change in behavior.

An alternative way would be to remove all the spaces in the tags before sorting. It also means that users who have templates etc which parse the tag using a period wouldn't have to update their code.

If the field is a custom column a "tag sort" field could be added to the table which could act in the same way as the 'author sort' and 'title sort' fields. In addition it should be possible to add a tweak which would allow users to decide how they want their tags to sort - existing or new.

Hopefully this should avoid all those bug reports for something that isn't a bug.

[chaley]

Originally Posted by MickiTee

An alternative way would be to remove all the spaces in the tags before sorting. It also means that users who have templates etc which parse the tag using a period wouldn't have to update their code.

This doesn't work. There are characters such as plus and parentheses that sort before period. The suggested change ensures that periods become something that sorts before all other printable characters.

If the field is a custom column a "tag sort" field could be added to the table which could act in the same way as the 'author sort' and 'title sort' fields. In addition it should be possible to add a tweak which would allow users to decide how they want their tags to sort - existing or new.

Hopefully this should avoid all those bug reports for something that isn't a bug.

Adding a "sort" would indeed permit people to do what they wanted with order but I think it is impractical. Tags and tags-like items come and go frequently, so maintaining the sort would be time consuming. It is also a database-incompatible change, which would break any plugin or app that thinks it knows the database schema.

The change I made is similar to the one suggested -- ensure that X.Y sorts below Xanything.Y. There was no need to change the value of the tag other than when computing the "sort key", and this only when the column is hierarchical. Templates etc don't need to be updated unless they depend on order, which isn't easy to do.

[kjdavies]

Originally Posted by chaley

This doesn't work. There are characters such as plus and parentheses that sort before period. The suggested change ensures that periods become something that sorts before all other printable characters.
Adding a "sort" would indeed permit people to do what they wanted with order but I think it is impractical. Tags and tags-like items come and go frequently, so maintaining the sort would be time consuming. It is also a database-incompatible change, which would break any plugin or app that thinks it knows the database schema.

The change I made is similar to the one suggested -- ensure that X.Y sorts below Xanything.Y. There was no need to change the value of the tag other than when computing the "sort key", and this only when the column is hierarchical. Templates etc don't need to be updated unless they depend on order, which isn't easy to do.

Right, that's why I suggested the change as I did, to be as minimally invasive as possible. It's not a zero-cost change, but I think it fairly small and limited scope. Ideally it happens in basically one place, whereas if I understand the other proposal it would have broader reach.

The change I really want is to be able to specify my own tag and hierarchy delimiters (comma and period both happen in real strings!) but that's a much scarier change.

(Another one I'd like, also intended for use with tags and hierarchies, is the option in the bulk regex editor to say "if the tag doesn't match the RE, remove it"... AKA "if you find a tag like this, change it in this way, else remove it". It would be very useful when the tags contain metadata I want in another field, but it might be entirely a 'me' problem.)

[chaley]

Originally Posted by kjdavies

The change I really want is to be able to specify my own tag and hierarchy delimiters (comma and period both happen in real strings!) but that's a much scarier change.

Yes it is and that's why it isn't going to happen, at least by me.

(Another one I'd like, also intended for use with tags and hierarchies, is the option in the bulk regex editor to say "if the tag doesn't match the RE, remove it"... AKA "if you find a tag like this, change it in this way, else remove it". It would be very useful when the tags contain metadata I want in another field, but it might be entirely a 'me' problem.)

Building this in is more than I want to do. It would be difficult to get "right" and cover all the corner cases.

You can do it yourself today with templates, either in bulk edit or action chains single field edit. In bulk edit you would use "template" as the source. In action chains you would use a template to compute the resulting value.
The template would fetch all the tags for a book, filter and modifiy them as desired using regexp or whatever, then return that result. Bulk edit would have the search regexp '^(.*)$', replace regexp '\1', and the destination field of tags. Action chains would be operating on tags.

A third way is to use action chains "python code" and write what you want directly against calibre's db. In this case you would fetch the tags from the db, filter and modify them as needed, then store them back into the db. This process processes tag-by-tag not book-by-book, which may or may not be what you want to do.

[kjdavies]

Originally Posted by chaley

Yes it is and that's why it isn't going to happen, at least by me.
Building this in is more than I want to do. It would be difficult to get "right" and cover all the corner cases.

You can do it yourself today with templates, either in bulk edit or action chains single field edit. In bulk edit you would use "template" as the source. In action chains you would use a template to compute the resulting value.
The template would fetch all the tags for a book, filter and modifiy them as desired using regexp or whatever, then return that result. Bulk edit would have the search regexp '^(.*)$', replace regexp '\1', and the destination field of tags. Action chains would be operating on tags.

A third way is to use action chains "python code" and write what you want directly against calibre's db. In this case you would fetch the tags from the db, filter and modify them as needed, then store them back into the db. This process processes tag-by-tag not book-by-book, which may or may not be what you want to do.

Right, I do have ways to do this latter, I just need to think through my order of operations.

I've added a custom column to most of my libraries, #scratch (long text, not in tag browser) specifically so I've got a place to mess it up a few times while I figure it out... I haven't added #scratchtag yet, but I'm thinking about it.

And I agree, getting this one right could be a bit fiddly. It's why I didn't submit a ticket for it.

What I really need to do is buckle down and get better at Python. It's on my List (of things to do), but it's not close enough to the top yet.

[kjdavies]

Originally Posted by chaley

The change is in calibre source.

I did not realize how good this would feel when I saw it.

THANK YOU @chaley!

My tags are still a mess, I'm working on that, but now they are a better-organized mess!