Question on metadata plugboard

[haertig]

I like books to display on my Paperwhite library page as "Series - Index - Title"

e.g., "Sigma Force - 01 - Sandstorm" and "Sigma Force - 07 - The Devil Colony"

This is easily accomplished with the "Metadata Plugboard" set to something like this:
{series}{series_index:0>2s| - | - }{title}

But for some books, either the series name or the book title is too long, and the result of the above plugboard ends up being truncated by the Paperwhite firmware when it is displayed.

My first thought to fix this was to use the function shorten() in the plugboard to truncate the series name and add an ellipsis on the end. But the Calibre documentation days that you should not use a function in a plugboard template. Plus, sometimes it is the title of the book that is too long, not the series. Additionally, a blind truncation of a field with an added ellipsis is sometimes awkward looking. On top of all this, we would be looking at fairly complex plugboard expression to handle all this.

So my proposed solution is a custom column which contains the string to display on my Paperwhite. I'll call it Alt ID, with a lookup name of #altid. Now the plugboard only needs to look for the existence of data in the #altid column. If found, use that. If not found, use {series}{series_index:0>2s| - | - }{title}

And I would manually populate this custom column when I add a new book to Calibre. Most of the time I don't need to bother with this column. But if I notice an extremely long series name or book title, I can optionally populate the column.

e.g., Say I had Series = "Harry Potter" and Title = "Harry Potter and the Chamber of Secrets". I would manually populate the #altid column with "Harry Potter - 02 - Chamber of Secrets" (altering the book title to fit)

e.g., Say I had Series = "Nina Wilde and Eddie Chase" and Title = "The Hunt for Atlantis". I would manually populate #altid with "Nina Wilde - 01 - The Hunt for Atlantis". (Altering the series name to fit ... poor Eddie got left out!)

This solution seems workable given that MOST book titles/series names in my collection seem to fit on the Paperwhite display using the simple plugboard {series}{series_index:0>2s| - | - }{title} without having to worry about populating #altid.

I have to research how to specify the plugboard that I want. One that will select #altid if it exists, and if not, will default to use {series}{series_index:0>2s| - | - }{title}

I have not yet approached the specification of this plugboard. But it sounds like something I might be able to accomplish. I'll have to read up a little more on the plugboard language though.

Does anybody else have a different/simpler way to handle this problem? Or know off the top of their head the plugboard specification that I would need to use? I don't like the manual #altid approach really, nor do I like the programmatic approach because it's just too hard to find an appropriate abbreviation/truncation of a long test string via algorithms. A programmatic approach might end up with something strange like, e.g.,

"Nina Wilde and E... - 01 - The Hunt for Atla..."

So I'm thinking the manual specification approach may be the better of two evils, given that the manual population route would rarely be needed, for most books.

Thanks in advance for any feedback/advice/help!

[ownedbycats]

Originally Posted by haertig

Does anybody else have a different/simpler way to handle this problem? Or know off the top of their head the plugboard specification that I would need to use? I don't like the manual #altid approach really, nor do I like the programmatic approach because it's just too hard to find an appropriate abbreviation/truncation of a long test string via algorithms. A programmatic approach might end up with something strange like, e.g.,

"Nina Wilde and E... - 01 - The Hunt for Atla..."

So I'm thinking the manual specification approach may be the better of two evils, given that the manual population route would rarely be needed, for most books.

Thanks in advance for any feedback/advice/help!

If I'm understanding your concerns right:

For something in-between "putting a difficult-to-adjust template in a composite column" and "entering everything manually by hand" I like to use Action Chains' single-field edit to populate the column. (You can also do this with the template mode in search-and-replace.)

Then it can be adjusted as needed. For a few chains, I like to run an ask-at-runtime SFE right after the template'd one so I can check and adjust it right away as needed.

[chaley]

Originally Posted by haertig

My first thought to fix this was to use the function shorten() in the plugboard to truncate the series name and add an ellipsis on the end. But the Calibre documentation says that you should not use a function in a plugboard template.

Where does it say that, because it is wrong and needs changing. Plugboards templates aren't special. There might be some restriction on using GUI-only functions in a plugboard, but I don't think so.

One possibility that extends on @ownedbycats idea is to use a combination of a stored template and a custom text column. The stored template first checks if the column contains a value, returning it if so. If the column doesn't contain a value then the template computes the "default" result however you wish.

Using a stored template makes it easier to test and maintain. Using a column as an override means you change only the books for which the template produces an undesired result. Removing the value from the column for a book automatically reverts back to the algorithm.

EDIT: Rereading I see you anticipated my idea. Sorry about the noise.

[haertig]

Originally Posted by chaley

Where does it say that, because it is wrong and needs changing.

https://manual.calibre-ebook.com/template_lang.html

[chaley]

Originally Posted by haertig

https://manual.calibre-ebook.com/template_lang.html

That is talking about a restriction in template Single Function Mode. And it is correct. However, templates can be written in Template Program Mode or General Program Mode, neither of which have that restriction.

Plugboards can use any of the above three modes, and also Python Template mode, should that be of interest.

[ownedbycats]

I see how the confusion happened. Perhaps additional documentation for plugboards?

[chaley]

Originally Posted by ownedbycats

I see how the confusion happened. Perhaps additional documentation for plugboards?

You are welcome to propose updates.

[haertig]

Originally Posted by chaley

...templates can be written in Template Program Mode or General Program Mode, neither of which have that restriction.

Thanks. I just looked into that (cursory glance). I'm pretty sure I can do what I want using those more advanced programming modes. Just a bit more reading required on my part to learn the specifics. Time will tell if it is worth the effort trying to develop the template that I want. I can always brute force things when I download a book to my Kindle. Go into Calibre and manually change a books Title and Series to what I want displayed on the Kindle, use a standard template to download that book to my Kindle, then return the books Title and Series to their original values. For someone like me who only transfers one or two books to my Kindle per month, this kludge may be good enough, even though it is not elegant in the slightest. My previous programmer self (before retirement) is screaming "No, don't go that kludge route!", but my practical self is countering with "Yeah, but it only takes a few seconds to do a kludge download, and then you're done with it (until the next book)."

One thing I haven't seen mentioned in the documentation though, which would be very nice, is some way to test your templates. Something where you could manually feed the fields to your template and observe what the template will make of those. As best I have been able to determine, the only way to test your template is to make it live in Calibre, then do a real transfer to your Kindle, and then observe on the Kindle what the result was. Then change the template, and repeat the process until you achieve success. It would be handy if there were a little "test window" in Calibre so you didn't have to keep plugging in your Kindle, downloading a book, deleting that book if your template failed, then twiddle the template and download to the Kindle again. Maybe I missed something, but this somewhat cumbersome process seems to be the only way to do any testing.

[chaley]

Originally Posted by haertig

One thing I haven't seen mentioned in the documentation though, which would be very nice, is some way to test your templates. Something where you could manually feed the fields to your template and observe what the template will make of those. As best I have been able to determine, the only way to test your template is to make it live in Calibre, then do a real transfer to your Kindle, and then observe on the Kindle what the result was. Then change the template, and repeat the process until you achieve success. It would be handy if there were a little "test window" in Calibre so you didn't have to keep plugging in your Kindle, downloading a book, deleting that book if your template failed, then twiddle the template and download to the Kindle again. Maybe I missed something, but this somewhat cumbersome process seems to be the only way to do any testing.

Use the "Template Tester", which is a dialog that does what you want. The tester is mentioned in the documentation, but not prominently.

You select some number of books then launch the tester. It runs the template and shows you the resulting value for each book. You can set breakpoints & step through templates, use stored templates, and save/reload templates.

I use the tester so much that I added it to the context menu for a book, and also gave it a shortcut (Ctrl+Alt+t). You can do that, put it on the tool bar, put it on the menu bar, or whatever. @ownedbycats uses them enough to have requested the ability to have multiple tester dialogs open at the same time.

[haertig]

Originally Posted by chaley

Plugboards can use any of the above three modes, and also Python Template mode, should that be of interest.

My programming career (in order) went C, Unix sysadmin, C++, Perl, Linux sysadmin, networking, security. And a bit of Java, but not enough to really count. I now am 100% Linux, and still use Perl for any scripting needs I have these days. I never did anything serious in Python, although it has since overtaken Perl in popularity. While I'm still pretty much a Perl guru, I'm only a Python newbie. But since I only need ONE template, I could probably muddle my way through a Python implementation without too much fuss. I'll do some more reading on the various programming modes that Calibre offers.

[ownedbycats]

Originally Posted by chaley

You are welcome to propose updates.

Pull request made.

Originally Posted by haertig

I now am 100% Linux, and still use Perl for any scripting needs I have these days. I never did anything serious in Python, although it has since overtaken Perl in popularity. While I'm still pretty much a Perl guru, I'm only a Python newbie.

The joke I've heard is that 99% of paint splatters count as valid perl programs

[chaley]

Originally Posted by haertig

My programming career (in order) went C, Unix sysadmin, C++, Perl, Linux sysadmin, networking, security. And a bit of Java, but not enough to really count. I now am 100% Linux, and still use Perl for any scripting needs I have these days. I never did anything serious in Python, although it has since overtaken Perl in popularity. While I'm still pretty much a Perl guru, I'm only a Python newbie. But since I only need ONE template, I could probably muddle my way through a Python implementation without too much fuss. I'll do some more reading on the various programming modes that Calibre offers.

Oh do I understand. I followed the same chain except that Pascal and C were intertwined while I was at Berkeley. I started with Python when I started working on calibre in 2010. My use of Android's version of Java started in 2012 when we started Calibre Companion. At this point my brain is so fuzzy I don't think I could do it again.

Suggestion: even though the GPM template language is a mixture of multiple languages (I made it up from lisp, Algol 68, and C), you might be better off starting there. GPM templates don't mess with calibre internal data and thus can't break much. Python templates can more-or-less do anything and can break everything, and they require more knowledge of the calibre database API. On the other hand, some things are really easier to express in Python, especially list (array) operations.

Feel free to ask questions in this thread or by PM. I am happy to support people who want to know how this stuff really works.

[haertig]

Originally Posted by ownedbycats

The joke I've heard is that 99% of paint splatters count as valid perl programs

True. The Perl motto "There's more than one way to do it" is the understatement of the year. You can write a one-liner that does some amazing stuff. But nobody else will understand it. Nor will you, 5 minutes after you wrote it.

Of course, after all my time using Perl, moving to Python would be like moving from running naked your entire life to being forced to wear a 3x undersized Speedo swimsuit. A bit ... confining.

[chaley]

Originally Posted by haertig

Of course, after all my time using Perl, moving to Python would be like moving from running naked your entire life to being forced to wear a 3x undersized Speedo swimsuit. A bit ... confining.

I wonder if you will still think this after exploring Python's nested generators, "partial" function variables, and "monkey patching". Kovid uses these a lot in calibre. I confess to often needing a lot of time to understand what the code actually does. I also confess to sometimes failing to reach understanding.

[ownedbycats]

Originally Posted by haertig

But nobody else will understand it. Nor will you, 5 minutes after you wrote it.

I've taken to heavily commenting my more complex templates after a few instances of having to completely rewrite them just to make a small adjustment because I wasn't sure what the code was doing.

(To be fair, GPM is very easy to understand so this is entirely a user problem.)