Path separator not working in templates?

[DavidTC]

So, here's a bug that I figured out how to get around while typing it up, (I was doing things the hard way, and now just use the Calibre path.), but it still seems like a bug and I feel I should report.

So, I have a field called #savepath that contains '{#genre}/{title} - {id}'.

If I put {#savepath} as the save template in Calibre, a book, for example, gets saved correctly to 'Science Fiction/Douglas Adams' Starship Titanic - 3209.epub' pretty much everywhere, including CC if it's using the Calibre file path. It works fine.

However, putting {#savepath} as the *CC* template, and using that, results in a file in the top CC directory named 'Science Fiction_Douglas Adams' Starship Titanic - 3209.pub'.

It doesn't notice I'm trying to make a subdirectory. And then, I presume, is forced to change the / it cannot put in a filename to a _.

This behavior, I guess, would be reasonable if CC just didn't want to put books in subdirectories, but it will happily put them there if *Calibre* tells it to. So I suspect it's a bug.

The 'Remove accents' option is off, just in case anyone is wondering if that's relevant.

I only had time to test this using the wireless device connection.

[DoctorOhh]

Originally Posted by DavidTC

So, here's a bug that I figured out how to get around while typing it up, (I was doing things the hard way, and now just use the Calibre path.), but it still seems like a bug and I feel I should report.

So, I have a field called #savepath that contains '{#genre}/{title} - {id}'.

I may be reading this CC FAQ entry wrong, but I don't believe this is a bug. CC's template does not have the full feature set of calibre and I don't think what you're attempting is supported in the manner your trying to accomplish it.

[chaley]

Actually, the fact that it works in a calibre template is probably a bug. Calibre removes slashes in template items, with the exception of a leading slash in a tag. The probable bug is that composite columns (columns built from other columns) don't have that processing (slash removal) applied to them. There isn't a good reason why composites should be treated differently from custom text or standard columns. However, I am not going to fix that probable bug because it has been there too long and people (like you) might depend upon it.

[CC stuff snipped out]

[DavidTC]

Calibre removes slashes in template items, with the exception of a leading slash in a tag.

I've been testing this since you posted, and I am in shock.

The behavior I was expecting, in fact, the behavior that I bet everyone was suspecting, that when you fill in a template, that template should come up with the same result everywhere. And then, *at the end*, if it's saving a file, it parse the directories and, at the end, right before it saves, it replaces invalid characters.

And I think most of us assume that if the result of the template evaluation is A/B, it would treat that the same if it got that from {#ab} or {#a}/{#b} or some program.

But apparently getting a metadata field, either with {} or field(), has some magical string processing during it that behaves differently if you're calculating a save template, and only a save template.

Not only is this an extremely inconsistent thing, it sorta breaks the entire concept of the language, if completely identical pieces of code can give different results depending on where they are.

It also appears to not be documented anywhere. The documentation says, literally, 'field(name) – returns the metadata field named by name.', This documentation is wrong.

Instead, what field() does is 'returns the metadata field named be name, unless it's called inside a save template, at which point it munges the metadata field to be more like a file name before returning it.' And that's just field(), there's probably others non-documented...how does lookup() behave? Does this affect first_non_empty()? author_links()?

Also, does raw_field() get the value *without* doing that? I probably should check, but I don't have the time, and having to constantly restart Calibre to change the save template is annoying. (Which is why I put that in a field in the first place.) Is there even a *way* to get the original field?

Seriously, the solution to 'newbies might have a / in their tags and accidentally make directories' was to change how some part of the language worked in a certain place. (And then not mention it anywhere.) Instead of just adding a strip_nonfile_chars() function and mentioning people should use {#fieldname:strip_nonfile_chars()} in the intro template documentation if that happens?

And at this point...yes, I know some people probably have existing stuff that assumes it, and oh noes, it can't ever be changed....but wow, this is such a huge and weird behavioral inconsistency that I argue it really should be dealt with. Give people a checkbox on the save template screen if they want the old way.

But that's not really you, I need to talk to kovidgoyal about that, I guess?

[chaley]

Moderator Notice
This is no longer a thread about CC. I am copying it to the calibre forum where I will respond.

[chaley]

@DavidTC: You are correct when you say that Kovid is the person who will make any decisions here. That said ...

I cannot imagine that he would agree to change the semantics of save/send templates in the way you describe (don't remove slashes). Doing so would probably break sending some books for every calibre user that uses save to disk/send to device, and that is millions of people. Consider what would happen if an author or a title contains a slash: seemingly random new directories would be created. Note that save/send templates existed long before I implemented the template language some 5 years ago. The decision was made way back then not to break existing templates.

I don't see any issues around template functions. They operate on their input, whatever that is. The semantics of the functions don't change.

The function "field" returns the "formatted" value (a "processed" value) in all contexts. When in the save/send context, that value will have slashes removed (except for composites), "title" becomes "title_sort" unless a tweak is set, custom number columns are unformatted, etc. The function "raw_field" returns the underlying value without any processing. For example, if the title of a book is "The Horror" then in save/send context {title} and a use of "field" will return the title sort "Horror, The" while "raw_field" will return "The Horror". In the GUI context, "{title}" returns the title, not the title_sort.

In your case, use this Template Program Mode template item

{:'raw_field('#savepath')'}

to get the value without the "remove slash" processing.

[kovidgoyal]

Yeah, breaking backwards compat is something I do only with overwhelming reason. This does not qualify.

[DavidTC]

Well, it's not worth arguing over, there are plenty of easy workarounds.

But this should be documented. There is currently no indication anywhere that:

a) fields in save/send templates operate differently from fields in composite columns templates and plugboard templates.

b) field() and {#fieldname} do any sort of 'formatting' besides turning timestamps into dates. Actually, even that's not really documented either, just implied with the format_date() stuff.

Incidentally, (a) is really weird, because it now appears there's no way to see in advance the results of the save/send templates. I had *thought* the way to do that was to cheat and write it in a composite column, and then move the template over, but apparently, nope, that's not always what you get.

[fidvo]

I do something similar, and I highly recommend using my method.

Instead of using a slash in the field value, use a period. Then modify your save template so it changes periods to slashes. You could even go one step further and set up that field as a hierarchy, and use the tag browser to literally browse your books as if they're in your genre folders.

The only drawback is if you have periods in your book titles.