custom column: count number of characters (in book path)

[unboggling]

Here's my first shot at a template for the full path of a book in a calibre library.

Edit:

Oops. Deleted. Discovered a problem in the template, back to the drawing board.

[chaley]

The question "how do I find the exact path to a book" has come up several times in the last month. I am wondering whether I should add a template function that returns it.

To be precise, I could add two functions, one that returns the path to the current library and one that returns the path to a given format. I don't see the use in a function that returns the path to the folder containing the formats, but I could be convinced otherwise.

It is worth noting that calibre creates paths in the library that save-to-disk cannot replicate. In particular, paths in the library can contain multiple consecutive spaces. Save to disk (and send to device) will reduce these to one space. Thus even if I add the function it won't be possible to recreate an identical folder hierarchy even with these functions. Also note that it is likely that these functions won't work in save templates. Reason: the template processor will need access to the database to evaluate these functions, and the DB is not available to the send/save jobs.

[unboggling]

Originally Posted by chaley

...To be precise, I could add two functions, one that returns the path to the current library and one that returns the path to a given format. I don't see the use in a function that returns the path to the folder containing the formats, but I could be convinced otherwise.

It is worth noting that calibre creates paths in the library that save-to-disk cannot replicate. In particular, paths in the library can contain multiple consecutive spaces. Save to disk (and send to device) will reduce these to one space. Thus even if I add the function it won't be possible to recreate an identical folder hierarchy even with these functions. Also note that it is likely that these functions won't work in save templates. Reason: the template processor will need access to the database to evaluate these functions, and the DB is not available to the send/save jobs.

Personally, I'm not concerned about OS/file-system limitations in my own use of calibre because I've never encountered any limits that annoyed me. But there is a large base of Windows users, some of whom seem concerned about length of entire path or filename alone. So I was responding to that, while using it as an excuse to learn a little more about template programming.

In the past few months (my memory may be faulty) I recall only 1 or 2 users wanting to exactly replicate the calibre library structure with Save to Disk, for some purpose of melding with a file-syncing application. Personally, I don't have a need for that.

But I recall more than a few users wanting to track string length of filename alone or entire path, particularly since someone mentioned the issue last month in "How I" thread, which brought me out of my self-inflicted OS X daze to look around. For them one or two special functions (that don't necessarily work in Save/Send/Plugboard templates) might be simpler to use. For their purposes using it just for a composite column would work fine. A special function such as "path()" returning entire path string from start to finish that automatically includes drive/volume designators and perhaps even an approximation of file extension (at book level of abstraction) might get used a lot more than a complex-seeming template program such as my construction above. A function including the exact format extension might be even better, though it seems difficult to implement due to the conflicting levels of abstraction between "book" and "format". Special functions such as strlen_path and strlen_just_the_filename might be even more frequently used, for composite column, if using them were as simple as clicking a quick-create button in "Add your own columns".

But I'm hazy about the various file-system path conventions and differences. I'm hazy about whether more users are concerned about limitations for filename component alone or for entire path. I'm hazy about calibre internals that truncate according to whatever criteria per whatever set of varying systems. And implementing such functions is probably a complicated endeavor. I don't know if your effort would be worth it. Maybe some of those users will chime in.

Edit: PS. I really like the template language. Thank you for creating it.

[theducks]

My 2p (thinking on what unBoggling said)

Maybe we just need is a (un-trimmed names) 'path length' warning indicator that appears (in the Metadata Edit Window) when the combination of Authors, Title and Library path start pushing the Limit.

[chaley]

Originally Posted by theducks

My 2p (thinking on what unBoggling said)

Maybe we just need is a (un-trimmed names) 'path length' warning indicator that appears (in the Metadata Edit Window) when the combination of Authors, Title and Library path start pushing the Limit.

I am not sure what purpose the warning would serve. Calibre deals with the limit by shortening the component parts of the name to make things fit. The warning would tell the user that the names of files and folders in the library might be truncated, but why should anyone care? It is hard to imagine that someone would shorten an author's name so that it fits within the limit.

[Man Eating Duck]

Originally Posted by chaley

I am not sure what purpose the warning would serve. Calibre deals with the limit by shortening the component parts of the name to make things fit. The warning would tell the user that the names of files and folders in the library might be truncated, but why should anyone care? It is hard to imagine that someone would shorten an author's name so that it fits within the limit.

Obviously. It would also be very confusing since users are (rightly) told they have no business nosing about in the folder structure

I must admit that I don't understand why people worry so much about the path length. Lots of entries in my library are cut off, but since calibre handles everything about it, why would I care?

However, if you are bored one day and decide to implement this function I guess that quite a few users would find a use for it.

[miquele]

Somehow calibre2odps is truncating the paths, at least to my understanding, thus rendering the download-links unusable. So the problem for me didn't sit with calibre, but since I cannot change the workings of calibre2opds either, I'm prepared to optimise the title a little bit if necessary. Otherwise I wouldn't care about it.

[unboggling]

Here was my second shot at a template for the full path of books in a calibre library.

Deleted.

Some of these templates became outdated due to new template functions recently added in calibre, which allow better path templates. For updated templates and discussion, please see Post #24 (the post following this post)

[unboggling]

Here's my third shot at a template for the full path of books in a calibre library, updated with some new template functions. Thank you, chaley and kovid. The full path templates using formats_paths() are easier to construct, with better performance, and (I assume) accuracy. The descriptions for the templates using new functions are dark blue.

Example Templates, Composite Columns, Filename or Path of Books in calibre Library. I am not concerned about path length or filename length in a library because calibre automatically stays within filesystem limitations by truncating filename or other path components when necessary, and the calibre library folder and filename structure are off-limits anyway. But some users are concerned enough to want short titles, filenames, or paths.

• Generalized Path. This first example uses no template language. It shows the exact generalized full path for any book in my primary calibre library named Prime, which is in a folder Libs in a folder Books in my home folder username on my system startup internal drive, which is formatted for HFS Plus (aka Mac OS Extended) Journaled and lives in OS X 10.8.x. My real username contains 7 characters so in the examples I shortened unboggling to 7 characters. The book id number in parentheses is automatically assigned when a book is added to a calibre library (and unique within that library). Author1 is the first author in the multi-value field Author(s).

  /Users/unboggl/Books/Libs/Prime/author1/title (id)/title - author1.extension

• Shortening the Path. The folder names leading to my library Prime are purposely short to maximize remaining string length within any path limitations, but the path length could be shortened more by moving the calibre library to the first level of my user account and eliminating the folders Books and Libs, or by moving the library to root level of a different drive volume with a short name, as in /Volumes/Bk/
• Differences for Other Operating Systems and File Systems. Windows uses a volume designator such as "C:\" and a backslash between path components, while on current versions of OS X, Linux, and other variants of Unix the path component separator is usually forwardslash (depending on specific file system).
• Constant, File Extension. For templates dealing with just the filename component of the path, the file extension at the end of the filename can be approximated by using a string constant, which can be customized per user. For precision it should include the preceding dot plus all the characters of the extension. But string length of potential extensions varies. Many are 4 characters such as epub, mobi, azw3, while some are 3 characters such as rtf, txt, lit. Some are 5 or more. Most of my books have only one format, EPUB, though there are a few exceptions that also have a PDF or MOBI. To cover the file extension for most of my cases, for filename templates I just use .epub as the constant, and do not worry about possible small differences.

  .epub

• First Author. In the author part of the filename for a book in a calibre library, calibre uses only author1 (the first author) out of potentially many co-authors. So author1 needs to be extracted from the list of co-authors in Author(s).

  {authors:list_item(0,&)}

• Filename String. This general program mode template returns the string for the filename component of the path, contatenating the string constant .epub with the title - author1 template. The extension may be wrong due to the use of .epub as a constant. That can be changed to a different extension in the template (within the single quotes).

  program: strcat(template('{title} - {authors:list_item(0,&)}'), '.epub')

• Filename Length. This general program mode template counts the number of characters in the filename component of the path. The count may be inexact due to the use of .epub as a constant. That can be changed to a different extension in the template (within the single quotes).

  program: strlen(strcat(template('{title} - {authors:list_item(0,&)}'), '.epub'))

• Note, Filename Template Inexactness. This applies only to the templates above that deal just with the Filename component. In addition to the inexactness pertaining to the extension constant, there may be other differences between those filename templates and the actual exact structure of filename in a calibre library. First, calibre automatically truncates a path component for a book in a library when string length exceeds a certain limit, which may be different depending on operating system or file system. Another difference mentioned in custom column: count number of characters, Post #17) is the actual path of a book in the library may include multiple sequential spaces, while the template processor reduces any multiple sequential spaces to one space. Compare the first example of a possible title in the actual filename of a book in the library with the second example showing that same title as rendered by a {title} template.

  The Spirit of    Spaciness

  The Spirit of Spaciness

• Path String. This general program mode template uses the new function formats_paths() and accurately (AFAIK) returns the string for the entire path, including all folder names, filename, and the extension for the format chosen as the selector key, for books in current library with that format. The selector key can be changed from EPUB to whatever format is preferred (within the single quotes).

  program: select(formats_paths(),'EPUB')

• Path Length. This general program mode template uses the new function formats_paths() and accurately (AFAIK) counts the number of characters in the entire path, for books in current library with that format. The selector key can be changed from EPUB to whatever format is preferred (within the single quotes).

  program: strlen(select(formats_paths(),'EPUB'))

• Current Library Path. This new template function returns the path for the current library, which in my case is /Users/unboggl/Books/Libs/Prime

  {:'current_library_path()'}

• Approximate Formats. This new template function returns the list of formats from the DB for each book in current library, without referring to any book files, so it is probably accurate rather than definitely accurate. A composite column based on {:'approximate_formats()'} might help calibre start up faster than a composite column based on {formats}.

  {:'approximate_formats()'}

[theducks]

Originally Posted by unboggling

[*]Differences for Other Operating Systems and File Systems. Windows uses a volume designator such as "C:\" and a backslash between path components, while on current versions of OS X, Linux, and other Unix-variants the path component separator is usually forwardslash (depending on specific file system). Also, I have not used Windows in years, so am unsure if Windows still requires a 3 character file extension following the dot.[*]Constant, File Extension. The file extension at the end of the path string for the filename component can be approximated by using a string constant, which can be customized per user. For precision it should include the preceding dot plus all the characters of the extension. But string length of potential extensions varies. Many are 4 characters such as epub, mobi, azw3, while some are 3 characters such as rtf, txt, lit. Some are 5 or more. Most of my books have only one format, EPUB, though there are a few exceptions that also have a PDF or MOBI. To cover the file extension for most of my cases, for filename templates I just use .epub as the constant, and do not worry about possible small differences.

.epub

Windows does care about extensions (you can have a naked file name). The ShellExt is the part that uses the Extension to launch the file/provide an Open With list).
Windows(NT Kernal on) also permits longer extension names. EPUB is 4, Mobi is 4, AZW3 is 4 'metadata.calibre' causes no grief

RIP 8 dot 3

[unboggling]

Originally Posted by theducks

Windows does care about extensions (you can have a naked file name). The ShellExt is the part that uses the Extension to launch the file/provide an Open With list).
Windows(NT Kernal on) also permits longer extension names. EPUB is 4, Mobi is 4, AZW3 is 4 'metadata.calibre' causes no grief

RIP 8 dot 3

RIP 8 dot 3. Yeah, I remember them fondly. 8.3 filenames were fun to construct.

Thanks. So I parse that to mean I can delete the sentence regarding being unsure about Windows extensions. (Done.)