Calibre beta available that fully integrates custom fields

[Kaitou Ace]

I'm trying this beta with the PRS-650, and using a custom field with the lookup of "dtags" of type tags
My goal is to use this field to create collections on the 650, so I went into preferences->plugins->Device interface plugins->SONY Device Interface, and put "#dtags" into the collections field. However when the books are sent to the device, the collections are blank. Is this because the 650 is new, or am I doing something wrong?

[chaley]

Originally Posted by Kaitou Ace

I'm trying this beta with the PRS-650, and using a custom field with the lookup of "dtags" of type tags
My goal is to use this field to create collections on the 650, so I went into preferences->plugins->Device interface plugins->SONY Device Interface, and put "#dtags" into the collections field. However when the books are sent to the device, the collections are blank. Is this because the 650 is new, or am I doing something wrong?

Either you are doing something wrong, or something is broken.

1) Are you sure you are running the beta? (Version number in lower left should be 0.7.902.)
2) Is #dtags is the lookup name for the custom field? In other words, if you search for #dtags:x, will you find things?
3) is #dtags a text/multiple field?
4) does the column actually have something in it?
5) Are you using automatic metadata management? (manual will work, but the results are very hard to interpret.)

If the answers to all 5 are yes, then you need to run calibre in debug mode to see what is going on. If you are running on windows, open a cmd box and type 'calibre-debug -g >caldebug.txt' (no quotes). Be sure to note what folder you are in. Zip and post the file caldebug.txt (you are welcome to look at it as well. ).

If we can't tell what is going on from the debug log, then we will need to get copies of the media.xml and metadata.calibre. However, lets wait on that.

[Kaitou Ace]

Originally Posted by chaley

Either you are doing something wrong, or something is broken.

1) Are you sure you are running the beta? (Version number in lower left should be 0.7.902.)
2) Is #dtags is the lookup name for the custom field? In other words, if you search for #dtags:x, will you find things?
3) is #dtags a text/multiple field?
4) does the column actually have something in it?
5) Are you using automatic metadata management? (manual will work, but the results are very hard to interpret.)

If the answers to all 5 are yes, then you need to run calibre in debug mode to see what is going on. If you are running on windows, open a cmd box and type 'calibre-debug -g >caldebug.txt' (no quotes). Be sure to note what folder you are in. Zip and post the file caldebug.txt (you are welcome to look at it as well. ).

If we can't tell what is going on from the debug log, then we will need to get copies of the media.xml and metadata.calibre. However, lets wait on that.

Well 1 - 4 were "yes."
The issue it looks like was the Automatic Metadata Management, once I switched that on, it started working. I couldn't get it to create the collections in manual, but so long as it works, I am satisfied. Much thanks!

[speakingtohe]

I installed calibre beta on a fresh Vista install because I was having so many weird problems. (plus I came back from a 5 month absence to find my desktop infested )

I cannot find the place where you turn of search bar auto complete in preferences?
Probably right in front of my face but I have looked 10 times and searched the help so if some kind soul could point it out.

Helen

[kovidgoyal]

If you mean "search as you type" it is in preferences->Interface->Look and Feel

[Kaitou Ace]

Now that it's creating the collections, it seems to be adding "(Collections)" at the end of every collection. That is the column heading of my custom field, so I can see where its coming from, but is there a way to disable or change that behavior?

[speakingtohe]

If you mean "search as you type" it is in preferences->Interface->Look and Feel

Thank you didn't look to the right. Stupid me! Whack! whack! whack!

[chaley]

Originally Posted by Kaitou Ace

Now that it's creating the collections, it seems to be adding "(Collections)" at the end of every collection. That is the column heading of my custom field, so I can see where its coming from, but is there a way to disable or change that behavior?

Yes. See the tweak sony_collection_renaming_rules.

Documentation is:

Spoiler:

# Specify renaming rules for sony collections. Collections on Sonys are named
# depending upon whether the field is standard or custom. A collection derived
# from a standard field is named for the value in that field. For example, if
# the standard 'series' column contains the name 'Darkover', then the series
# will be named 'Darkover'. A collection derived from a custom field will have
# the name of the field added to the value. For example, if a custom series
# column named 'My Series' contains the name 'Darkover', then the collection
# will be named 'Darkover (My Series)'. If two books have fields that generate
# the same collection name, then both books will be in that collection. This
# tweak lets you specify for a standard or custom field the value to be put
# inside the parentheses. You can use it to add a parenthetical description to a
# standard field, for example 'Foo (Tag)' instead of the 'Foo'. You can also use
# it to force multiple fields to end up in the same collection. For example, you
# could force the values in 'series', '#my_series_1', and '#my_series_2' to
# appear in collections named 'some_value (Series)', thereby merging all of the
# fields into one set of collections. The syntax of this tweak is
# {'field_lookup_name':'name_to_use', 'lookup_name':'name', ...}
# Example 1: I want three series columns to be merged into one set of
# collections. If the column lookup names are 'series', '#series_1' and
# '#series_2', and if I want nothing in the parenthesis, then the value to use
# in the tweak value would be:
# sony_collection_renaming_rules={'series':'', '#series_1':'', '#series_2':''}
# Example 2: I want the word '(Series)' to appear on collections made from
# series, and the word '(Tag)' to appear on collections made from tags. Use:
# sony_collection_renaming_rules={'series':'Series', 'tags':'Tag'}
# Example 3: I want 'series' and '#myseries' to be merged, and for the
# collection name to have '(Series)' appended. The renaming rule is:
# sony_collection_renaming_rules={'series':'Series', '#myseries':'Series'}
sony_collection_renaming_rules={}

[chaley]

Originally Posted by Kaitou Ace

Well 1 - 4 were "yes."
The issue it looks like was the Automatic Metadata Management, once I switched that on, it started working. I couldn't get it to create the collections in manual, but so long as it works, I am satisfied. Much thanks!

It should work with manual management, but you would need to resend all the books. It will not update the collections for books already there. Does this match with your experience?

[chaley]

The next beta contains an improved template and composite field mechanism. You can now conditionally introduce slashes into path names, so you can create variable-depth folder hierarchies. You can also conditionally replace field values with other field values.

The FAQ that discusses this is below (in the markdown that calibre uses for such things).

Spoiler:

The |app| template language
================================================== =====

The |app| template language is used in various places. It is used to control the folder structure and file name when saving files from the |app| library to the disk or eBook reader.
It is also used to define "virtual" columns that contain data from other columns and so on.

The basic template language is very simple, but has very powerful advanced features. The basic idea is that a template consists of names in curly brackets that are then replaced by the corresponding metadata from the book being processed. So, for example, the default template used for saving books to device in |app| is::

{author_sort}/{title}/{title} - {authors}

For the book "The Foundation" by "Isaac Asimov" it will become::

Asimov, Isaac/The Foundation/The Foundation - Isaac Asimov

You can use all the various metadata fields available in calibre in a template, including any custom columns you have created yourself. To find out the template name for a column simply hover your mouse over the column header. Names for custom fields (columns you have created yourself) always have a # as the first character. For series type custom fields, there is always an additional field named ``#seriesname_index`` that becomes the series index for that series. So if you have a custom series field named #myseries, there will also be a field named #myseries_index.

In addition to the column based fields, you also can use::

{formats} - A list of formats available in the calibre library for a book
{isbn} - The ISBN number of the book

If a particular book does not have a particular piece of metadata, the field in the template is automatically removed for that book. So for example::

{author_sort}/{series}/{title} {series_index}

If a book has a series, the template will produce::

{Asimov, Isaac}/Foundation/Second Foundation - 3

and if a book does not have a series::

{Asimov, Isaac}/Second Foundation

(|app| automatically removes multiple slashes and leading or trailing spaces).


Advanced formatting
----------------------

You can do more than just simple substitution with the templates. You can also conditionally include text and control how the substituted data is formatted.

First, conditionally including text. There are cases where you might want to have text appear in the output only if a field is not empty. A common case is series and series_index, where you want either nothing or the two values with a hyphen between them. Calibre handles this case using a special field syntax.

For example, assume you want to use the template

{series} - {series_index} - {title}

If the book has no series, the answer will be '- - title'. Many people would rather the result be simply 'title', without the hyphens. To do this, use the extended syntax `{field:|prefix_text|suffix_text}`. When you use this syntax, if field has the value SERIES then the result will be prefix_textSERIESsuffix_text. If field has no value, then the result will be the empty string (nothing). The prefix and suffix can contain blanks.

Using this syntax, we can solve the above series problem with the template:

{series}{series_index:| - | - }{title}

The hyphens will be included only if the book has a series index.

Notes: you must include the : character if you want to use a prefix or a suffix. You must either use no | characters or both of them; using one, as in `{field:| - }`, is not allowed. It is OK not to provide any text for one side or the other, such as in `{series:|| - }`. Using `{title:||}` is the same as using `{title}`.

Second: formatting. Suppose you wanted to ensure that the series_index is always formatted as three digits with leading zeros. This would do the trick::

{series_index:0>3s} - Three digits with leading zeros

If instead of leading zeros you want leading spaces, use:

{series_index:>3s} - Three digits with leading spaces

For trailing zeros, use:

{series_index:0<3s} - Three digits with trailing zeros


If you want only the first two letters of the data, use::

{author_sort:.2} - Only the first two letter of the author sort name

The |app| template language comes from python and for more details on the syntax of these advanced formatting operations, look at the `Python documentation <http://docs.python.org/library/string.html#format-string-syntax>`_.

Advanced features
------------------

Using templates in custom columns
----------------------------------

There are sometimes cases where you want to display metadata that |app| does not normally display, or to display data in a way different from how |app| normally does. For example, you might want to display the ISBN, a field that |app| does not display. You can use custom columns for this. To do so, you create a column with the type 'column built from other columns' (hereafter called composite columns), enter a template, and |app| will display in the column the result of evaluating that template. To display the isbn, create the column and enter `{isbn}` into the template box. To display a column containing the values of two series custom columns separated by a comma, use `{#series1:||,}{#series2}`.

Composite columns can use any template option, including formatting.

You cannot change the data contained in a composite column. If you edit a composite column by double-clicking on any item, you will open the template for editing, not the underlying data. Editing the template on the GUI is a quick way of testing and changing composite columns.

Using functions in templates
-----------------------------

Suppose you want to display the value of a field in upper case, when that field is normally in title case. You can do this (and many more things) using the functions available for templates. For example, to display the title in upper case, use `{title:uppercase()}`. To display it in title case, use `{title:titlecase()}`.

Function references replace the formatting specification, going after the : and before the first `|` or the closing `}`. Functions must always end with `()`. Some functions take extra values (arguments), and these go inside the `()`.

The syntax for using functions is `{field:function(arguments)}`, or `{field:function(arguments)|prefix|suffix}`. Argument values cannot contain a comma, because it is used to separate arguments. Functions return the value of the field used in the template, suitably modified.

The functions available are:

* `lowercase()` -- return value of the field in lower case.
* `uppercase()` -- return the value of the field in upper case.
* `titlecase()` -- return the value of the field in title case.
* `capitalize()` -- return the value as capitalized.
* `ifempty(text)` -- if the field is not empty, return the value of the field. Otherwise return `text`.
* `test(text if not empty, text if empty)` -- return `text if not empty` if the field is not empty, otherwise return `text if empty`.
* `contains(pattern, text if match, text if no match)` -- return `text if match` if any part of the field is matched by the regular expression, otherwise return `text if no match`.
* `shorten(left chars, middle text, right chars)` -- Return a shortened version of the field, consisting of `left chars` characters from the beginning of the field, followed by `middle text`, followed by `right chars` characters from the end of the string. `Left chars` and `right chars` must be integers. For example, assume the title of the book is `Ancient English Laws in the Times of Ivanhoe`, and you want it to fit in a space of at most 15 characters. If you use `{title:shorten(9,-,5)}, the result will be `Ancient E-nhoe`. If the field's length is less than `left chars` + `right chars` + the length of `middle text`, then the field will be used intact. For example, the title `The Dome` would not be changed.
* `lookup(field if not empty, field if empty)` -- like test, except the arguments are field (metadata) names, not text. The value of the appropriate field will be fetched and used. Note that because composite columns are fields, you can use this function in one composite field to use the value of some other composite field. This is extremely useful when constructing variable save paths (more later).
* `re(pattern, replacement)` -- return the field after applying the regular expression. All instances of `pattern` are replaced with `replacement`. As in all of |app|, these are python-compatible regular expressions.

Special notes for save/send templates
-------------------------------------

Special processing is applied when a template is used in a `save to disk` or `send to device` template. The values of the fields are cleaned, replacing characters that are special to file systems with underscores, including slashes. This means that field text cannot be used to create folders. However, slashes are not changed in prefix or suffix strings, so slashes in these strings will cause folders to be created. Because of this, you can create variable-depth folder structure.

For example, assume we want the folder structure `series/series_index - title`, with the caveat that if series does not exist, then the title should be in the top folder. The template to do this is

{series:||/}{series_index:|| - }{title}

The slash and the hyphen appear only if series is not empty.

The lookup function lets us do even fancier processing. For example, assume we want the following: if a book has a series, then we want the folder structure `series/series index - title.fmt`. If the book does not have a series, then we want the folder structure `genre/author_sort/title.fmt`. If the book has no genre, use 'Unknown'. We want two completely different paths, depending on the value of series.

To accomplish this, we:
1. Create a composite field (call it AA) containing `{series:||}/{series_index} - {title'}`. If the series is not empty, then this template will produce `series/series_index - title`.
2. Create a composite field (call it BB) containing `{#genre:ifempty(Unknown)}/{author_sort}/{title}`. This template produces `genre/author_sort/title`, where an empty genre is replaced wuth `Unknown`.
3. Set the save template to `{series:lookup(AA,BB)}`. This template chooses composite field AA if series is not empty, and composite field BB if series is empty. We therefore have two completely different save paths, depending on whether or not `series` is empty.

[Starson17]

Originally Posted by chaley

The next beta contains an improved template

OK, I already used COOL! (squared), so normally, I'd go up to COOL!(cubed). In this case, however, in view of all the creativity, let's allow COOL! to become imaginary and still cube it!

Great work, Charles!

[N13L5]

that's awesome!

will this great formatting work for book formats too, like pdf, epub, fb2 etc?

So you could get pdf's to be saved in a different folder from epub?

Just wondering, cause the formats the books are available in aren't in columns or metadata per se, since Calibre keeps track of that.

[theducks]

Originally Posted by Starson17

OK, I already used COOL! (squared), so normally, I'd go up to COOL!(cubed). In this case, however, in view of all the creativity, let's allow COOL! to become imaginary and still cube it!

Great work, Charles!

You Math Majors
Can we keep it simple please.

I flunked Calculus

[chaley]

Originally Posted by N13L5

will this great formatting work for book formats too, like pdf, epub, fb2 etc?

No, not yet.

So you could get pdf's to be saved in a different folder from epub?

Just wondering, cause the formats the books are available in aren't in columns or metadata per se, since Calibre keeps track of that.

You can get the formats using {formats}, but you can't get the specific format being sent. It isn't in the metadata for the book.

EDIT: of course, if your books have only one format, then {formats} is enough.

I am considering adding a secondary dictionary containing things like device, extension (format), cardname, and possibly others. If I do this, then you will be able to do tests on them. I am thinking the field names will be $X, so (for example) {$ext}/{series}... would do what I think you want. But this isn't here yet.

[Timoleon]

Originally Posted by Starson17

OK, I already used COOL! (squared), so normally, I'd go up to COOL!(cubed). In this case, however, in view of all the creativity, let's allow COOL! to become imaginary and still cube it!

Great work, Charles!

Hey! I used it first!!