Calibre beta available that fully integrates custom fields

[itimpi]

Originally Posted by N13L5

How portable is a windows installation of Calibre, do you go by "microsoft best err.. worst practice" and write everything to the registry, or can I install the beta copy of Calibre on another drive to mess around on, without killing my first installation?

Calibre does not write anything to the registry, and a portable installation is definitely possible. Look here on how to achieve this.

The one caution is that if anything changes in the database schema between releases (although this is rare) you may find that the metadata.db file is not compatible with both releases (calibre only provides forwards compatibility).

[N13L5]

Originally Posted by itimpi

The one caution is that if anything changes in the database schema between releases (although this is rare) you may find that the metadata.db file is not compatible with both releases (calibre only provides forwards compatibility).

tyvm for the info,

in regards to the metadata.db, does this mean both installations would share the same database, or are you simply telling me that I might not be able to use the old database on the new beta install?

my intention would be to install a separate Calibre with its own (throwaway)database solely for testing...

[chaley]

@theducks & @Manichean: Any chance you have books in your library that have no formats? Kovid found a bug in get_metadata last night that explains Manichean's exception. I don't know about theducks' case, because the exception text is missing.

@Manichean: exceptions during set_books_in_library will cause no end of troubles. Lets wait to trace down the second issue (add_book exception), because it will probably go away. I too use folder devices for lots of testing, for exactly the same reason. However, I can't test collections that way.

@theducks: how big is your library and on your device? Assuming the problem is missing formats, then 55 seconds in set_books_in_library is a bit excessive.

[itimpi]

Originally Posted by N13L5

tyvm for the info,

in regards to the metadata.db, does this mean both installations would share the same database, or are you simply telling me that I might not be able to use the old database on the new beta install?

my intention would be to install a separate Calibre with its own (throwaway)database solely for testing...

If you look at the batch file it allows you to control the placement of all calibre related files, so if you are keeping them separate there should be no issues.

It would be interesting to see if you can run two versions of calibre in parallel if they are using completely different sets of files. I have never tried it, but it may well work. However if you decide to try it i would strongly recomment having backups of your library before trying in case it goes wrong and you end up with a corrupt library.

[Manichean]

Originally Posted by chaley

@theducks & @Manichean: Any chance you have books in your library that have no formats? Kovid found a bug in get_metadata last night that explains Manichean's exception. I don't know about theducks' case, because the exception text is missing.

None that I know of. A search for "formats:false", which I assume should find books with no format, as "formats:true" finds every book in my library, didn't find any books.

Originally Posted by chaley

@Manichean: exceptions during set_books_in_library will cause no end of troubles. Lets wait to trace down the second issue (add_book exception), because it will probably go away. I too use folder devices for lots of testing, for exactly the same reason. However, I can't test collections that way.

I assumed as much. My device hasn't got collections or anything, but categorizes books using the folder structure. That actually leads to another question/ feature request I have, but I'll discuss that in another thread, I think.

[chaley]

Originally Posted by Manichean

None that I know of. A search for "formats:false", which I assume should find books with no format, as "formats:true" finds every book in my library, didn't find any books.

Could you run a database integrity check? The code that is blowing up is indeed the formats check. Either a book has no formats, or a book has formats that don't actually exist.

[Manichean]

Originally Posted by chaley

Could you run a database integrity check? The code that is blowing up is indeed the formats check. Either a book has no formats, or a book has formats that don't actually exist.

Of course. Since I blew up my testing database (entirely through faults of my own), I've ran the check on my "production" installation (0.7.19). Completed without error. I'm currently restoring the testing database on this computer from the "production" one and will re-run the check once it's completed using 0.7.901. I'll post again once that's done.

[theducks]

Originally Posted by chaley

@theducks & @Manichean: Any chance you have books in your library that have no formats? Kovid found a bug in get_metadata last night that explains Manichean's exception. I don't know about theducks' case, because the exception text is missing.

@Manichean: exceptions during set_books_in_library will cause no end of troubles. Lets wait to trace down the second issue (add_book exception), because it will probably go away. I too use folder devices for lots of testing, for exactly the same reason. However, I can't test collections that way.

@theducks: how big is your library and on your device? Assuming the problem is missing formats, then 55 seconds in set_books_in_library is a bit excessive.

Lots of "place holders" (no formats) in my Library.

I had been using the "Get by ISBN function with a Bar-code scanner, to document my paperback library (of which some are so old as to not have either Bookman (978...) nor ISBN codes) . formats:false count : 36

I have to use a USB 1 connection as the V5 hates Intel's USB2-enhanced driver (verified 4 different make-models of computers)
1100+ books. I ran a disc-repair (no errors) just in case (Calibre has not dealt well with FS errors in the past)
Get Meta-data has always been slow, used to take 5Mins with 800 books until the cached data

[Manichean]

This is weird: I just finished restoring my test library, ran a database integrity check (which didn't report any errors) and then was unable to replicate the error I reported earlier.

Edit: Oh, and I took a bit longer because totally unrelated issues interfered.

[Manichean]

I just uninstalled Calibre, removed any folders left, re-copied the library and reinstalled the beta. Again, I'm unable to reproduce the error. I'm assuming that, since I performed a database integrity check on the library, Calibre did find some errors and silently corrected them without notice. Can this happen?

[kovidgoyal]

A db integrity check dumps the db to sql and re-creates it. This can often fix small problems in the db.

[chaley]

Originally Posted by Manichean

I just uninstalled Calibre, removed any folders left, re-copied the library and reinstalled the beta. Again, I'm unable to reproduce the error. I'm assuming that, since I performed a database integrity check on the library, Calibre did find some errors and silently corrected them without notice. Can this happen?

Also, you restored the testing database. Perhaps that fixed the problem.

In any event, Kovid is making new binaries. I will make an announcement when they are ready. In addition to the fix(es), there are some new features.

[Manichean]

Originally Posted by kovidgoyal

A db integrity check dumps the db to sql and re-creates it. This can often fix small problems in the db.

So there might have been errors...
Though this might be insignificant by now, there still aren't any entries without formats in the library.

Originally Posted by chaley

Also, you restored the testing database. Perhaps that fixed the problem.

In any event, Kovid is making new binaries. I will make an announcement when they are ready. In addition to the fix(es), there are some new features.

Yes, I did restore it, but I was only trying to replicate conditions as they were before I noticed the error. Still, I'm excited for those features

[chaley]

A new binary release of this beta has been uploaded to http://calibre-ebook.com/downloads/betas/.

Changes in this version:

1) Fixed problem where books with no formats caused an exception when a device was connected.
2) Several performance enhancements
3) In RE mode, composite fields can now be used as a search/replace source.
4) Bulk edit can now renumber series starting at an arbitrary integer.
5) Bulk edit can now force titles to title case.
6) Template values can now include text that will be included only if the referenced field is not empty. (See the soon-to-be template language faq entry below for details.)

Calibre Template Language:

Spoiler:

The Calibre template language
================================================== =====

The Calibre template language is used in various places. It is used to control the folder structure and file name when saving files from the Calibre library to the disk or eBook reader.
It is used to define "virtual" columns that contain data from other columns and so on.

In essence, the template language is very simple. The basic idea is that a template consists of names in curly brackets that are then replaced by the corresponding metadata from the book being processed. So, for example, the default template used for saving books to device in Calibre is::

{author_sort}/{title}/{title} - {authors}

For the book "The Foundation" by "Isaac Asimov" it will become::

Asimov, Isaac/The Foundation/The Foundation - Isaac Asimov

You can use all the various metadata fields available in calibre in a template, including the custom columns you have created yourself. To find out the template name for a column sinply hover your mouse over the column header. Names for custom fields (columns you have created yourself) are always prefixed by an #. For series type fields, there is always an additional field named ``series_index`` that becomes the series index for that series. So if you have a custom series field named #myseries, there will also be a field named #myseries_index. In addition to the column based fields, you also can use::

{formats} - A list of formats available in the Calibre library for a book
{isbn} - The ISBN number of the book

If a particular book does not have a particular piece of metadata, the field in the template is automatically removed for that book. So for example::

{author_sort}/{series}/{title} {series_index}

will become::

{Asimov, Isaac}/Foundation/Second Foundation - 3

and if a book does not have a series::

{Asimov, Isaac}/Second Foundation

(Calibre automatically removes multiple slashes and leading or trailing spaces).


Advanced formatting
----------------------

You can do more than just simple substitution with the templates. You can also conditionally include text and control how the substituted data is formatted.

Regarding conditionally including text: there are cases where you might want to have text appear in the output only if a field is not empty. A common case is series and series_index, where you want either nothing or the two values with a hyphen between them. Calibre handles this case using a special field syntax.
For example, assume you want to use the template

{series} - {series_index} - {title}

Unfortunately, if the book has no series, the answer will be '- - title'. Many people would rather it be simply 'title', without the hyphens. To do this, use the extended syntax {some_text|field|other_text}. When you use this syntax, if field has the value SERIES then the result will be some_textSERIESother_text. If field has no value, then the result will be the empty string (nothing). Using this syntax, we can solve the above series problem with the template

{series}{ - |series_index| - }{title}

The hyphens will be included only if the book has a series index. Note: you must either use no | characters or both of them. Using one, such as in {field| - }, is not allowed. It is OK to not provide any text for one side or the other, such as in {|series| - }. Using {|title|} is the same as using {title}.

Now to formatting. Suppose you wanted to ensure that the series_index is always formatted as three digits with leading zeros. This would do the trick::

{series_index:0>3s} - Three digits with leading zeros

If instead of leading zeros you want leading spaces, use::

{series_index:>3s} - Thre digits with leading spaces

For trailing zeros, use::

{series_index:0<3s} - Three digits with trailing zeros


If you want only the first two letters of the data to be rendered, use::

{author_sort:.2} - Only the first two letter of the author sort name

The Calibre template language comes from python and for more details on the syntax of these advanced formatting operations, look at the `Python documentation <http://docs.python.org/library/string.html#format-string-syntax>`_.

[Manichean]

Originally Posted by chaley

A new binary release of this beta has been uploaded to http://calibre-ebook.com/downloads/betas/.

Changes in this version:

1) Fixed problem where books with no formats caused an exception when a device was connected.
2) Several performance enhancements
3) In RE mode, composite fields can now be used as a search/replace source.
4) Bulk edit can now renumber series starting at an arbitrary integer.
5) Bulk edit can now force titles to title case.
6) Template values can now include text that will be included only if the referenced field is not empty. (See the soon-to-be template language faq entry below for details.)

Calibre Template Language:
[...]

That is SO COOL! There I am, ranting about wanting to ask about a new feature, and then you guys go ahead and implement it without me even asking! I'm talking (mainly) about the conditional fields in templates. You guys rock.