[Library Closed Plugin] Save Composite Custom Columns

[chaley]

This plugin saves the values of custom columns "made from other columns" ("composite" columns) for use by apps that directly read the calibre database, for example the Calibre Companion Cloud Connection.

Special Notes:

• Requires Calibre 2.54 or later
• CC V5.1.1 and later can use the information saved by this plugin

Configuration:

• To choose whether or not this plugin should process the current library, go to calibre's Preferences / Plugins / Library Closed plugins. Select this plugin, press the "Customize plugin" button, and set the check box appropriately.

Installation Notes:

• Download the attached zip file and install the plugin/restart Calibre as described in the Introduction to plugins thread.
• Note that this is not a GUI plugin so it is not intended/cannot be added to context menus/toolbars etc.

Version History:

Spoiler:

Version 1.1.1 (11/April/2020)
* Python 3 compatibility
* This version is compatible with calibre 6.

Version 1.1.0 (4/April/2016)
* Added the ability to choose on a per-library basis whether or not to run the plugin

Version 1.0.0: (1/April/2016:
* Initial release.

[Terisa de morgan]

Hi,

I've been using this plugin since it appeared but today I've got to run calibre in debug mode, and looking at the debug log I've found a lot of lines like this:

Spoiler:

Traceback (most recent call last):
File "site-packages\calibre\utils\formatter.py", line 524, in safe_format
File "site-packages\calibre\utils\formatter.py", line 495, in evaluate
File "string.py", line 543, in vformat
File "string.py", line 576, in _vformat
File "site-packages\calibre\utils\formatter.py", line 442, in format_field
File "site-packages\calibre\utils\formatter.py", line 400, in _eval_program
File "site-packages\calibre\utils\formatter.py", line 195, in program
File "site-packages\calibre\utils\formatter.py", line 208, in statement
File "site-packages\calibre\utils\formatter.py", line 251, in expr
File "site-packages\calibre\utils\formatter.py", line 45, in error
ValueError: Formatter: función desconocida obt_review junto a obt_review

And all of them are related to functions I've defined in calibre but only in the library I'm leaving (where I'm running the plugin). I don't know if it is important or not, but I wanted to comment it.

[chaley]

Originally Posted by Terisa de morgan

Hi,

I've been using this plugin since it appeared but today I've got to run calibre in debug mode, and looking at the debug log I've found a lot of lines like this:
[...]

As far as I can tell that means that the user-defined template function obt_review is used in a composite column (built from other columns) but no longer exists in that library. It could also be used in some other template context such as column icons or coloring.

If I am right then the messages should be appearing while you use the library. They should not be appearing after the "plugin Save Composite Custom Columns library ..." message.

[Terisa de morgan]

The function exists, as I've checked it. Anyway, as they're hidden columns for other uses, I'll show them and check (I'll be back to you with the answer).

[Terisa de morgan]

The function exists and the column is showing right in the library, so I don't know which the problem is.

[chaley]

Originally Posted by Terisa de morgan

The function exists and the column is showing right in the library, so I don't know which the problem is.

Are the messages appearing before or after the "plugin Save Composite Custom Columns library ..." shows (before closing your library)? If before then they are coming from somewhere in calibre not related to the plugin. If after then the plugin might be involved.

Perhaps I can tell something from the metadata.db. Zip it up and send it to me at calibre_bugs at charles dot haleys dot org.

[Terisa de morgan]

Originally Posted by chaley

Are the messages appearing before or after the "plugin Save Composite Custom Columns library ..." shows (before closing your library)? If before then they are coming from somewhere in calibre not related to the plugin. If after then the plugin might be involved.

Perhaps I can tell something from the metadata.db. Zip it up and send it to me at calibre_bugs at charles dot haleys dot org.

After but, for being sure, I'm going to deactivate the plugin and check the log. It it isn't there... I'll send you all the log (or perhaps only for a book )

[chaley]

Originally Posted by Terisa de morgan

After but, for being sure, I'm going to deactivate the plugin and check the log. It it isn't there... I'll send you all the log (or perhaps only for a book )

Could you also send me the source for the custom function? Thanks.

[chaley]

@Terisa de morgan: I found the problem. It requires two conditions to fail.

1. You use "change library" instead of closing calibre.
2. Your calibre library is large enough so that the composite column cache is overrun. And no, I don't know how large that is.

The fix is in calibre, not the plugin. I am submitting it now.

[Terisa de morgan]

Originally Posted by chaley

@Terisa de morgan: I found the problem. It requires two conditions to fail.

1. You use "change library" instead of closing calibre.
2. Your calibre library is large enough so that the composite column cache is overrun. And no, I don't know how large that is.

The fix is in calibre, not the plugin. I am submitting it now.

Trust in my hand to find bugs And yes, I don't usually close calibre in that library.. although is not too big in terms of books... but I'm afraid it is in terms of columns.

Thank you very much for you fix.

[Terisa de morgan]

@chaley, with this calibre release there is a different problem:

Spoiler:

Traceback (most recent call last):
File "site-packages\calibre\utils\formatter.py", line 524, in safe_format
File "site-packages\calibre\utils\formatter.py", line 495, in evaluate
File "string.py", line 543, in vformat
File "string.py", line 576, in _vformat
File "site-packages\calibre\utils\formatter.py", line 480, in format_field
File "site-packages\calibre\utils\formatter_functions.py", line 128, in eval_
TypeError: evaluate() takes exactly 5 arguments (7 given)

[chaley]

Originally Posted by Terisa de morgan

@chaley, with this calibre release there is a different problem

That says that you have a composite column containing a standard template (not program that is making an illegal function call. In particular it is calling a function that requires one argument but is being given 3. Of course, the problem could instead be another bug.

If you send me a zip of the metadata.db I can almost certainly find either the bug or the problem.

[Terisa de morgan]

Originally Posted by chaley

That says that you have a composite column containing a standard template (not program that is making an illegal function call. In particular it is calling a function that requires one argument but is being given 3. Of course, the problem could instead be another bug.

If you send me a zip of the metadata.db I can almost certainly find either the bug or the problem.

But... it wasn't there before and I haven't changed the template... well, I'll send you the metadata.db, sorry for being an annoyance.

[chaley]

Originally Posted by Terisa de morgan

But... it wasn't there before and I haven't changed the template... well, I'll send you the metadata.db, sorry for being an annoyance.

Not at all an annoyance. The last inquiry lead to finding a very subtle bug in calibre. Who knows what we will find this time?

If it is not a bug then it is probably a template that calls one of your user-defined template functions. The problem wouldn't have been visible until we fixed the bug this week that removed all user-template functions before the plugin was run. Before you saw a general no-function error. Now you see a specific too-many-arguments error.

If the column is hidden then you wouldn't see the error while using the GUI, even in debug mode, unless its value was used in some other composite column that isn't hidden.

Thanks for sending me the db. That will make finding the problem possible.

[chaley]

Originally Posted by Terisa de morgan

But... it wasn't there before and I haven't changed the template... well, I'll send you the metadata.db, sorry for being an annoyance.

Whew (at least from my point of view), this one isn't a calibre bug.

The column #sak has the Single Function Mode template

{series:test(S:{series},{author})}

Templates embedded into templates are to be avoided in Single Function Mode because the inner templates are expanded before the template is parsed. This evaluation order means that if an item such as {series} contains a comma then that comma becomes an argument separator. That is why the documentation has the big bold sentence Do not use subtemplates (`{ ... }`) as function arguments. In that metadata.db there are 5 series names that contain a comma.

You can avoid this problem by using Template Program Mode

{series:'test($, strcat('S:', $), field('authors'))'}

or the equivalent in General Program Mode.