[GUI Plugin] Consolidate All Library Metadata (CALM)

[DaltonST]

[1] Run in DEBUG mode, then post the entire debug log.

[2] The harmonize_enumeration_displays error hints that you have not yet harmonized all of your custom columns yet, especially valid values for Enumeration datatypes. Read the tool-tips. Use the Job Spy tool for creating a matrix of all custom columns in all of your libraries.

I will look at it after you post your debug log.


Added: the sorting of the column by clicking the column header is a Qt 6 error. I worked fine in Qt 5, but Qt 6 requires a more specific instruction. You are the first person to report this (maybe the first to try to sort it since Calibre 6.0.0?). Anyway, the next release of CALM will sort it again, but as always in earlier releases, it will only sort by the Name column.

Added: I have bullet-proofed the error you got re: harmonizing your enumeration custom column values. I will release a new version of CALM soon. You still have to harmonize your enumerations, however; it just won't throw an error. In debug mode, it will print a message of exactly what was wrong with your custom column.


Debug mode via command line: calibre-debug -g



DaltonST

[DaltonST]

Version 2.0.45 -2023-01-26 Qt6 compatibility. Improve DEBUG for enumeration harmonization.


DaltonST

[Sue_]

I have my calibre libraries on an internal HDD, with opensuse installed on the SSD.
My libraries are in a folder on the HDD, Collections/Calibre/
The collections folder is symlinked to home, and at first I thought maybe that was the issue, so I changed the path for every library in calibre to the direct path, but the issue persists.

When I try to run the get columns, the window pops up for every library - could not open.

The debug: Everything seems fine for all libraries until this point:

Attaching to New Source Library: /datahdd/@/mydata/media/collections/calibre/main library/metadata.db
NOT Attached: /datahdd/@/mydata/media/collections/calibre/main library/metadata.db
CantOpenError: unable to open database: /datahdd/@/mydata/media/collections/calibre/main library/metadata.db
^^^^^^^ Source Library was NOT found: /datahdd/@/mydata/media/collections/calibre/main library


I have used the check function, and the dbs are not corrupt, so I don't know what is causing this issue.

[DaltonST]

Wild guess: the '/@/' in the path?

Try copying a small test library to a vanilla path for testing.



DaltonST

[Sue_]

Originally Posted by DaltonST

Wild guess: the '/@/' in the path?

Try copying a small test library to a vanilla path for testing.



DaltonST

I hope thats not it, @ is the filesystem, btrfs subvolume

To move it away, I will have to move huge libraries onto a different drive (so a "real" move)

But the symlink version of the file path, no @, didn't work either....

I will try a library on the other drive, but is there another way to fix it if this is the issue?

[Sue_]

Nope, put a couple libraries into a folder on home, same errors.
Very confused, direct doesn't work, symlinked doesn't work, actually in the home folder doesn't work.
Libraries are all checked, not corrupted.
CALM has no trouble creating/deleting/refreshing its own library with all of the filepaths, so why would it not be able to open the other libraries to get the columns?

[DaltonST]

Idea: the library where you run CALM can be totally random (just not the target CALM library itself). So, switch to one of the source libraries before configuring it for a generation. If it was already, switch it to a library on the same drive as Calibre's executable.

APSW, the SQLite package, appears to have a .db connection failure due to permissions or other basic but annoying issue.

DaltonST

[Sue_]

Already was in a Non- CALM library on the same drive

Permissions shouldn't be an issue either, but I checked and as it should be.

[DaltonST]

Try creating a new, empty library in a new local drive directory by having Calibre copy the structure of a real one. In CALM, remove all source libraries, then add just the new empty one. Save. See what happens.

I assume LAN drives are not involved, since Calibre doesn't support them.

[Sue_]

No change
It's going to be something stupid and small, but those are the hardest to find sometimes............

[DaltonST]

I added some new debug statements to the apsw_attach_to_source function, and increased the busy timeout from 2000 to 4000 milliseconds.

Example of its debug log:

ATTACH DATABASE 's:/calibre/calibrelibrarycodes/metadata.db'  AS 'SOURCE' ;
length of path:  42
Attaching to New Source Library: s:/calibre/calibrelibrarycodes/metadata.db
Now Attached to New Source Library: s:/calibre/calibrelibrarycodes/metadata.db
PRAGMA SOURCE.busy_timeout = 4000;
Finished: apsw_attach_to_source

Try the attached .zip by loading it from the file in Preferences > Plugins, and restarting.

Random thoughts: path length too long; OS case sensitivity; busy timeout was too short for your hardware; something that makes your flavor of Linux different from all others; using a server Linux instead of a desktop Linux?


DaltonST

[Sue_]

Longest was 53, is that ok?
Opensuse Tumbleweed is intended for desktop.
Computer not that old, plus a little overloaded with ram, os installed on SSD, speed shouldn't be an issue.

Going to delete all and try again, just in case it decides to like me.

[Sue_]

Deleted all and started over, same as before.

[DaltonST]

So, to summarize, it fails even if 100%, all, of the source libraries are on the same drive as Calibre is installed and with full read/write/execute permissions for Calibre? And the debug log via calibre-debug -g shows the 'attaching to...' message, but then jumps directly to the fatal error message of 'cannot open'?



DaltonST

[Sue_]

The libraries were all on the main drive where os and applications are installed, direct - no symlink.
Calibre itself has no issue interacting with the libraries at all, permissions are normal and as expected in the user home folder.

The same thing happened:

ATTACH DATABASE '/home/susan/calibre/testlibrary/metadata.db' *AS 'SOURCE' ;
length of path: *43
Attaching to New Source Library: /home/susan/calibre/testlibrary/metadata.db
NOT Attached: /home/susan/calibre/testlibrary/metadata.db
CantOpenError: unable to open database: /home/susan/calibre/testlibrary/metadata.db
^^^^^^^ Source Library was NOT found: */home/susan/calibre/testlibrary

This happens on all libraries, but for everything else it seems fine, after these errors it goes on to “source library path added to set" and then “adding source to table _source_libraries” for each library.

The only thing that really makes my system different from most Linux users would be that most Linux installs haven't used btrfs as the default option, but that shouldn't cause an issue?