[GUI Plugin] Consolidate All Library Metadata (CALM)

[Sue_]

I created another user, fresh new home folder, new libraries etc....
Didn't help any

[DaltonST]

I am thinking that btrfs could be the reason that apsw cannot connect even though the plugin use of the python os.path.isdir() etc. and shutil seem to work fine. Obviously, Calibre itself uses apsw and it seems to work properly for you, yes? You can copy books among different libraries, etc. yes?

How long have you been using Calibre with btrfs?


DaltonST

[Sue_]

I've had Opensuse Tumbleweed for quite awhile, and calibre has never had an issue. I copied books between libraries quite alot as I am reorganizing, moved/copied/created libraries....everything has been working fine. It is very confusing, there must be one little thing somewhere, that doesn't affect calibre itself, or which calibre would only rarely use?

[DaltonST]

https://btrfs.wiki.kernel.org/index.php/Gotchas


Fragmentation issue causing apsw to give up on attaching to another db due to timeout?


See this thread I posted in Calibre. https://www.mobileread.com/forums/sh...d.php?t=351904

[Sue_]

Defrag on /home, no change:

ATTACH DATABASE '/home/susan/calibre/testlibrary/metadata.db' *AS 'SOURCE' ;
length of path: *43
Attaching to New Source Library: /home/susan/calibre/testlibrary/metadata.db
NOT Attached: /home/susan/calibre/testlibrary/metadata.db
CantOpenError: unable to open database: /home/susan/calibre/testlibrary/metadata.db
^^^^^^^ Source Library was NOT found: */home/susan/calibre/testlibrary

[DaltonST]

One important thing that I haven't seen is the actual, full, detailed debug log from running from the command line via calibre-debug -g .

DaltonST

[DaltonST]

Also, where did the asterisks * come from in the *43 and in */home/susan/calibre/testlibrary ??


Also, the * just before *AS 'SOURCE' ; ??

ATTACH DATABASE '/home/susan/calibre/testlibrary/metadata.db' *AS 'SOURCE' ; 
length of path: *43 
Attaching to New Source Library: /home/susan/calibre/testlibrary/metadata.db 
NOT Attached: /home/susan/calibre/testlibrary/metadata.db 
CantOpenError: unable to open database: /home/susan/calibre/testlibrary/metadata.db 
 ^^^^^^^ Source Library was NOT found: */home/susan/calibre/testlibrary

The debug log should be similar to this from my Windows 11 testing:

ATTACH DATABASE 's:/calibre/calibrelibrarycodes/metadata.db'  AS 'SOURCE' ; 
length of path:  42 
Attaching to New Source Library: s:/calibre/calibrelibrarycodes/metadata.db 
Now Attached to New Source Library: s:/calibre/calibrelibrarycodes/metadata.db
PRAGMA SOURCE.busy_timeout = 4000;  
Finished: apsw_attach_to_source

[Sue_]

The asterisks appear when I paste into here, no idea why, they are not shown in Konsole.
Tho, they do not show up until the post is previewed, or posted.

Sorry, I didn't keep any of the logs
I should have, but there were no other issues that I saw in them.
I am wiping my calibre install again, with drill to insure absolutely nothing remains before trying again.
That there was no issue before must mean something, somewhere has changed, so maybe I missed something on my last attempt.

[Sue_]

Ok, total wipe & reinstall, even recreated the test libraries.

I tried a couple of times, just to be sure I didn't miss anything.

No change.

Here is the full log:
susan@localhost:~> calibre-debug --gui
calibre 6.11 embedded-python: False
Linux-6.1.8-1-default-x86_64-with-glibc2.36 Linux ('64bit', 'ELF')
('Linux', '6.1.8-1-default', '#1 SMP PREEMPT_DYNAMIC Tue Jan 24 06:29:25 UTC 2023 (baebfe0)')
Python 3.10.9
Interface language: None
Successfully initialized third party plugins: Consolidate All Library Metadata (2, 0, 45) && Job Spy (1, 0, 202) && MultiColumnSearch (1, 0, 96) && View Manager (1, 10, 2)
QPA platform: xcb
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 102.23899371069182 x 102.35820895522387
ApplicationPaletteChange event ignored
Using calibre Qt style: True
[0.00] Starting up...
[0.00] Showing splash screen...
[0.04] splash screen shown
[0.04] Initializing db...
[0.06] db initialized
[0.06] Constructing main UI...
Screens currently in system:
{'depth': 24,
'device_pixel_ratio': 1.0,
'geometry_in_logical_pixels': {'height': 1080, 'width': 1920, 'x': 0, 'y': 0},
'index_in_screens_list': 0,
'manufacturer': 'ViewSonic Corporation',
'model': 'VX2250 SERIES',
'name': 'DVI-I-1',
'serial': 'RWS102406078-',
'size_in_logical_pixels': {'height': 1080, 'width': 1920},
'virtual_geometry': {'height': 1080, 'width': 1920, 'x': 0, 'y': 0}}
Restoring geometry for: calibre_main_window_geometry
{'frame_geometry': {'height': 1036, 'width': 1920, 'x': 0, 'y': 0},
'full_screened': False,
'geometry': {'height': 1005, 'width': 1920, 'x': 0, 'y': 31},
'maximized': True,
'normal_geometry': {'height': 487, 'width': 1322, 'x': 454, 'y': 160},
'screen': {'depth': 24,
'device_pixel_ratio': 1.0,
'geometry_in_logical_pixels': {'height': 1080,
'width': 1920,
'x': 0,
'y': 0},
'index_in_screens_list': 0,
'manufacturer': 'ViewSonic Corporation',
'model': 'VX2250 SERIES',
'name': 'DVI-I-1',
'serial': 'RWS102406078-',
'size_in_logical_pixels': {'height': 1080, 'width': 1920},
'virtual_geometry': {'height': 1080,
'width': 1920,
'x': 0,
'y': 0}}}
Matching screen: {'depth': 24,
'device_pixel_ratio': 1.0,
'geometry_in_logical_pixels': {'height': 1080, 'width': 1920, 'x': 0, 'y': 0},
'index_in_screens_list': 0,
'manufacturer': 'ViewSonic Corporation',
'model': 'VX2250 SERIES',
'name': 'DVI-I-1',
'serial': 'RWS102406078-',
'size_in_logical_pixels': {'height': 1080, 'width': 1920},
'virtual_geometry': {'height': 1080, 'width': 1920, 'x': 0, 'y': 0}}
Setting widget geometry to: {'height': 1005, 'width': 1920, 'x': 0, 'y': 31}
Restoring widget to maximized
CALM: initialization_complete............
Job Spy has begun initialization...
Calibre, and hence Job Spy, was gracefully shut down last time? True
Last time daemon started: never
Last time daemon failed: never
Total daemon starts inception_to_date: 0
Total daemon failures inception-to-date: 0
qt.gui.imageio: libpng warning: iCCP: known incorrect sRGB profile
JS: ===>>> change_gui_alternating_row_colors()
Job Spy has finished initialization...
[0] un-optimized number of undesirable characters accumulated: 756
[1] optimized number of undesirable characters accumulated: 596
MCS has finished initialization...
[1.53] main UI initialized...
[1.53] Hiding splash screen
Starting QuickView
[1.69] splash screen hidden
[1.69] Started up in 1.69 seconds with 467 books
CALM_DB_VERSION_METADATA_DB 2
CALM_DB_VERSION_METADATA_TOOLS_DB 2
CALM_DB_VERSION_UPGRADE_FORCE_UPGRADE False
CALM_DB_VERSION_UPGRADE_SUPPRESS_INFO_DIALOGS False
CALM_FORCE_RECONSOLIDATION False
CALM_LAST_CONSOLIDATION_JOB_DATETIME ...NEVER...
CALM_LAST_CONSOLIDATION_NUMBER_BOOKS 0
CALM_LAST_CONSOLIDATION_NUMBER_LIBRARIES 0
CALM_LAST_CONSOLIDATION_TARGET_DB_FULL_PATH ZZ:
CALM_MCS_INDEX_CONSOLIDATION False
CALM_TARGET_DB_FULL_PATH /home/susan/Calibre/CALM/metadata.db
CALM_TARGET_DB_GENERATING_NONCALM_LIBRARY_PATH ZZ:||>>>||ZZ:
CALM_TARGET_DB_LAST_CHOSEN_LIBRARY_DIRECTORY ZZ:
CALM_TARGET_DB_MUST_BE_REFRESHED False
CALM_TARGET_PARENT_DIRECTORY /home/susan/Calibre
CALM_USER_STATUS USER
Number of keys in self.mysourceprefs in 'load_source_prefs_init' : 128
Number of keys in self.mysourceprefs in 'load_source_prefs' : 128
my_db.library_path: /home/susan/Calibre/LanguageandWriting
self.target_db: /home/susan/Calibre/CALM/metadata.db
Number of keys in self.mysourceprefs in 'save_source_preferences' : 128
Number of keys in self.myparentprefs in 'child_invocation_of_save_real_prefs_parent' : 128
Number of keys in self.mysourceprefs in 'optimize_source_prefs' [1] : 128
Saved: LIBRARY_PATH_03 /home/susan/Calibre/LanguageandWriting
Saved: LIBRARY_PATH_04 /home/susan/Calibre/New Library
tmp_was_checked_dict[k] = unicode_type("True") LIBRARY_PATH_04_IS_ACTIVE True
Saved: LIBRARY_PATH_05 /home/susan/Calibre/NewBooksForCalibre
tmp_was_checked_dict[k] = unicode_type("True") LIBRARY_PATH_05_IS_ACTIVE True
Saved: LIBRARY_PATH_06 /home/susan/Calibre/Testlibrary
tmp_was_checked_dict[k] = unicode_type("True") LIBRARY_PATH_06_IS_ACTIVE True
path_checked_status_dict[v1] = v2 : /home/susan/Calibre/New Library True
path_checked_status_dict[v1] = v2 : /home/susan/Calibre/NewBooksForCalibre True
path_checked_status_dict[v1] = v2 : /home/susan/Calibre/Testlibrary True
for row in tmp_sorted_list: !00
for row in tmp_sorted_list: !01
for row in tmp_sorted_list: !02
for row in tmp_sorted_list: /home/susan/Calibre/LanguageandWriting
for row in tmp_sorted_list: /home/susan/Calibre/New Library
for row in tmp_sorted_list: /home/susan/Calibre/NewBooksForCalibre
for row in tmp_sorted_list: /home/susan/Calibre/Testlibrary
Number of keys in self.mysourceprefs in 'optimize_source_prefs' [2] : 128
Number of keys in self.mysourceprefs in 'save_real_prefs_source': 128
Number of keys in self.mysourceprefs in 'load_source_prefs' : 128
Number of keys in self.mysourceprefs in 'save_source_preferences' : 128
Number of keys in self.myparentprefs in 'child_invocation_of_save_real_prefs_parent' : 128
Number of keys in self.myparentprefs in 'child_invocation_of_save_real_prefs_parent' : 128
n_tools_size = os.path.getsize(tools_path): 2260992
Number of keys in self.myparentprefs in 'child_invocation_of_save_real_prefs_parent' : 128
LIBRARY_PATH_03 : /home/susan/Calibre/LanguageandWriting
LIBRARY_PATH_04 : /home/susan/Calibre/New Library
LIBRARY_PATH_05 : /home/susan/Calibre/NewBooksForCalibre
LIBRARY_PATH_06 : /home/susan/Calibre/Testlibrary
number_source_libraries: 4
n_last_to_show: 6
n_tools_size = os.path.getsize(tools_path): 2260992
metadata_tools.db is good to go.
CALM_DB_VERSION_METADATA_DB 2
CALM_DB_VERSION_METADATA_TOOLS_DB 2
CALM_DB_VERSION_UPGRADE_FORCE_UPGRADE False
CALM_DB_VERSION_UPGRADE_SUPPRESS_INFO_DIALOGS False
CALM_FORCE_RECONSOLIDATION False
CALM_LAST_CONSOLIDATION_JOB_DATETIME ...NEVER...
CALM_LAST_CONSOLIDATION_NUMBER_BOOKS 0
CALM_LAST_CONSOLIDATION_NUMBER_LIBRARIES 0
CALM_LAST_CONSOLIDATION_TARGET_DB_FULL_PATH ZZ:
CALM_MCS_INDEX_CONSOLIDATION False
CALM_TARGET_DB_FULL_PATH /home/susan/Calibre/CALM/metadata.db
CALM_TARGET_DB_GENERATING_NONCALM_LIBRARY_PATH ZZ:||>>>||ZZ:
CALM_TARGET_DB_LAST_CHOSEN_LIBRARY_DIRECTORY ZZ:
CALM_TARGET_DB_MUST_BE_REFRESHED False
CALM_TARGET_PARENT_DIRECTORY /home/susan/Calibre
CALM_USER_STATUS USER
Number of keys in self.mysourceprefs in 'load_source_prefs_init' : 128
Number of keys in self.mysourceprefs in 'load_source_prefs' : 128
my_db.library_path: /home/susan/Calibre/LanguageandWriting
self.target_db: /home/susan/Calibre/CALM/metadata.db
Number of keys in self.mysourceprefs in 'save_source_preferences' : 128
Number of keys in self.myparentprefs in 'child_invocation_of_save_real_prefs_parent' : 128
Number of keys in self.mysourceprefs in 'optimize_source_prefs' [1] : 128
Saved: LIBRARY_PATH_03 /home/susan/Calibre/LanguageandWriting
Saved: LIBRARY_PATH_04 /home/susan/Calibre/New Library
tmp_was_checked_dict[k] = unicode_type("True") LIBRARY_PATH_04_IS_ACTIVE True
Saved: LIBRARY_PATH_05 /home/susan/Calibre/NewBooksForCalibre
tmp_was_checked_dict[k] = unicode_type("True") LIBRARY_PATH_05_IS_ACTIVE True
Saved: LIBRARY_PATH_06 /home/susan/Calibre/Testlibrary
tmp_was_checked_dict[k] = unicode_type("True") LIBRARY_PATH_06_IS_ACTIVE True
path_checked_status_dict[v1] = v2 : /home/susan/Calibre/New Library True
path_checked_status_dict[v1] = v2 : /home/susan/Calibre/NewBooksForCalibre True
path_checked_status_dict[v1] = v2 : /home/susan/Calibre/Testlibrary True
for row in tmp_sorted_list: !00
for row in tmp_sorted_list: !01
for row in tmp_sorted_list: !02
for row in tmp_sorted_list: /home/susan/Calibre/LanguageandWriting
for row in tmp_sorted_list: /home/susan/Calibre/New Library
for row in tmp_sorted_list: /home/susan/Calibre/NewBooksForCalibre
for row in tmp_sorted_list: /home/susan/Calibre/Testlibrary
Number of keys in self.mysourceprefs in 'optimize_source_prefs' [2] : 128
Number of keys in self.mysourceprefs in 'save_real_prefs_source': 128
Number of keys in self.mysourceprefs in 'load_source_prefs' : 128
Number of keys in self.mysourceprefs in 'save_source_preferences' : 128
Number of keys in self.myparentprefs in 'child_invocation_of_save_real_prefs_parent' : 128
Number of keys in self.myparentprefs in 'child_invocation_of_save_real_prefs_parent' : 128
Number of keys in self.myparentprefs in 'child_invocation_of_save_real_prefs_parent' : 128
Number of keys in self.myparentprefs in 'save_real_prefs_parent' : 128
autopopulate: k, : /home/susan/calibre/languageandwriting
autopopulate: k, : /home/susan/calibre/new library
autopopulate: k, : /home/susan/calibre/newbooksforcalibre
autopopulate: k, : /home/susan/calibre/testlibrary
Attaching to New Source Library: /home/susan/calibre/newbooksforcalibre/metadata.db
NOT Attached: /home/susan/calibre/newbooksforcalibre/metadata.db
CantOpenError: unable to open database: /home/susan/calibre/newbooksforcalibre/metadata.db
^^^^^^^ Source Library was NOT found: /home/susan/calibre/newbooksforcalibre
Attaching to New Source Library: /home/susan/calibre/languageandwriting/metadata.db
NOT Attached: /home/susan/calibre/languageandwriting/metadata.db
CantOpenError: unable to open database: /home/susan/calibre/languageandwriting/metadata.db
^^^^^^^ Source Library was NOT found: /home/susan/calibre/languageandwriting
Attaching to New Source Library: /home/susan/calibre/new library/metadata.db
NOT Attached: /home/susan/calibre/new library/metadata.db
CantOpenError: unable to open database: /home/susan/calibre/new library/metadata.db
^^^^^^^ Source Library was NOT found: /home/susan/calibre/new library
Attaching to New Source Library: /home/susan/calibre/testlibrary/metadata.db
NOT Attached: /home/susan/calibre/testlibrary/metadata.db
CantOpenError: unable to open database: /home/susan/calibre/testlibrary/metadata.db
^^^^^^^ Source Library was NOT found: /home/susan/calibre/testlibrary
source library path added to set: /home/susan/Calibre/LanguageandWriting
source library path added to set: /home/susan/Calibre/New Library
source library path added to set: /home/susan/Calibre/NewBooksForCalibre
source library path added to set: /home/susan/Calibre/Testlibrary
adding source to table _source_libraries: /home/susan/Calibre/NewBooksForCalibre
adding source to table _source_libraries: /home/susan/Calibre/LanguageandWriting
adding source to table _source_libraries: /home/susan/Calibre/New Library
adding source to table _source_libraries: /home/susan/Calibre/Testlibrary
number_source_libraries_0: 4
number_source_libraries = unicode_type(col): 0
number_source_libraries = unicode_type(col): 0

[Sue_]

Huh, no asterisks appeared in the post this time, so one difference I guess.....

[BetterRed]

Couple of idle observations, ignore if irrelevant:

1. Linux file systems I've used (Ext3/4, NFS etc) are case sensitive. i.e. 'File.Txt', 'file.txt' and 'FILE.TXT' are distinct. In the log I see:

Attaching to New Source Library: /home/susan/calibre/languageandwriting/metadata.db

And a further down I see:

source library path added to set: /home/susan/Calibre/LanguageandWriting

2. I recall seeing problems (20+ years ago) with file names that had 'foreign' characters on an IBM Linux system - and if memory serves me right IBM had its fingers in the btfs pie.

BR

[Sue_]

I didn't even notice the case was being changed, why would that be happening?

[DaltonST]

Originally Posted by Sue_

I didn't even notice the case was being changed, why would that be happening?

Possibly you have 2 different Libraries, each with different cases.

CALM reads the list that Calibre keeps of known Libraries along with the number of times they have been used. CALM loads the Source Library Tabs automatically when the pushbutton is clicked using that saved list, sorting descending by the usage count. See the attached example image.

You can find that file via grep in the Calibre configuration directory looking for text of one of your Library's name and path.

Question: did you manually key the Source Library name? Or use the pushbutton?

Also, I still have not seen a full, detailed debugging log at all. Only your excerpts, including the odd asterisks. Please post a full, real log, and let me review the real thing, unedited. Thanks.



DaltonST

[Sue_]

I did post a full log? A few posts above.

I don't have multiple libraries with the same name in calibre, when I recreated or copied a library to test this issue, I used forget library in calibre on all other libraries, they were all in the same case anyways, I didn't change that, and the filepaths would be different also as I am using a folder on the main drive and my "real" libraries are on the HDD. CALM' s list of libraries only showed the ones it was supposed to, none of the ones calibre was told to forget about.

[DaltonST]

Your log shows you have Job Spy installed. It has a utility to create a matrix of all custom columns & their attributes in all known Calibre libraries. So, it has to connect/attach to each one sequentially to do so. Just like CALM does (but APSW fails).

Please run that utility. If it has the same APSW errors as CALM, then we know it is either an OS/filesystem conflict with APSW, or some other system issue.

DaltonST