Stopping Calibre rechecking library metadata

[Mythlandia]

I have a number of different Calibre libraries which I swap between.
Every few times I swap libraries Calibre will decide to reload info from the directories & metadata (at least it reads thousands of files). This takes ten minutes to an hour depending upon the size of the library. Sometimes I will lose patience, kill Calibre, go back in and the same library opens immediately.

Is there anyway to get Calibre just to trust it's own data?

The 45mb metadata database could be read in less than a second, but the actual library is 65gb and just touching the metadata files means reading gigabytes of data.

[theducks]

Not Normal (unless you have custom column (composite or coloring) that requires file info).
Calibre wants will backup its metadata in each folder. That happens once, then only if the book data changes in the DB.
Kill that process and it will start over where it thinks it was. :footshoot:

[Mythlandia]

Up to 4096 bytes are being read for each book directory in the library (sometimes) when swapping from one library toanother using Quick Switch.
Contrary to your assertion, some times when killing Calibre and restarting the library goes in immediately.
Perhap the simplest question is what causes Calibre to read the actual book directories when opening a library?

[PeterT]

If I remember correctly, having a column that displays the formats that a book has will cause this to happen as well.

See this post for ideas..

[theducks]

Originally Posted by Mythlandia

Up to 4096 bytes are being read for each book directory in the library (sometimes) when swapping from one library toanother using Quick Switch.
Contrary to your assertion, some times when killing Calibre and restarting the library goes in immediately.
Perhap the simplest question is what causes Calibre to read the actual book directories when opening a library?

My first statement applies. You have a custom column that needs filesystem info. Why aborting speeds that up ???

[DoctorOhh]

Originally Posted by PeterT

If I remember correctly, having a column that displays the formats that a book has will cause this to happen as well.

See this post for ideas..

+1 Read this thread before proceeding.

[Mythlandia]

It appears that color coloring of columns based upon formats has the same bad effect as a custom column using formats. However, it appears (not proven) that when you start calibre it assumes the metadatabase is correct (at least some of the time) and does not force a recheck of the directories, thus when you abort and restart calibre it goes in quickly without a check of the directories; when you do a quick switch of libraries, it appears that directories are then sometimes (not always) forced to be rechecked.

The good news is that in the few minutes since I turned off column coloring, all the library switches have been measured in seconds, not minutes.
The bad news is that my library isn't as pretty...

Thanks theducks, PeterT and DoctorOhh for your help in solving this for me.

[chaley]

Originally Posted by Mythlandia

It appears that color coloring of columns based upon formats has the same bad effect as a custom column using formats. However, it appears (not proven) that when you start calibre it assumes the metadatabase is correct (at least some of the time) and does not force a recheck of the directories, thus when you abort and restart calibre it goes in quickly without a check of the directories; when you do a quick switch of libraries, it appears that directories are then sometimes (not always) forced to be rechecked.

The good news is that in the few minutes since I turned off column coloring, all the library switches have been measured in seconds, not minutes.
The bad news is that my library isn't as pretty...

Thanks theducks, PeterT and DoctorOhh for your help in solving this for me.

In principle, coloring should not cause the delays you are seeing. Of course, practice might be different, but I have looked at this code several times (I wrote it) and I don't see why it should be so.

My guess is that you have a custom column "built from other columns" in your library that shows formats, and that at some time in the past you sorted on that column. By default calibre remembers the last N sorts (IIRC N equals 5). If the formats column is in the last N, calibre will look at every book and start slowly. If it is not, then calibre will start much more quickly.

You can control this behavior explicitly by using the tweak "Specify columns to sort the booklist by on startup (ID: sort_columns_at_startup)". I think that the tweak is also applied when switching libraries.

[Mythlandia]

Charley - Just saw your November reply. At the time deleting the color coloring of the columns cured my problem (even though, from your answer it may just have been a side effect), but recently Calibre has taken to re-checking the disk on startup. I found your post and did your tweaks, which worked and I can now get into Calibre in about 25-30 seconds (with a 47000 book library).

The two values I set were:

"Specify columns to sort the booklist by on startup"
sort_columns_at_startup = [('authors', 0), ('title', 0)]

"Set the maximum number of sort 'levels'"
maximum_resort_levels = 2

[theducks]

Originally Posted by Mythlandia

Charley - Just saw your November reply. At the time deleting the color coloring of the columns cured my problem (even though, from your answer it may just have been a side effect), but recently Calibre has taken to re-checking the disk on startup. I found your post and did your tweaks, which worked and I can now get into Calibre in about 25-30 seconds (with a 47000 book library).

The two values I set were:

"Specify columns to sort the booklist by on startup"
sort_columns_at_startup = [('authors', 0), ('title', 0)]

"Set the maximum number of sort 'levels'"
maximum_resort_levels = 2

I don't remember seeing a stored (between sessions) Index for any 'Sort'

Calibre needs to generate a 'Startup sort' on the fly.

the sort needs to process 47K Authors with 47K Author entries.

If startup time is important... Don't sort until you need to.