Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 11-26-2012, 02:28 PM   #1
Mythlandia
Reader
Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.
 
Posts: 13
Karma: 6184
Join Date: Jun 2009
Location: Celebration, FL
Device: iPad
Stopping Calibre rechecking library metadata

I have a number of different Calibre libraries which I swap between.
Every few times I swap libraries Calibre will decide to reload info from the directories & metadata (at least it reads thousands of files). This takes ten minutes to an hour depending upon the size of the library. Sometimes I will lose patience, kill Calibre, go back in and the same library opens immediately.

Is there anyway to get Calibre just to trust it's own data?

The 45mb metadata database could be read in less than a second, but the actual library is 65gb and just touching the metadata files means reading gigabytes of data.
Mythlandia is offline   Reply With Quote
Old 11-26-2012, 03:16 PM   #2
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,860
Karma: 5654321
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Not Normal (unless you have custom column (composite or coloring) that requires file info).
Calibre wants will backup its metadata in each folder. That happens once, then only if the book data changes in the DB.
Kill that process and it will start over where it thinks it was. :footshoot:
theducks is online now   Reply With Quote
Old 11-26-2012, 07:36 PM   #3
Mythlandia
Reader
Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.
 
Posts: 13
Karma: 6184
Join Date: Jun 2009
Location: Celebration, FL
Device: iPad
Up to 4096 bytes are being read for each book directory in the library (sometimes) when swapping from one library toanother using Quick Switch.
Contrary to your assertion, some times when killing Calibre and restarting the library goes in immediately.
Perhap the simplest question is what causes Calibre to read the actual book directories when opening a library?
Mythlandia is offline   Reply With Quote
Old 11-26-2012, 08:37 PM   #4
PeterT
Taking a break; Fed up
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 6,982
Karma: 44005669
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
If I remember correctly, having a column that displays the formats that a book has will cause this to happen as well.

See this post for ideas..
PeterT is offline   Reply With Quote
Old 11-26-2012, 08:55 PM   #5
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,860
Karma: 5654321
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by Mythlandia View Post
Up to 4096 bytes are being read for each book directory in the library (sometimes) when swapping from one library toanother using Quick Switch.
Contrary to your assertion, some times when killing Calibre and restarting the library goes in immediately.
Perhap the simplest question is what causes Calibre to read the actual book directories when opening a library?
My first statement applies. You have a custom column that needs filesystem info. Why aborting speeds that up ???
theducks is online now   Reply With Quote
Old 11-26-2012, 09:13 PM   #6
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,861
Karma: 12755553
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by PeterT View Post
If I remember correctly, having a column that displays the formats that a book has will cause this to happen as well.

See this post for ideas..
+1 Read this thread before proceeding.
DoctorOhh is offline   Reply With Quote
Old 11-27-2012, 04:30 PM   #7
Mythlandia
Reader
Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.
 
Posts: 13
Karma: 6184
Join Date: Jun 2009
Location: Celebration, FL
Device: iPad
It appears that color coloring of columns based upon formats has the same bad effect as a custom column using formats. However, it appears (not proven) that when you start calibre it assumes the metadatabase is correct (at least some of the time) and does not force a recheck of the directories, thus when you abort and restart calibre it goes in quickly without a check of the directories; when you do a quick switch of libraries, it appears that directories are then sometimes (not always) forced to be rechecked.

The good news is that in the few minutes since I turned off column coloring, all the library switches have been measured in seconds, not minutes.
The bad news is that my library isn't as pretty...

Thanks theducks, PeterT and DoctorOhh for your help in solving this for me.
Mythlandia is offline   Reply With Quote
Old 11-27-2012, 05:13 PM   #8
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,637
Karma: 1137414
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by Mythlandia View Post
It appears that color coloring of columns based upon formats has the same bad effect as a custom column using formats. However, it appears (not proven) that when you start calibre it assumes the metadatabase is correct (at least some of the time) and does not force a recheck of the directories, thus when you abort and restart calibre it goes in quickly without a check of the directories; when you do a quick switch of libraries, it appears that directories are then sometimes (not always) forced to be rechecked.

The good news is that in the few minutes since I turned off column coloring, all the library switches have been measured in seconds, not minutes.
The bad news is that my library isn't as pretty...

Thanks theducks, PeterT and DoctorOhh for your help in solving this for me.
In principle, coloring should not cause the delays you are seeing. Of course, practice might be different, but I have looked at this code several times (I wrote it) and I don't see why it should be so.

My guess is that you have a custom column "built from other columns" in your library that shows formats, and that at some time in the past you sorted on that column. By default calibre remembers the last N sorts (IIRC N equals 5). If the formats column is in the last N, calibre will look at every book and start slowly. If it is not, then calibre will start much more quickly.

You can control this behavior explicitly by using the tweak "Specify columns to sort the booklist by on startup (ID: sort_columns_at_startup)". I think that the tweak is also applied when switching libraries.
chaley is offline   Reply With Quote
Old 02-28-2013, 01:28 PM   #9
Mythlandia
Reader
Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.Mythlandia got an A in P-Chem.
 
Posts: 13
Karma: 6184
Join Date: Jun 2009
Location: Celebration, FL
Device: iPad
Charley - Just saw your November reply. At the time deleting the color coloring of the columns cured my problem (even though, from your answer it may just have been a side effect), but recently Calibre has taken to re-checking the disk on startup. I found your post and did your tweaks, which worked and I can now get into Calibre in about 25-30 seconds (with a 47000 book library).

The two values I set were:

"Specify columns to sort the booklist by on startup"
sort_columns_at_startup = [('authors', 0), ('title', 0)]

"Set the maximum number of sort 'levels'"
maximum_resort_levels = 2
Mythlandia is offline   Reply With Quote
Old 02-28-2013, 02:58 PM   #10
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,860
Karma: 5654321
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by Mythlandia View Post
Charley - Just saw your November reply. At the time deleting the color coloring of the columns cured my problem (even though, from your answer it may just have been a side effect), but recently Calibre has taken to re-checking the disk on startup. I found your post and did your tweaks, which worked and I can now get into Calibre in about 25-30 seconds (with a 47000 book library).

The two values I set were:

"Specify columns to sort the booklist by on startup"
sort_columns_at_startup = [('authors', 0), ('title', 0)]

"Set the maximum number of sort 'levels'"
maximum_resort_levels = 2
I don't remember seeing a stored (between sessions) Index for any 'Sort'

Calibre needs to generate a 'Startup sort' on the fly.

the sort needs to process 47K Authors with 47K Author entries.

If startup time is important... Don't sort until you need to.
theducks is online now   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Stopping Calibre from checking when device=hard drive Taka B. Library Management 13 01-06-2012 01:34 PM
Differences between library and reader: Force calibre to write metadata into ebook durtreg Calibre 17 01-28-2011 04:30 AM
Stopping Calibre from resorting source docs MacEachaidh Calibre 11 05-13-2010 03:11 PM
eBook Library (Mac) and metadata Samcxjo Sony Reader 2 10-12-2009 05:26 PM
Vanishing Changes in Library Metadata rasbadar Calibre 7 09-15-2008 12:04 PM


All times are GMT -4. The time now is 03:00 PM.


MobileRead.com is a privately owned, operated and funded community.