Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 09-18-2012, 10:10 AM   #16
electronicfur
Enthusiast
electronicfur is on a distinguished road
 
Posts: 26
Karma: 50
Join Date: Aug 2011
Device: Samsung Note FBReader, N900 FBReader, Nook Simple Touch (Kindle3 died)
Quote:
Originally Posted by chaley View Post
The only code I can find in calibre that looks at the attributes of a format file is called when a template referring to a format is evaluated. Are you certain that you don't have a hidden composite column, and in particular a "formats" column? Or perhaps you have installed a plugin that walks through your books?

Copying a library folder will keep many of the preferences, and in particular all the custom columns, hidden or not. There are a large number of preferences stored outside the db.
I was just coming on to post that I figured out the differences between my two test libraries having been pouring through all the preference settings.

But you beat me to it, thank you.

There was a hidden formats column, as you guessed. The problem was that it was hidden, and also that it had "show in tags browser" ticked. So this caused the reading of all file attributes in the library. Even if the tag browser is hidden.

Thanks for all the help.
electronicfur is offline   Reply With Quote
Old 09-18-2012, 10:43 AM   #17
electronicfur
Enthusiast
electronicfur is on a distinguished road
 
Posts: 26
Karma: 50
Join Date: Aug 2011
Device: Samsung Note FBReader, N900 FBReader, Nook Simple Touch (Kindle3 died)
It may be worth adding a warning about this on the "Create a custom columns" screen, especially as Formats is a "Quick create" template.

The Formats column causing the reading of every file in the library at startup really makes a massive difference. With 10,915 books my startup times have been about 4 minutes. Now it starts up in seconds.

It also seems to make a big difference editing metadata. I'm guessing because after editing it's refreshing part of the list, and so goes and reads a bunch of files again.

I think I added this column to my library right at the start but subsequently hid it. To be honest I forgot I had done so, and assumed it was just a default Calibre column when I was looking for custom columns today

So ever since using Calibre it really has given me a false impression as I've just assumed that Calibre was very slow starting up and slow editing metadata.

Now it seems like a speed demon
electronicfur is offline   Reply With Quote
Old 09-18-2012, 10:52 AM   #18
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
@chaley - you are probably already considering it but in you want prompting - would you consider changing the calls behind the formats column so it doesn't touch the disk and instead just uses the formats calibre has recorded in the database for each book? I too have a formats column as it is a darned handy one to have. I don't have enormous startup times by any means, but if I can shave time off without losing functionality that sounds a win to me.

BTW - you are probably also aware that the worst possible thing you can do in a large calibre library is click on the column heading of a formats column to sort it - it takes "forever".
kiwidude is offline   Reply With Quote
Old 09-18-2012, 10:54 AM   #19
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,861
Karma: 12755553
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by electronicfur View Post
I think I added this column to my library right at the start but subsequently hid it. To be honest I forgot I had done so, and assumed it was just a default Calibre column when I was looking for custom columns today
When I saw you were still in denial (post 13) responding to chaley's mention of the "Formats" column as if it didn't apply, I decided to attach the default columns as an image so you could see the real default columns.

I'm glad you got everything straightened out.
DoctorOhh is offline   Reply With Quote
Old 09-18-2012, 11:24 AM   #20
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,639
Karma: 1137414
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by kiwidude View Post
@chaley - you are probably already considering it but in you want prompting - would you consider changing the calls behind the formats column so it doesn't touch the disk and instead just uses the formats calibre has recorded in the database for each book? I too have a formats column as it is a darned handy one to have. I don't have enormous startup times by any means, but if I can shave time off without losing functionality that sounds a win to me.
Actually I am not considering it.

The problem is that the DB can be wrong. When getting the definitive list of formats, db2 checks that the files are really there. This code was added at some point to fix a user-reported bug. Later the code was added to delay evaluation of the check until the formats info is actually used.

If the existence check is removed, then it would be very possible that mi.formats would contain items that do not really exist. This isn't something I want to do. If someone else wants to make such a change, and assuming Kovid would accept the change, then well and good. That person would own the consequences.
chaley is offline   Reply With Quote
Old 09-18-2012, 11:32 AM   #21
electronicfur
Enthusiast
electronicfur is on a distinguished road
 
Posts: 26
Karma: 50
Join Date: Aug 2011
Device: Samsung Note FBReader, N900 FBReader, Nook Simple Touch (Kindle3 died)
Quote:
Originally Posted by DoctorOhh View Post
When I saw you were still in denial (post 13) responding to chaley's mention of the "Formats" column as if it didn't apply, I decided to attach the default columns as an image so you could see the real default columns.

I'm glad you got everything straightened out.
Yes, much appreciated DoctorOhh. I was too busy diff-ing other setting, files, etc and totally missed the formats column being a custom column first time round

Spent about 4 hours today trying to track it all down, as I finally cracked and couldnt handle the long startup times anymore, and I think it was a case of not seeing the wood for the trees!

Also I just noticed that Formats is shown in the tag browser regardless of that column being present, so as Kiwidude suggests, the data is clearly available without hitting the disk. It also makes the "show in tag browser" somewhat redundant for that column. And it gives me a handy excuse for not realising Formats was a custom column

Cheers,
EF
electronicfur is offline   Reply With Quote
Old 09-18-2012, 11:34 AM   #22
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
@chaley - but does the formats formula actually have to get its values from the mi object? I've not looked into the code one iota around this btw, just speculating out loud and presuming your formula parsing at some point translates the word "formats" into some function call?

I'm just wondering whether a special case can be made of it, since I would suggest it is one of the most desirable custom columns to have (for users who store more than one format) but can have such a negative startup time impact. I'm aware of the possibilities of a user deleting stuff out of the library folders directly getting it out of sync, but if it offers the possibility of a fast startup to reward those who don't do anything so stupid that gets my vote
kiwidude is offline   Reply With Quote
Old 09-18-2012, 12:11 PM   #23
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,639
Karma: 1137414
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by kiwidude View Post
@chaley - but does the formats formula actually have to get its values from the mi object? I've not looked into the code one iota around this btw, just speculating out loud and presuming your formula parsing at some point translates the word "formats" into some function call?
The basic list of formats comes from the DB. The accuracy of that list for a given book is checked when you refer to the list in a template such as "{formats}" or use one of the formatter functions to get info about the formats.
Quote:
I'm just wondering whether a special case can be made of it, since I would suggest it is one of the most desirable custom columns to have (for users who store more than one format) but can have such a negative startup time impact. I'm aware of the possibilities of a user deleting stuff out of the library folders directly getting it out of sync, but if it offers the possibility of a fast startup to reward those who don't do anything so stupid that gets my vote
The problem is that people forget promises "not to do anything stupid" and they file a bug report saying that the list is wrong. At that point one of the two of us needs to take 15 to 30 minutes to figure out what happened and reply, and even longer if the necessary info isn't in the report.

That said, the change isn't hard. If Kovid agrees, I can make the change. I will ask him. However, I really don't want to own the certain-to-happen bug reports, and I suspect that he won't want to either.
chaley is offline   Reply With Quote
Old 09-18-2012, 12:14 PM   #24
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,106
Karma: 5168844
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
One serious problem with not scanning the disc is that it may delay the discovery that all or parts of the library is missing.

Perhaps it could be a tweak? Disable check of existing book formats on boot. But perhaps add code to do the check like when metadata is backed up in the background to the folders?
Adoby is offline   Reply With Quote
Old 09-18-2012, 01:02 PM   #25
electronicfur
Enthusiast
electronicfur is on a distinguished road
 
Posts: 26
Karma: 50
Join Date: Aug 2011
Device: Samsung Note FBReader, N900 FBReader, Nook Simple Touch (Kindle3 died)
Quote:
Originally Posted by chaley View Post
The problem is that people forget promises "not to do anything stupid" and they file a bug report saying that the list is wrong. At that point one of the two of us needs to take 15 to 30 minutes to figure out what happened and reply, and even longer if the necessary info isn't in the report.
Isn't there already an issue now as things stand currently? Because the tag browser has Formats in it as standard, which are retrieved from the database, and so this info would be out of sync as well if a user has manually deleted files.

As a user it doesnt make much sense to me to have the tag browser using Formats data from the database, but the Formats column retrieving data from the file system.

I primarily used the format column when I was organising my library, so handy in that circumstance. Personally I can live without it, but am more concerned that other users will also unknowingly slow down their Calibre by having the column.
electronicfur is offline   Reply With Quote
Old 09-18-2012, 01:32 PM   #26
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,639
Karma: 1137414
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by electronicfur View Post
Isn't there already an issue now as things stand currently? Because the tag browser has Formats in it as standard, which are retrieved from the database, and so this info would be out of sync as well if a user has manually deleted files.

As a user it doesnt make much sense to me to have the tag browser using Formats data from the database, but the Formats column retrieving data from the file system.

I primarily used the format column when I was organising my library, so handy in that circumstance. Personally I can live without it, but am more concerned that other users will also unknowingly slow down their Calibre by having the column.
The formats category is aggregated. It is right if any book has that format, not if a particular book has the format.

The column slows down calibre if you have a very large library and you do what you did (check show in tag browser) or sort on it. If it is 'just there' it has almost no impact.

My position is that I don't want to spend time on something that gives me negative benefit (extra support load). My 2500 book library doesn't have the problem, and in any event I don't have a formats column. Someone else who might benefit is welcome to take this on.
chaley is offline   Reply With Quote
Old 09-18-2012, 02:03 PM   #27
Ripplinger
350 Hoarder
Ripplinger ought to be getting tired of karma fortunes by now.Ripplinger ought to be getting tired of karma fortunes by now.Ripplinger ought to be getting tired of karma fortunes by now.Ripplinger ought to be getting tired of karma fortunes by now.Ripplinger ought to be getting tired of karma fortunes by now.Ripplinger ought to be getting tired of karma fortunes by now.Ripplinger ought to be getting tired of karma fortunes by now.Ripplinger ought to be getting tired of karma fortunes by now.Ripplinger ought to be getting tired of karma fortunes by now.Ripplinger ought to be getting tired of karma fortunes by now.Ripplinger ought to be getting tired of karma fortunes by now.
 
Ripplinger's Avatar
 
Posts: 2,281
Karma: 2299170
Join Date: Dec 2010
Location: Midwest USA
Device: Sony PRS-350: 3 Silver, 2 Blue!; Kobo Glo
I remember another post about this quite awhile ago. What helped me a lot was to add Calibre to the trusted list of my AV program. After doing that, the load time is negligible for me, no more than any other program starting up. There will always be a bit of a delay collecting all the info and various formats of your library, but this made it like 20x faster than what it used to be for me.
Ripplinger is offline   Reply With Quote
Old 09-18-2012, 02:35 PM   #28
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,639
Karma: 1137414
Join Date: Jan 2010
Location: France
Device: Many android devices
After discussion with Kovid, we have decided to leave {formats} as it is, but also to add a function "approximate_formats" that returns the list of formats from the db. This function can be referenced in template program mode using the template
Code:
{:'approximate_formats()'}
The change should be in trunk tomorrow and released this week.
chaley is offline   Reply With Quote
Old 09-18-2012, 03:03 PM   #29
electronicfur
Enthusiast
electronicfur is on a distinguished road
 
Posts: 26
Karma: 50
Join Date: Aug 2011
Device: Samsung Note FBReader, N900 FBReader, Nook Simple Touch (Kindle3 died)
Cool, I was just thinking about suggesting an additional function.

Quote:
Originally Posted by chaley View Post
The column slows down calibre if you have a very large library and you do what you did (check show in tag browser) or sort on it. If it is 'just there' it has almost no impact.
Also just to clarify for your info, if the formats column is "just there" showing, then it does causes every file in the library to be accessed on startup and has the impact I described.

I've tested all the cases pretty extensively today, so I'm pretty sure the above is the case.

If the formats column is hidden on the other hand (ie unticked but still present in the "add your own columns" screen), then not every file is accessed, unless the "show in tag browser" option is checked. With that "show in tag browser" option checked it then also causes every file in the library to be accessed.

Cheers,
EF
electronicfur is offline   Reply With Quote
Old 09-18-2012, 03:11 PM   #30
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Thanks chaley. I had hoped there might be some sort of compromise
kiwidude is offline   Reply With Quote
Reply

Tags
open slow

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre Portable - Which Library Opens? nynaevelan Calibre 5 09-21-2011 12:58 PM
Best way to get a large Calibre library into the PE library? Filark enTourage Archive 0 04-20-2011 10:18 PM
Kindle and Calibre user with problem importing large library into Calibre pleabargain Calibre 1 12-07-2010 10:19 AM
Large library/calibre interface StanByk Calibre 12 10-19-2009 10:20 AM
How large is your Calibre library? radius Calibre 28 10-12-2009 01:14 AM


All times are GMT -4. The time now is 04:33 PM.


MobileRead.com is a privately owned, operated and funded community.