calibre program launch speed

[cybmole]

well I discovered the preference setting that tells calibre to "exit" only to system tray, so when I reuse it during the day it returns instantly. 1 slowish start up per day is fine. ( less actually 'cos mostly I have the PC sleep overnight, like I do )

[Manichean]

Originally Posted by blastyblast

Manichean, I agree with the spirit of pollito pito's post. I think the community will benefit more from viewing concerns like this objectively, without defending why the program has issues.

Yes, you're right, although I stand by my opinion that Calibre, in its current state, isn't too slow for what it does.

By the way, I'm a software designer... is there anything I can do to help the project?

I tip my hat to you for jumping in on the first post and direct you to this page.

[chaley]

Originally Posted by blastyblast

By the way, I'm a software designer... is there anything I can do to help the project?

Sure. Lots of things. One would be to work out where in the SW your installation is spending its time.

I have test libraries of 2000 books, and have never seen a startup time longer than 30 seconds to the 'hi there' message (usually 20 after a reboot, usually 5-10 thereafter) and 20 seconds more to the GUI (usually 5 to 10). On my production library (1000 books) I see the similar numbers. These are eminently explainable, given the SW and data storage architecture.

The wild variance in reported timings are probably configuration specific (O/S, version, ulimit settings, caching setup on storage, network latencies, anti-virus, what-have-you). What these are and their effects are next to impossible to work out remotely. And in my case, not having the problem, I don't have much incentive to look for solutions.

I don't doubt that you are seeing something different. The question is why, a question you are best equipped to answer. The next question will be whether something can/should be done about 'it', but that question must wait until the reasons are well understood.

Edit: in reference to configuration-specific factors, one user reported dramatic performance improvements by defragging his disk. Thread is here.

Edit 2: Just to beat a dead horse: I created a 5000 book library by copying my 1000 book library 5 times without merging enabled. I checked to see that I indeed did have 5000 books. I rebooted to eliminate all caching behavior, waited for the machine was fully booted (dropbox done etc), then started calibre.

Time to splash: 9 seconds. Time to GUI running: 28 seconds. Reported startup time: 18.5 seconds, consistent with the 28 - 9.

Quit and immediately started again, to check cache effectiveness. Time to splash: 3 seconds. Time to start: 7 seconds. Reported startup time: 3.25 seconds.

This test demonstrates two things. 1) there is nothing fundamentally wrong, and 2) your mileage will vary.

Configuration: O/S: Win7 Pro x32, Mem: 4GB, Proc: Intel i5/750 at 2.6GHz (not overclocked), Disk: WD 500gb SATA II @ 7200rpm. Graphics, dvd, case, cooling, power, & network are irrelevant.

[Lady Fitzgerald]

I'm having better results. Running 0.7.19, it takes 18 seconds to start up to a 1248 e-book library after having done a reboot of my machine. After the first start, it only takes 10 seconds to start up. I have a total of three libraries right now: e-books (e-pub and PDF), Comics (232 of them), and Songbooks and Sheet Music (empty except for a test file). I'm running three different libraries because each one displays different columns and has different custom columns. The e-book library uses only the Title, Author, and Series columns. I also have a Sub-series, Unread, and On JBL custom columns. The comic library only uses the Title column (which includes the number of the comic). I use a custom Series column (to avoid the series number since I didn't need it; I got the name "Series" to take by putting a space in front of it which isn't noticeable in the column heading) and a custom Unread column. Songbooks and Sheet Music hasn't really been set up yet. It takes only 2 seconds to switch between libraries.

Curiously (and somewhat annoyingly) enough, on the 18 second start up, it takes 13 seconds before the splash screen shows up. Odd that.

[kacir]

Originally Posted by Manichean

Okay, either your computer is really, really fast, or mine is slow, or my programs are really slow, or the Office suite has improved it's raw performance dramatically.

Some programs, notably MSOffice, are set up by default to load large chunks of program into memory as the computer starts, so if you start the program 10 minutes after your PC has started it loads lightning fast.

[kacir]

Originally Posted by cybmole

I am concerned that if I dump in a collection of 3000+ sci fi that I found recently then I'd be looking at 5-6 MINUTES for start-up, which would be annoying.

It will slow down, but not that drastically.
I haven't timed the startup since I have added 3000 books at once, but it might be something like 20 seconds. I will time it in the evening.
By the way, I do not use antivirus on computer with my main Calibre Library. It runs on Mint Linux.

[Manichean]

Originally Posted by kacir

Some programs, notably MSOffice, are set up by default to load large chunks of program into memory as the computer starts, so if you start the program 10 minutes after your PC has started it loads lightning fast.

I thought so. OpenOffice, by it's default settings, tries to do the same thing.

[blastyblast]

Originally Posted by Manichean

Yes, you're right, although I stand by my opinion that Calibre, in its current state, isn't too slow for what it does.


I tip my hat to you for jumping in on the first post and direct you to this page.

I'm more familiar with Calibre now than I was when I made my first post... I completely agree with you, Calibre's speed is satisfactory. There are a few things I did to speed up performance on my computer (I'm sure I'll get some comments, like "those things are obvious", and they are to some degree, but hey at least I'm trying lol...).

One, I had my library on a network drive. But before you comment, know that I have approx. 30 terabytes of storage on three servers in my house. It's been a hobby for more than 10 years, and it's super high speed; BUT my focus has been on streaming media, like high def movies. Which means that it's not optimized for small files like ebooks. It's obvious in retrospect that Calibre will run slow when the library is stored on a network drive, but I've seen chatter from other folks about doing the same thing... Switching to a local drive cut initial load time in half, and saving changes by probably even more than that.

Two, I realized that I had more than twice the number of books in my library than I suggested in my first post. I forgot to count all of my books with 3 or 4 formats... So I just started clicking through and removing books in formats that will never get used. That took quite a while, but seems to have paid off as well.

Three, I took Chaley's advice (by proxy) and defragged my server and local drives... Not kidding when I say it ran for 3 days non stop. (30 TB takes a while lol).

And four, I seem to have found some extra patience, lol. I love the app so much that I really don't care how long it takes to load, although Calibre is significantly faster now that I've cleaned house... It loads >5,000 books in less than 15 seconds. Nice.

Thanks for the dialog folks, it helped.

[blastyblast]

Originally Posted by chaley

Sure. Lots of things. One would be to work out where in the SW your installation is spending its time.

Thanks for the reply and ideas, chaley, very helpful.

To clarify my interest in helping out, I do software UI design, not development. I'm afraid I'm pretty useless when it comes to programming. I'll keep checking the 'get involved' forums to see if I can be of value somehow.

[Manichean]

Originally Posted by blastyblast

Two, I realized that I had more than twice the number of books in my library than I suggested in my first post. I forgot to count all of my books with 3 or 4 formats... So I just started clicking through and removing books in formats that will never get used. That took quite a while, but seems to have paid off as well.

Just as a usage hint: You could do this by searching for "formats:X", wherein X would be the format you want to remove from your library, and then remove all occurences of this format at once using the bulk edit metadata feature.

[chaley]

Originally Posted by blastyblast

(I'm sure I'll get some comments, like "those things are obvious", and they are to some degree, but hey at least I'm trying lol...).

Its obvious that you will get comments that its obvious that you will get comments that it is obvious that (stack overflow!)

[...]Switching to a local drive cut initial load time in half, and saving changes by probably even more than that.

One of the things that Kovid and I discovered some time back while doing work on bulk edit was that SQLite creates a file per transaction to keep its journals. On Kovid's linux installation, this didn't seem to matter. On my Win7 installation, life was not good. We got a 2 order of magnitude improvement in bulk edit performance by grouping transactions. Evidence pointed toward rapid file creation/deletion being very expensive on NTFS (Windows journalling file system), and this can't help but be worse over a network.

So I just started clicking through and removing books in formats that will never get used. That took quite a while, but seems to have paid off as well.

This one surprises me. The only thing in calibre's startup that looks at formats is building the tag browser format category, and that is a single query on a view. Calibre doesn't look at the actual files until you ask it to do something to one. Do you have an idea how much this cleanup helped?

Three, I took Chaley's advice (by proxy) and defragged my server and local drives... Not kidding when I say it ran for 3 days non stop. (30 TB takes a while lol).

Do you have before and after times? The reason I ask is that suggesting defragging has occasionally met with derision. It would be nice to know what level of difference it made in your case.

And four, I seem to have found some extra patience, lol. I love the app so much that I really don't care how long it takes to load, although Calibre is significantly faster now that I've cleaned house... It loads >5,000 books in less than 15 seconds. Nice.

Going from 120-180 seconds to 15 seconds is good work. It is hard not to like factors of 10.

Thanks for the dialog folks, it helped.

Thank you for taking the time to test, and (even more) taking the time to report.

[itimpi]

One thing that can be done when using network drives is to have the metadata.db file held locally even if the books are held on the network drive. This provides a good compromise between performance and having the main library on the location with lots of space.

It does mean that you may want to periodically copy the metadata.db file to the location of the books folder, but this is a relatively small file to have to copy.

[blastyblast]

Originally Posted by Manichean

Just as a usage hint: You could do this by searching for "formats:X", wherein X would be the format you want to remove from your library, and then remove all occurences of this format at once using the bulk edit metadata feature.

I started out doing that, however in most cases I had to take some time figure out which format was the "most correct" version before I deleted anything.

[blastyblast]

Originally Posted by chaley

Its obvious that you will get comments that its obvious that you will get comments that it is obvious that (stack overflow!)

Lol...

(to my statement on cleaning out books...) This one surprises me. The only thing in calibre's startup that looks at formats is building the tag browser format category, and that is a single query on a view. Calibre doesn't look at the actual files until you ask it to do something to one. Do you have an idea how much this cleanup helped?

Thanks for clarifying... To your point, this particular action may not have had any impact at all on load speed, I'm afraid I didn't do a very good job of doing before/after comparisons with each action I took. I cleaned out ~2,000 out of 6,000 books, some individually, some through format lookup. I'll try to pay more attention to how usage patterns and optimization efforts impact performance for now on. However, cleaning up formats *seemed* to impact search speeds while using calibre. Although my perception here is extremely subjective... I suppose it could have been the same part of my brain that makes me think my car runs better after it gets washed

(to my comments about defragging...) Do you have before and after times? The reason I ask is that suggesting defragging has occasionally met with derision. It would be nice to know what level of difference it made in your case.

Suggesting defragging should not be met with derision out of hand, for a few reasons. There is a really arbitrary urban myth that NFTS-based OS's don't need to be defragged, so if you're running Vista, Windows 7, or even Windows Home Server, you might not think about defragging often. I'm not sure why, but I think this view was probably founded on the misconception that defragging primarily eliminates what is called "free space fragmentation", which some folks might not think is necessary with the massive hard drives available to use today. This is quite wrong. Free space fragmentation on it's own is a very big cause of sluggish read/write operations on massive hard drives. But defragging can also improves file fragementation, directory fragmentation, metadata/MFT fragmentation. In my case, for instance, I haven't had defragging my servers at the top of my mind for the last 6 to 9 months, so when I checked the other day I found that I had from 3 to 6% free space fragmentation on many of the 2 TB drives on my server, but I had as high as 15% to 42% fragmentation in the directories, and metadata/MFTs of almost all the drives. Without getting too technical here, let's say you're moving very small files around quite a bit, AND you're changing names, tags, etc. Over time you will notice a gradual performance tax if not only the files themselves are becoming fragmented, but the "compounded issue" of having fragmented directories and metadata that the hard drive uses to point to these files.

That said, the noticable impact of defragging with be like testing for low blood sugar. You only get a good read when you're testing at the right time, under the right circumstances. But, regardless of what you observe, the read/write heads on your HDs will thank you (and possibly live longer) for making their job easier.

DISCLAIMER: DO NOT, EVER, EVER... use the standard Microsoft dafragging tool on Windows Home Server. Don't log in through remote desktop and start defragging drives this way. You need to use a specialized defragging tool (like DiskKeeper or PerfectDisk) that knows how to work with shadow copies and other things that make WHS different from your desktop,

Going from 120-180 seconds to 15 seconds is good work. It is hard not to like factors of 10.

Which reminds me, I did one other very important thing to speed up performance. I rebuilt the search index on both my local PC and the server. Before doing that, however, I checked the search options and found that several file types were not even being indexed. Such as epub and mobi. That's a problem because Windows had to find those files over again every time a search was performed. But also, over time Windows Search builds up artifacts, like bad links to files that were deleted long ago... Rebuilding the index had a tremendous impact on how fast searches were taking in Explorer. Prior to rebuilding some searches were instant, but in other cases I would often sit there for several minutes waiting for Windows to deliver results. Now, most searches are instant. Calibre seems a bit snappier too. My *theory* is that despite the fact that Calibre doesn't touch Windows Search, Windows is working to index the directories (libraries) that Calibre uses to store books and other files; which is normal, but (again my theory) without certain file extensions being added to the index, Windows 7 seems to want to rediscover them every time you browse your library. Just an observation. [/QUOTE]

Thank you for taking the time to test, and (even more) taking the time to report.

Absolutely my pleasure. Let me say again how much I appreciate this application and its developers.

[speakingtohe]

My startup is pretty long more than a minute at the best of times and up to 5 minutes if I start it immediately after rebooting. I have 8000+ books and a lot of formats though. 15 gig and 44.245 files so one can't wonder.

My cookbook library takes a very short time with less than 400 titles

Helen