[Old Thread] Large collection problem

[cybmole]

Originally Posted by jbcohen

I believe that the way Calibre does its cataloging is via Random Access Memory or ROM for short.

it's RAM for short. ROM is short for read only memory

I only have 1,200 entires in my main library but I see no noticeable RAM usage change when closing/ launching calibre .it only takes 91MB RAM once loaded -

but if scaling is linear then 12,000 entries could take 910MB; 20,000+ entries will need ~2GB or will start to use page file ?

[oventura]

I have done some tests with your answers:
- I have deleted the calculated columns -> I got no difference
- I checked the use of CPU and RAM during the opening but they are not the problems. The CPU just go until 20%, and it only uses about 300MB of RAM (It leaves 900MB of physical RAM free).
I will try a defrag of the hard drive

[kiwidude]

Noooooo.... calibre does *not* need 2GB of ram for 20000 titles.

I've got a library here with 33K titles in it that takes under 250kb and another of 10K which is about 220kb. Both start up easily under 15 seconds. When this issue has come up in the past it has been because of:
(1) disk speed / fragmentation
(2) custom columns
(3) having the tag browser visible at startup

I know chaley has done a lot of work at tuning calibre at various points and for most people it has made a massive difference (certainly did for me) so some of those issues are not as bad as they were in the past.

[theducks]

Originally Posted by kiwidude

Noooooo.... calibre does *not* need 2GB of ram for 20000 titles.

I've got a library here with 33K titles in it that takes under 250kb and another of 10K which is about 220kb. Both start up easily under 15 seconds. When this issue has come up in the past it has been because of:
(1) disk speed / fragmentation
(2) custom columns
(3) having the tag browser visible at startup

I know chaley has done a lot of work at tuning calibre at various points and for most people it has made a massive difference (certainly did for me) so some of those issues are not as bad as they were in the past.

And let us not forget the big CPU hog: Antivirus scanning of all files being touched at startup.

[PeterT]

Originally Posted by theducks

And let us not forget the big CPU hog: Antivirus scanning of all files being touched at startup.

Actually, I don't think many files get touched at startup. I just used ProcMon from SysInternals to monitor and besides the access of the programs executable files, the majority of all accesses were to the metadata.db file; in my case only a few (13 books) seemed to have covers accessed and none of the actual ePub files.

[RolandD]

Leave calibre open.

[oventura]

After the defrag the problem persists.
I have been doing some more test and I have seen that after reboot the PC when I open calibre it takes a lot of time. But if I close Caliber and open again it opens in just one minute.
I have used the system tools of windows 7 and I have detected that when it takes a lot of time, the process calibre.exe is accessing to all the directories and subdirectories of the collection (about 20.000 directories)
But when it opens quickly it doesn't access to any of those directories.
Why calibre needs to access to all the directories? Is any way of avoiding it?
Thanks!

[PeterT]

That is strange... I see no sign at all of my collection being scanned on startup.

[BetterRed]

Could it be that the library disk has gone into power save mode ?

BTW running defrag on W7 isn't really necessary. W7 will defrag drives when your system isn't busy, unless you've deliberately disabled it. Running 2 Pay4Me defraggers on my W7 system makes not a scrap of difference; whereas on my old XP box they made a lot of difference.

BR

[oventura]

Originally Posted by BetterRed

Could it be that the library disk has gone into power save mode ?

BTW running defrag on W7 isn't really necessary. W7 will defrag drives when your system isn't busy, unless you've deliberately disabled it. Running 2 Pay4Me defraggers on my W7 system makes not a scrap of difference; whereas on my old XP box they made a lot of difference.

BR

The defrag is done. I have tested the same in a different computer and I have the same problem

[DoctorOhh]

Originally Posted by oventura

The defrag is done. I have tested the same in a different computer and I have the same problem

Ensure that the tag browser is closed prior to exiting calibre. That way the open tag browser doesn't slow down the startup of calibre.

Be aware of sorting on composite columns. Composite columns may touch every item/book during startup to populate the custom columns. If a composite column is also expected to be sorted on startup you could have delays in starting calibre.

Ensure that you exempt your library from your A/V program or the fact that the composite column above touching every book folder with A/V will cause an even longer delay.

If your library is running from a NAS or another machine this could slow startup.

If you are running from a USB drive there could be some slowing on startup.

These are off the top of my head, I'm sure there are other tips folks will post.

Good Luck.