Is Calibre slowing down?

[hidden.platypus]

@chaley

but I would be surprised if 2GB is enough for a library that size.

I borrowed a larger library than mine because that statement intrigued me. Why would the size of the calibre library have an effect on how much RAM calibre uses? I used calibre and copied the borrowed library and it made no noticeable difference to RAM usage.

Just curious what that meant.

[chaley]

Originally Posted by hidden.platypus

@chaley



I borrowed a larger library than mine because that statement intrigued me. Why would the size of the calibre library have an effect on how much RAM calibre uses? I used calibre and copied the borrowed library and it made no noticeable difference to RAM usage.

Just curious what that meant.

Because calibre reads the entire database into memory when it starts and processes it from there. The database is used mainly as storage.

[DNSB]

Originally Posted by chaley

Because calibre reads the entire database into memory when it starts and processes it from there. The database is used mainly as storage.

Not sure if I am looking at the right file, but for my library which runs about 2GB of epubs, the metadata.db file is around 4.8MB. I can't see that making much of a difference on most computers.

Regards,
David

[hidden.platypus]

Originally posted by chaley
Because calibre reads the entire database into memory when it starts and processes it from there. The database is used mainly as storage.

@chaley
I tried to replicate this, but even with my borrowed library I can't get Calibre to show a meaningful increase in RAM usage.

@DNSB

Not sure if you're looking at the right file, or making a different point. chaley's argument seems to be that the greater the amount the files in your calibre library the more RAM calibre will need. My personal library is only 300 books large. I borrowed one that has 800 titles, each with duplicate formats. Haven't seen any noticeable RAM increase.

[BetterRed]

I just flipped between a few libraries - Private/Working Set/Virtual were

KB 141,976/165.836/350,576 - 0 books
KB 197,474/223,552/409,800 - 3481 books
KB 246,324/271,232/459.400 - 7562 books

If I go back to the 0 books library its 222,832/247,192/458,376

I draw no conclusions or inferences from that.

I know people who are running calibre libraries of up to a couple of thousand books on 2G Vista laptops - sure its not fast, nor is watching a slideshow of 50Mp images of their cat - but they're happy with both

BR

[Hoods7070]

I don't think it's any of the above. I have done everything suggested in this thread, to no avail. I even tried uninstalling the latest version of #2 - going back as far as #1 - no appreciable difference. I checked all drivers, blah blah. I've been working with PCs forever (since pre-DOS) so I know my way around and rarely get defeated by a system issue. I HATE that I can't figure this out. Everything else runs brilliantly on the laptop - multitasking with huge Photoshop files, word processors etc. without a hint of slowness. I can only conclude it's Calibre, and apparently Calibre only, not liking SOMETHING about my system.

This following bit should probably be in a separate thread, and I will probably duplicate it. At this stage I feel darned near suicidal, LOL. Not that suicide is a laughing matter, of course. Calibre is just behaving extremely strangely. My usual workflow is:

Enter the data I want, either from the booklist of the "info" screen. This comprises nothing but Series and Country of the Author. I also check the publication date for the original release date. Anal huh??

Next, run Epub modify, to insert a jacket into the book.

Run Quality Check to swap Author FN/LN

Delete epub original files.

I have been following that same pattern for years!

So, today, I move -- painstakingly and VERY, VERY slowly, a few at a time so I don't rip what remains of my hair out -- all books missing tags in the booklist to a temporary library. (Why were they missing in the first place, I ask?? I figured it was just some aberration/my fault/whatever.) Then I painstakingly and VERY, VERY slowly correct the booklist data for all the books that had mysteriously lost their tags and comments and move them back to my main library. About 500 books -- it took me all day because of aforementioned speed issues. Interestingly, things did NOT appreciably speed up as the number of books in the temp library dwindled. The last move of about 25 books took as long as the first one. Possibly because the holdup is at the other end, in the main library.

11:30 tonight I finished and switched to the main library only to find that every single book I modified today had no tags, no comments. GEEZ. I truly could cry. In fact I think I will. I couldn't duplicate this on my PC. I give up.

[mrmikel]

If you are running the latest version, Calibre has shifted to QT5. Could that be the issue?

[DNSB]

Originally Posted by hidden.platypus

@chaley
I tried to replicate this, but even with my borrowed library I can't get Calibre to show a meaningful increase in RAM usage.

@DNSB

Not sure if you're looking at the right file, or making a different point. chaley's argument seems to be that the greater the amount the files in your calibre library the more RAM calibre will need. My personal library is only 300 books large. I borrowed one that has 800 titles, each with duplicate formats. Haven't seen any noticeable RAM increase.

The file I was looking at is the database.db file that, AFAIK, is the only file loaded into memory when Calibre starts up. My point was that with >2000 books, the database is less than 5MB. My point was that with such a small file (well, by today's standards -- I remember using 8" single density floppy disks -- it's memory usage is going to be lost in the round off errors calculating memory use.

Regards,
David

[chaley]

Originally Posted by DNSB

The file I was looking at is the database.db file that, AFAIK, is the only file loaded into memory when Calibre starts up. My point was that with >2000 books, the database is less than 5MB. My point was that with such a small file (well, by today's standards -- I remember using 8" single density floppy disks -- it's memory usage is going to be lost in the round off errors calculating memory use.

Regards,
David

Calibre doesn't simply read the file into memory. Instead it builds a large number of data structures that increase memory usage by orders of magnitude. I have personal experience of a laptop with 1.5GB of RAM thrashing when I opened my 2400 book calibre library, taking minutes to start. A 200 book test library opened in seconds. Of course calibre wasn't the only thing running, but then calibre is *never* the only thing running. I ended up replacing that laptop. YMMV.

[DNSB]

Originally Posted by chaley

Calibre doesn't simply read the file into memory. Instead it builds a large number of data structures that increase memory usage by orders of magnitude. I have personal experience of a laptop with 1.5GB of RAM thrashing when I opened my 2400 book calibre library, taking minutes to start. A 200 book test library opened in seconds. Of course calibre wasn't the only thing running, but then calibre is *never* the only thing running. I ended up replacing that laptop. YMMV.

Just opened Calibre with my standard library, took about 2 seconds to open (i7-3820QM@2.7GHz laptop, 32GB RAM, Samsung 840 EVO 500GB Win7 x64). I use this laptop to do quite a bit of virtual machine work so needed the RAM and HD space.

I created a new empty library, closed Calibre and re-opened.

Old: 142,316KB memory, 50MB IO Read, 36KB IO write, 518KB IO Other

New: 104,372KB memory, 32MB IO Read, 40KB IO Write, 518KB IO Other

Not what I would call orders of magnitude increase in memory usage. 38MB of memory going from a 4.8MB to a 127KB database file.

Trying to measure the difference in startup times showed about .5 seconds more to open the full library compared to the empty library averaged over 5 trials.

Regards,
David

[chaley]

aYou said your DB was 5MB (I assume your 48 is really 4.8). Using your numbers the memory usage grew by 38 MB, which is in fact almost an order of magnitude.

You can draw any conclusions you want from your data. I will draw my own from 4 years of experience developing calibre. I am out of here.

[Adoby]

Originally Posted by Hoods7070

11:30 tonight I finished and switched to the main library only to find that every single book I modified today had no tags, no comments. GEEZ. I truly could cry. In fact I think I will. I couldn't duplicate this on my PC. I give up.

Can you duplicate it on the laptop? That is, when updating a single tag on a single book, does it work on the PC but the exact same steps do not work on the laptop?

If there is a difference, then you can go on looking from there. And let us know the minimal steps needed to repeat the problem you see.

If you update books in calibre, only the database is updated, not the books. You either have to save the changes to the books, using save to disk or do a copy to another library using the calibre GUI, so the database for the other library is updated. If you simply copy the books, then you don't copy the updated metadata.

[hidden.platypus]

Originally posted by Hoods7070
Enter the data I want, either from the booklist of the "info" screen. This comprises nothing but Series and Country of the Author. I also check the publication date for the original release date. Anal huh??

Next, run Epub modify, to insert a jacket into the book.

Run Quality Check to swap Author FN/LN

Delete epub original files.

I have been following that same pattern for years!

Wow. That is a lot of work. But, all of us I'm sure are guilty of massaging our collections in like fashion. I know I spend more time editing metadata on my music library than I should. But hey, we got computers. That's what they're for right?

Just out of curiosity. When was the last time you ran Calibre's Library Maintenance? Just you know. . . databases do need integrity checked once in a while.

[BetterRed]

@Hoods7070 - maybe you already tried these, apologies if you already said so

• run calibre on your laptap [sic] in minimal safe mode
• backup library & rebuild the database via Library maintenance->Restore database
• rename the calibre config folder, calibre will create a new one, when it starts you'll have to use Create/switch library to tell it where you library is - if no effect then revert to prior calibre config folder.
• run a system file checker /scannow

BR

[Hoods7070]

Hmm - I posted at length here but a mod must have moved or deleted the post.

I have fixed all my missing tags - which I lost again !! no idea how - but managed to retrieve the newly added tags with a database repair left running overnight.

FWIW after more uninstalling of everything, things have improved a great deal. The only thing operating unbearably slowly now is the plugin for reversing Author FN/LN. I have asked the author about that.

Thanks for all assistance!