Library Size and Performance

[Hampshire Nanny]

Now that I've played around with Calibre to become a little bit familiar with how to use it, I'm beginning to populate its library with the hundreds of books I've downloaded from various places such as MobileRead, Feedbooks, Baen, and Project Gutenberg. At this time, I have 432 books in my library -- including the supplied Quick Start Guide.

I've noticed that any interaction with the library -- even if it's only editing the metadata for a single book -- goes abysmally slowly now. Much more slowly than when I first started loading books into the library. Opening a small ePub can take minutes. Adding 10 books may take as much as 20 minutes.

Some info: using Calibre Portable 1.39.0 on a 128G USB 3.0 thumb drive. I have some plug-ins installed ... will be glad to list them if someone thinks that the plug-ins could be causing the performance problem.

On the Windows 7 system I am using, all drives are connected via USB except the {very small} boot drive. I wouldn't expect to get any better performance by moving the program and library onto another drive ... unless Windows 7 uses different rules for buffering drives it perceives as regular drives versus those it identifies as "devices with removable storage" -- that is, the thumb drive.

I'd appreciate any help -- from Calibre experts and/or Windows experts.

[theducks]

Originally Posted by Hampshire Nanny

Now that I've played around with Calibre to become a little bit familiar with how to use it, I'm beginning to populate its library with the hundreds of books I've downloaded from various places such as MobileRead, Feedbooks, Baen, and Project Gutenberg. At this time, I have 432 books in my library -- including the supplied Quick Start Guide.

I've noticed that any interaction with the library -- even if it's only editing the metadata for a single book -- goes abysmally slowly now. Much more slowly than when I first started loading books into the library. Opening a small ePub can take minutes. Adding 10 books may take as much as 20 minutes.

Some info: using Calibre Portable 1.39.0 on a 128G USB 3.0 thumb drive. I have some plug-ins installed ... will be glad to list them if someone thinks that the plug-ins could be causing the performance problem.

On the Windows 7 system I am using, all drives are connected via USB except the {very small} boot drive. I wouldn't expect to get any better performance by moving the program and library onto another drive ... unless Windows 7 uses different rules for buffering drives it perceives as regular drives versus those it identifies as "devices with removable storage" -- that is, the thumb drive.

I'd appreciate any help -- from Calibre experts and/or Windows experts.

1) USB Flash drives are slow
2) Is the drive connecting at the old USB1 speed
3) is your anti-virus scanning

I have used a USB Hard drive (old Laptop drive in a caddy), Performance is only a tiny bit slower than a Local SATA drive. 4000 books

10 books on an old P4 running XP is under a minute

[BetterRed]

@Hampshire Nanny - Presumably you have more than calibre on the 128GB flash drive. How is performance of other applications on other data on the same drive.

FWIW : I have a library of ~40,000 books (20G), and another of ~8,000 books (600GB), mirrored to a USB3.0 spinning disk drive. Calibre performance on those libraries is faster than the same libraries on an internal SATA2.0 spinning disk drive . That said, the performance on the internal SATA 2.0 disk is more than good enough.

Have you checked the speed of the USB3.0 flash drive. Here is a review of some speed testers ==>> 6 Tools to Test Read and Write Speed of USB Flash Drives

Try copying the 400+ books to an empty library without any existing custom columns and see how that goes. I have known composite columns to slow things down, but I hasten to add a) only on large libraries, and b) since Calibre 1.0 even those situations have gone away.

BR

[Tatjana]

Originally Posted by BetterRed

@Hampshire Nanny - Presumably you have more than calibre on the 128GB flash drive. How is performance of other applications on other data on the same drive.

FWIW : I have a library of ~40,000 books (20G), and another of ~8,000 books (600GB), mirrored to a USB3.0 spinning disk drive. Calibre performance on those libraries is faster than the same libraries on an internal SATA2.0 spinning disk drive . That said, the performance on the internal SATA 2.0 disk is more than good enough.

Have you checked the speed of the USB3.0 flash drive. Here is a review of some speed testers ==>> 6 Tools to Test Read and Write Speed of USB Flash Drives

Try copying the 400+ books to an empty library without any existing custom columns and see how that goes. I have known composite columns to slow things down, but I hasten to add a) only on large libraries, and b) since Calibre 1.0 even those situations have gone away.

BR

I have 2 huge Calibre libraries, 168 211 books on main one (181 GB), all books from Gutenberg.org in epub and mobi formats (I update it weekly, so around 45 500 books right now (I am using the main one, can't switch to check (83 GB) )

Both libraries are on a SAMSUNG HD103UJ S-ATA II (7200 rpm)
I back them up to an external USB drive and what takes around 19 hours.

I Have libraries databases and Calibre temp folder in a "persistent" RamDisk of 4GB (using symlinks to fool Calibre). These databases are backed up every 60 minutes to another local disk (not the usb one)

The software I use for the ramdisk is freeware, and no nags or anything annoying, SoftPerfect RamDisk (http://www.softperfect.com/)

The only slow down i notice is when Calibre starts up. It will take about 2 minutes to show the user interface. After that it is fast even when I ask it to do nested searches. or update metadata in bulk.

Once a week, I use MyDefrag do defragment this partition, with the preset "Data disk Monthly" (MyDefrag is a free scriptable defragmenter for windows)

And I am very happy with the overall performance of Calibre.

I am sorry if I didn't explain something properly, I am not a native English speaker and I am still learning English at school.

Tatjana

[BetterRed]

@Tatjana - you might want to consider using a folder and file synchronize tool to mirror the libraries to the external usb. I have about 1.3TB of data on my internal 2TB SATA drive, takes a few minutes to mirror the changes to a usb at the end of the day, most of which is me eyeballing the preview to make sure there are no surprises.

This is a fairly comprehensive list of available alternatives ==>> Comparison of file synchronization software

BR

[Tatjana]

Originally Posted by BetterRed

@Tatjana - you might want to consider using a folder and file synchronize tool to mirror the libraries to the external usb. I have about 1.3TB of data on my internal 2TB SATA drive, takes a few minutes to mirror the changes to a usb at the end of the day, most of which is me eyeballing the preview to make sure there are no surprises.

This is a fairly comprehensive list of available alternatives ==>> Comparison of file synchronization software

BR

BetterRed, thank you very much for the suggestion. I am already syncing instead of backing up the whole libraries with great results, 32 minutes to sync When I read your post I ! I am actually syncing one of the libraries with several schools, using BitTorrent Sync, so I should know better.

Thank you again!

[BetterRed]

@Tatjana - it was the 19 hours to do a backup that you mentioned in your earlier post that made me sit up and take notice. I've thought about using BT Synch, but I can't think of anyone I trust enough to give them the key

BR

[Tatjana]

Originally Posted by BetterRed

@Tatjana - it was the 19 hours to do a backup that you mentioned in your earlier post that made me sit up and take notice. I've thought about using BT Synch, but I can't think of anyone I trust enough to give them the key

BR

You can always trust me

Actually, you can "give" 3 kinds of keys, the most restrictive will allow you to sync with another computer, but on this last one all files will be encrypted

Tat