Best type of system for calibre

[LadyKate]

I am looking at new laptops and am wondering about a few things.

Does the processor make much difference to the running of Calibre? The speed of adding books in large batches?

Is it the speed of the hard drive that is important when adding books or is it the processor or a combination of the two?

What hardware change would speed up the adding of books to Calibre?

Just figured that I might as well ask these questions when looking at a new system rather than later.

[BetterRed]

@LadyKate - for adding books, my rankings would be disk speed (eg SSD v HDD), memory capacity, and CPU a distant third.

I add books to an Intake library which is on an SSD, after getting the basics right - e.g. authors, title, and ISBN and doing any 'quick' conversions (e.g. AZW to ePub) I move them to their permanent libraries on an HDD.

But dual disk (SSD and HDD) laptops are few and far between. There are hybrid dual drives. i.e separate SSD and HDD drives ina single 'package'. You might be able to get that in a Dell Alienware laptop, I recently put one into my Dell Alien Alpha.

BR

[theducks]

Come on! How often do you add enough books to make FAST a significant need? You HAVE your Library. Adding books now is probably less than a dozen at a shot. (It takes me more time to download and UnZip a Baen Monthly Bundle, than Add it to Calibre and do my added housekeeping on those.

Converting is where the Horses are needed.
Start up Task Manager or Process Explorer. You can watch the various demands: CPU, RAM, I/O, Net

#1 Lots of RAM to give things room to happen without memory caching
#2 almost any modern CPU that is not rock bottom. This is not Gaming...You do not need the hottest i7 or AMD to run Calibre. an i3 should do the job.
#3 a decent HD. While SSD will get fast access to files... but other than startup... Calibre chugs along in mostly a single thread (a quad core does not gain a whole bunch here)

[BetterRed]

@theducks - some fan fiction enthusiasts can download hundreds of story's a day, I'm aware of one who uses a dedicated PC for it, I scrape dozens of items a day from RSS feeds, not into calibre though.

If you only have one computer speed of getting things done can be issue. Most people are not comfortable with multi-tasking, irrespective of gender. They watch 200MB downloads chugging along at 40KB/sec, while scrolling twitter their phones.

BR

[LadyKate]

For one library I maintain I add large batches of several thousand books (well, perhaps documents would be a better description) a couple of times a year. I guess it's time to spring for a large ssd for my desktop system and watch the prices on laptops.

Although I have 32 gig of ram on my desktop i7 I don't find that it goes above 35 percent when adding files. On the other hand my disk is running at 95 to 100 percent.

I am wondering, how much does the size of the library being added to affect the speed of the process?

[LadyKate]

Originally Posted by BetterRed

@theducks - some fan fiction enthusiasts can download hundreds of story's a day, I'm aware of one who uses a dedicated PC for it, I scrape dozens of items a day from RSS feeds, not into calibre though.

If you only have one computer speed of getting things done can be issue. Most people are not comfortable with multi-tasking, irrespective of gender. They watch 200MB downloads chugging along at 40KB/sec, while scrolling twitter their phones.

BR

So true about the way things can add up. With me being lazy and not keeping up for months on end, it leaves me with thousands of files to be added.

While I'm find with multitasking, I do find it necessary to break the adds into numbers that will complete in reasonable (two or three hour) batches. I've had the unpleasant occurrence of my power going out in the middle of a large add and having to make the decision of "continue on or scrap this and start over".

I so often wish that we could have two instances of Calibre running at the same time. One of the main reasons I really want another computer/laptop.

Pity I find i3's to be so sluggish.

[theducks]

Originally Posted by LadyKate

For one library I maintain I add large batches of several thousand books (well, perhaps documents would be a better description) a couple of times a year. I guess it's time to spring for a large ssd for my desktop system and watch the prices on laptops.

Although I have 32 gig of ram on my desktop i7 I don't find that it goes above 35 percent when adding files. On the other hand my disk is running at 95 to 100 percent.

I am wondering, how much does the size of the library being added to affect the speed of the process?

Ah the plague of 'I/O bound"
Maybe a 2+ drive, striped RAID would help or 10K RPM drives
Rotational Latency and SATA bus speed are limiting. a Stripe spreads the load

[BetterRed]

Originally Posted by LadyKate

I am wondering, how much does the size of the library being added to affect the speed of the process?

Not a lot - in my experience the number of and the size of the files being added is the main factor - big files take longer because calibre copies them. By big I mean things like 400MB video files.

The reason I have my Intake library on SSD is because of these big files, on an HDD a single file can take so long to add that the window goes into a not responding state for a while, on the SDD that doesn't happen. I add 10-20 items a day one at a time, fixing the essential metadata as I go, and once they're all done I move them to the HDD libraries - by then it's usually time to get breakfast, go for a run/walk/stroll, whatever.

If you only have the mass updates a couple of times a year, I suggest you do them overnight, if necessary in 'batches' over a few nights.

BR

[theducks]

Watching the Task Manager while running various Quality Check PI tasks, I noticed RAM usage grows significantly the bigger the 'found' group.
If you were RAM Limited, I would have suggested smaller batches.

16G (I have 8G on W10 and 2G on XP )

The only time I see Disc I/O in the 90's, is when I apply the fixes to the 'found list'

@BR many years ago I was working at a cable TV head end developer where we had a 'streamer' set up to simulate channel line up. The boss bought a hot P4 'gamer' Mobo and multi disk SAS I/O. It kept crashing the stream because the disk I/O could not keep up with more than 1 stream.
The streamer boar manufacturer could not duplicate our issue. They had 4 Streams running 24/7 on a Dell P3 based server .
We got a used Dell P3-800 rack server and all was well.

Bottom line was I/O intensive servers can not be replaced by a Process Intensive desktop model.

[JSWolf]

I have a laptop with 8GB of ram and it runs pretty good. I don't need (yet) any more ram than that. It does have a 1TB hard drive and no SSD. But remember that an SSD of a decent size is going to be expensive. Someday I'll get an SSD but not now. Too many other uses for my money.

When you buy a laptop, buy one that when the warranty is over you can easily upgrade. Mine is easy to add ram and it's easy to add an SSD. It does have a 5th gen quad core i7. One thing you do want to get in a laptop is good Wifi with WiFi AC even if your router does not support AC, that's something you could get for much better WiFi.

[kacir]

Originally Posted by LadyKate

Does the processor make much difference to the running of Calibre? The speed of adding books in large batches?

There was an update to Calibre a couple of years ago that sped up adding of large numbers of books *very* significantly.
https://www.mobileread.com/forums/sho...d.php?t=251076
In my experience, a decent processor and 6GB of RAM is plenty for adding books. I think (when looking at system monitor) that Calibre does use several separate threads when adding books, so it can benefit from multi-core, multi-thread processor.

Originally Posted by LadyKate

Is it the speed of the hard drive that is important when adding books or is it the processor or a combination of the two?

I do not think that the speed of the HDD is that important. There are other things you might want to do in Calibre where the speed difference between spinning rust and SSD is significant. Such as: bulk editing metadata(*), creating backups of your library, ... anything where you want to write thousands of small files.

(*) Calibre keeps a backup of metadata for each individual book in an *.opf file in the directory where the book is stored. But writting to those files is staggered, so a bulk metadata edit doesn't overwhelm the system with a slower disk.

You haven't asked about operating system.
I personally have a very good results with using Mint Linux as operating system to host my library (and many other things I need to do at home). In Linux you do not have to have an antivirus that will want to examine each of thousands little files before they are written to or read from the disk.

[JSWolf]

I run Windows and I've had no issues with Windows from Windows 7, 8.1, & 10. So going with Windows will not be an issue. As Kovid has said, (paraphrased) Linux has the least users, but the biggest support issues.

[LadyKate]

Thanks everyone. I'm thinking maybe I should be looking for a 1TB SSD for my db. Thank goodness they are going down. Maybe they will drop a bit more in the next six months.

I guess I should be looking at a decent speed (i5 minimum) with at least 8 gig ram (hopefully with a spare slot) and as much hard drive as can find. Although the drive isn't most important with a laptop for me, I can always use a large USB drive as the laptop is more for cleanup etc than for adding. I basically wanted a laptop as a second system and for use away from home.

I'm thinking that if I made note in a custom column that mods had been made then I could quickly make the changes to my main db when I return home.

While my dream laptop is way different than what I can afford at least I can watch for a sale to get something that will be workable.

[MinnieHeather]

Originally Posted by LadyKate

Thanks everyone. I'm thinking maybe I should be looking for a 1TB SSD for my db. Thank goodness they are going down. Maybe they will drop a bit more in the next six months.

I guess I should be looking at a decent speed (i5 minimum) with at least 8 gig ram (hopefully with a spare slot) and as much hard drive as can find. Although the drive isn't most important with a laptop for me, I can always use a large USB drive as the laptop is more for cleanup etc than for adding. I basically wanted a laptop as a second system and for use away from home.

I'm thinking that if I made note in a custom column that mods had been made then I could quickly make the changes to my main db when I return home.

While my dream laptop is way different than what I can afford at least I can watch for a sale to get something that will be workable.

Thanks to you also who have started this conversation ... I get the information from this group
Once again thank you

[JSWolf]

Originally Posted by LadyKate

Thanks everyone. I'm thinking maybe I should be looking for a 1TB SSD for my db. Thank goodness they are going down. Maybe they will drop a bit more in the next six months.

You don't need a 1TB SSD. 1 TB HD yes. Also if you can, get a 256GB SSD for use with the OS and programs. That way the system will run faster especially when booting and patching/updating.

I guess I should be looking at a decent speed (i5 minimum) with at least 8 gig ram (hopefully with a spare slot) and as much hard drive as can find. Although the drive isn't most important with a laptop for me, I can always use a large USB drive as the laptop is more for cleanup etc than for adding. I basically wanted a laptop as a second system and for use away from home.

Yes, I agree that an i5 at minimum and 8GB at minimum.

Also, you should get an HD screen (1920x1080) at 15.6" or 17.3".