MobileRead Forums
Register Guidelines E-Books Today's Posts Search

Go Back   MobileRead Forums > E-Book Software > Calibre
Reload this Page Slow with large library

Notices

Reply
Page 1 of 2 1 2
 
Thread Tools Search this Thread
Old 02-05-2020, 09:36 AM   #1
luoto
Member
luoto began at the beginning.
 
Posts: 15
Karma: 10
Join Date: Jun 2012
Device: ipad
Slow with large library
Hello.

Is there any way to optimize existing indices as Calibre (latest version on Mac OSX 10.14) is now quite slow with a very large library of various books. Moving to SSD is no option for the storage side of things, but if Calibre's indices can live on an SSD that may make things faster?!?

Adding new books has a noticeable delay whilst the computer (an otherwise powerful Mac Pro with lots of memory) does its stuff.

I did not find any obivous pointers with search at least !
luoto is offline   Reply With Quote
Old 02-05-2020, 09:55 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,364
Karma: 27230406
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
calibre keeps its indice in RAM. Hide the Tag browser. Dont use columns built from other columns.
kovidgoyal is offline   Reply With Quote
Old 02-05-2020, 10:02 AM   #3
luoto
Member
luoto began at the beginning.
 
Posts: 15
Karma: 10
Join Date: Jun 2012
Device: ipad
Quote:
Originally Posted by kovidgoyal View Post
calibre keeps its indice in RAM. Hide the Tag browser. Dont use columns built from other columns.
Thank you for the prompt response. Blindingly fast.

I do not have tag browser visible. I use only default columns and have not tried to make any custom views. Tags is shown as one of the (default) view columns in the view where you see what books are available (but I only have a few different tag names, I don't extensively use the tagging feature/more for internal source signposting).

I have 128Gb of RAM on this Mac Pro so it shouldn't be complaining about memory fortunately.

Is there anything else I can do/or a report to submit that may help you with further development of this excellent application?

Thank you!
luoto is offline   Reply With Quote
Old 02-05-2020, 10:04 AM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 45,364
Karma: 27230406
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The tag browser is the panel on the left of the book list. Have you hidden that? And how large is "large library". i.e. how many books.
kovidgoyal is offline   Reply With Quote
Old 02-05-2020, 10:12 AM   #5
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 31,067
Karma: 60358908
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A


and Tame your Anti-whatever. Those (book) files have been already scanned, so turn off Real time scanning on your Library folder (I leave it for the whole system scan)
FWIW I just got a new PC. I copied my Library over my LAN. Pausing the A/V on the new PC DOUBLED THE TRANSFER SPEED
theducks is offline   Reply With Quote
Old 02-05-2020, 10:40 AM   #6
luoto
Member
luoto began at the beginning.
 
Posts: 15
Karma: 10
Join Date: Jun 2012
Device: ipad
Quote:
Originally Posted by kovidgoyal View Post
The tag browser is the panel on the left of the book list. Have you hidden that? And how large is "large library". i.e. how many books.
Yes it is hidden. I just have the text menu on the left-hand side. 53k-ish books in the library.
luoto is offline   Reply With Quote
Old 02-05-2020, 03:29 PM   #7
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 21,736
Karma: 29711016
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by luoto View Post
Yes it is hidden. I just have the text menu on the left-hand side. 53k-ish books in the library.
"text menu on the left-hand side" - not sure what you mean ? Can you post a screen shot

53,000+ items is not a particularly big library, I have one that's 180,000+, I've seen others users mention libraries of a million items.

What sort of files (formats) are you adding, and how big are they? The files added get copied into the library sub-folders. My experience is that the time taken to add an item is a function of the size of the file being added, e.g. a 100K epub novel/biography takes a couple of seconds, whilst a 1G video clip might take 10-20 seconds.

BR
BetterRed is offline   Reply With Quote
Old 02-05-2020, 03:38 PM   #8
Sirtel
Grand Sorcerer
Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.
 
Sirtel's Avatar
 
Posts: 13,485
Karma: 239219543
Join Date: Jan 2014
Location: Estonia
Device: Kobo Sage & Libra 2
Well, HDD is slow, nothing to be done about that. Why not move to SSD? I have over 20 000 books in my library and it takes less than a tenth of my 500 GB SSD. SSD is not so expensive today as it once was.
Sirtel is offline   Reply With Quote
Old 02-05-2020, 06:11 PM   #9
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 79,775
Karma: 145864619
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3
@luoto how any eBooks do you have in this slow library?
JSWolf is online now   Reply With Quote
Old 02-05-2020, 07:00 PM   #10
DNSB
Bibliophagist
DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.
 
DNSB's Avatar
 
Posts: 46,288
Karma: 169098402
Join Date: Jul 2010
Location: Vancouver
Device: Kobo Sage, Libra Colour, Lenovo M8 FHD, Paperwhite 4, Tolino epos
Quote:
Originally Posted by luoto View Post
Yes it is hidden. I just have the text menu on the left-hand side. 53k-ish books in the library.
Quote:
Originally Posted by JSWolf View Post
@luoto how any eBooks do you have in this slow library?
Sheesh.... He already said 53K-ish books.
DNSB is offline   Reply With Quote
Old 02-06-2020, 02:16 AM   #11
luoto
Member
luoto began at the beginning.
 
Posts: 15
Karma: 10
Join Date: Jun 2012
Device: ipad
Thanks for the other comments folks, that are noted. The books vary in size from maybe a megabyte for an ePub or less to say 20-50 megabytes if a PDF, taking about 680Gb of disk space. Moving to SSD for that drive is not presently a viable option as it is a much larger HDD with other items on that would need rehoming and already the drive bays are maxed out space-wise.
luoto is offline   Reply With Quote
Old 02-06-2020, 03:16 AM   #12
Sirtel
Grand Sorcerer
Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.Sirtel ought to be getting tired of karma fortunes by now.
 
Sirtel's Avatar
 
Posts: 13,485
Karma: 239219543
Join Date: Jan 2014
Location: Estonia
Device: Kobo Sage & Libra 2
Quote:
Originally Posted by luoto View Post
Thanks for the other comments folks, that are noted. The books vary in size from maybe a megabyte for an ePub or less to say 20-50 megabytes if a PDF, taking about 680Gb of disk space. Moving to SSD for that drive is not presently a viable option as it is a much larger HDD with other items on that would need rehoming and already the drive bays are maxed out space-wise.
Yes, if you have a lot of PDFs, then the storage space will be a problem. My library is less than 50 GB, but I have very few PDFs.

Your only option would seem to be splitting the library. Put the books you rarely need on an external drive and the rest on a SSD.
Sirtel is offline   Reply With Quote
Old 02-06-2020, 04:04 AM   #13
luoto
Member
luoto began at the beginning.
 
Posts: 15
Karma: 10
Join Date: Jun 2012
Device: ipad
Thank you all, once again, for the advice given.
luoto is offline   Reply With Quote
Old 02-06-2020, 04:26 AM   #14
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 21,736
Karma: 29711016
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by luoto View Post
Thanks for the other comments folks, that are noted. The books vary in size from maybe a megabyte for an ePub or less to say 20-50 megabytes if a PDF, taking about 680Gb of disk space. Moving to SSD for that drive is not presently a viable option as it is a much larger HDD with other items on that would need rehoming and already the drive bays are maxed out space-wise.
You've not told us how long it actually takes to Add a 1MB epub and a 50MB PDF.

BR
BetterRed is offline   Reply With Quote
Old 02-06-2020, 04:34 AM   #15
luoto
Member
luoto began at the beginning.
 
Posts: 15
Karma: 10
Join Date: Jun 2012
Device: ipad
Quote:
Originally Posted by BetterRed View Post
You've not told us how long it actually takes to Add a 1MB epub and a 50MB PDF.

BR
About 14-20 seconds for a 25mb PDF (just tested one) and about 25 seconds for a 40mb ePuB. The status bar sits at zero percent for a time then rushes to 100 per cent and sits there for a second or two. All other file accesses on the system seem to be normal, e.g. opening Word documents, moving the files with Finder and the like. I guess I just must split the library until a move to a larger capacity SSD is possible and practical (the last time I looked multiple tb drives were very expensive).
luoto is offline   Reply With Quote
Reply
Page 1 of 2 1 2

« Previous Thread | Next Thread »

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Large Calibre Library Slow - Solved Mythlandia Library Management 0 09-09-2013 03:46 AM
Large Library working slow bigbird1227 Library Management 3 11-21-2011 05:14 PM
Calibre a bit slow editing in large Collections Evilwarning Calibre 13 11-04-2010 04:27 AM
Large FB2s slow? edwlarkey Astak EZReader 2 10-09-2009 12:29 AM
Sony PRS-505 slow whilst changing Chapters. XHTML too large? pato1 Workshop 3 06-08-2009 03:05 PM


All times are GMT -4. The time now is 09:58 AM.

Contact Us - FAQ - Guidelines - Privacy - Top

MobileRead.com is a privately owned, operated and funded community.