10-02-2012, 09:32 AM | #1 |
Handy Elephant
Posts: 1,736
Karma: 26785668
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus
|
Calibre Speedup
Sometimes people complain about calibre being slow. And it is indeed possible to make calibre slow down to a crawl by unknowingly asking calibre to do things that are very demanding if you have a large library.
Here are some things I have done to speed up calibre on my computer. A lot if this information comes from other threads here on the forum. I haven't run any benchmarks, so I can't say exactly how much faster things are. But I think calibre is fast enough for me now... Custom columns (See: https://www.mobileread.com/forums/sho...d.php?t=188312)
Custom columns that depend on other columns have to be "calculated" when you use them. For example when searching, sorting or using them in the tag browser. If you have a lot of books this can be very slow. A special case is the "Formats" column many have and need. If you search or sort on formats calibre will scan every folder and book file in your library to figure out exactly what formats you actually have. It can be VERY time consuming if you have a large library on a slow disk. Instead you can ask calibre to use stored information about the formats already available in the database. It will be much faster. But it may also be unsafe, since you then may not discover that files for some formats are missing, until it is too late. To use the already existing stored data about the formats, in the database, change the custom column for formats to use {:'approximate_formats()'} instead of {formats}. If you often sort on formats then it will speed up calibre a lot. But at the cost of not warning when files are missing. You can perhaps compensate by checking the library more often. Temp folder on the fastest disk (See: https://www.mobileread.com/forums/sho...d.php?t=189593) A more advanced speed-up is to make sure that the temp-folder for your operating system is placed on your fastest disk. Not everyone is comfortable to do this, it may make your OS unusable if you make mistakes... As I understand it, when calibre converts formats, updates metadata, saves books to disk or sends books to a device, it first creates a temporary copy of the book in the temp folder. Depending on the format, the book may also be "exploded/unzipped," metadata changed, and finally the book may be zipped again. Then the book is copied to the new destination and finally the book in the temp folder is deleted. Also a temporary metadata.db may be created and updated in the temp folder. Obviously it is best if all this happens on the fastest disk available. I use Linux and have created a RAM-disk using tempfs and mount \tmp there. I have 16GB of RAM and use up to 8 GB of that as temp folder. It seems to speed up saves, send to device and bulk updates of metadata quite a bit. Place metadata.db on a fast disk (See: https://www.mobileread.com/forums/sho...d.php?t=189714) Again this is not something everyone is comfortable with. It can be risky, and you may loose metadata, if you make mistakes. I have my calibre library on a NAS. It is noticiable slower than my local SSD-disk. So I moved my metadata.db on to my local SSD and placed a symlink in the calibre library, where metadata.db used to be, to the moved metadata.db. ---- Do you know some other ways to speed up calibre? Please add more suggestions! Last edited by Adoby; 10-02-2012 at 09:36 AM. |
10-02-2012, 11:42 AM | #2 | |
Wizard
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
|
Quote:
Last edited by unboggling; 10-19-2012 at 09:04 PM. Reason: clarify |
|
Advert | |
|
10-02-2012, 12:24 PM | #3 |
Guru
Posts: 614
Karma: 1164945
Join Date: Jan 2012
Location: SF Bay Area, California, USA
Device: iPhone 7+, iPad Mini4, Kindle Voyage
|
Adoby and unboggling, those are some very helpful suggestions. Thanks for posting.
|
10-11-2012, 01:51 PM | #4 | |
Wizard
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
|
Quote:
There is some discussion in various forums that symlinks in OS X often get corrupted/broken and cause more problems than they're worth. I tried a script for setting up RAMdisk for temp files at startup. Tried various ramdisk sizes. There was no noticable speedup of calibre bulk conversions. The ramdisk-tempfile-startup script in these blog articles doesn't use symlinks:
Exactly which temp folders on OS X Mountain Lion could be put on ramdisk to speed up various calibre operations? Do all calibre operations that use temp files use the same temp folder(s)? Has anyone successfully sped up calibre operations such as bulk conversions by putting relevant temp folders used by calibre onto ramdisk, specifically on OS X Mountain Lion? At this point, I don't have enough information about the temp folders used by calibre - or enough Unix knowledge/skill - to proceed. Since calibre is already fast enough for me, I don't have much motivation to proceed either. But when I get a new computer with SSD or Flash internal drive, I'll have more motivation, to extend the life of the drive. Perhaps a Unix-smart MountainLion-savvy calibre superuser or developer may have some suggestions. Last edited by unboggling; 10-11-2012 at 06:49 PM. |
|
10-12-2012, 08:04 PM | #5 |
null operator (he/him)
Posts: 20,678
Karma: 26966376
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
@unboggling - Symlinks (known as Softlinks in Unix) break if the name of the target or its location is changed - in Unix and in Windows. Given that OS/X is BSD Unix at its core, then I assume its the same.
But if you only use them for files such as metadata.db's then you're unlikely to change the name or location, and if you do, then hopefully you wouldn't do it unconsciously - i.e. you would remove the old symlinks and create a new ones referencing new targets on your SSD in the Calibre library directories (folders). I suspect the optimal performance related preferences will depend on how one uses Calibre and one's overall work patterns. My settings (also quad core) are - Priority=Low, Max Jobs=1 and Use all cores=unchecked. My perception is that things go faster - overall. To improve start up times I've seen the following recommended a couple of times - 'close the tag browser before closing Calibre' . But I've never figured out how to do that - does anyone know what it means? Start up times aren't an issue for me - so I never bothered to ask. BR |
Advert | |
|
10-12-2012, 09:08 PM | #6 | |||
Wizard
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
|
@BR, Thanks for responding.
Quote:
Quote:
Quote:
Last edited by unboggling; 10-12-2012 at 10:36 PM. Reason: clarify |
|||
10-12-2012, 10:11 PM | #7 | |||
null operator (he/him)
Posts: 20,678
Karma: 26966376
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
Quote:
I'm no great fan of Apple, but even I don't believe they would have broken softlinks - I think they've been in Unix since the get-go, like most of Unix probably inherited from Multics - internally they're just a text file. Quote:
Re performance settings : I often have other heavy duty things going on - Matlab models, software builds, video conversions. When I do a bulk operations in Calibre I am happy to go away and do something else, read newsfeeds, debug code, watch stock tickers - have lunch. If the Calibre job takes 20 minutes rather than 10 that's cool, there's always another fish to fry Quote:
I was looking for something on the wrong side, near the tag browser - which on my setup is on the left. Its got a keyboard shortcut toggle too - very handy I'm always wishing the book list was wider A couple of days ago I discovered that Word has keyboard shortcuts for ellipsis and em - I've been using Word for 20+ years - I'm a slow learner BR |
|||
10-12-2012, 10:32 PM | #8 | |||
Wizard
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
|
Quote:
Quote:
Quote:
|
|||
10-12-2012, 11:04 PM | #9 | |
null operator (he/him)
Posts: 20,678
Karma: 26966376
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
Quote:
They're like the people who say that Windows Named Streams (known as file forks in Unix) are places where virus's hide out - show me one I say - never got to see one. But so what if a symlink/softlink gets broken, its only a link, the data's still there, just make a new link. See attachment - its from Word 2007 Insert Symbols->Special Characters Tab BR |
|
10-13-2012, 12:52 AM | #10 |
Wizard
Posts: 4,812
Karma: 26912940
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
|
In April I noticed searchbar searches went from seconds to a minute or more.
Now I really try to spell it right the first time. I am not sure whether it was an update or library size thing although I know library size contributes. Search is much faster on a smaller library, but I only added 6 books and deleting several hundred had no effect. I also deleted (from a copy of library) all custom columns. no difference. Library loads in seconds, custom columns built on other columns calculate instantly. I recently split my library by genre as I am tryng to get a better grip on the genre thing. (I am sure there are better ways to do this but was easiest on my somewhat elderly brain) But after applying a base genre to all books I rebuilt to one library. Better to have one slower library than 10 fast ones Pretty sure it is not hardware related as works the same with I7 desktop 12G Ram and I5 laptop 8 G Ram. Any insight appreciated. Helen |
10-13-2012, 02:49 AM | #11 |
null operator (he/him)
Posts: 20,678
Karma: 26966376
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
Helen
I've always noticed that if I do an unqualified search for say "quantitative" then it takes a few minutes with a lot of disk I/O, but it seems to be a one time delay. If after that I do another search for say "subprime" its back in a flash with very little disk I/O. Tomorrow the first search will be 'slow', so I'm guessing that SQLLite builds a temporary index for unqualified searches - i.e. when its not searching via Tag Browser items (author, publisher etc) only. My unqualified search terms are usually in the summary metadata and titles, I don't bother much with Keyword Tags - whats in the summary is usually good enough for me. Beyond that I use Windows Search. I have a library of 3500 documents - not many books as such, mainly research papers from governments, academe, consultants, ngo's etc. BR |
10-13-2012, 03:11 AM | #12 | |
US Navy, Retired
Posts: 9,865
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
You could limit your general searches by going to Preferences - Searching and editing the "Columns that non-prefixed searches are limited to" box. I limit my searches to title, authors, tags, series, and publisher. By removing comments you can drastically increase the speed of your non-prefixed searches. Last edited by DoctorOhh; 10-13-2012 at 03:14 AM. |
|
10-13-2012, 07:25 AM | #13 |
Well trained by Cats
Posts: 29,974
Karma: 56143930
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
|
I use non-qualified searches all the time and don't see 'Minutes' go by
on a lowly P4 with 4K books in the Library I do have the setting mentioned by DoctorOhh down to 3 fields |
10-13-2012, 10:13 AM | #14 |
Wizard
Posts: 4,812
Karma: 26912940
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
|
Thanks for the suggestion. I will try it.
I unwittingly exagerated the time it is usually only about 30 seconds, sometimes longer. Seems longer Helen Tried it and at least twice as fast Thanks Helen Last edited by speakingtohe; 10-13-2012 at 10:20 AM. |
10-13-2012, 10:19 AM | #15 | |
Wizard
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
|
Quote:
|
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
How to speedup page turn on sideloaded epub on Kobo Touch with Firmware 2.0 | MisterMax | Kobo Reader | 40 | 12-06-2012 05:31 AM |
Patch: Calibre adds tags to identify ebook formats created by calibre. | siebert | Calibre | 1 | 07-18-2011 02:07 PM |
Speedup Metadata download | libertytrooper | Library Management | 4 | 07-04-2011 09:40 AM |
iLiad Some speedup ideas.. | henris42 | iRex Developer's Corner | 2 | 05-28-2008 07:59 AM |
Processor Speedup? | sirmaru | Sony Reader Dev Corner | 4 | 01-05-2008 06:23 PM |