Thread: Calibre Speedup
View Single Post
Old 10-27-2012, 10:12 PM   #44
by the bootstraps
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
Posts: 1,052
Karma: 858115
Join Date: Jan 2011
Location: Southeast US
Device: PRS-T2, Nexus 7, KindleT, iPad1, Kindle3KB
@ cybmole & all,
I've been thinking about this a lot the past couple days. I want to gain more familiarity with Unix and even learn some superuser kinds of things with Unix. I care about learning how to do things I didn't previously know how to do, and I also care about applying the KISS Principle to my computer usage. I don't care much about the life of the SSD, although even I recently used that as an excuse, and I don't care much about further improving performance past SSD speed, within the constraints of current speeds of processor, bus's, caches, RAM, storage in/out, etc.

Further discussion in spoiler:

Life of SSD. Over time the technology improves, SSDs get cheaper and better. The hype about SSD wear and SSD life originates from cost of and problems with early SSDs, what, starting about 10 years ago? At this point, the persisting hype does not apply for the normal user, when combined with wear leveling and keeping the SSD less than half full. As itimpi pointed out, SSD-lifetime may affect heavy users. On the other hand I question even that because most heavy users probably trade up to better tech every 2 years or so anyway. For me the life of this SSD doesn't matter much because I am a relatively normal user, a hobbyist. My most intensive storage i/o is for the OS, several apps including calibre, and the files generated with those apps. I don't do any video editing. I don't do anything with audio except listen to some tunes once in awhile. I am not a "heavy" user such as itimpi mentioned. I am not a company with lots of storage i/o. I am not a server farm. I will replace this 18 month old computer (including this new SSD) with a new faster computer in a year or so. Even if I transfer this same SSD to the new computer this SSD should function well for at least 4 years, which is plenty long enough. For me just lasting 2 years is long enough.

Speed Improvements. Any speed improvements I achieve past SSD speed are just gravy. Speed improvements in performance of calibre or other applications are not necessary because everything is already fast enough for me. When I want to speed things up further I'll buy a new computer with faster processer, caches, RAM, and bus's, along with higher capacity caches, additional cores and processors, more max addressable RAM and at least 64GB RAM, at least 2 internal SSDs, max'd thunderbolt capability, etc. Until then anything else for maximizing this computer probably isn't worth the expense or hassle of implementing or maintaining the changes.

Paging. I turned off paging briefly as an experiment. Otherwise I keep it turned on. I am not advocating turning it off, on Windows or any Unix variant including Linux or OS X. I also keep the paging files (swapfiles on OS X) on the SSD. I never said anything about moving paging files to RAM.

Keep It Simple. When I apply the KISS Principle to my computing situation, I arrive at some KISSi-ized conclusions, recently arrived at in the last couple of days, which is why I didn't mention them up front before asking about using ramdisks for temp files:
  • Don't use ramdisks for anything, not even temp files, unless you are a heavy user with intensive daily i/o usage. Which I'm not.
  • Let the OS handle everything as it was designed to handle everything for most users excluding superusers. Superusers will do whatever they want anyway. Basically what an OS is supposed to do is manage the details for memory, i/o, storage locations, etc. I believe Apple abandoned easy GUI ramdisks on OS X for a reason a couple years ago. Now as far as I can tell, memory management problems that plagued MacOS and early versions of OS X aren't as bad as they used to be.
  • If using SSD(s), keep all files on SSD(s) except backups. Even if preserving the life of the SSDs were important, these considerations apply. Where should the normal user, most users, draw the lines, if in a superuser-type effort to maximize performance, the user starts putting specific directories and files (such as calibre conversion temp files) on ramdisk, SSD, or HDD, depending on various reasons for each type of storage for each type of file? There is an almost never-ending list of specific things to deal with, and then symlinks or other kludges to support each of those many decisions when the decision varies from the "main OS way / most users way" of doing things. Each decision in this arena adds complications and some solutions come laden with specific performance or maintenance issues. It is much KISSier to just keep all files on one or more internal SSDs, with each SSD more than twice the size capacity of all OS, application, and user files on it (including space used by page files aka swapfiles). In addition keep at least 1 preferably 2 hourly (or even daily) backups of each SSD on HDD(s). If an SSD fails, buy a new SSD and restore from backup. No muss, no fuss, easy.

Why learn Unix? Unix has evolved during the 20 years since I was relatively familiar with it, and during that time my Unix skills have atrophied. I've been considering buying a PC rather than a Mac for my next computer. If I get a Windows PC I'll probably add a Linux partition and run mostly from Linux. So I am refamiliarizing with Unix and determined to learn some superuser-type things before then. Which will be valuable even if the next computer is a Mac because OS X is also based on a Unix variant. Plus there is the possibility of putting whatever OS on a partition, or running that OS under emulation, on whichever hardware platform. So current Unix skills will be worthwhile unless all of a sudden I decide I like MS Windows, which didn't happen the last couple of times I had Windows PCs, and I don't envision that starting to happen now. But if that does happen, great, I'll run Windows from a partition or emulation and have some Unix variant in another partition or emulation.

Why learn superuser things? Such as how to use ramdisks, or turn paging on or off, or whatever, even if that knowledge isn't really necessary or won't specifcally be applied? Because its fun.

Why document these kinds of things here? There are some superusers of various OSs that hang out on MobileRead. Hopefully I'll learn something from them, as I recently learned about using CALIBRE_TEMP_DIR from itimpi and kovid. In the case of OS X, many of these types of things are poorly documented, such as using ramdisks. The best documentation is often just the man pages for specific commands of a particular Unix flavor, but first the user must have enough familiarity to know even what to man. Many calibre users on various OSs have SSDs, may be curious how to do ramdisks and what types of files are suitable for ramdisks. Since Adobe and others discussed speeding calibre up on Linux and Windows, I decided to mention similar things for calibre users on OS X, as I learn about such things. That helps me and may help some other users too.

@ itimpi & kovid,
Thanks again for your help.

Last edited by unboggling; 10-28-2012 at 02:24 AM. Reason: clarify
unboggling is offline   Reply With Quote