10-23-2012, 07:22 AM | #31 |
Wizard
Posts: 4,552
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
|
On Windows and Linux (and I presume on MacOS as well) you can over-ride Calibre using the system /tmp by setting the CALIBRE_TEMP environment variable to point to where you want Calibre to put its temporary files. This might save you doing some research and shoudl certainly be quick to try?
|
10-23-2012, 05:08 PM | #32 | |
Wizard
Posts: 4,812
Karma: 26912940
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
|
Quote:
No I think 20 or 30 seconds is the norm and that is better than good enough. But... speed of converting books is slightly more important to me than how fast my OS loads or a program loads. I boot my computer, maybe once a day, more like three times a week and start most programs at the same or lower frequency. An SSD could probably save me a good hour or two a year. I still have a lot of library cleanup to do so calibre performance is to me more interesting, well it is all interesting if not really significant from my POV, and thanks again for posting your findings/updates. Helen |
|
10-23-2012, 09:19 PM | #33 | ||
Wizard
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
|
Quote:
Edit: But I'm Unix-ignorant and possibly missed something. Quote:
I haven't done shell programming for 20 years, so it may take me awhile but it seems doable. I can try to adapt from these calibre startup batch files a search of the forum turned up, previously posted by itimpi and DoctorOhh. calibre temp folders, yet again (startup batch file, itimpi) 100,000 temp files (startup batch file, DoctorOhh) Edit: What is an appropriate size for a ramdisk devoted to calibre temp files? You're welcome. My conversions EPUB to MOBI average about 25 seconds too, with SSD, but without the temp files to ramdisk, using the performance settings mentioned in post #2 of this thread. The primary reason I upgraded to SSD and more RAM was that I decided to delay buying a new computer for a year or so, so this was just to tide me over, as a psychological sop so I wouldn't buy a new computer now. The upgrades really weren't necessary for performance. Last edited by unboggling; 10-23-2012 at 10:24 PM. Reason: clarify |
||
10-24-2012, 12:02 AM | #34 |
creator of calibre
Posts: 43,858
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
There are various environment variables your system can have that influence where the temp files end up. itimpi told you of the calibre specific one. There are lots more, http://docs.python.org/library/tempf...mpfile.tempdir
Presumably one of those is set on your system to something other than /tmp. |
10-24-2012, 12:49 AM | #35 | |
Wizard
Posts: 3,720
Karma: 1759970
Join Date: Sep 2010
Device: none
|
Quote:
consider LOWERing the priority of calibre, let it run in the BACKground, & multitask/get on with your life while it does its thing. I don't get how watching the progress bar of a book conversion can be the most interesting thing in anyone's day |
|
10-24-2012, 02:31 PM | #36 | |
Wizard
Posts: 4,812
Karma: 26912940
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
|
Quote:
I find it interesting and maybe some others do as well. Perhaps we deserve to be belittled because of this, but speaking for myself only, I find it a bit offensive to see two non contributing belittling posts . Just don't understand your motive here. Enlighten me. Helen |
|
10-24-2012, 07:47 PM | #37 | |
Wizard
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
|
Quote:
TMPDIR=/var/folders/x3/2x81rgjj3lq_2r6wv3_xdkrm0000gn/T/ When I mount a RAMdisk on that directory that RAMdisk doesn't get created. When I reset TMPDIR to a different directory that directory doesn't get used. After installing calibre command line tools and setting CALIBRE_TEMP_DIR manually in the Terminal, CALIBRE_TEMP_DIR doesn't get used after starting the calibre GUI, not even when resetting CALIBRE_TEMP_DIR after calibre startup. What gets used in all cases so far is TMPDIR as listed above. I assume the calibre environment variable CALIBRE_TEMP_DIR requires calibre command line tools first installed (on OS X). So far I haven't tried this with the calibre development environment which is not yet installed. Is setting CALIBRE_TEMP_DIR manually before or after launching the calibre GUI supposed to work? Or is that just with the calibre development environment? Maybe I'm trying to set the ramdisk on /var/folders/x3/2x81rgjj3lq_2r6wv3_xdkrm0000gn/T too early in the OS X startup and it gets overwritten later by automatic user login initialization. I'll work on this more tomorrow, including testing a timing delay. The thought has crossed my mind that figuring this out would be simpler on Windows or Linux. Perhaps when I finally get a new computer it won't be a Mac. Last edited by unboggling; 10-24-2012 at 09:13 PM. Reason: clarify |
|
10-24-2012, 10:40 PM | #38 |
creator of calibre
Posts: 43,858
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
If you set CALIBRE_TEMP_DIR so that it exists before the calibre GUI is launched it will be used in preference to all other variables. I think on OS X that means you have to set it for lauchd via the launchd environment plist file, a bit of googling should tell you how to do that.
|
10-26-2012, 07:59 PM | #39 | |
Wizard
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
|
Quote:
Thank you, Kovid and itimpi, for the clues above, and thank you, GRiker, for additional clues in the thread: https://www.mobileread.com/forums/sho...d.php?t=186069 which included a link to: http://www.dowdandassociates.com/con...etclaunchdconf explaining how to set a system-wide environment variable during OS X startup, per the launchd configuration file /etc/launchd.conf. That works for CALIBRE_TEMP_DIR, but it does not work for the environment variable TMPDIR. (The older method of setting environment variables in ~/.MacOSX/environment.plist doesn't work in Mountain Lion, at least for me. And apparently launchd.conf per user is not yet supported.) I modified the startup script to mount a ramdisk on a new directory CalibreTempDir dedicated to calibre temporary files, set as CALIBRE_TEMP_DIR per /etc/launchd.conf. The script also mounts ramdisks on /private/tmp and /var/run. I was possibly wrong earlier when I said a ramdisk didn't work for /var/folders or any of its subdirectories, including the directory pointed to by that other environment variable TMPDIR. OS X doesn't show the contents of a ramdisk mounted on var/folders or its subdirectories, while OS X does show the contents of ramdisks mounted on /private/tmp, /var/run, and my new CalibreTempDir, so I assumed either it didn't work or wasn't working right. I figure that messing with /var/folders or its subdirectories is too dangerous, until I learn more, such as how to successfully redirect the directory pointed to by TMPDIR to a ramdisk'd directory I can see the contents of. I was mostly concerned about the calibre conversion temp files, because calibre generates and deletes a lot of them during conversions (deleting any remaining temp files at calibre Quit). So redirecting those temp files to ramdisk preserves the life of the SSD. This kludge uses hdik and mounts ramdisks on directories. Surprisingly putting calibre conversion temp files on ramdisk results in the same conversion times as with no ramdisk. I suspect the lack of speedup may have something to do with memory management in OS X, though turning dynamic pager off or on didn't affect the speed of calibre conversions. The script I'm using is in this spoiler: Spoiler:
Last edited by unboggling; 10-28-2012 at 07:10 PM. Reason: clarify |
|
10-27-2012, 02:40 AM | #40 | |
Wizard
Posts: 3,720
Karma: 1759970
Join Date: Sep 2010
Device: none
|
Quote:
see previous post conclusion: "Surprisingly putting calibre conversion temp files on ramdisk results in the same conversion times as with no ramdisk." don't you get that if having temp files in RAM was such a smart move it would be in Windows alreay. Microsoft have good reason for not doing that which google will explain to you if interested. now I return to to how many angels can I balance on this pin / how many nanoseconds can I shave from this process. have a nice day & don't waste too much time composing a retort - you may need those seconds Last edited by cybmole; 10-27-2012 at 02:43 AM. |
|
10-27-2012, 05:48 AM | #41 | |||
Wizard
Posts: 4,552
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
|
Quote:
However it is not good as a generalization. If a program creates lots of temporary files then using RAM for temporary files can make a bid difference. An example of this that I am aware of is calibre2opds. That creates LOTS of temporary files while running and using a RAM disk can easily give something like a 30% improvement in run time on. In fact the main use that I had for the CALIBRE_TEMP_DIR setting was the fact that Calibre was prone to leaving the temp files around when it had finished rather than removing them. Giving Calibre its own private temp folder mad it easy to develop a batch file for launching Calibre that tidied up any files left from previous runs. Quote:
It is only recently that RAM has become cheap enough that the average user was not better using additional RAM to reduce Windows use of a swap file. However now RAM is cheap enough that it is quite feasible to have more than enough RAM to run without a swap file and still have enough free for uses such as temporary files in RAM. I found that switching off the Windows swap file and using a RAM disk for temporary files made my Windows system much more responsive. I did have 16GB of RAM available in this case which was why I thought it was worth experimenting. Quote:
Then there is simply the interest factor in knowing how to do something like this. It might not be cost-effective but it can still be stimulating to do this sort of investigation Last edited by itimpi; 10-27-2012 at 05:51 AM. |
|||
10-27-2012, 06:59 AM | #42 |
Wizard
Posts: 3,720
Karma: 1759970
Join Date: Sep 2010
Device: none
|
what you risk is all sort of paging error / out of space issues in windows because you have to guess and set a Max size RAM disc, which will be far smaller than what windows could have used on your HD / SSD. if your temp files don't all fit into your allocated RAM disc space, expect hours of entertainment? as you try to debug the program failures
& there are specific extra reasons why you should never assign your system page file to a RAM disc, as Google can explain. if you have an SSD , you almost certainly have a secondary HDD for data. Just assign TMP to that drive to prolong SSD life |
10-27-2012, 09:26 AM | #43 | |||
Wizard
Posts: 4,552
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
|
Quote:
Quote:
Quote:
|
|||
10-27-2012, 10:12 PM | #44 |
Wizard
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
|
@ cybmole & all,
I've been thinking about this a lot the past couple days. I want to gain more familiarity with Unix and even learn some superuser kinds of things with Unix. I care about learning how to do things I didn't previously know how to do, and I also care about applying the KISS Principle to my computer usage. I don't care much about the life of the SSD, although even I recently used that as an excuse, and I don't care much about further improving performance past SSD speed, within the constraints of current speeds of processor, bus's, caches, RAM, storage in/out, etc. Further discussion in spoiler: Spoiler:
@ itimpi & kovid, Thanks again for your help. Last edited by unboggling; 10-28-2012 at 02:24 AM. Reason: clarify |
10-27-2012, 11:13 PM | #45 |
Wizard
Posts: 3,720
Karma: 1759970
Join Date: Sep 2010
Device: none
|
the discussion is improving!
I do buy into the "it's not about speed, it's about understanding the tech" argument, & calibre conversions are as good as any other testbed. When I tried to set up ram discs for other reasons, win 7 fought back fiercely, hence googling, hence earlier comments. PS I think that folks rich enough to try SSD are rich enough to replace the if TMP usage wears them out. Me, I'm waiting until they get much bigger/cheaper. |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
How to speedup page turn on sideloaded epub on Kobo Touch with Firmware 2.0 | MisterMax | Kobo Reader | 40 | 12-06-2012 05:31 AM |
Patch: Calibre adds tags to identify ebook formats created by calibre. | siebert | Calibre | 1 | 07-18-2011 02:07 PM |
Speedup Metadata download | libertytrooper | Library Management | 4 | 07-04-2011 09:40 AM |
iLiad Some speedup ideas.. | henris42 | iRex Developer's Corner | 2 | 05-28-2008 07:59 AM |
Processor Speedup? | sirmaru | Sony Reader Dev Corner | 4 | 01-05-2008 06:23 PM |