Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 10-23-2012, 07:22 AM   #31
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,552
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by unboggling View Post
Thanks.

I guess I need to buy a book with discussion of temp files, symlinks, and ramdisks on OS X Mountain Lion.

For now I'm not using ramdisk until I know what I'm doing.
On Windows and Linux (and I presume on MacOS as well) you can over-ride Calibre using the system /tmp by setting the CALIBRE_TEMP environment variable to point to where you want Calibre to put its temporary files. This might save you doing some research and shoudl certainly be quick to try?
itimpi is offline   Reply With Quote
Old 10-23-2012, 05:08 PM   #32
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,812
Karma: 26912940
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
Quote:
Originally Posted by unboggling View Post
I convert all books that I add to calibre. But I'm trying to speed up conversions with temp files on ramdisks mainly out of curiosity, not necessity.
Same with me. I started using calibre on an old Dell 9300 (R.I.P) and I think a minute a book was the norm. It worked, I was happy. I actually thought it was awesome and amazing.

No I think 20 or 30 seconds is the norm and that is better than good enough.

But... speed of converting books is slightly more important to me than how fast my OS loads or a program loads.

I boot my computer, maybe once a day, more like three times a week and start most programs at the same or lower frequency. An SSD could probably save me a good hour or two a year.

I still have a lot of library cleanup to do so calibre performance is to me more interesting, well it is all interesting if not really significant from my POV, and thanks again for posting your findings/updates.

Helen
speakingtohe is offline   Reply With Quote
Advert
Old 10-23-2012, 09:19 PM   #33
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
Quote:
Originally Posted by kovidgoyal View Post
I'm not an OS X user so I can't help. Basically you need to find out where /tmp points to and ensure that location is mounted in RAM.
On OS X Mountain Lion 10.8.2 the symlink /tmp points to /private/tmp. Another symlink /var points to /private/var. The shell script discussed above mounts /private/tmp as ramdisk. So if calibre conversion temp files go to /tmp and /tmp is really /private/tmp, that means they should have gone to the /private/tmp ramdisk and sped up the conversion. But the only calibre temp files I noticed end up in directories in /private/var/folders/2x81rgjj3lq_2r6wv3_xdkrm0000gn/T/

Edit: But I'm Unix-ignorant and possibly missed something.

Quote:
Originally Posted by itimpi View Post
On Windows and Linux (and I presume on MacOS as well) you can over-ride Calibre using the system /tmp by setting the CALIBRE_TEMP environment variable to point to where you want Calibre to put its temporary files. This might save you doing some research and shoudl certainly be quick to try?
Thanks!!! Hopefully that will work. So the setup would be as follows. At computer startup create a ramdisk mounted on a directory with appropriate permissions. Startup calibre with another shell script that sets CALIBRE_TEMP environment variable to that same directory.

I haven't done shell programming for 20 years, so it may take me awhile but it seems doable. I can try to adapt from these calibre startup batch files a search of the forum turned up, previously posted by itimpi and DoctorOhh.

Edit: What is an appropriate size for a ramdisk devoted to calibre temp files?

Quote:
Originally Posted by speakingtohe View Post
...I think 20 or 30 seconds is the norm and that is better than good enough.

But... speed of converting books is slightly more important to me than how fast my OS loads or a program loads….
You're welcome. My conversions EPUB to MOBI average about 25 seconds too, with SSD, but without the temp files to ramdisk, using the performance settings mentioned in post #2 of this thread. The primary reason I upgraded to SSD and more RAM was that I decided to delay buying a new computer for a year or so, so this was just to tide me over, as a psychological sop so I wouldn't buy a new computer now. The upgrades really weren't necessary for performance.

Last edited by unboggling; 10-23-2012 at 10:24 PM. Reason: clarify
unboggling is offline   Reply With Quote
Old 10-24-2012, 12:02 AM   #34
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,773
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
There are various environment variables your system can have that influence where the temp files end up. itimpi told you of the calibre specific one. There are lots more, http://docs.python.org/library/tempf...mpfile.tempdir

Presumably one of those is set on your system to something other than /tmp.
kovidgoyal is offline   Reply With Quote
Old 10-24-2012, 12:49 AM   #35
cybmole
Wizard
cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.
 
Posts: 3,720
Karma: 1759970
Join Date: Sep 2010
Device: none
Quote:
Originally Posted by unboggling View Post
I convert all books that I add to calibre. But I'm trying to speed up conversions with temp files on ramdisks mainly out of curiosity, not necessity.
so do I, but that is ONCE per book & I add, on average, maybe one book per day.

consider LOWERing the priority of calibre, let it run in the BACKground, & multitask/get on with your life while it does its thing.

I don't get how watching the progress bar of a book conversion can be the most interesting thing in anyone's day
cybmole is offline   Reply With Quote
Advert
Old 10-24-2012, 02:31 PM   #36
speakingtohe
Wizard
speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.speakingtohe ought to be getting tired of karma fortunes by now.
 
Posts: 4,812
Karma: 26912940
Join Date: Apr 2010
Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet
Quote:
Originally Posted by cybmole View Post
so do I, but that is ONCE per book & I add, on average, maybe one book per day.

consider LOWERing the priority of calibre, let it run in the BACKground, & multitask/get on with your life while it does its thing.

I don't get how watching the progress bar of a book conversion can be the most interesting thing in anyone's day
I doubt that it is the most interesting thing in anyones day, just as squashing another person's enthusiasm is possibly not the highlight of yours.

I find it interesting and maybe some others do as well. Perhaps we deserve to be belittled because of this, but speaking for myself only, I find it a bit offensive to see two non contributing belittling posts . Just don't understand your motive here. Enlighten me.

Helen
speakingtohe is offline   Reply With Quote
Old 10-24-2012, 07:47 PM   #37
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
Quote:
Originally Posted by kovidgoyal View Post
There are various environment variables your system can have that influence where the temp files end up. itimpi told you of the calibre specific one. There are lots more, http://docs.python.org/library/tempf...mpfile.tempdir

Presumably one of those is set on your system to something other than /tmp.
Thanks. I looked at that python tempdir info. On my system the OS X environment variable TMPDIR is set at startup, before opening calibre. That is what Python is using, but I'm not sure if that is set automatically by OS X or by Python as part of OS X startup.

TMPDIR=/var/folders/x3/2x81rgjj3lq_2r6wv3_xdkrm0000gn/T/

When I mount a RAMdisk on that directory that RAMdisk doesn't get created. When I reset TMPDIR to a different directory that directory doesn't get used. After installing calibre command line tools and setting CALIBRE_TEMP_DIR manually in the Terminal, CALIBRE_TEMP_DIR doesn't get used after starting the calibre GUI, not even when resetting CALIBRE_TEMP_DIR after calibre startup. What gets used in all cases so far is TMPDIR as listed above.

I assume the calibre environment variable CALIBRE_TEMP_DIR requires calibre command line tools first installed (on OS X). So far I haven't tried this with the calibre development environment which is not yet installed. Is setting CALIBRE_TEMP_DIR manually before or after launching the calibre GUI supposed to work? Or is that just with the calibre development environment?

Maybe I'm trying to set the ramdisk on /var/folders/x3/2x81rgjj3lq_2r6wv3_xdkrm0000gn/T too early in the OS X startup and it gets overwritten later by automatic user login initialization.

I'll work on this more tomorrow, including testing a timing delay.

The thought has crossed my mind that figuring this out would be simpler on Windows or Linux. Perhaps when I finally get a new computer it won't be a Mac.

Last edited by unboggling; 10-24-2012 at 09:13 PM. Reason: clarify
unboggling is offline   Reply With Quote
Old 10-24-2012, 10:40 PM   #38
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,773
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
If you set CALIBRE_TEMP_DIR so that it exists before the calibre GUI is launched it will be used in preference to all other variables. I think on OS X that means you have to set it for lauchd via the launchd environment plist file, a bit of googling should tell you how to do that.
kovidgoyal is offline   Reply With Quote
Old 10-26-2012, 07:59 PM   #39
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
Quote:
Originally Posted by kovidgoyal View Post
If you set CALIBRE_TEMP_DIR so that it exists before the calibre GUI is launched it will be used in preference to all other variables. I think on OS X that means you have to set it for lauchd via the launchd environment plist file, a bit of googling should tell you how to do that.
Got it! With CALIBRE_TEMP_DIR set, all the calibre conversion temp files go to that directory. It has to be set by setenv in a suitable launchd configuration file at login, or later using launchctl setenv in Terminal.

Thank you, Kovid and itimpi, for the clues above, and thank you, GRiker, for additional clues in the thread:
https://www.mobileread.com/forums/sho...d.php?t=186069
which included a link to:
http://www.dowdandassociates.com/con...etclaunchdconf
explaining how to set a system-wide environment variable during OS X startup, per the launchd configuration file /etc/launchd.conf. That works for CALIBRE_TEMP_DIR, but it does not work for the environment variable TMPDIR. (The older method of setting environment variables in ~/.MacOSX/environment.plist doesn't work in Mountain Lion, at least for me. And apparently launchd.conf per user is not yet supported.)

I modified the startup script to mount a ramdisk on a new directory CalibreTempDir dedicated to calibre temporary files, set as CALIBRE_TEMP_DIR per /etc/launchd.conf. The script also mounts ramdisks on /private/tmp and /var/run. I was possibly wrong earlier when I said a ramdisk didn't work for /var/folders or any of its subdirectories, including the directory pointed to by that other environment variable TMPDIR. OS X doesn't show the contents of a ramdisk mounted on var/folders or its subdirectories, while OS X does show the contents of ramdisks mounted on /private/tmp, /var/run, and my new CalibreTempDir, so I assumed either it didn't work or wasn't working right. I figure that messing with /var/folders or its subdirectories is too dangerous, until I learn more, such as how to successfully redirect the directory pointed to by TMPDIR to a ramdisk'd directory I can see the contents of.

I was mostly concerned about the calibre conversion temp files, because calibre generates and deletes a lot of them during conversions (deleting any remaining temp files at calibre Quit). So redirecting those temp files to ramdisk preserves the life of the SSD.

This kludge uses hdik and mounts ramdisks on directories. Surprisingly putting calibre conversion temp files on ramdisk results in the same conversion times as with no ramdisk. I suspect the lack of speedup may have something to do with memory management in OS X, though turning dynamic pager off or on didn't affect the speed of calibre conversions. The script I'm using is in this spoiler:
Spoiler:
Code:
#!/bin/bash

# +----------------------------------------------------------------------+
# |                                                                      |
# |  Set up Mac OS X to store temporary files in RAM rather than on disk.|
# |                                                                      |
# |  By Philipp Klaus <http://blog.philippklaus.de>                      |
# |                                                                      |
# |  Originally by Ricardo Gameiro <http://blogs.nullvision.com/?p=357>  |
# |  Changes by Daniel Jenkins                                           |
# |     <http://blogs.nullvision.com/?p=357#comment-1140>                |
# |                                                                      |
# +----------------------------------------------------------------------+

cd /System/Library/StartupItems
sudo mkdir RamFS
sudo chown -R root:wheel RamFS
sudo chmod -R u+rwX,g+rX,o+rX RamFS
cat << "EOF" | sudo tee RamFS/RamFS > /dev/null
#!/bin/sh
# Create a RAM disk with same perms as mountpoint

RAMDisk() {
    mntpt=$1
    rdsize=$(($2*1024*1024/512))
    echo "Creating RamFS for $mntpt"
    # Create the RAM disk.
    dev=`hdik -drivekey system-image=yes -nomount ram://$rdsize`
    # Successfull creation...
    if [ $? -eq 0 ] ; then
        # Create HFS on the RAM volume.
        newfs_hfs $dev
        # Store permissions from old mount point.
        eval `/usr/bin/stat -s $mntpt`
        # Mount the RAM disk to the target mount point.
        mount -t hfs -o union -o nobrowse $dev $mntpt
        # Restore permissions like they were on old volume.
        chown $st_uid:$st_gid $mntpt
        chmod $st_mode $mntpt
    fi
}

# Test for arguments.
if [ -z $1 ]; then
    echo "Usage: $0 [start|stop|restart] "
    exit 1
fi

# Source the common setup functions for startup scripts
test -r /etc/rc.common || exit 1 
. /etc/rc.common

StartService () {
    ConsoleMessage "Starting RamFS disks..."
    RAMDisk /private/tmp 1024
    RAMDisk /var/run 64
    # Next two lines added by unboggling ZZZZZZZZZZZZZZZZZZZZZZZ
    RAMDisk /Users/unboggling/Books/CalibreTempDir 1024
    # RAMDisk /var/folders/ 1024 ZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
    # RAMDisk /var/db 1024
    # mkdir -m 1777 /var/db/mds
}
StopService () {
    ConsoleMessage "Stopping RamFS disks, nothing will be done here..."
    # diskutil unmount /private/tmp /private/var/run
    # diskutil unmount /private/var/run
}

RestartService () {
    ConsoleMessage "Restarting RamFS disks, nothing will be done here..."
}

RunService "$1"
EOF
sudo chmod u+x,g+x,o+x RamFS/RamFS


cat << EOF | sudo tee RamFS/StartupParameters.plist > /dev/null
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist SYSTEM "file://localhost/System/Library/DTDs/PropertyList.dtd">
<plist version="0.9">
    <dict>
        <key>Description</key>
        <string>RamFS Disks Manager</string>
        <key>OrderPreference</key>
        <string>Early</string>
        <key>Provides</key>
        <array>
                <string>RamFS</string>
        </array>
        <key>Uses</key>
        <array>
                <string>Disks</string>
        </array>
    </dict>
</plist>
EOF

Last edited by unboggling; 10-28-2012 at 07:10 PM. Reason: clarify
unboggling is offline   Reply With Quote
Old 10-27-2012, 02:40 AM   #40
cybmole
Wizard
cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.
 
Posts: 3,720
Karma: 1759970
Join Date: Sep 2010
Device: none
Quote:
Originally Posted by speakingtohe View Post
. Just don't understand your motive here. Enlighten me.

Helen
just indicating that you all are wasting hours/days failing to save even seconds:-)

see previous post conclusion:
"Surprisingly putting calibre conversion temp files on ramdisk results in the same conversion times as with no ramdisk."

don't you get that if having temp files in RAM was such a smart move it would be in Windows alreay. Microsoft have good reason for not doing that which google will explain to you if interested.

now I return to to how many angels can I balance on this pin / how many nanoseconds can I shave from this process.
have a nice day & don't waste too much time composing a retort - you may need those seconds

Last edited by cybmole; 10-27-2012 at 02:43 AM.
cybmole is offline   Reply With Quote
Old 10-27-2012, 05:48 AM   #41
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,552
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by cybmole View Post
just indicating that you all are wasting hours/days failing to save even seconds:-)

see previous post conclusion:
"Surprisingly putting calibre conversion temp files on ramdisk results in the same conversion times as with no ramdisk."
That is true in the case of Calibre as it is CPU bound during conversion and does most of the work in RAM (which is why converting large books can be RAM intensive.

However it is not good as a generalization. If a program creates lots of temporary files then using RAM for temporary files can make a bid difference. An example of this that I am aware of is calibre2opds. That creates LOTS of temporary files while running and using a RAM disk can easily give something like a 30% improvement in run time on.

In fact the main use that I had for the CALIBRE_TEMP_DIR setting was the fact that Calibre was prone to leaving the temp files around when it had finished rather than removing them. Giving Calibre its own private temp folder mad it easy to develop a batch file for launching Calibre that tidied up any files left from previous runs.

Quote:
don't you get that if having temp files in RAM was such a smart move it would be in Windows alreay. Microsoft have good reason for not doing that which google will explain to you if interested.
Which is probably why Microsoft are now adding features to Windows that support just this mode of working I suppose!

It is only recently that RAM has become cheap enough that the average user was not better using additional RAM to reduce Windows use of a swap file. However now RAM is cheap enough that it is quite feasible to have more than enough RAM to run without a swap file and still have enough free for uses such as temporary files in RAM. I found that switching off the Windows swap file and using a RAM disk for temporary files made my Windows system much more responsive. I did have 16GB of RAM available in this case which was why I thought it was worth experimenting.

Quote:
now I return to to how many angels can I balance on this pin / how many nanoseconds can I shave from this process.
have a nice day & don't waste too much time composing a retort - you may need those seconds
You seem to have lost site of a secondary objective to this activity. That is if you have a SSD then having the temp files in RAM rather than on the SSD can reduce wear on the SSD. For many who are light users of their machines this will not matter as the SSD will outlast their PC, but if you are a heavy user it can definitely become a factor. For this use knowing how to use RAM for temporary files has a wider benefit than just Calibre.

Then there is simply the interest factor in knowing how to do something like this. It might not be cost-effective but it can still be stimulating to do this sort of investigation

Last edited by itimpi; 10-27-2012 at 05:51 AM.
itimpi is offline   Reply With Quote
Old 10-27-2012, 06:59 AM   #42
cybmole
Wizard
cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.
 
Posts: 3,720
Karma: 1759970
Join Date: Sep 2010
Device: none
what you risk is all sort of paging error / out of space issues in windows because you have to guess and set a Max size RAM disc, which will be far smaller than what windows could have used on your HD / SSD. if your temp files don't all fit into your allocated RAM disc space, expect hours of entertainment? as you try to debug the program failures

& there are specific extra reasons why you should never assign your system page file to a RAM disc, as Google can explain.

if you have an SSD , you almost certainly have a secondary HDD for data. Just assign TMP to that drive to prolong SSD life
cybmole is offline   Reply With Quote
Old 10-27-2012, 09:26 AM   #43
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,552
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by cybmole View Post
what you risk is all sort of paging error / out of space issues in windows because you have to guess and set a Max size RAM disc, which will be far smaller than what windows could have used on your HD / SSD. if your temp files don't all fit into your allocated RAM disc space, expect hours of entertainment? as you try to debug the program failures
True - however I have enough RAM that can set the RAM disk to be an order of magnitude larger than the space that has ever been used on HD (I have a job that has been monitoring that) so this is unlikely to ever be an issue.

Quote:
& there are specific extra reasons why you should never assign your system page file to a RAM disc, as Google can explain.
I agree with that - but as I run without a paging file this is not relevant. If you find you are paging with any frequency then reducing your free RAM to put temp files on a RAM disk would be false economy.

Quote:
if you have an SSD , you almost certainly have a secondary HDD for data. Just assign TMP to that drive to prolong SSD life
Actually I do not any more! Most external data of any size is held on a NAS device so will never be used for TMP. I also have some USB 3 HD but these are unsuitable for TMP as they cannot guaranteed to be there.
itimpi is offline   Reply With Quote
Old 10-27-2012, 10:12 PM   #44
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
@ cybmole & all,
I've been thinking about this a lot the past couple days. I want to gain more familiarity with Unix and even learn some superuser kinds of things with Unix. I care about learning how to do things I didn't previously know how to do, and I also care about applying the KISS Principle to my computer usage. I don't care much about the life of the SSD, although even I recently used that as an excuse, and I don't care much about further improving performance past SSD speed, within the constraints of current speeds of processor, bus's, caches, RAM, storage in/out, etc.

Further discussion in spoiler:
Spoiler:

Life of SSD. Over time the technology improves, SSDs get cheaper and better. The hype about SSD wear and SSD life originates from cost of and problems with early SSDs, what, starting about 10 years ago? At this point, the persisting hype does not apply for the normal user, when combined with wear leveling and keeping the SSD less than half full. As itimpi pointed out, SSD-lifetime may affect heavy users. On the other hand I question even that because most heavy users probably trade up to better tech every 2 years or so anyway. For me the life of this SSD doesn't matter much because I am a relatively normal user, a hobbyist. My most intensive storage i/o is for the OS, several apps including calibre, and the files generated with those apps. I don't do any video editing. I don't do anything with audio except listen to some tunes once in awhile. I am not a "heavy" user such as itimpi mentioned. I am not a company with lots of storage i/o. I am not a server farm. I will replace this 18 month old computer (including this new SSD) with a new faster computer in a year or so. Even if I transfer this same SSD to the new computer this SSD should function well for at least 4 years, which is plenty long enough. For me just lasting 2 years is long enough.

Speed Improvements. Any speed improvements I achieve past SSD speed are just gravy. Speed improvements in performance of calibre or other applications are not necessary because everything is already fast enough for me. When I want to speed things up further I'll buy a new computer with faster processer, caches, RAM, and bus's, along with higher capacity caches, additional cores and processors, more max addressable RAM and at least 64GB RAM, at least 2 internal SSDs, max'd thunderbolt capability, etc. Until then anything else for maximizing this computer probably isn't worth the expense or hassle of implementing or maintaining the changes.

Paging. I turned off paging briefly as an experiment. Otherwise I keep it turned on. I am not advocating turning it off, on Windows or any Unix variant including Linux or OS X. I also keep the paging files (swapfiles on OS X) on the SSD. I never said anything about moving paging files to RAM.

Keep It Simple. When I apply the KISS Principle to my computing situation, I arrive at some KISSi-ized conclusions, recently arrived at in the last couple of days, which is why I didn't mention them up front before asking about using ramdisks for temp files:
  • Don't use ramdisks for anything, not even temp files, unless you are a heavy user with intensive daily i/o usage. Which I'm not.
  • Let the OS handle everything as it was designed to handle everything for most users excluding superusers. Superusers will do whatever they want anyway. Basically what an OS is supposed to do is manage the details for memory, i/o, storage locations, etc. I believe Apple abandoned easy GUI ramdisks on OS X for a reason a couple years ago. Now as far as I can tell, memory management problems that plagued MacOS and early versions of OS X aren't as bad as they used to be.
  • If using SSD(s), keep all files on SSD(s) except backups. Even if preserving the life of the SSDs were important, these considerations apply. Where should the normal user, most users, draw the lines, if in a superuser-type effort to maximize performance, the user starts putting specific directories and files (such as calibre conversion temp files) on ramdisk, SSD, or HDD, depending on various reasons for each type of storage for each type of file? There is an almost never-ending list of specific things to deal with, and then symlinks or other kludges to support each of those many decisions when the decision varies from the "main OS way / most users way" of doing things. Each decision in this arena adds complications and some solutions come laden with specific performance or maintenance issues. It is much KISSier to just keep all files on one or more internal SSDs, with each SSD more than twice the size capacity of all OS, application, and user files on it (including space used by page files aka swapfiles). In addition keep at least 1 preferably 2 hourly (or even daily) backups of each SSD on HDD(s). If an SSD fails, buy a new SSD and restore from backup. No muss, no fuss, easy.

Why learn Unix? Unix has evolved during the 20 years since I was relatively familiar with it, and during that time my Unix skills have atrophied. I've been considering buying a PC rather than a Mac for my next computer. If I get a Windows PC I'll probably add a Linux partition and run mostly from Linux. So I am refamiliarizing with Unix and determined to learn some superuser-type things before then. Which will be valuable even if the next computer is a Mac because OS X is also based on a Unix variant. Plus there is the possibility of putting whatever OS on a partition, or running that OS under emulation, on whichever hardware platform. So current Unix skills will be worthwhile unless all of a sudden I decide I like MS Windows, which didn't happen the last couple of times I had Windows PCs, and I don't envision that starting to happen now. But if that does happen, great, I'll run Windows from a partition or emulation and have some Unix variant in another partition or emulation.

Why learn superuser things? Such as how to use ramdisks, or turn paging on or off, or whatever, even if that knowledge isn't really necessary or won't specifcally be applied? Because its fun.

Why document these kinds of things here? There are some superusers of various OSs that hang out on MobileRead. Hopefully I'll learn something from them, as I recently learned about using CALIBRE_TEMP_DIR from itimpi and kovid. In the case of OS X, many of these types of things are poorly documented, such as using ramdisks. The best documentation is often just the man pages for specific commands of a particular Unix flavor, but first the user must have enough familiarity to know even what to man. Many calibre users on various OSs have SSDs, may be curious how to do ramdisks and what types of files are suitable for ramdisks. Since Adobe and others discussed speeding calibre up on Linux and Windows, I decided to mention similar things for calibre users on OS X, as I learn about such things. That helps me and may help some other users too.


@ itimpi & kovid,
Thanks again for your help.

Last edited by unboggling; 10-28-2012 at 02:24 AM. Reason: clarify
unboggling is offline   Reply With Quote
Old 10-27-2012, 11:13 PM   #45
cybmole
Wizard
cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.
 
Posts: 3,720
Karma: 1759970
Join Date: Sep 2010
Device: none
the discussion is improving!

I do buy into the "it's not about speed, it's about understanding the tech" argument, & calibre conversions are as good as any other testbed.
When I tried to set up ram discs for other reasons, win 7 fought back fiercely, hence googling, hence earlier comments.
PS I think that folks rich enough to try SSD are rich enough to replace the if TMP usage wears them out. Me, I'm waiting until they get much bigger/cheaper.
cybmole is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
How to speedup page turn on sideloaded epub on Kobo Touch with Firmware 2.0 MisterMax Kobo Reader 40 12-06-2012 05:31 AM
Patch: Calibre adds tags to identify ebook formats created by calibre. siebert Calibre 1 07-18-2011 02:07 PM
Speedup Metadata download libertytrooper Library Management 4 07-04-2011 09:40 AM
iLiad Some speedup ideas.. henris42 iRex Developer's Corner 2 05-28-2008 07:59 AM
Processor Speedup? sirmaru Sony Reader Dev Corner 4 01-05-2008 06:23 PM


All times are GMT -4. The time now is 10:10 AM.


MobileRead.com is a privately owned, operated and funded community.