Thread: Calibre Speedup
View Single Post
Old 10-26-2012, 07:59 PM   #39
unboggling
Wizard
unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.unboggling ought to be getting tired of karma fortunes by now.
 
Posts: 1,065
Karma: 858115
Join Date: Jan 2011
Device: Kobo Clara, Kindle Paperwhite 10
Quote:
Originally Posted by kovidgoyal View Post
If you set CALIBRE_TEMP_DIR so that it exists before the calibre GUI is launched it will be used in preference to all other variables. I think on OS X that means you have to set it for lauchd via the launchd environment plist file, a bit of googling should tell you how to do that.
Got it! With CALIBRE_TEMP_DIR set, all the calibre conversion temp files go to that directory. It has to be set by setenv in a suitable launchd configuration file at login, or later using launchctl setenv in Terminal.

Thank you, Kovid and itimpi, for the clues above, and thank you, GRiker, for additional clues in the thread:
https://www.mobileread.com/forums/sho...d.php?t=186069
which included a link to:
http://www.dowdandassociates.com/con...etclaunchdconf
explaining how to set a system-wide environment variable during OS X startup, per the launchd configuration file /etc/launchd.conf. That works for CALIBRE_TEMP_DIR, but it does not work for the environment variable TMPDIR. (The older method of setting environment variables in ~/.MacOSX/environment.plist doesn't work in Mountain Lion, at least for me. And apparently launchd.conf per user is not yet supported.)

I modified the startup script to mount a ramdisk on a new directory CalibreTempDir dedicated to calibre temporary files, set as CALIBRE_TEMP_DIR per /etc/launchd.conf. The script also mounts ramdisks on /private/tmp and /var/run. I was possibly wrong earlier when I said a ramdisk didn't work for /var/folders or any of its subdirectories, including the directory pointed to by that other environment variable TMPDIR. OS X doesn't show the contents of a ramdisk mounted on var/folders or its subdirectories, while OS X does show the contents of ramdisks mounted on /private/tmp, /var/run, and my new CalibreTempDir, so I assumed either it didn't work or wasn't working right. I figure that messing with /var/folders or its subdirectories is too dangerous, until I learn more, such as how to successfully redirect the directory pointed to by TMPDIR to a ramdisk'd directory I can see the contents of.

I was mostly concerned about the calibre conversion temp files, because calibre generates and deletes a lot of them during conversions (deleting any remaining temp files at calibre Quit). So redirecting those temp files to ramdisk preserves the life of the SSD.

This kludge uses hdik and mounts ramdisks on directories. Surprisingly putting calibre conversion temp files on ramdisk results in the same conversion times as with no ramdisk. I suspect the lack of speedup may have something to do with memory management in OS X, though turning dynamic pager off or on didn't affect the speed of calibre conversions. The script I'm using is in this spoiler:
Spoiler:
Code:
#!/bin/bash

# +----------------------------------------------------------------------+
# |                                                                      |
# |  Set up Mac OS X to store temporary files in RAM rather than on disk.|
# |                                                                      |
# |  By Philipp Klaus <http://blog.philippklaus.de>                      |
# |                                                                      |
# |  Originally by Ricardo Gameiro <http://blogs.nullvision.com/?p=357>  |
# |  Changes by Daniel Jenkins                                           |
# |     <http://blogs.nullvision.com/?p=357#comment-1140>                |
# |                                                                      |
# +----------------------------------------------------------------------+

cd /System/Library/StartupItems
sudo mkdir RamFS
sudo chown -R root:wheel RamFS
sudo chmod -R u+rwX,g+rX,o+rX RamFS
cat << "EOF" | sudo tee RamFS/RamFS > /dev/null
#!/bin/sh
# Create a RAM disk with same perms as mountpoint

RAMDisk() {
    mntpt=$1
    rdsize=$(($2*1024*1024/512))
    echo "Creating RamFS for $mntpt"
    # Create the RAM disk.
    dev=`hdik -drivekey system-image=yes -nomount ram://$rdsize`
    # Successfull creation...
    if [ $? -eq 0 ] ; then
        # Create HFS on the RAM volume.
        newfs_hfs $dev
        # Store permissions from old mount point.
        eval `/usr/bin/stat -s $mntpt`
        # Mount the RAM disk to the target mount point.
        mount -t hfs -o union -o nobrowse $dev $mntpt
        # Restore permissions like they were on old volume.
        chown $st_uid:$st_gid $mntpt
        chmod $st_mode $mntpt
    fi
}

# Test for arguments.
if [ -z $1 ]; then
    echo "Usage: $0 [start|stop|restart] "
    exit 1
fi

# Source the common setup functions for startup scripts
test -r /etc/rc.common || exit 1 
. /etc/rc.common

StartService () {
    ConsoleMessage "Starting RamFS disks..."
    RAMDisk /private/tmp 1024
    RAMDisk /var/run 64
    # Next two lines added by unboggling ZZZZZZZZZZZZZZZZZZZZZZZ
    RAMDisk /Users/unboggling/Books/CalibreTempDir 1024
    # RAMDisk /var/folders/ 1024 ZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
    # RAMDisk /var/db 1024
    # mkdir -m 1777 /var/db/mds
}
StopService () {
    ConsoleMessage "Stopping RamFS disks, nothing will be done here..."
    # diskutil unmount /private/tmp /private/var/run
    # diskutil unmount /private/var/run
}

RestartService () {
    ConsoleMessage "Restarting RamFS disks, nothing will be done here..."
}

RunService "$1"
EOF
sudo chmod u+x,g+x,o+x RamFS/RamFS


cat << EOF | sudo tee RamFS/StartupParameters.plist > /dev/null
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist SYSTEM "file://localhost/System/Library/DTDs/PropertyList.dtd">
<plist version="0.9">
    <dict>
        <key>Description</key>
        <string>RamFS Disks Manager</string>
        <key>OrderPreference</key>
        <string>Early</string>
        <key>Provides</key>
        <array>
                <string>RamFS</string>
        </array>
        <key>Uses</key>
        <array>
                <string>Disks</string>
        </array>
    </dict>
</plist>
EOF

Last edited by unboggling; 10-28-2012 at 07:10 PM. Reason: clarify
unboggling is offline   Reply With Quote