Calibre Speedup

[theducks]

The Tag browser (usually) remembers those pesky custom column names
I just open the tag browser, select my choices, the close the browse to get my screen width back

[unboggling]

Originally Posted by theducks

The Tag browser (usually) remembers those pesky custom column names
I just open the tag browser, select my choices, the close the browse to get my screen width back

Yeah. But it's faster to just type the lookup names in search box, if no mistakes are made.

[lbik]

The best way to speedup Calibre, ist an SSD for the Library.

[ilovejedd]

Originally Posted by lbik

The best way to speedup Calibre, ist an SSD for the Library.

I completely agree. Well, it does help for the library folder itself but if space is a premium, then you can use it just for metadata.db. Of course, RAM disk is so much faster.

[unboggling]

Originally Posted by lbik

The best way to speedup Calibre, ist an SSD for the Library.

Originally Posted by ilovejedd

I completely agree. Well, it does help for the library folder itself but if space is a premium, then you can use it just for metadata.db. Of course, RAM disk is so much faster.

FYI.

I upgraded startup drive to SSD from hard disk, and RAM to 16 GB from 8 GB, in an early 2011 Macbook Pro 17 inch notebook that has a 2.2 GHz quad core i7 processor. Did it myself.

Apple says this model only can be upgraded to 8 GB maximum RAM, but some research showed the real max is 16 GB. I haven't had any problems since installing the 16 GB RAM last week.

The SSD has 6 gigabit link speed, while the old hard disk has 3 gigabit. To preserve the life of the SSD, I turned on trim support and noatime, turned off local Time Machine snapshots, turned off sleep mode memory backup to persistent storage, and disabled sleep disk image.

Speedup results, without doing anything with temp folders and ramdisk:

• OS X startup time: more than twice as fast on SSD, a significant difference.
• calibre startup time: 13 seconds on SSD, 18 seconds on hard drive, 25 to 30% faster with calibre app, libraries, and config dir all on SSD instead of hard drive. That is with startup restriction off. (Starting up in restriction requiress extra time for the restriction search).
• bulk conversion time: shaved 1 or 2 seconds off each conversion during bulk conversion operation, approx 5% faster on SSD. Not as much speedup as I expected.

[speakingtohe]

Originally Posted by unboggling

FYI.

I upgraded startup drive to SSD from hard disk, and RAM from 8 GB to 16 GB, in an early 2011 Macbook Pro 17 inch notebook that has a 2.2 GHz quad core i7 processor. Did it myself.

Apple says this model only can be upgraded to 8 GB maximum RAM, but some research showed the real max is 16 GB. I haven't had any problems since installing the 16 GB RAM last week.

The SSD has 6 gigabit link speed, while the old hard disk has 3 gigabit. To preserve the life of the SSD, I turned on trim support and noatime, turned off local Time Machine snapshots, turned off sleep mode memory backup to persistent storage, and disabled sleep disk image.

Speedup results, without doing anything with temp folders and ramdisk:

• OS X startup time: over 500% faster with SSD, a significant difference.
• calibre startup time: faster by approx 1 second, approx 5%, with calibre app, libraries, and config dir all on SSD. Not as much speedup as I expected.
• bulk conversion time: shaved 1 or 2 seconds off each conversion during bulk conversion operation, approx 5% faster on SSD. Not as much speedup as I expected.

Interesting.

I am not really surprised that calibre does not convert faster as it is pretty cpu intensive and you have lots of RAM.

My calibre startup time seems to be about 10 seconds (I have way too many books in there) with SATA HD and 8G RAM, so even 5% does not excite me.

Nice to see some actual numbers though.

Thanks

Helen

[itimpi]

On Windows I found that using a RAM disk for temporary files helped speed thins up. I presume the same an be done on MacOS ? It also has the additional benefit of reducing wear on the SSD.

[unboggling]

Originally Posted by speakingtohe

Interesting.

I am not really surprised that calibre does not convert faster as it is pretty cpu intensive and you have lots of RAM.

My calibre startup time seems to be about 10 seconds (I have way too many books in there) with SATA HD and 8G RAM, so even 5% does not excite me.

Nice to see some actual numbers though.

Thanks

Helen

Oops, I made some mistakes earlier so retested with variables more leveled, and revised my prior post with the corrections. OS X startup after shutdown is twice as fast, not five times as fast. And I'd previously tested with calibre startup restriction off, but on SSD with startup restriction on. I turned off the startup restriction, removed some recently added books from library that were added since testing with the hard disk drive, and got speedier results, over 25% faster without restriction than with restriction.

[unboggling]

Originally Posted by itimpi

On Windows I found that using a RAM disk for temporary files helped speed thins up. I presume the same an be done on MacOS ? It also has the additional benefit of reducing wear on the SSD.

Regarding temp files and RAMdisk, I'm still researching and testing exactly which temp folders would be appropriate. Apparently on OS X, not all temp folders should be put on RAMdisk, such as /var/db, while /private/tmp and /var/run are okay on RAMdisk. So I'm still trying to determine which specific temp folders appropriate for RAMdisk would speed up various specific calibre operations. I noticed that on OS X, conversion uses /var/folders. Does anyone know all the other specific temp folders that various calibre operations use on OS X, or where I could look to determine that?

What size RAMdisk do you use?

[kovidgoyal]

calibre uses only /tmp

On OS X, /tmp is usually a symlink pointing elsewhere.

[unboggling]

Originally Posted by kovidgoyal

calibre uses only /tmp

On OS X, /tmp is usually a symlink pointing elsewhere.

Thanks. So if I understand correctly how symlinks work, that means I still don't know which temp folders to mount as ramdisks. I've done some testing, trying to adapt the script in the spoiler to mount relevant temp folders each to its own ramdisk at startup. Script is from a blog article (which took it from GitHub):

http://blog.alutam.com/2012/04/01/op.../#time-machine

https://raw.github.com/gist/931579/1...disk_MacOSX.sh

It works but doesn't speed up calibre startup or bulk conversions. Tried various ramdisk sizes. I added a line for another ramdisk for /var/folders which is where calibre conversions seem to happen on OS X. /var/folders actually mounts on ramdisk as /private/var/folders. But that didn't speed up bulk conversions either. So I commented out my extra added line.

Obviously I'm missing or misunderstanding something. I'm new to command line and symlink. Is there a way with command line to figure out exactly which folders the symlink /tmp points to?

Spoiler:

#!/bin/bash

# +----------------------------------------------------------------------+
# |                                                                      |
# |  Set up Mac OS X to store temporary files in RAM rather than on disk.|
# |                                                                      |
# |  By Philipp Klaus <http://blog.philippklaus.de>                      |
# |                                                                      |
# |  Originally by Ricardo Gameiro <http://blogs.nullvision.com/?p=357>  |
# |  Changes by Daniel Jenkins                                           |
# |     <http://blogs.nullvision.com/?p=357#comment-1140>                |
# |                                                                      |
# +----------------------------------------------------------------------+

cd /System/Library/StartupItems
sudo mkdir RamFS
sudo chown -R root:wheel RamFS
sudo chmod -R u+rwX,g+rX,o+rX RamFS
cat << "EOF" | sudo tee RamFS/RamFS > /dev/null
#!/bin/sh
# Create a RAM disk with same perms as mountpoint

RAMDisk() {
    mntpt=$1
    rdsize=$(($2*1024*1024/512))
    echo "Creating RamFS for $mntpt"
    # Create the RAM disk.
    dev=`hdik -drivekey system-image=yes -nomount ram://$rdsize`
    # Successfull creation...
    if [ $? -eq 0 ] ; then
        # Create HFS on the RAM volume.
        newfs_hfs $dev
        # Store permissions from old mount point.
        eval `/usr/bin/stat -s $mntpt`
        # Mount the RAM disk to the target mount point.
        mount -t hfs -o union -o nobrowse $dev $mntpt
        # Restore permissions like they were on old volume.
        chown $st_uid:$st_gid $mntpt
        chmod $st_mode $mntpt
    fi
}

# Test for arguments.
if [ -z $1 ]; then
    echo "Usage: $0 [start|stop|restart] "
    exit 1
fi

# Source the common setup functions for startup scripts
test -r /etc/rc.common || exit 1 
. /etc/rc.common

StartService () {
    ConsoleMessage "Starting RamFS disks..."
    RAMDisk /private/tmp 1024
    RAMDisk /var/run 128
    # RAMDisk /var/folders added by unboggling. 
    # RAMDisk /var/folders 1024
    # RAMDisk /var/db 1024
    # mkdir -m 1777 /var/db/mds
}
StopService () {
    ConsoleMessage "Stopping RamFS disks, nothing will be done here..."
    # diskutil unmount /private/tmp /private/var/run
    # diskutil unmount /private/var/run
}

RestartService () {
    ConsoleMessage "Restarting RamFS disks, nothing will be done here..."
}

RunService "$1"
EOF
sudo chmod u+x,g+x,o+x RamFS/RamFS


cat << EOF | sudo tee RamFS/StartupParameters.plist > /dev/null
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist SYSTEM "file://localhost/System/Library/DTDs/PropertyList.dtd">
<plist version="0.9">
    <dict>
        <key>Description</key>
        <string>RamFS Disks Manager</string>
        <key>OrderPreference</key>
        <string>Early</string>
        <key>Provides</key>
        <array>
                <string>RamFS</string>
        </array>
        <key>Uses</key>
        <array>
                <string>Disks</string>
        </array>
    </dict>
</plist>
EOF

[kovidgoyal]

I'm not an OS X user so I can't help. Basically you need to find out where /tmp points to and ensure that location is mounted in RAM.

[cybmole]

sheesh - how often do you guys NEED to re-convert your books ?

what do you plan to do with the saved seconds , apart from contribute even more to this thread

[unboggling]

Originally Posted by kovidgoyal

I'm not an OS X user so I can't help. Basically you need to find out where /tmp points to and ensure that location is mounted in RAM.

Thanks.

I guess I need to buy a book with discussion of temp files, symlinks, and ramdisks on OS X Mountain Lion.

For now I'm not using ramdisk until I know what I'm doing.

[unboggling]

Originally Posted by cybmole

sheesh - how often do you guys NEED to re-convert your books ?

what do you plan to do with the saved seconds , apart from contribute even more to this thread

I convert all books that I add to calibre. But I'm trying to speed up conversions with temp files on ramdisks mainly out of curiosity, not necessity.