ebook-convert doesn't add converted version to library?

[Ackis]

I decided to write a quick bash script to convert my ebook formats into epub which Marvin can read (bloody wife is using the Android so I can't use moonreader). When I converted a few files, they weren't automatically added to the Calibre library. Is that intended behaviour, and if so what's the best way to work around it?

As an aside, should I run ebook-polish on newly converted files (e.g. remove unused CSS)?

#! /bin/bash

fullfilename=${0##*/}
filename="${fullfilename%.*}"
logfile="/var/log/scripts/$filename.log"
convertpath="/tmp/ebookconvert"

echo "Script output will be logged to $logfile"

#exec > "$logfile" 2>&1

/opt/scripts/scriptheader.sh "eBook Convert"

test -e "$convertpath" || mkdir "$convertpath"

if ! [[ -f "/usr/bin/ebook-convert" ]]; then
        echo "ebook-convert not found. Aborting."
        exit 1
fi

if [[ -z $1 ]]; then
        echo "No argument supplied.  Please enter the path you which to convert."
        exit 1
elif ! [[ -d "$1" ]]; then
        echo "Argument passed is not a directory."
        exit 1
fi

echo "Scanning $1 for ebooks files..."

# List the types of files in the directory
echo "eBook file types detected:"
find "$1" -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u

# Find the media types that will be converted.
# Useful for verifying after the fact.
echo "Files will be copied to '$convertpath' before conversion."

find "$1" -type f -name "*.mobi" -o -type f -name "*.lit" -o -type f -name "*.pdf" > "$convertpath/ebooklist.txt"

echo "Converting MOBI Files."

find "$1" -type f -name "*.mobi" | while read line; do
        # Check to see if we already have the format.
        if ! [[ -f "${line%.mobi}.epub" ]]; then
                /usr/bin/ebook-convert "$line" "${line%.mobi}.epub";
                if [ $? -ne 0 ]; then
                        echo "Error converting $line.";
                fi
        else
                echo "ePub for $line exists already, no conversion needed.";
        fi
done

[twowheels]

IME it does not. I think that it's just the tool that the GUI application calls to perform the translation step, but the GUI is responsible for all of the "bookkeeping" related to the db.

I personally like it that way as I don't use any of the features of Calibre beyond that one tool -- in fact, I wish I could download the backend CLI tools separate from the GUI.

[eschwartz]

You can use

calibredb add

to add books to calibre's library.

[Ackis]

Originally Posted by eschwartz

You can use

calibredb add

to add books to calibre's library.

I had tried that already actually. If it's a net new book that will work, but if it's a conversion of an existing book format it didn't work. There's an option to add a book format with the calibredb program, however you need to know the ID of the book, which you only can get via the list command.

[eschwartz]

Are you converting the books from the calibre directory? If so, you can extract the id from the folder location. Or parse the metadata.opf

It is also possible to use calibre-debug to run calibre's code to trigger a library convert, but that is above my paygrade -- I don't know how.

[eschwartz]

Just curious -- is there a reason you use find and while read line, instead of "for file in *.mobi"

And instead of

fullfilename=${0##*/}
filename="${fullfilename%.*}"

you can use

filename="$(basename $0)"

And

test -e "$convertpath" || mkdir "$convertpath"

could be changed to

mkdir -p "$convertpath"

[kovidgoyal]

You're doing it wrong

You want to start with calibredb list to get a list of all books and their ids,

calibredb list -f formats

Then iterate over the books, converting them, sending the converted output to a temp file ad add the temp file like this

calibredb add_format book_id tmpfile

[eschwartz]

#! /bin/bash

filename="$(basename $0)"
logfile="/var/log/scripts/$filename.log"
convertpath="/tmp/ebookconvert"

echo "Script output will be logged to $logfile"

#exec > "$logfile" 2>&1

/opt/scripts/scriptheader.sh "eBook Convert"

mkdir -p "$convertpath" 

if ! [[ -f "/usr/bin/ebook-convert" ]]; then
        echo "ebook-convert not found. Aborting."
        exit 1
fi

if [[ -z $1 ]]; then
        echo "No argument supplied.  Please enter the path you which to convert."
        exit 1
elif ! [[ -d "$1" ]]; then
        echo "Argument passed is not a directory."
        exit 1
fi

# Entering the directory
cd $1

echo "Scanning $1 for ebooks files..."

# List the types of files in the directory, with frequency
echo "eBook file types detected:"
find . -type f | sed -rn 's|.*/[^/]+\.([^/.]+)$|\1|p' | sort | uniq -c

# Find the media types that will be converted.
# Useful for verifying after the fact.
echo "Files will be copied to '$convertpath' before conversion."

find . -type f -name "*.mobi" -o -type f -name "*.lit" -o -type f -name "*.pdf" > "$convertpath/ebooklist.txt"

echo "Converting MOBI Files."

# IFS change for file names with spaces in them
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")

for file in *.mobi; do
    basefile=$(basename "$file")
    # Check to see if we already have the format.
    if ! [[ -f "$basefile.epub" ]]; then
        /usr/bin/ebook-convert "$file" "$basefile.epub";

            if [ $? -ne 0 ]; then
                echo "Error converting $file.";
            fi

        /usr/bin/calibredb add $file
    else
        echo "ePub for $file exists already, no conversion needed.";
    fi
done

# restore default IFS
IFS=$SAVEIFS

[Ackis]

Originally Posted by eschwartz

Just curious -- is there a reason you use find and while read line, instead of "for file in *.mobi"

The main reason why I've done them that way is because I think this is the second script I've written. Just lack of knowledge.

[eschwartz]

Originally Posted by Ackis

The main reason why I've done them that way is because I think this is the second script I've written. Just lack of knowledge.

Been there, done that. It gets better.

A general recommendation seems to be, use as few commands to get a job done. "for" can do quite a few interesting things on its own.
It took me a while to catch on to basename.

Happy scripting!

... trying to figure out how to parse calibredb list output...

[eschwartz]

Originally Posted by kovidgoyal

You're doing it wrong

You want to start with calibredb list to get a list of all books and their ids,

calibredb list -f formats

Then iterate over the books, converting them, sending the converted output to a temp file ad add the temp file like this

calibredb add_format book_id tmpfile

...

something like this...

#!/bin/bash

# Search for which books we want. Find books that aren't in EPUB yet.
calibre-search='not formats:"=EPUB"'
regex="(\d+) \[([^\]]*)\]"
# Search calibredb, iterate over each result.
calibredb list -s "$calibre-search" -f formats | while read line; do
# Extract info from lines into bash_rematch
    [[ "$line" =~ "$regex" ]]
    ebook-convert "${BASH_REMATCH[2]}" "/tmp/converted.epub"
    calibredb add_format "${BASH_REMATCH[1]}" "/tmp/converted.epub"
done

[kovidgoyal]

Better, but it wont handle books that have multiple formats, for which there will be multiple entries inside the [] brackets.

And now that I think about it, there is actually a bug in calibredb list. It should not be separating multiple path by commas, as commas can occur in filenames. Now fixed: https://github.com/kovidgoyal/calibr...4e95f15101ec6c

[kovidgoyal]

Since I was here in any case, I decided to add a --for-machine option to calibredb list that makes it easier to parse the output.

And here's a one liner (in python because I hate sh) that demonstrates how to get a list of all ids and formats for ever id in the library

calibredb list --for-machine -f formats | python2 -c "import sys; print '\n\n'.join('id:%s formats:%r' % (r.partition(chr(29))[0], r.partition(chr(29))[-1].split(chr(28))) for r in sys.stdin.read().split(chr(30)) if r.strip())"

And the help of --for-machine

  --for-machine         Generate output that is more suitable for machine
                        parsing. Book entries are separates using the ASCII
                        Record Separator character (30), individual fields in
                        an entry using the Group Separator (29) and individual
                        filenames using the File Separator (28)

[eschwartz]

Thanks! I will look into using this...

[Ackis]

Originally Posted by eschwartz

Been there, done that. It gets better.

A general recommendation seems to be, use as few commands to get a job done. "for" can do quite a few interesting things on its own.
It took me a while to catch on to basename.

Happy scripting!

... trying to figure out how to parse calibredb list output...

Thanks. When I'm learning something new I tend to be as verbose as possible with the code and even the comments just to make it a bit easier for me to understand.

Originally Posted by kovidgoyal

Better, but it wont handle books that have multiple formats, for which there will be multiple entries inside the [] brackets.

And now that I think about it, there is actually a bug in calibredb list. It should not be separating multiple path by commas, as commas can occur in filenames. Now fixed: https://github.com/kovidgoyal/calibr...4e95f15101ec6c

Yay, at least something good came out of my mucking around.

Originally Posted by kovidgoyal

Since I was here in any case, I decided to add a --for-machine option to calibredb list that makes it easier to parse the output.

And here's a one liner (in python because I hate sh) that demonstrates how to get a list of all ids and formats for ever id in the library

calibredb list --for-machine -f formats | python2 -c "import sys; print '\n\n'.join('id:%s formats:%r' % (r.partition(chr(29))[0], r.partition(chr(29))[-1].split(chr(28))) for r in sys.stdin.read().split(chr(30)) if r.strip())"

And the help of --for-machine

  --for-machine         Generate output that is more suitable for machine
                        parsing. Book entries are separates using the ASCII
                        Record Separator character (30), individual fields in
                        an entry using the Group Separator (29) and individual
                        filenames using the File Separator (28)

Thank you for adding that in. Python is on my to-do list. :P