Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Conversion

Notices

Reply
 
Thread Tools Search this Thread
Old 07-22-2014, 05:12 PM   #1
Ackis
Member
Ackis began at the beginning.
 
Posts: 17
Karma: 10
Join Date: Sep 2013
Location: Canada
Device: Galaxy Tab 2 10.1, iPad 2
ebook-convert doesn't add converted version to library?

I decided to write a quick bash script to convert my ebook formats into epub which Marvin can read (bloody wife is using the Android so I can't use moonreader). When I converted a few files, they weren't automatically added to the Calibre library. Is that intended behaviour, and if so what's the best way to work around it?

As an aside, should I run ebook-polish on newly converted files (e.g. remove unused CSS)?

Code:
#! /bin/bash

fullfilename=${0##*/}
filename="${fullfilename%.*}"
logfile="/var/log/scripts/$filename.log"
convertpath="/tmp/ebookconvert"

echo "Script output will be logged to $logfile"

#exec > "$logfile" 2>&1

/opt/scripts/scriptheader.sh "eBook Convert"

test -e "$convertpath" || mkdir "$convertpath"

if ! [[ -f "/usr/bin/ebook-convert" ]]; then
        echo "ebook-convert not found. Aborting."
        exit 1
fi

if [[ -z $1 ]]; then
        echo "No argument supplied.  Please enter the path you which to convert."
        exit 1
elif ! [[ -d "$1" ]]; then
        echo "Argument passed is not a directory."
        exit 1
fi

echo "Scanning $1 for ebooks files..."

# List the types of files in the directory
echo "eBook file types detected:"
find "$1" -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u

# Find the media types that will be converted.
# Useful for verifying after the fact.
echo "Files will be copied to '$convertpath' before conversion."

find "$1" -type f -name "*.mobi" -o -type f -name "*.lit" -o -type f -name "*.pdf" > "$convertpath/ebooklist.txt"

echo "Converting MOBI Files."

find "$1" -type f -name "*.mobi" | while read line; do
        # Check to see if we already have the format.
        if ! [[ -f "${line%.mobi}.epub" ]]; then
                /usr/bin/ebook-convert "$line" "${line%.mobi}.epub";
                if [ $? -ne 0 ]; then
                        echo "Error converting $line.";
                fi
        else
                echo "ePub for $line exists already, no conversion needed.";
        fi
done
Ackis is offline   Reply With Quote
Old 07-22-2014, 05:29 PM   #2
twowheels
Wizard
twowheels ought to be getting tired of karma fortunes by now.twowheels ought to be getting tired of karma fortunes by now.twowheels ought to be getting tired of karma fortunes by now.twowheels ought to be getting tired of karma fortunes by now.twowheels ought to be getting tired of karma fortunes by now.twowheels ought to be getting tired of karma fortunes by now.twowheels ought to be getting tired of karma fortunes by now.twowheels ought to be getting tired of karma fortunes by now.twowheels ought to be getting tired of karma fortunes by now.twowheels ought to be getting tired of karma fortunes by now.twowheels ought to be getting tired of karma fortunes by now.
 
twowheels's Avatar
 
Posts: 1,171
Karma: 8269419
Join Date: Nov 2010
Device: KPW/iPad2/5S/Tab3 Past:NewtonOMP>N120> Palm3>3c> iPAQ3850>Droid>K3>DX
IME it does not. I think that it's just the tool that the GUI application calls to perform the translation step, but the GUI is responsible for all of the "bookkeeping" related to the db.

I personally like it that way as I don't use any of the features of Calibre beyond that one tool -- in fact, I wish I could download the backend CLI tools separate from the GUI.
twowheels is offline   Reply With Quote
 
Enthusiast
Old 07-22-2014, 10:09 PM   #3
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
Posts: 6,691
Karma: 11161085
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
You can use
Code:
calibredb add
to add books to calibre's library.
eschwartz is offline   Reply With Quote
Old 07-22-2014, 11:44 PM   #4
Ackis
Member
Ackis began at the beginning.
 
Posts: 17
Karma: 10
Join Date: Sep 2013
Location: Canada
Device: Galaxy Tab 2 10.1, iPad 2
Quote:
Originally Posted by eschwartz View Post
You can use
Code:
calibredb add
to add books to calibre's library.
I had tried that already actually. If it's a net new book that will work, but if it's a conversion of an existing book format it didn't work. There's an option to add a book format with the calibredb program, however you need to know the ID of the book, which you only can get via the list command.
Ackis is offline   Reply With Quote
Old 07-23-2014, 12:04 AM   #5
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
Posts: 6,691
Karma: 11161085
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Are you converting the books from the calibre directory? If so, you can extract the id from the folder location. Or parse the metadata.opf

It is also possible to use calibre-debug to run calibre's code to trigger a library convert, but that is above my paygrade -- I don't know how.
eschwartz is offline   Reply With Quote
Old 07-23-2014, 12:08 AM   #6
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
Posts: 6,691
Karma: 11161085
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Just curious -- is there a reason you use find and while read line, instead of "for file in *.mobi"

And instead of

Code:
fullfilename=${0##*/}
filename="${fullfilename%.*}"
you can use

Code:
filename="$(basename $0)"
And
Code:
test -e "$convertpath" || mkdir "$convertpath"
could be changed to
Code:
mkdir -p "$convertpath"
eschwartz is offline   Reply With Quote
Old 07-23-2014, 12:10 AM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,154
Karma: 5381911
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You're doing it wrong

You want to start with calibredb list to get a list of all books and their ids,

calibredb list -f formats

Then iterate over the books, converting them, sending the converted output to a temp file ad add the temp file like this

calibredb add_format book_id tmpfile
kovidgoyal is offline   Reply With Quote
Old 07-23-2014, 12:11 AM   #8
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
Posts: 6,691
Karma: 11161085
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Code:
#! /bin/bash

filename="$(basename $0)"
logfile="/var/log/scripts/$filename.log"
convertpath="/tmp/ebookconvert"

echo "Script output will be logged to $logfile"

#exec > "$logfile" 2>&1

/opt/scripts/scriptheader.sh "eBook Convert"

mkdir -p "$convertpath" 

if ! [[ -f "/usr/bin/ebook-convert" ]]; then
        echo "ebook-convert not found. Aborting."
        exit 1
fi

if [[ -z $1 ]]; then
        echo "No argument supplied.  Please enter the path you which to convert."
        exit 1
elif ! [[ -d "$1" ]]; then
        echo "Argument passed is not a directory."
        exit 1
fi

# Entering the directory
cd $1

echo "Scanning $1 for ebooks files..."

# List the types of files in the directory, with frequency
echo "eBook file types detected:"
find . -type f | sed -rn 's|.*/[^/]+\.([^/.]+)$|\1|p' | sort | uniq -c

# Find the media types that will be converted.
# Useful for verifying after the fact.
echo "Files will be copied to '$convertpath' before conversion."

find . -type f -name "*.mobi" -o -type f -name "*.lit" -o -type f -name "*.pdf" > "$convertpath/ebooklist.txt"

echo "Converting MOBI Files."

# IFS change for file names with spaces in them
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")

for file in *.mobi; do
    basefile=$(basename "$file")
    # Check to see if we already have the format.
    if ! [[ -f "$basefile.epub" ]]; then
        /usr/bin/ebook-convert "$file" "$basefile.epub";

            if [ $? -ne 0 ]; then
                echo "Error converting $file.";
            fi

        /usr/bin/calibredb add $file
    else
        echo "ePub for $file exists already, no conversion needed.";
    fi
done

# restore default IFS
IFS=$SAVEIFS
eschwartz is offline   Reply With Quote
Old 07-23-2014, 12:42 AM   #9
Ackis
Member
Ackis began at the beginning.
 
Posts: 17
Karma: 10
Join Date: Sep 2013
Location: Canada
Device: Galaxy Tab 2 10.1, iPad 2
Quote:
Originally Posted by eschwartz View Post
Just curious -- is there a reason you use find and while read line, instead of "for file in *.mobi"
The main reason why I've done them that way is because I think this is the second script I've written. Just lack of knowledge.
Ackis is offline   Reply With Quote
Old 07-23-2014, 12:58 AM   #10
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
Posts: 6,691
Karma: 11161085
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Quote:
Originally Posted by Ackis View Post
The main reason why I've done them that way is because I think this is the second script I've written. Just lack of knowledge.
Been there, done that. It gets better.

A general recommendation seems to be, use as few commands to get a job done. "for" can do quite a few interesting things on its own.
It took me a while to catch on to basename.

Happy scripting!

... trying to figure out how to parse calibredb list output...
eschwartz is offline   Reply With Quote
Old 07-23-2014, 01:47 AM   #11
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
Posts: 6,691
Karma: 11161085
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Quote:
Originally Posted by kovidgoyal View Post
You're doing it wrong

You want to start with calibredb list to get a list of all books and their ids,

calibredb list -f formats

Then iterate over the books, converting them, sending the converted output to a temp file ad add the temp file like this

calibredb add_format book_id tmpfile


...

something like this...
Code:
#!/bin/bash

# Search for which books we want. Find books that aren't in EPUB yet.
calibre-search='not formats:"=EPUB"'
regex="(\d+) \[([^\]]*)\]"
# Search calibredb, iterate over each result.
calibredb list -s "$calibre-search" -f formats | while read line; do
# Extract info from lines into bash_rematch
    [[ "$line" =~ "$regex" ]]
    ebook-convert "${BASH_REMATCH[2]}" "/tmp/converted.epub"
    calibredb add_format "${BASH_REMATCH[1]}" "/tmp/converted.epub"
done
eschwartz is offline   Reply With Quote
Old 07-23-2014, 02:01 AM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,154
Karma: 5381911
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Better, but it wont handle books that have multiple formats, for which there will be multiple entries inside the [] brackets.

And now that I think about it, there is actually a bug in calibredb list. It should not be separating multiple path by commas, as commas can occur in filenames. Now fixed: https://github.com/kovidgoyal/calibr...4e95f15101ec6c
kovidgoyal is offline   Reply With Quote
Old 07-23-2014, 03:35 AM   #13
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,154
Karma: 5381911
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Since I was here in any case, I decided to add a --for-machine option to calibredb list that makes it easier to parse the output.

And here's a one liner (in python because I hate sh) that demonstrates how to get a list of all ids and formats for ever id in the library

Code:
calibredb list --for-machine -f formats | python2 -c "import sys; print '\n\n'.join('id:%s formats:%r' % (r.partition(chr(29))[0], r.partition(chr(29))[-1].split(chr(28))) for r in sys.stdin.read().split(chr(30)) if r.strip())"
And the help of --for-machine

Code:
  --for-machine         Generate output that is more suitable for machine
                        parsing. Book entries are separates using the ASCII
                        Record Separator character (30), individual fields in
                        an entry using the Group Separator (29) and individual
                        filenames using the File Separator (28)
kovidgoyal is offline   Reply With Quote
Old 07-23-2014, 08:28 AM   #14
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
Posts: 6,691
Karma: 11161085
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Thanks! I will look into using this...
eschwartz is offline   Reply With Quote
Old 07-23-2014, 02:32 PM   #15
Ackis
Member
Ackis began at the beginning.
 
Posts: 17
Karma: 10
Join Date: Sep 2013
Location: Canada
Device: Galaxy Tab 2 10.1, iPad 2
Quote:
Originally Posted by eschwartz View Post
Been there, done that. It gets better.

A general recommendation seems to be, use as few commands to get a job done. "for" can do quite a few interesting things on its own.
It took me a while to catch on to basename.

Happy scripting!

... trying to figure out how to parse calibredb list output...
Thanks. When I'm learning something new I tend to be as verbose as possible with the code and even the comments just to make it a bit easier for me to understand.

Quote:
Originally Posted by kovidgoyal View Post
Better, but it wont handle books that have multiple formats, for which there will be multiple entries inside the [] brackets.

And now that I think about it, there is actually a bug in calibredb list. It should not be separating multiple path by commas, as commas can occur in filenames. Now fixed: https://github.com/kovidgoyal/calibr...4e95f15101ec6c
Yay, at least something good came out of my mucking around.

Quote:
Originally Posted by kovidgoyal View Post
Since I was here in any case, I decided to add a --for-machine option to calibredb list that makes it easier to parse the output.

And here's a one liner (in python because I hate sh) that demonstrates how to get a list of all ids and formats for ever id in the library

Code:
calibredb list --for-machine -f formats | python2 -c "import sys; print '\n\n'.join('id:%s formats:%r' % (r.partition(chr(29))[0], r.partition(chr(29))[-1].split(chr(28))) for r in sys.stdin.read().split(chr(30)) if r.strip())"
And the help of --for-machine

Code:
  --for-machine         Generate output that is more suitable for machine
                        parsing. Book entries are separates using the ASCII
                        Record Separator character (30), individual fields in
                        an entry using the Group Separator (29) and individual
                        filenames using the File Separator (28)
Thank you for adding that in. Python is on my to-do list. :P
Ackis is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
ebook-meta command doesn't add cover to ePub and mobi EbokJunkie Conversion 8 04-28-2014 03:13 AM
Version 8.41 doesn't add book killa Calibre 0 02-28-2012 11:53 AM
ebook-convert vs windows version zhoujianfu Calibre 12 05-21-2010 01:08 PM
Calibredb add after ebook-convert fails spullar Calibre 5 03-04-2010 01:14 PM
Why doesn't eBook library 3.0 work? mgmueller Sony Reader 30 09-20-2009 12:38 PM


All times are GMT -4. The time now is 01:40 PM.


MobileRead.com is a privately owned, operated and funded community.