Scripting ebook-convert for batch parallel processing?

[joewiz]

Hi! First post here. I'm interested in using the bash shell to script the conversion of my epub files to mobi. Right now I have this working:

for f in *.epub; do ebook-convert "$f" "`basename "$f" .epub`.mobi"; done;

This works fine, but it only operates on one epub at a time. In the Calibre GUI it seems to run up to 3 conversions at once. Is there a way using the command line to run ebook-convert in parallel? I have a quad-core iMac, which could blow through these much faster in parallel.

Thanks for any suggestions,
Joe

p.s. I'm on Mac OS X 10.7.4, using Calibre 0.8.53.

[DoctorOhh]

Welcome to Mobileread.

Moderator Notice
Moved thread out of development sub-forum. Read this sticky post before posting again in the development sub-forum.

[joewiz]

@dwanthny: Thanks for putting my question in the right sub-forum!

I look forward to any ideas people here in the Conversion sub-forum might have.

[kovidgoyal]

Bash is so last century Save the following as batch.py in the directory wth your epub files. Then run it as either

python batch.py
or
calibre-debug -e batch.py

#!/usr/bin/python
import os, time, glob, subprocess

files = glob.glob('*.epub')

workers = []
while files or workers:
    while len(workers) < 4 and files:
        f = files[0]
        files = files[1:]
        w = subprocess.Popen(['ebook-convert', f,
            os.path.splitext(f)[0]+'.mobi'])
        workers.append(w)
    for w in list(workers):
        if w.poll() is not None:
            workers.remove(w)
    time.sleep(0.1)

Change 4 to the number of worker processes you want.

[joewiz]

Originally Posted by kovidgoyal

Bash is so last century Save the following as batch.py in the directory wth your epub files.

Awesome! Thanks so much. I'll give it a try.

[Dyspeptica]

Joewiz, Don't believe him. Bash is still now!

Jump over to here: https://www.mobileread.com/forums/sho...124219&page=17

At the bottom of the page is a chunk of script which I happened to post earlier today (well, it's still Wed. here).

Pure bash setup with a call to ebook-convert and all of the command line switches available.

For *multi* use, I suppose you could open another terminal, and start another thread there. It *should* work...but that could depend on how your OS provides library (.so) access to calling processes. I have an idea that the memory space for running processes is kept separate. But MMMV. (I'm going to try it just to see).

Otherwise, you will need to split your source files: can't have two 'for file in *;' processes looking at the same folder!

[PeterT]

I'm willing to bet xpath could also be used with the --max-procs=4 parameter...

[ilovejedd]

Originally Posted by kovidgoyal

Bash is so last century Save the following as batch.py in the directory wth your epub files. Then run it as either

python batch.py
or
calibre-debug -e batch.py

#!/usr/bin/python
import os, time, glob, subprocess

files = glob.glob('*.epub')

workers = []
while files or workers:
    while len(workers) < 4 and files:
        f = files[0]
        files = files[1:]
        w = subprocess.Popen(['ebook-convert', f,
            os.path.splitext(f)[0]+'.mobi'])
        workers.append(w)
    for w in list(workers):
        if w.poll() is not None:
            workers.remove(w)
    time.sleep(0.1)

Change 4 to the number of worker processes you want.

Question, would this work under Windows, too?

[kovidgoyal]

Yes, it should work in windows.

[Foster82]

HI will this also scan sub directory's too ?