Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Conversion

Notices

Reply
 
Thread Tools Search this Thread
Old 05-29-2012, 07:51 PM   #1
joewiz
Junior Member
joewiz began at the beginning.
 
Posts: 3
Karma: 10
Join Date: May 2012
Device: iPad
Scripting ebook-convert for batch parallel processing?

Hi! First post here. I'm interested in using the bash shell to script the conversion of my epub files to mobi. Right now I have this working:

Code:
for f in *.epub; do ebook-convert "$f" "`basename "$f" .epub`.mobi"; done;
This works fine, but it only operates on one epub at a time. In the Calibre GUI it seems to run up to 3 conversions at once. Is there a way using the command line to run ebook-convert in parallel? I have a quad-core iMac, which could blow through these much faster in parallel.

Thanks for any suggestions,
Joe

p.s. I'm on Mac OS X 10.7.4, using Calibre 0.8.53.
joewiz is offline   Reply With Quote
Old 05-29-2012, 08:18 PM   #2
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,909
Karma: 12755553
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Welcome to Mobileread.

Moderator Notice
Moved thread out of development sub-forum. Read this sticky post before posting again in the development sub-forum.
DoctorOhh is offline   Reply With Quote
Old 05-29-2012, 08:51 PM   #3
joewiz
Junior Member
joewiz began at the beginning.
 
Posts: 3
Karma: 10
Join Date: May 2012
Device: iPad
@dwanthny: Thanks for putting my question in the right sub-forum!

I look forward to any ideas people here in the Conversion sub-forum might have.
joewiz is offline   Reply With Quote
Old 05-30-2012, 01:33 AM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,465
Karma: 5383257
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Bash is so last century Save the following as batch.py in the directory wth your epub files. Then run it as either

python batch.py
or
calibre-debug -e batch.py

Code:
#!/usr/bin/python
import os, time, glob, subprocess

files = glob.glob('*.epub')

workers = []
while files or workers:
    while len(workers) < 4 and files:
        f = files[0]
        files = files[1:]
        w = subprocess.Popen(['ebook-convert', f,
            os.path.splitext(f)[0]+'.mobi'])
        workers.append(w)
    for w in list(workers):
        if w.poll() is not None:
            workers.remove(w)
    time.sleep(0.1)
Change 4 to the number of worker processes you want.

Last edited by kovidgoyal; 05-30-2012 at 05:29 AM.
kovidgoyal is offline   Reply With Quote
Old 05-30-2012, 09:13 PM   #5
joewiz
Junior Member
joewiz began at the beginning.
 
Posts: 3
Karma: 10
Join Date: May 2012
Device: iPad
Quote:
Originally Posted by kovidgoyal View Post
Bash is so last century Save the following as batch.py in the directory wth your epub files.
Awesome! Thanks so much. I'll give it a try.
joewiz is offline   Reply With Quote
Old 05-30-2012, 09:35 PM   #6
Dyspeptica
Member
Dyspeptica began at the beginning.
 
Posts: 20
Karma: 10
Join Date: May 2012
Device: iPad
Joewiz, Don't believe him. Bash is still now!

Jump over to here: http://www.mobileread.com/forums/sho...124219&page=17

At the bottom of the page is a chunk of script which I happened to post earlier today (well, it's still Wed. here).

Pure bash setup with a call to ebook-convert and all of the command line switches available.

For *multi* use, I suppose you could open another terminal, and start another thread there. It *should* work...but that could depend on how your OS provides library (.so) access to calling processes. I have an idea that the memory space for running processes is kept separate. But MMMV. (I'm going to try it just to see).

Otherwise, you will need to split your source files: can't have two 'for file in *;' processes looking at the same folder!
Dyspeptica is offline   Reply With Quote
Old 05-30-2012, 10:27 PM   #7
PeterT
Taking a break; Fed up
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 7,433
Karma: 46459035
Join Date: Nov 2007
Location: Toronto
Device: Wife: Touch, Arc, Vox Me: Nexus 7, Glo
I'm willing to bet xpath could also be used with the --max-procs=4 parameter...
PeterT is offline   Reply With Quote
Old 06-12-2012, 02:03 PM   #8
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 2,312
Karma: 5761596
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PRS-350, Nexus S, Galaxy S, Nook Color, iPhone4, iPT4, iPad 2012
Quote:
Originally Posted by kovidgoyal View Post
Bash is so last century Save the following as batch.py in the directory wth your epub files. Then run it as either

python batch.py
or
calibre-debug -e batch.py

Code:
#!/usr/bin/python
import os, time, glob, subprocess

files = glob.glob('*.epub')

workers = []
while files or workers:
    while len(workers) < 4 and files:
        f = files[0]
        files = files[1:]
        w = subprocess.Popen(['ebook-convert', f,
            os.path.splitext(f)[0]+'.mobi'])
        workers.append(w)
    for w in list(workers):
        if w.poll() is not None:
            workers.remove(w)
    time.sleep(0.1)
Change 4 to the number of worker processes you want.
Question, would this work under Windows, too?
ilovejedd is offline   Reply With Quote
Old 06-13-2012, 12:38 AM   #9
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,465
Karma: 5383257
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Yes, it should work in windows.
kovidgoyal is offline   Reply With Quote
Old 03-15-2013, 06:34 AM   #10
Foster82
Junior Member
Foster82 began at the beginning.
 
Posts: 1
Karma: 10
Join Date: Mar 2013
Device: kindle
HI will this also scan sub directory's too ?
Foster82 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Batch breaks after ebook-convert.exe j0s3f Calibre 1 05-04-2011 06:27 PM
How to batch-convert with ebook-convert? cypresstwist Conversion 8 02-22-2011 10:28 AM
ebook-convert won't work in a batch file since 0.6.17 mbovenka Calibre 20 01-13-2011 08:03 PM
Batch processing of PDB files? Asterra iRex 6 12-04-2007 02:10 PM
Batch-convert Rocket Ebook format for GEB2150? Fauve Fictionwise eBookwise 1 04-28-2007 06:12 PM


All times are GMT -4. The time now is 07:02 PM.


MobileRead.com is a privately owned, operated and funded community.