Concurrent Calibre CLI conversion jobs

Atarian · 04-20-2013, 07:12 AM

Hi all; at my job we have been requested to investigate something that I thought calibre would be fantastic for, but we need a little help.

Some background first: there are a lot of users which have each a lot of manuals, procedures and other documents in different formats (doc, txt, rtf, pdf, etc), and we want to convert all that into ePub or pdf, so the documentation can then be easily read by them in mobility using tablets or even standard eReaders (this BYOD thing is running strong). We want to put a simple service in place to have users uploading all their stuff, having it converted in background by calibre running in CLI mode, then dumped in a folder so the user can then transfer to his device as sees fit. We're using an Ubuntu Linux box for this. We have run some test successfully but, since the conversion jobs are being queued and one user cannot have his own queue processed until the former one is done, we're afraid that if we run this for everybody what will happen is that the last users requesting the conversion will take ages until they get their files converted, so a better solution has to be devised.

So, here's (at last) the question: we thought that we could greatly improve the process by giving each user its own instance of calibre to run his conversion. This way we could have concurrent users running concurrent calibres thus making the conversion time a breeze (or at least will depend only on how much stuff an individual user has, not having to wait on others). We're not going to use calibre for managing the documents or anything, only to run the conversion, so we're not concerned about calibre's db integrity or anything.

Has anyone tried to do this somehow? Is it even possible?

Any hint will be greatly appreciated! Thanks in advance!

kovidgoyal · 04-20-2013, 07:25 AM

Umm, what's preventing you from running as many instances of ebook-convert as you want at the same time?

Atarian · 04-20-2013, 10:59 AM

Quote:

Originally Posted by kovidgoyal

Umm, what's preventing you from running as many instances of ebook-convert as you want at the same time?

Wow, thanks Kovid, great to get an answer from the Man himself!

No, we have actually not tried it yet; just had the idea and before even trying we wanted to know if there was any known obstacle to it, to save some time; many apps don't support multiple instances, didn't know if this could be the case.
From your answer I understand that we should be able to run simultaneously as many ebook-convert instances as wished, providing the server has the needed resources and so on, right? So, a script with something as simple as the following:

ebook-convert file1.doc file1.pdf &
ebook-convert file2.txt file2.pdf &
ebook-convert file3.rtf file3.epub &
ebook-convert file4.doc file4.mobi &

would run four simultaneous conversion jobs, and end of the story, is that correct?

kovidgoyal · 04-20-2013, 11:22 AM

yes .

Atarian · 04-20-2013, 11:43 AM

Quote:

Originally Posted by kovidgoyal

yes .

Short and sweet, just the way I like it

Thanks a million; time to do some coding, yay!

04-20-2013, 07:12 AM	#1
Atarian Junior Member Posts: 4 Karma: 10 Join Date: Apr 2013 Device: Kindle Paperwhite	Concurrent Calibre CLI conversion jobs Hi all; at my job we have been requested to investigate something that I thought calibre would be fantastic for, but we need a little help. Some background first: there are a lot of users which have each a lot of manuals, procedures and other documents in different formats (doc, txt, rtf, pdf, etc), and we want to convert all that into ePub or pdf, so the documentation can then be easily read by them in mobility using tablets or even standard eReaders (this BYOD thing is running strong). We want to put a simple service in place to have users uploading all their stuff, having it converted in background by calibre running in CLI mode, then dumped in a folder so the user can then transfer to his device as sees fit. We're using an Ubuntu Linux box for this. We have run some test successfully but, since the conversion jobs are being queued and one user cannot have his own queue processed until the former one is done, we're afraid that if we run this for everybody what will happen is that the last users requesting the conversion will take ages until they get their files converted, so a better solution has to be devised. So, here's (at last) the question: we thought that we could greatly improve the process by giving each user its own instance of calibre to run his conversion. This way we could have concurrent users running concurrent calibres thus making the conversion time a breeze (or at least will depend only on how much stuff an individual user has, not having to wait on others). We're not going to use calibre for managing the documents or anything, only to run the conversion, so we're not concerned about calibre's db integrity or anything. Has anyone tried to do this somehow? Is it even possible? Any hint will be greatly appreciated! Thanks in advance!

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Can I use Calibre's CLI on a cloud-based app?	MortAngstrom	Development	1	08-15-2012 11:42 PM
Can I use Calibre's CLI tools for an app?	markvb	Conversion	5	04-15-2012 10:00 PM
how to set the index of the file using calibre CLI	kindleren	Conversion	3	02-21-2012 09:53 PM

04-20-2013, 07:25 AM	#2
kovidgoyal creator of calibre Posts: 43,860 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Umm, what's preventing you from running as many instances of ebook-convert as you want at the same time?

04-20-2013, 11:22 AM	#4
kovidgoyal creator of calibre Posts: 43,860 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	yes .

Advert

Advert