Running Calibre on Google

Mixx · 07-08-2012, 05:52 AM

Hi,

could anyone please explain in plain terms what would it take to run Calibre on some Google engine (app engine or compute engine?) in a way that is simple to install/configure/maintain?

What I am trying to achieve: I'd like to use the recipes feature of calibre on a regular basis. I am interested in lot's of diverse fields and read in a (small) number of languages. Thus, I'd download on the week-end some 10-12 feeds, maybe 3-4 on a daily basis. I am not very disturbed by the fact that many recipes are not working perfectly anymore and exhibit quirks.

I can not really subscribe to them on a commercial basis, as such a service doe snot currently exist (after readbeam.com has closed down) or is just a courtesy of somebody on best effort basis and is therefore not very reliable or easily configurable (I am very grateful to reddit.kindle, though). In any case, commercial services are likely to charge "by recipe" (as readbeam.com was planning) and that would be very un-economical for me.

If it was easy to install Calibre on "Google" (whatever that means) and just run it scheduled daily for 30 mins. or so, that would be sufficient for me and probably very inexpensive (if not for free for such a minimal usage). I'd have to do the admin, but that's all right.

I have some knowledge how to write a script to run the recipes from command line (headless?) and I think I'd be bale to figure out how to schedule it.

I have done a setup under Windows using scheduled wakeup and hibernation, but on my Vista PC (vintage) it is not reliable at all as a service.

I'd appreciate any how-to-s, elaborations or pointers.

Cheers, Mixx

chaley · 07-08-2012, 06:42 AM

My opinion: abandon hope all ye who enter here. I don't see it being possible, much less easy, to run calibre on a google engine.

You would have much better luck installing it on a linux virtual server hosted somewhere. You could also use scheduled tasks (cron or what-have-you) on your home machine if it us up 24/24. Set up a job that runs ebook-convert for each recipe that you want to download.

PeterT · 07-08-2012, 11:58 AM

I wonder if it could be done with Google Cloud Compute Engine ?

chaley · 07-08-2012, 12:35 PM

Quote:

Originally Posted by PeterT

I wonder if it could be done with Google Cloud Compute Engine ?

To the extent that compute engine is a vanilla VPS, it should be possible, where I don't think it can be done without massive effort on app engine. That said, my guess is that pricing would be prohibitive once out of demo mode. I found this to be true with Amazon's computing products. Also, instance management could be a bother.

One can get a suitable VPS for under $10/month. Working out the price for a compute engine is hard, but assuming a fulltime instance it seems to be at least $30/month.

theimeto · 03-11-2013, 01:02 AM

It's very doable actually. As a complete Google App Engine n00b + far from python wizardy I took me two weekends to hack the calibre code and to do the necessary plumbing.

I was interested in fetching books from my Google Drive and converting them (from EPUB, FB2) to MOBI so I can download them directly from the Kindle Web browser. YMMV. The following rough notes are from the diff of my hacks to stock calibre.

1) drop (rename) db, devices, gui2, trac, translations, web, most of the utils subdirectories
2) ebooks/oeb/transforms/rasterize.py - make it always raise exception
3) create python implementations plugins/magick.py (use PIL) and plugins/cPalmdoc.py (forward back to ebooks.compression.palmdoc.py_compress_doc)
4) some hacks in ptempfile.py; Very small changes in lock.py, constants.py and config_base.py

I used http://code.google.com/p/pyfilesystem to create virtual filesystem.

Before importing calibre code:

1) set environment variables - "CALIBRE_WORKER_TEMP_DIR", "CALIBRE_CONFIG_DIRECTORY", set sys.resources_location and sys.extensions_location
2) create fake 'os' module, wire relevant methods (mkdir, chdir, stat, etc) to your code; wire fake_os.path.os and fake_os.path.genericpath.os back to fake_os. replace 'os' in sys.modules
3) replace builtin open with your code.

About 300 LOC total.

Then you can import calibre code; use in-memory file system for input, output files for calibre.
Once calibre finish the ebook conversion and created the output file, I copy it to blobstore so it can be downloaded.

Stanbey · 07-06-2014, 02:04 PM

Hi,

I just stumbled across this thread because I was trying to do something very similar. The first thing I noticed when I looked into this was that Calibre seems to use some things that aren't supported on App Engine (e.g. the ICU stuff). Did you manage to work around that?

Would you be willing to share the changes you made to get this to work?

Kind regards,
Stanbey

Quote:

Originally Posted by theimeto

It's very doable actually. As a complete Google App Engine n00b + far from python wizardy I took me two weekends to hack the calibre code and to do the necessary plumbing.
I was interested in fetching books from my Google Drive and converting them (from EPUB, FB2) to MOBI so I can download them directly from the Kindle Web browser. YMMV. The following rough notes are from the diff of my hacks to stock calibre.
1) drop (rename) db, devices, gui2, trac, translations, web, most of the utils subdirectories
2) ebooks/oeb/transforms/rasterize.py - make it always raise exception
3) create python implementations plugins/magick.py (use PIL) and plugins/cPalmdoc.py (forward back to ebooks.compression.palmdoc.py_compress_doc)
4) some hacks in ptempfile.py; Very small changes in lock.py, constants.py and config_base.py
I used http://code.google.com/p/pyfilesystem to create virtual filesystem.
Before importing calibre code:
1) set environment variables - "CALIBRE_WORKER_TEMP_DIR", "CALIBRE_CONFIG_DIRECTORY", set sys.resources_location and sys.extensions_location
2) create fake 'os' module, wire relevant methods (mkdir, chdir, stat, etc) to your code; wire fake_os.path.os and fake_os.path.genericpath.os back to fake_os. replace 'os' in sys.modules
3) replace builtin open with your code.
About 300 LOC total.
Then you can import calibre code; use in-memory file system for input, output files for calibre.
Once calibre finish the ebook conversion and created the output file, I copy it to blobstore so it can be downloaded.

07-08-2012, 05:52 AM	#1
Mixx Zealot Posts: 143 Karma: 387 Join Date: Sep 2010 Device: Kindle 3	Running Calibre on Google Hi, could anyone please explain in plain terms what would it take to run Calibre on some Google engine (app engine or compute engine?) in a way that is simple to install/configure/maintain? What I am trying to achieve: I'd like to use the recipes feature of calibre on a regular basis. I am interested in lot's of diverse fields and read in a (small) number of languages. Thus, I'd download on the week-end some 10-12 feeds, maybe 3-4 on a daily basis. I am not very disturbed by the fact that many recipes are not working perfectly anymore and exhibit quirks. I can not really subscribe to them on a commercial basis, as such a service doe snot currently exist (after readbeam.com has closed down) or is just a courtesy of somebody on best effort basis and is therefore not very reliable or easily configurable (I am very grateful to reddit.kindle, though). In any case, commercial services are likely to charge "by recipe" (as readbeam.com was planning) and that would be very un-economical for me. If it was easy to install Calibre on "Google" (whatever that means) and just run it scheduled daily for 30 mins. or so, that would be sufficient for me and probably very inexpensive (if not for free for such a minimal usage). I'd have to do the admin, but that's all right. I have some knowledge how to write a script to run the recipes from command line (headless?) and I think I'd be bale to figure out how to schedule it. I have done a setup under Windows using scheduled wakeup and hibernation, but on my Vista PC (vintage) it is not reliable at all as a service. I'd appreciate any how-to-s, elaborations or pointers. Cheers, Mixx

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Google Market running on Kobo Vox	danifunker	Kobo Tablets	122	02-24-2012 07:56 AM
Running Calibre on Server	godlich	Recipes	2	02-09-2012 12:57 PM
Running Calibre on apache..	mnsweeps	Calibre	3	03-28-2011 02:32 AM
calibre running slowly OS X??	tm3	Calibre	3	01-14-2011 10:31 AM
Running Calibre (0.4.77)	dsb66	Calibre	9	07-18-2008 04:42 AM

07-08-2012, 06:42 AM	#2
chaley Grand Sorcerer Posts: 11,741 Karma: 6997045 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	My opinion: abandon hope all ye who enter here. I don't see it being possible, much less easy, to run calibre on a google engine. You would have much better luck installing it on a linux virtual server hosted somewhere. You could also use scheduled tasks (cron or what-have-you) on your home machine if it us up 24/24. Set up a job that runs ebook-convert for each recipe that you want to download.

07-08-2012, 11:58 AM	#3
PeterT Grand Sorcerer Posts: 12,167 Karma: 73448616 Join Date: Nov 2007 Location: Toronto Device: Nexus 7, Clara, Touch, Tolino EPOS	I wonder if it could be done with Google Cloud Compute Engine ?

03-11-2013, 01:02 AM	#5
theimeto Junior Member Posts: 1 Karma: 10 Join Date: Jul 2008 Device: Sony PRS505	It's very doable actually. As a complete Google App Engine n00b + far from python wizardy I took me two weekends to hack the calibre code and to do the necessary plumbing. I was interested in fetching books from my Google Drive and converting them (from EPUB, FB2) to MOBI so I can download them directly from the Kindle Web browser. YMMV. The following rough notes are from the diff of my hacks to stock calibre. 1) drop (rename) db, devices, gui2, trac, translations, web, most of the utils subdirectories 2) ebooks/oeb/transforms/rasterize.py - make it always raise exception 3) create python implementations plugins/magick.py (use PIL) and plugins/cPalmdoc.py (forward back to ebooks.compression.palmdoc.py_compress_doc) 4) some hacks in ptempfile.py; Very small changes in lock.py, constants.py and config_base.py I used http://code.google.com/p/pyfilesystem to create virtual filesystem. Before importing calibre code: 1) set environment variables - "CALIBRE_WORKER_TEMP_DIR", "CALIBRE_CONFIG_DIRECTORY", set sys.resources_location and sys.extensions_location 2) create fake 'os' module, wire relevant methods (mkdir, chdir, stat, etc) to your code; wire fake_os.path.os and fake_os.path.genericpath.os back to fake_os. replace 'os' in sys.modules 3) replace builtin open with your code. About 300 LOC total. Then you can import calibre code; use in-memory file system for input, output files for calibre. Once calibre finish the ebook conversion and created the output file, I copy it to blobstore so it can be downloaded.