Calibre as a Web Application for Periodicals?

[ChuckEggDotCom]

Kovid,

Have you considered Calibre as a web application for periodicals?

Reasons:
1) On my Sony Reader, I want to read Daily Newspapers, Weekly Magazines, Monthly Magazines.

2) Although I'm impressed by the complexity of CALIBRE, as a user, I don't really care about it. I simply want to quickly download some current LRF files for my reader. It would make much more sense for a server to process daily, monthly, weekly feeds and publish them for download automatically.

3) I could use my online CALIBRE like a feed reader, except instead of reading the feeds online, I could download multiple LRF files directly to my SD card.

4) I could use other peoples recipies rather than reinventing the wheel or trying to find where they've uploaded the recipies on the web.

5) A web based CALIBRE would be much easier to patch than trying to support and update hundreds of client applications.

I speak from experience. Over the past 10 years I've converted a few of my client applications to web based applications - with many good results.

[kovidgoyal]

Sure, I've thought about it. Problem is bandwidth costs and writing the server app (I prefer hacking on calibre itself). Also eventually, I will add the ability to schedule automatic downloads to the calibre GUI, which would make a server app somewhat redundant.

[ChuckEggDotCom]

Originally Posted by kovidgoyal

Sure, I've thought about it. Problem is bandwidth costs and writing the server app (I prefer hacking on calibre itself). Also eventually, I will add the ability to schedule automatic downloads to the calibre GUI, which would make a server app somewhat redundant.

Okay.

I can understand you liking to hack on calibre itself. For more than 20 years I've been an independent Windows application programmer (but for the past 10 years I've been a web applications manager).

Would your mind be changed if I offered server capacity?

Converting books with a desktop application makes sense because the files are typically downloaded to a local computer, anyway. Converting dynamic periodicals for which there is fresh content every day does not make sense as a desktop application since your computer may not be available every day to automatically process it.

A server app may be redundant, but it's a better choice than a client program because if you leave my PC off for several days while you travel, it will be impossible for me to generate periodical .LRF files after the fact. If you want to log in from an airport or hotel room and grab periodical files to-go, it's nearly impossible with a web application that may take too long to run.

The number of popular periodicals is somewhat finite. Just like www.TinyURL, if users have the same requests for periodicals, a properly designed web app would not store redundant copies of the same publication, but point to the same single copy of the LRF.

Currently I operate three servers and I disagree about the bandwidth. It seems most .LRF BBeB files are 800k or less. That's about the size of 10 average web images. It's not an unmanageable traffic volume even with a couple hundred users.

Also, in my experience, it's much faster and easier to update one web application than update 300 desktop client applications.

You've done a great job with Calibre, but I may have some of my programming staff look into writing a web application because I don't necessarily want to install or run Calibre every night.

For now, I've created a batch file that simply calls each periodical and writes it to my computer's SD card. I can run this by batch using a cron every night, but I don't want to leave on my computer every night and waste power. Some nights I can't leave my computer on because I travel.

Step 1: Insert your SD card to your computer.
Step 2: Change the bottom batch command to copy the files to your SD card (mine is F: drive, yours may be some other letter)
Step 3: Copy the lines below to a batch file (mags.bat) and run it to create a collection of periodicals to read on your reader.


WEB2LRF atlantic
WEB2LRF ap
WEB2LRF barrons --username=blah --password=mypass
WEB2LRF bbc
WEB2LRF chr_mon
WEB2LRF cnn
WEB2LRF economist
WEB2LRF faznet
WEB2LRF jpost
WEB2LRF jutarnji
WEB2LRF nasa
WEB2LRF newsweek
WEB2LRF newyorker
WEB2LRF newyorkreview
WEB2LRF nytimes --username=blah --password=mypass
WEB2LRF upi
WEB2LRF usatoday
WEB2LRF portfolio
WEB2LRF reuters
WEB2LRF spiegelde
WEB2LRF wsj --username=blah --password=mypass
WEB2LRF wash_post
FEEDS2LRF bw_news.py --author='news'
copy *.lrf f:*.* /Y

[ChuckEggDotCom]

Kovid,

By the way. You rock. I'm completely serious when I say that.

Chuck

[kovidgoyal]

Well calibre currently has about 10k users, and the number is growing. Assuming half use the webapp and download an average of 3 periodicals a week that's 3*5000*800 which is about 50GB a month. That's tolerable, but say a year down the road if the number of users/periodicals increases this could become 500GB.

I agree that a webapp is more suitable for people that don't have a computer they keep switched on 24/7 with broadband access, especially for daily publications.

But if you're willing to tolerate the bandwidth costs, I'll be happy to help develop a webapp. Technically it would be pretty simple to do.

[athlonkmf]

/me signs up for potential hoster too.

It'd be nice to have such a service. And my websites are already chugging out 4 TB of bandwidth monthly. A 500GB extra wouldn't even be noticed (and then I still have 6TB monthly left)

[kovidgoyal]

I find designing websites to be rather tedious, but I'm willing to write a backend that will fetch the recipes and store the last n in a simple sqlite database. Writing a front-end that serves up the downloaded e-books on demand should be easy enough.

[banjopicker]

I would think that publishers of newspapers and magazines would take legal action if there was a website that served up repackaged full-text versions of their content. Whereas with Calibre, the user does the repackaging themselves, so lawyers wouldn't know where to send the Cease and Desist letter.

[yagiz]

Kovid,

You're already developing in Python. You should definitely have a look at Google App Engine.

[Stingo]

Guys, I'm an IP attorney and I can tell you that you are putting a big bulls-eye on your back by setting up a web site which repackages and redistributes content. At least in the US you would run into potentially serious issues.
Stick to the lower profile desktop app.

[kovidgoyal]

True, that is one of the reasons I haven't done this yet.

[astrodad]

Does Python support a Java Web Start kind of thing?

[kovidgoyal]

What's a Java Web Start?

[pilotbob]

Originally Posted by kovidgoyal

What's a Java Web Start?

A hacky kludge at best.

BOb

[kovidgoyal]

Ah I see its the technology for installing and launching an app by clicking in a browser. No python, doesn't have this (to the best of my knowledge).