Wikipedia on Calibre

[franc]

Hello,

i found on:

http://www.axelschaefermdb.de/service/wikipedia.html

the wikipedia (2008) for mobipocket (pcr) and tried to open it with Calibre 0.6.54 for MAC OSX 10.6.3. But this doesn't work. The machine (a new one) gets busy and busier and after a good while i have to close calibre with force.

Is it possible though to open this wpmp in Calibre?

Thank you,

franc

[kovidgoyal]

No, if you find the wikipedia in EPUB format it will probably work, but not mobi.

[franc]

Oh, what a pitty. There is no epub-Wikipedia, unfortunately.
Calibre is such a nice program and should be worse than mobiPocket Reader in this matter?

[Stinger]

You could always make your own ePub from the official Wikipedia database download:
http://en.wikipedia.org/wiki/Wikiped...abase_download

Obviously there might be some issues with the dbase being so friggen huge:
'Current revisions only, no talk or user pages. (This is probably the one you want. WARNING: 5.6 GB compressed, up to 20 times that size uncompressed.)"

If you want to do this with Calibre, you might need to use the tools mentioned on the above page to cook the database into HTML.

[franc]

Originally Posted by Stinger

You could always make your own ePub from the official Wikipedia ...

I heard that this work takes even with many computers several weeks...

[Stinger]

Definitely not weeks.

Last time I looked into downloading the snapshots so I could have a offline Wikipedia version for an old laptop without Wifi, I read it usually took several hours to render the entire English articles into static HTML.

I didn't need static HTML, so I only installed the MediaWiki software and used the downloaded database in its current format, so I can't personally attest to how long it would take. But if you have a modern processor, I don't think you need to worry about the HTML rendering taking weeks.

I found this on the Wikipedia help page on downloading dumps:

MediaWiki 1.5 includes routines to dump a wiki to HTML, rendering the HTML with the same parser used on a live wiki. As the following page states, putting one of these dumps on the web unmodified will constitute a trademark violation. They are intended for private viewing in an intranet or desktop installation.

Since this functionality is available in the official MediaWiki software package, I would assume the code is relatively well written/optimized. This means you don't have to rely on a 3rd-party tool that might use a rough and dirty solution to get the job done. This might have been the case with where you heard the several weeks estimate maybe?


ALSO:
Last time I looked, I found a few download links to Static HTML dumps that other people have already rendered, and made available for download. You're probably going to be stuck with old snapshots if you use these sources, but it will save you the step of rendering the HTML yourself.

In any case, the resulting ePub file is going to be friggen MASSIVE, and you might run into problems with eReader software and mobile readers even being able to use it meaningfully in the best case, and not even able to open it due to sheer size in the worst case.

[franc]

Originally Posted by Stinger

... This means you don't have to rely on a 3rd-party tool that might use a rough and dirty solution to get the job done. This might have been the case with where you heard the several weeks estimate maybe?

Yes, indeed, this could have been the thing.
But tell me, why isn't there any epub-Wiki to download in the net, if it would be easily done?

[Stinger]

I think the answer to this is mainly in my ALSO: edit above.

The size of the file could be a big problem on many systems. Fat32 is a pretty common native filesystem for mobile readers so they can be accessed through windows, linux and mac easily and the maximum filesize limit would probably be exceeded by this ePub (4GB).

You also lose a lot of the 'magic' of Wikipedia by not being able to jump around easily via the internal hyperlinks (at least on most eReader devices today). I could never use Wikipedia to my liking without some kind of tabbed browsing functionality. My wikipedia jump tree usually has MANY branches (as in, I start with reading one article and end up with 27 open firefox tabs only loosly related to my original article )

On another note, lets not forget the very dynamic nature of the project itself. Sure, there are many articles that are in great shape already, but there is always work going on in adding/overhauling much of the content. So there is the issue of continuously having to update the ePub, and I don't think there is any easy way to do this. We'd either have to periodically re-download the entire ePub to keep it current, or someone would have to devise a tool to grab updated data and inject it into the current ePub. Even with the periodic injections, these updates would likely still be pretty large given the scope and number of daily changes to Wikipedia.

eReading in its current form is pretty friendly towards linear reading, maybe some simple content searching and jumping around with a TOC, but definitely not friendly towards dynamic content such as Wikipedia.