I was traveling around the world for the last 8 months and now that I'm back I finally got to hack together the one thing I was missing during my travels... an offline wikipedia browser for the Iliad.

It's still quite quirky, but working. It's quite easy to install the "Simple English Wikipedia" on your flash card in less than 100MB, but the full one is a little bit tricky at 4GB.

I hope you guys enjoy it.

The project is hosted at:
http://code.google.com/p/pyoffwiki/

And here is the page to get you started:
http://code.google.com/p/pyoffwiki/wiki/GettingUpAndRunning


Adam B. Edit:
I've made an easy install package for this Wikipedia port.

I'm not a fan of Dillo, so I've included Midori as well. I've also created an html document with a link so that it can be browsed with the iLiad's html viewer. I've had mixed results with this, but it can be nicer than the 3rd party web browsers sometimes.

Sreenshots on this post (http://www.mobileread.com/forums/showpost.php?p=187061&postcount=7).

You can find the all in one installer here: http://projects.mobileread.com/iliad/users/yokos/Wikipedia.0.1.1.zip

To Install, simply unzip the folder and copy to your iLiad's memory card or USB drive. It will use about 140MB, so it won't fit on internal memory. This should be able to run from any location.

Edit 2: Yokos has updated the all in one package with the following changes:

fixed sh scripts [line feed encoding]
./xset +fp $scriptdir/fonts/TTF [dillo]
international version of dillo [öäüß etc.]
dillo home/start up page with hyperlinks to Offline Wikipedia and some sites
missing icon of "stopping wikipedia server" Please re-download (http://projects.mobileread.com/iliad/users/yokos/Wikipedia.0.1.1.zip) if you were having problems with the original.

Oh this is so cool. I will try it immediately. :2thumbsup:2thumbsup:2thumbsup

I was waiting for this one!

Happy hour! :iloveyou: (sorry ;)

Gogolo

One question: I want to get the full wiki German. Can you help me find the file? Thanks.
Edit2: Think I found it now: http://download.wikimedia.org/dewiki/latest/dewiki-latest-pages-articles.xml.bz2

Sorry, another problem: You have a Index file here (http://code.google.com/p/pyoffwiki/downloads/detail?name=simplewiki-20080316-pages-articles.zip). Did you convert this? Or can I find a similar for german wiki here (http://download.wikimedia.org/dewiki/latest/)? I would be happy for a little help. Thanks

It works like a charm!!! Oh my god a full encyclopedia on iLiad :D

Now I'd like to get the german version running...

Thanks for your work!
Gogolo

Gogolo - would you mind to share your knowledge with us and let us know how you have managed to do a german version. A nice little "how to" would be great!!

Regards

:thanks:
"Danke im Voraus"
:thanks:

Sorry, no German Version yet.

I think this is the German database:

http://download.wikimedia.org/dewiki/latest/dewiki-latest-pages-articles.xml.bz2

But the Index is needed, and I just cant find it, nor do I know if this is running only with simple Wiki.

@amirshim: How can we create our own Index files? Thank you.

Greetings
Gogolo

Brilliant!

I'm doing to do some testing, and try the full dump. That's very cool! :2thumbsup

I've made an easy install package for this Wikipedia port.

I'm not a fan of Dillo, so I've included Midori as well. I've also created an html document with a link so that it can be browsed with the iLiad's html viewer. I've had mixed results with this, but it can be nicer than the 3rd party web browsers sometimes.

Sreenshots attached.

You can find the installer in the first post of this thread.

To Install, simply unzip the folder and copy to your iLiad's memory card or USB drive. It will use about 140MB, so it won't fit on internal memory. This should be able to run from any location.

This sounds very good. I'll try it out :)

Hmmm... Couldn't quite get the AIO package to work.
Dillo and Midori start then die (the busy-bars appear for about two seconds then disappear) and index.html freezes and keeps blinking the busy LED after tapping on the link :(

What am i doing wrong?

PS: I've decompressed the zip into Programs on my SD card. I already have Midori and Minimo installed, if that makes any difference...

Have you tried the version of Dillo and Midori you already have installed?

Try pointing them to http://localhost:8080

Great package Adam. I also like Midori better, but I had problems with it in the past, so I was testing with Dillo, but now I'll switch back to it.

I just have one problem with it. The run.sh files in the wiki and midori directory seem to be causing problems (at least for me) so here are the fixed up ones.

Copy the attached "wiki.run.sh" file to \Wikipedia\wiki\ and rename it to run.sh (delete the old one)
Copy the attached "midori.run.sh" file to \Wikipedia\apps\3_midori\ and rename it to run.sh (delete the old one)
Copy the attached "killwiki.run.sh" file to Wikipedia\apps\z_killwiki\ and rename it to run.sh (delete the old one)

That made the package work for me.

Have you tried the version of Dillo and Midori you already have installed?

Try pointing them to http://localhost:8080

No luck :/

Output.txt is empty and error.txt contains your log :)

Great package Adam. I also like Midori better, but I had problems with it in the past, so I was testing with Dillo, but now I'll switch back to it.

I just have one problem with it. The run.sh files in the wiki and midori directory seem to be causing problems (at least for me) so here are the fixed up ones.

Copy the attached "wiki.run.sh" file to \Wikipedia\wiki\ and rename it to run.sh (delete the old one)
Copy the attached "midori.run.sh" file to \Wikipedia\apps\3_midori\ and rename it to run.sh (delete the old one)

That made the package work for me.

That's odd, I've only edited those files from vi on my iLiad. Perhaps it's because of the version of busybox I'm running.

Thanks for the fix. :thumbsup:

Can you integrate them into the package... that is... if it works for other people.
Thanks, Amir

BTW: It's not a port (as you stated before)... this viewer is brand new development for the Iliad... that obviously uses many other [ported] open source packages... which wouldn't be there if it wasn't for you. Thanks for all the hard work.

Works with Amir's script.
Dillo is still dead, Midori won't refresh all of a sudden for some reason, but iliad's HTML viewer seems to work nicely, so the server's running now.

Edit: now the stop script doesn't work :)
Edit2: Oh, and thanks to both of you for an awesome package!

I updated my last post for the kill problem:
http://www.mobileread.com/forums/showpost.php?p=187304&postcount=11

@amirshim: Sorry if I ask again: Can you explain how you found or created the Index file for the database?

German (and probably other) users would be highly happy to get their language wiki on iLiad. Can you give us a hint? (As I see it we only need Index file and database to get our language... is it?)

Thanks for this nice application. It is very fast and smooth.

Sorry for my impatience - but the goal is so near now :rolleyes:

Gogolo

Thank you very much! :2thumbsup

The code to generate the Indexes is not ready for public consumption, but I generated two more indexes... check them out here:

http://code.google.com/p/pyoffwiki/wiki/OtherIndexes

I also got the Hebrew Wikipedia working, but none of the Iliad browsers (Dillo, Midori or Minimo) are able to display hebrew fonts, so it's not of much use. Any other requests?

Thank you very much for your work!

Edit: I deleted my post because IT IS RUNNING NOW in German with full Wikipedia!

I should have read your advices on OtherIndexes properly...

Only problem is, that the umlaute are not displayed - only weird characters. Is this a dillo problem?

This is extremly nice and I thank you another time! :-))))
Gogolo

Im editing all the time :-) Adams Hmtl link to internal browser did it. This one shows the umlaute very nice. Great!

Edit again: Dillo International also does the job for German umlaute, get it here (http://www.mobileread.com/forums/showpost.php?p=75928&postcount=44)

http://hardware.slashdot.org/article.pl?sid=08/05/22/2127225

People who use Linux will run into a problem of the run.sh being NOT executable (at least for the all-in-one zip). This is due to wrong EOF symbol used in them. fixing them manually (delete the ^M symbol at every end-of-line) should work. If you try to be smart and use "unzip -a" to let it does the ASCII conversion for you, sorry it won't work! This is because the original zip file wasn't packed with proper ASCII flags for ASCII file.

But it'll be most beneficial if somebody can fix the Shell scripts in the original zip.

Also I noted that in dillo/run.sh, it says "./xset +fp $scriptdir/fonts/TTF", but the directory fonts/ttf actually appears to be all in lowercase. So this line never works. There could be similiar issues elsewhere.

May this be a bug?: Random Page runs only one time - you'd have to restart dillo to run it another time.

Wiki is the killer app for iLiad :thumbsup:
Gogolo

I have uploaded a new version of the package
some of the changes
# fixed sh scripts [line feed encoding]
# ./xset +fp $scriptdir/fonts/TTF [dillo]
# international version of dillo [öäüß etc.]
# dillo home/start up page with hyperlinks to Offline Wikipedia and some sites
# missing icon of "stopping wikipedia server"
...
[EDIT] # stripped dillo libs

http://projects.mobileread.com/iliad/users/yokos/Wikipedia.0.1.1.zip

I have uploaded a new version of the package
some of the changes
# fixed sh scripts [line feed encoding]
# ./xset +fp $scriptdir/fonts/TTF [dillo]
# international version of dillo [öäüß etc.]
# dillo home/start up page with hyperlinks to Offline Wikipedia and some sites
# missing icon of "stopping wikipedia server"
...

http://projects.mobileread.com/iliad/users/yokos/Wikipedia.0.1.1.zip

Great! I've updated the first post.

Also I noted that in dillo/run.sh, it says "./xset +fp $scriptdir/fonts/TTF", but the directory fonts/ttf actually appears to be all in lowercase. So this line never works. There could be similiar issues elsewhere.

Hmmm ... I wonder if there's any connection between Wikipedia not working on my Iliad and the fact that I never got the Fontswitcher to work, either...

Will try the updated Wikipedia later. (Much later - I'm the last man on dial-up.)

Dillo caches everything, so you have to click the refresh button on the top (4th from the left) to get random to give you a new page.
-Amir

you have to click the refresh button on the top (4th from the left) to get random to give you a new page

Thanks for this one. I like to sit, letting me surprise and reading...;)

Gogolo

Cool, thanks guys! Now all I need is a bigger flash card -- the Simple English version of Wikipedia makes me feel like I'm being patronised by a primary school teacher :)

Now all I need is a bigger flash card

Yes, but everybody thinking about buying a new flashcard for full english wikipedia please read this (http://code.google.com/p/pyoffwiki/wiki/LargeFileProblems) first.

Gogolo

Yes, but everybody thinking about buying a new flashcard for full english wikipedia please read this (http://code.google.com/p/pyoffwiki/wiki/LargeFileProblems) first.

Thanks for the info! I'm using linux, so I'm OK from that point of view.

First i like to thank amirshim for his great work, as all others for building and fixing the package.
:2thumbsup
It's great to have wikipedia offline on iLiad.
:book2:
I have only one problem with dillo, as i can not enter german umlauts as äöü.
But midori have no problem with this.

Nice weekend

Oliver

I think there is another bug: If you click a link that begins with a Umlaut (äöü) it gives back a error - in all three browsers.
Gogolo

This is so not working. Copied the all-in-one package over to my CF card.
On the iLiad I run "start wikipedia server." tried the wikipedia.html and dillo


wikipedia.html : LCD blinks like 5-10 minuttes, gives an error - Connection refused

Dillo: Unable to connect to host.


*****
Error.txt:

'import site' failed; use -v for traceback
Traceback (most recent call last):
File "web.py", line 13, in <module>
from mwlib import uparser, htmlwriter
File "/media/cf/Wikipedia/exe/python/lib/python2.5/site-packages/mwlib/__init__.py", line 5, in <module>
import pkg_resources
ImportError: No module named pkg_resources

***

I have a iLiad Book edition (NO WIFI) OS vers. 2.12.1 . with IPDF fullscreen, etc. installed


One more thing, i don't know if it's because i have the ipdf fullscreen hack installed, but how do i get dillo to show a keyboard so that I can type in a url ? - when connected to a wired network. I know wikipedia works offline.

Yes, but everybody thinking about buying a new flashcard for full english wikipedia please read this (http://code.google.com/p/pyoffwiki/wiki/LargeFileProblems) first.

Gogolo

Hi,
Thanks for this information and to all the people involved in bringing the wikipedia to the iliad. Do you know if this same problem/fix applies to USB drives as well? I was planning on getting a tiny 4GB USB drive for wikipedia instead of buying a 4GB CF card, since it would be cheaper.
Thanks for any help.
Tom

I just ordered a USB-CF-reader to reformat my 8GB card to ext2 to prepare it to receive the english wikipedia. I just don't dare to call any createfs on the iLiad :) I got one for 7,- Euro, thats not even a pizza+drink.

And a big THANK YOU! for amirshim!

4gb usb key should work.

amirshim, if you have the time, could you look to the code once, if the problem with umlaute (öäü) in the beginning of words can be solved?

Every link beginning with a äöü like ägypten gives a error: AssertionError: cannot get article '\xc3\x84gypten'

Sometimes this happens even, when the umlaut is within the word, but sometimes not :chinscratch:

Thanks for your work!
Gogolo

hI Tom , just to confirm , i have tried usb stick 4 gb and it works.
Following are some problems which i faced.
- you need to format usb with ext2 , now how you manage it if you are windows users like me this is another story :)
- when your usb stick is ext2 formated then your iliad does not recognize it like normal usb with FAT so always you have to mount it yourself which i handled by copying onescript with "mount /dev/sda1 /mnt/usb -t ext2"
- thrild problem comes that when you use ext2 then linux permission gets applied so in first shot it does not work as phython directory etc. don't have execute permission so i manage that by "chmod 777 " etc. etc.
- though dillo is not liked by AdamB and other fellow but reality is it browse more smoothly than madori or minimo atleast for wikipedia ( or alteast my experience ) :)
- last and not least , it is wonderfull to have wikipedia under my pillow :P

I got it working too. Thanks for this really nice application amirshim :)

hI Tom , just to confirm , i have tried usb stick 4 gb and it works.
Following are some problems which i faced.
- you need to format usb with ext2 , now how you manage it if you are windows users like me this is another story :)
- when your usb stick is ext2 formated then your iliad does not recognize it like normal usb with FAT so always you have to mount it yourself which i handled by copying onescript with "mount /dev/sda1 /mnt/usb -t ext2"
- thrild problem comes that when you use ext2 then linux permission gets applied so in first shot it does not work as phython directory etc. don't have execute permission so i manage that by "chmod 777 " etc. etc.
- though dillo is not liked by AdamB and other fellow but reality is it browse more smoothly than madori or minimo atleast for wikipedia ( or alteast my experience ) :)
- last and not least , it is wonderfull to have wikipedia under my pillow :P

Thank you very much for the detailed information. Its nice to hear someone has got it working from the USB drive-I ordered one of the tiny Kingston Data traveler Mini 4GB drives to use. I have already downloaded a Linux for Dummies eBook since to be honest I don't really know anything about the operating system, but the chance to have Wikipedia on the iliad provides a great incentive to learn. Thanks again for your help.

Delete this.

Gogolo,
I've never really had to deal with unicode, especially in python, but I think I might have a solution to your problem. But I don't have time to test it, so I guess you're my new beta tester :cool:

Anyway... download the following file, and replace the one in the "wiki/" directory with it. Update me (and the forum) on what happens.

http://pyoffwiki.googlecode.com/svn-history/r45/trunk/wiki/web.py

-Amir

so I guess you're my new beta tester

:) I am pleased to be your beta tester.

And YES! it works just fine!!! :thumbsup:

Thank you very much for your effort! Now german speaking users: Get your wikipedia on iLiad, its way cool :book2:

Greetings
Gogolo

Okay I just got my CF-card reader and fdisk'ed and makefs.ext'ed my 8GB CF-card.

I wondered why the CF card isn't mounted and found out that the default /etc/fstab is wrong. So i tried to make this "simple change" directly within mxrcvt.. It was a pain. I just couldn't do it. So I installed this http://www.mobileread.com/forums/showpost.php?p=144039&postcount=46 ssh access package that is done pretty well. Just its named so badly, this is hardly an "unbrickable iliad now". :)

Okay CF mounts now, I played around with the wikipedia stuff and stop at following error.txt message:

File "web.py", line 22, in <module>
db = bz2wiki.Bz2WikiDB(mainfilename)
File "/media/cf/wiki/bz2wiki.py", line 41, in __init__
self.cdb_file = seekbzip2.SeekableBzip2File(basefn+'.cdb.bz2',base fn+'.cdb.bzt')
File "/media/cf/wiki/seekbzip2.py", line 20, in __init__
self.init_table()
File "/media/cf/wiki/seekbzip2.py", line 34, in init_table
for line in open( self.table_filename ):
IOError: [Errno 2] No such file or directory: '../wikidata/enwiki-20080312-pages-articles.xml.cdb.bzt'


Okay I'm stupid I forgot copy over the other wikidata file. I did this now, no error message. But also no wikipedia. The processor light just keeps blinking after I clicked the link in Wikipedia.html but nothing happens.

Strange enough If I point my normal notebook at the (the Iliads IP) 192.168.1.104:8080 I *DO* get a wikipedia! Seems to be a problem with the buildin browser then...

Oh it works now... strange.... but COOOOOOOLLLLLL! :) :) :) :) :) :) :)

Okay the buildin browser is a pain, I cannot normally search a single side, or select the search box... I'll have to install one of the other browsers... but not today anymore :)

Okay I installed dilo, definitely better than the default browser... but still everything extremely slow. And the scrollbar is almost out of the screen! Nor do the up/down buttons work at all. *Sigh* Iliad, its all so much beta ...its all so much slow... I fear in this state I cannot use it "to show off", since when I demonstrate it, I will likely tear my hair...

Many thanks to amirshim, adam, and atk. I have the full-length wikipedia working nicely on my iliad-its great! I think having the full wikipedia with you whenever you want really is another big plus for having an iliad. The default iliad browser seems to work quite well for me for browsing the offline.

Hi amirshim,

Your (Adam's?) wikipedia.html it's a stripped-down web page with a link to the actual wiki.


<a href=http://localhost:8080>Open Wikipedia</a>

why not just a redirect?


<html>
<head>
<meta http-equiv="refresh" content="0;url=http://localhost:8080">
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
</body>
</html


You know... one less link to follow to get to the meat. :bookworm:

Michele

Hi all,

Is there any description available for the index format for pyoffwiki? I'd like to work on further wikipedia snapshots, but it sounds like we're stuck without the index generation code.

Amir, if you have any hacky index code you'd be willing to share, I think I could help clean it up. I've worked on offline wikireaders before.

Thanks!

Is there a way to have an italian index? Actually this stops me from have a wikipedia on iLiad...

Thank you


f.

Hello

:thanks:

Thanks Amir and others. Clearly a contender for _the_ killer app for the iLiad . . . but we really need the index generation code for the latest snapshots (its grown from 3.5 to 5GB!) and all the other languages . . . or maybe I should learn how and do it myself!

Davros