beta offline(ish) Wiktionary viewer for DR

Mr. X · 03-16-2010, 07:19 PM

I managed to port http://users.softlab.ece.ntua.gr/~tt...iaOffline.html to the DR 3.0 architecture. It allows you to browse a wikimedia site off of local storage with semi-marked up text but no graphics (see pic for example).

Unfortunately, the web browser on the DR goes online when there's any network access, even to localhost (127.0.0.1). It won't open local URLs at all in airplane mode, or presumably on an 800S... Until there's a solution to this running web-based apps locally (was also thinking of porting a little wiki till we get note taking) is suboptimal

For the adventurous/curious:

Install the following support packages onto your SD card (each is self-contained and creates a single folder in Programs/. To remove just delete):

Then grab the Wiktionary program from http://www.cis.upenn.edu/~ecronin/tmp/Wiktionary.zip (warning, 350MB) and decompress it to the SD card as well. Browse the SD card and run it from Programs. All in all it takes about 1.2GB of storage

For me this was a fun excuse to get a handle on the quirks of porting existing stuff to the iRex, but I don't plan to polish it up any further unless erbrowser gets an offline mode that would make it practical. If anyone's interested in hacking on it the source is all in that Wiktionary.zip file.

Aard (http://aarddict.org/) looks more promising as an offline wiktionary/wikipedia viewer, but would require someone writing a different GUI to it in pyGTK, the current one is pyQT...

badbob001 · 03-17-2010, 07:54 AM

From the source, it looks like if the URL starts with 'file://', then it will not access the network:

"\erbrowser-3.0+svnr5508\erbrowser\src\view retrieved 1.c"

Code:

    else if (is_local_file(uri))
    {
        // local file, no network needed
    }

...

Code:

static gboolean is_local_file(const gchar *uri)
{
    return uri && g_str_has_prefix(uri, "file://");
}

Shaggy · 03-17-2010, 10:52 AM

Quote:

Originally Posted by badbob001

From the source, it looks like if the URL starts with 'file://', then it will not access the network:

Yeah, that was my first thought as well. I wonder if it would be difficult to convert any http://127.0.0.1... references to file://... instead. That should take care of it.

Mr. X · 03-18-2010, 09:23 AM

file:// won't help in this situation as there is actually a small web server running locally which generates the pages on the fly (so you can do search for example)... Patching is_local_file() would be simple enough to do I suppose

Shaggy · 03-18-2010, 10:16 AM

Quote:

Originally Posted by Mr. X

file:// won't help in this situation as there is actually a small web server running locally which generates the pages on the fly (so you can do search for example)... Patching is_local_file() would be simple enough to do I suppose

Ah, didn't realize it was a web server. I assumed it was just a bunch of html files. Yeah, patching is_local_file() to also check for a localhost URL would probably work.

Mr. X · 03-18-2010, 03:22 PM

Yep, one new subroutine and a one-line patch and the DR won't go online accessing the local wiktionary server... Patch for erbrowser is attached, and the zipfile for wiktionary referenced above has been updated. Because erbrowser itself is tiny I just include the patched version in the _offline.wiktionary directory so it doesn't overwrite any system files.

I will also post a _offline.wikipedia zipfile with the index but not the actual 6 gigs of pages, and some instructions on how to split up the data yourself.

Mr. X · 03-19-2010, 09:47 AM

OK, final update of the files for real this time I think...

New instructions:

You still need the five support libraries from the first message. Then decompress OfflineWiki.zip from this message. This completes the install of the program, but you still need the wiki data and the index for it... See the documentation in the original offline.wikipedia package at http://users.softlab.ece.ntua.gr/~tt...iaOffline.html for how to build these yourself. I've put the index files for English Wiktionary and Wikipedia up on my personal web space for the March 2010 dumps of the respective sites. You can grab them to avoid needing to compile and install anything on your local machine (other than bzip2 which most people should have):

Wiktionary:
http://download.wikimedia.org/enwikt...ticles.xml.bz2 -> _offline.wiki/wiktionary/wiki-splits
http://www.cis.upenn.edu/~ecronin/tmp/wiktionary-db.zip -> _offline.wiki/wiktionary/db
cd _offline.wiki/wiktionary and run "make splits"
rm _offline.wiki/wiktionary/wiki-splits/enwiktionary-20100310-pages-articles.xml.bz2

Wikipedia:
http://download.wikimedia.org/enwiki...ticles.xml.bz2 -> _offline.wiki/wikipedia/wiki-splits
http://www.cis.upenn.edu/~ecronin/tmp/wikipedia-db.zip -> _offline.wiki/wikipedia/db
cd _offline.wiki/wikipedia and run "make splits"
rm _offline.wiki/wikipedia/wiki-splits/enwiki-20100312-pages-articles.xml.bz2

Now the bad news. Wikipedia is 5.8GB for the data itself, and another 3GB or so for the index of articles... Meaning it won't fit on an 8GB MicroSD card... So until 16GB cards come down in price a full wikipedia on the DR is probably not practical for most people.

KarlSchroeder · 03-26-2010, 07:57 PM

The Wikpedia-db.zip link above is broken. How can I get this file? (I've downloaded the rest and bought a SD card specifically to try loading wikipedia on. Without the index file, though, I'm out of luck as far as I know.)

Mr. X · 03-27-2010, 05:21 AM

Sorry about that, I forgot to upload it. It should be there in a few hours...

KarlSchroeder · 03-27-2010, 04:33 PM

Thanks. It appears to be there now, but I get an "Access Forbidden" message when I try to get it. A permissions issue, I guess...

Mr. X · 03-27-2010, 04:50 PM

Upload finished, should be good to go now.

fekhner · 11-06-2011, 08:56 PM

Hi Mr. X, first of all, thanks a lot for sharing!

Unfortunately, I am not able to make it work so there is something I must be doing wrong: when I try to run 'make split' (by doing a cd to the directory on the DR from the computer), it says " make: *** No rule to make target `splits'. Stop. "

(i) Am I suppose to run it like this ?
(ii) I am a little puzzled by the content of the _/offline.wiki/Wikipedia.desktop
which contains Exec=./_offline.wiktionary.sh and not Exec=./_offline.wikipedia.sh
(iii) Has anybody else managed to get this to work ?

03-16-2010, 07:19 PM	#1
Mr. X Connoisseur Posts: 68 Karma: 64 Join Date: Mar 2010 Location: Philadelphia Device: iRex DR800SG	beta offline(ish) Wiktionary viewer for DR I managed to port http://users.softlab.ece.ntua.gr/~tt...iaOffline.html to the DR 3.0 architecture. It allows you to browse a wikimedia site off of local storage with semi-marked up text but no graphics (see pic for example). Unfortunately, the web browser on the DR goes online when there's any network access, even to localhost (127.0.0.1). It won't open local URLs at all in airplane mode, or presumably on an 800S... Until there's a solution to this running web-based apps locally (was also thinking of porting a little wiki till we get note taking) is suboptimal For the adventurous/curious: Install the following support packages onto your SD card (each is self-contained and creates a single folder in Programs/. To remove just delete): https://dsl-scm.cis.upenn.edu:8888/s....1.1-DR800.tbz https://dsl-scm.cis.upenn.edu:8888/s...+gtk-DR800.tbz https://dsl-scm.cis.upenn.edu:8888/s...DR800-mini.tbz https://dsl-scm.cis.upenn.edu:8888/s...DR800-full.tbz https://dsl-scm.cis.upenn.edu:8888/s...0.18-DR800.tbz Then grab the Wiktionary program from http://www.cis.upenn.edu/~ecronin/tmp/Wiktionary.zip (warning, 350MB) and decompress it to the SD card as well. Browse the SD card and run it from Programs. All in all it takes about 1.2GB of storage For me this was a fun excuse to get a handle on the quirks of porting existing stuff to the iRex, but I don't plan to polish it up any further unless erbrowser gets an offline mode that would make it practical. If anyone's interested in hacking on it the source is all in that Wiktionary.zip file. Aard (http://aarddict.org/) looks more promising as an offline wiktionary/wikipedia viewer, but would require someone writing a different GUI to it in pyGTK, the current one is pyQT...

03-17-2010, 07:54 AM	#2
badbob001 Fanatic Posts: 556 Karma: 1102020 Join Date: Sep 2009 Device: Kindle Keyboard (rip), Kindle Voyage, Fire Tablet 10 '17, iPad '19	From the source, it looks like if the URL starts with 'file://', then it will not access the network: "\erbrowser-3.0+svnr5508\erbrowser\src\view retrieved 1.c" Code: else if (is_local_file(uri)) { // local file, no network needed } ... Code: static gboolean is_local_file(const gchar *uri) { return uri && g_str_has_prefix(uri, "file://"); }

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[old-topic]ebook-viewer: Open source, crossplatform viewer for EPUB, LIT, MOBI, etc	kovidgoyal	Calibre	68	05-30-2011 08:46 PM
Hello from Seattle(ish)	jrgorman	Introduce Yourself	4	05-29-2008 04:06 PM
Full Wiktionary Newly Available for Download (And new IMDB files)	MatYadabyte	News	0	03-22-2007 10:52 AM
New PDF Viewer beta for Cybook available	Colin Dunstan	Legacy E-Book Devices	0	10-02-2005 01:14 PM
Cybook PDF viewer beta	Colin Dunstan	Legacy E-Book Devices	2	09-30-2005 02:45 PM

03-18-2010, 09:23 AM	#4
Mr. X Connoisseur Posts: 68 Karma: 64 Join Date: Mar 2010 Location: Philadelphia Device: iRex DR800SG	file:// won't help in this situation as there is actually a small web server running locally which generates the pages on the fly (so you can do search for example)... Patching is_local_file() would be simple enough to do I suppose

03-26-2010, 07:57 PM	#8
KarlSchroeder Junior Member Posts: 8 Karma: 10 Join Date: May 2009 Device: iRex DR1000s	The Wikpedia-db.zip link above is broken. How can I get this file? (I've downloaded the rest and bought a SD card specifically to try loading wikipedia on. Without the index file, though, I'm out of luck as far as I know.)

03-27-2010, 05:21 AM	#9
Mr. X Connoisseur Posts: 68 Karma: 64 Join Date: Mar 2010 Location: Philadelphia Device: iRex DR800SG	Sorry about that, I forgot to upload it. It should be there in a few hours...

03-27-2010, 04:33 PM	#10
KarlSchroeder Junior Member Posts: 8 Karma: 10 Join Date: May 2009 Device: iRex DR1000s	Thanks. It appears to be there now, but I get an "Access Forbidden" message when I try to get it. A permissions issue, I guess...

03-27-2010, 04:50 PM	#11
Mr. X Connoisseur Posts: 68 Karma: 64 Join Date: Mar 2010 Location: Philadelphia Device: iRex DR800SG	Upload finished, should be good to go now.

11-06-2011, 08:56 PM	#12
fekhner Addict Posts: 326 Karma: 6908 Join Date: Sep 2010 Device: DRs, EE, ipad/iphone, Helix	Hi Mr. X, first of all, thanks a lot for sharing! Unfortunately, I am not able to make it work so there is something I must be doing wrong: when I try to run 'make split' (by doing a cd to the directory on the DR from the computer), it says " make: *** No rule to make target `splits'. Stop. " (i) Am I suppose to run it like this ? (ii) I am a little puzzled by the content of the _/offline.wiki/Wikipedia.desktop which contains Exec=./_offline.wiktionary.sh and not Exec=./_offline.wikipedia.sh (iii) Has anybody else managed to get this to work ?