hacker
02-11-2005, 12:15 AM
As promised over here (http://www.mobileread.com/forums/showthread.php?t=2235), I've taken some time to clean up and cannibalize part of a spider I use daily for Plucker (http://www.plkr.org), and modified it to allow you to run it and spider MapQuest (http://www.mapquest.com/atlas/?region=index) to build yourself a World Atlas with images, country data, and lots of other bits.
The whole script is only 87 lines of actual code! (161 with comments and liberal spacing for readibility) I prefer writing clean, tight, well-commented code. My code is my business card, and this is no exception.
The script is written in Perl, my language of choice, but all modules used are either in core, or available via CPAN (http://search.cpan.org/) (perl -MCPAN -e 'install "Module::Name"'). It should be easy to run and figure out. I've commented it where required. My only requirement of using this, is that you don't rip off the code and claim you wrote it, or parts of it, and that you provide some feedback so I can improve it; good, bad, feature requests, bugs you find, whatever. I'd like to know!
Unfortunately, I cannot redistribute the completed version of the maps in mobile format, because that would violate MapQuest's copyright and Terms of Use, but you can see how good it looks in the screenshots below.
The entire script is attached below. Just grab the script and run it in an empty directory. It will spider and fetch the 238-or-so separate pages from MapQuest's World Atlas pages, strip out the unnecessary HTML, Javascript, stylesheets, and other non-visible bits, and write each country to its own file. All of the external links to country data is rewritten to reference the local copies. The only pieces fetched remotely are the images themselves.
When the spidering is complete, it outputs a top-level index file for you to point your mobile creation tool towards, so you can then spider the content yourself, and convert it to the format of your choice.
Hopefully many users will find this useful. Enjoy!
The whole script is only 87 lines of actual code! (161 with comments and liberal spacing for readibility) I prefer writing clean, tight, well-commented code. My code is my business card, and this is no exception.
The script is written in Perl, my language of choice, but all modules used are either in core, or available via CPAN (http://search.cpan.org/) (perl -MCPAN -e 'install "Module::Name"'). It should be easy to run and figure out. I've commented it where required. My only requirement of using this, is that you don't rip off the code and claim you wrote it, or parts of it, and that you provide some feedback so I can improve it; good, bad, feature requests, bugs you find, whatever. I'd like to know!
Unfortunately, I cannot redistribute the completed version of the maps in mobile format, because that would violate MapQuest's copyright and Terms of Use, but you can see how good it looks in the screenshots below.
The entire script is attached below. Just grab the script and run it in an empty directory. It will spider and fetch the 238-or-so separate pages from MapQuest's World Atlas pages, strip out the unnecessary HTML, Javascript, stylesheets, and other non-visible bits, and write each country to its own file. All of the external links to country data is rewritten to reference the local copies. The only pieces fetched remotely are the images themselves.
When the spidering is complete, it outputs a top-level index file for you to point your mobile creation tool towards, so you can then spider the content yourself, and convert it to the format of your choice.
Hopefully many users will find this useful. Enjoy!