Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > Amazon Kindle > Kindle Developer's Corner

Notices

Reply
 
Thread Tools Search this Thread
Old 09-21-2012, 09:20 AM   #1
silver18
THE NOOB
silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.
 
silver18's Avatar
 
Posts: 701
Karma: 1545649
Join Date: Jan 2012
Location: Italy
Device: Kindle Touch 5.3.2
Wget & recursive download

Good afternoon!!!

I'm here to ask a noobish question (once again)...
I'm using
Code:
wget
to download a page but it seems our Kindle can't manage the
Code:
-r
option to retrieve all images (and links)...

Is there a workaround?
Thanks to everyone!!
silver18 is offline   Reply With Quote
Old 09-21-2012, 09:38 AM   #2
geekmaster
Carpe diem, c'est la vie.
geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.
 
geekmaster's Avatar
 
Posts: 6,433
Karma: 10773668
Join Date: Nov 2011
Location: Multiverse 6627A
Device: K1 to PW3
You could use a standalone "full featured" wget. There may be one in twobob's collection...
geekmaster is offline   Reply With Quote
Advert
Old 09-21-2012, 11:04 AM   #3
silver18
THE NOOB
silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.
 
silver18's Avatar
 
Posts: 701
Karma: 1545649
Join Date: Jan 2012
Location: Italy
Device: Kindle Touch 5.3.2
I tried downloading this wget, I extracted the package and placed in /mnt/us.

Then I tried:
Code:
/mnt/us/wget http://www.google.com
And I got:
Code:
-sh: /mnt/us/wget: not found
I'm really a PITA with unix and packages...
silver18 is offline   Reply With Quote
Old 09-21-2012, 11:23 AM   #4
knc1
Going Viral
knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.
 
knc1's Avatar
 
Posts: 17,212
Karma: 18210809
Join Date: Feb 2012
Location: Central Texas
Device: No K1, PW2, KV, KOA
Hmm...
/mnt/us is FAT32 - -
Doesn't FAT32 support execute permission?

Do an:
ls -l /mnt/us/wget
And see if it is marked executable.

Also do a:
mount
and see if the mount parameters given when mounting the FAT32 was given the "exec" permission or was given the "noexec" permission.

Note:
My fingers and my keyboard are not getting along today.
Always wait at least 10 minutes after this post time before quoting or replying - it can take me that long to spot errors and typos.
knc1 is offline   Reply With Quote
Old 09-21-2012, 11:33 AM   #5
geekmaster
Carpe diem, c'est la vie.
geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.
 
geekmaster's Avatar
 
Posts: 6,433
Karma: 10773668
Join Date: Nov 2011
Location: Multiverse 6627A
Device: K1 to PW3
Quote:
Originally Posted by knc1 View Post
Doesn't FAT32 support execute permission? ...
EVERYTHING on the USB drive "automagically" has execute permission (not a FAT attribute, so assumed).
Quote:
Originally Posted by silver18 View Post
I tried downloading this wget, I extracted the package and placed in /mnt/us. ...
You probably need "armel" instead of "armhf". Here is the armel wget:
http://packages.debian.org/sid/armel/wget/download

It is the same URL that you provided (http://packages.debian.org/sid/armhf/wget/download), but with "armhf" changed to "armel". Try that one.

Last edited by geekmaster; 09-21-2012 at 11:46 AM.
geekmaster is offline   Reply With Quote
Advert
Old 09-21-2012, 11:38 AM   #6
baf
Evangelist
baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.
 
Posts: 404
Karma: 2200000
Join Date: May 2012
Device: kt
You may use wget from here, which I built for KT. At the bottom of the webpage.
baf is offline   Reply With Quote
Old 09-21-2012, 12:00 PM   #7
geekmaster
Carpe diem, c'est la vie.
geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.
 
geekmaster's Avatar
 
Posts: 6,433
Karma: 10773668
Join Date: Nov 2011
Location: Multiverse 6627A
Device: K1 to PW3
Quote:
Originally Posted by baf View Post
You may use wget from here, which I built for KT. At the bottom of the webpage.
Thanks! I know there were a few versions floating around.
geekmaster is offline   Reply With Quote
Old 09-23-2012, 07:56 AM   #8
silver18
THE NOOB
silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.
 
silver18's Avatar
 
Posts: 701
Karma: 1545649
Join Date: Jan 2012
Location: Italy
Device: Kindle Touch 5.3.2
Thanks Baf, that wget works really well!!!
It's a good starting point for my future WAF (if it will ever become real...)
silver18 is offline   Reply With Quote
Old 09-23-2012, 07:59 AM   #9
twobob
( ͡° ͜ʖ ͡°){ʇlnɐɟ ƃǝs}Týr
twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.
 
twobob's Avatar
 
Posts: 6,586
Karma: 6299991
Join Date: Jun 2012
Location: uti gratia usura (Yao ying da ying; Mo ying da yieng)
Device: PW-WIFI|K5-3G+WIFI| K4|K3-3G|DXG|K2| Rooted Nook Touch
yeah I was gonna point him at bafs. my work is already done.

There is another one linked to a static curl ligging about somewhere IIRC also.

Baf's will do the job fine (@ GM I didn't even bother building one, nor a locate, nor an updatedb, nor anything baf had already done. his versions are shiny and tidy, I use them)
twobob is offline   Reply With Quote
Old 09-28-2012, 12:00 PM   #10
silver18
THE NOOB
silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.
 
silver18's Avatar
 
Posts: 701
Karma: 1545649
Join Date: Jan 2012
Location: Italy
Device: Kindle Touch 5.3.2
Ok, here I am again...

Baf's wget works really fine (thanks a lot!!!) but I really don't know how to set it up nicely...
Here the situation:
I would like to download a web page (single web page) from Wikipedia, so I run this:
Code:
export PAGES="/mnt/us/WebPages"

webpage="http://it.wikipedia.org/wiki/Wget"

/mnt/us/extensions/offlinepages/bin/resource/Wget/wget -e robots=off --wait=20 --limit-rate=20K --quiet --no-parent --page-requisites --convert-links --adjust-extension -U Mozilla -P "$PAGES" "$webpage"
It goes past the robot.txt exclusions and, thanks to -U, --wait and --limit-rate it isn't detected as a site crawler.
BUT it doesn't download images...
It retrieves only Wget.html and favicon.ico!!
Isn't it supposed to download all the necessary to display the page?

If I turn on the --recursive, I get all the linked html to the wget page but no images again!!

Any help?
Thanks guys!!!
silver18 is offline   Reply With Quote
Old 09-28-2012, 12:37 PM   #11
knc1
Going Viral
knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.
 
knc1's Avatar
 
Posts: 17,212
Karma: 18210809
Join Date: Feb 2012
Location: Central Texas
Device: No K1, PW2, KV, KOA
Images are probably protected from off-site references.

Try having wget pass that web-site's set of cookies so that it thinks the image references are coming from a browser viewing that site.

You might also have to fake up a "refer" header for wget.

Both described in the wget documentation available - I don't know the details off hand (I would have to look them up like any other mere moral.)
knc1 is offline   Reply With Quote
Old 09-28-2012, 01:30 PM   #12
silver18
THE NOOB
silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.
 
silver18's Avatar
 
Posts: 701
Karma: 1545649
Join Date: Jan 2012
Location: Italy
Device: Kindle Touch 5.3.2
Thanks for the hint...

I tried this:

Code:
/mnt/us/extensions/offlinepages/bin/resource/Wget/wget --random-wait -e robots=off --limit-rate=20K --page-requisites --no-parent --convert-links --adjust-extension --restrict-file-names=windows --cookies=on --load-cookies=/var/local/waf/browser/cookieJar --keep-session-cookies -U mozilla -P "$PAGES" "$webpage"
to use the browser's cookies (I visited the webpage using the browser before doing it).
I even tried with:
Code:
--referer=http://it.wikipedia.org
But no luck....
silver18 is offline   Reply With Quote
Old 09-28-2012, 01:35 PM   #13
twobob
( ͡° ͜ʖ ͡°){ʇlnɐɟ ƃǝs}Týr
twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.
 
twobob's Avatar
 
Posts: 6,586
Karma: 6299991
Join Date: Jun 2012
Location: uti gratia usura (Yao ying da ying; Mo ying da yieng)
Device: PW-WIFI|K5-3G+WIFI| K4|K3-3G|DXG|K2| Rooted Nook Touch
hmm. possibly do a bit of playing around with fiddler then. in firefox.
twobob is offline   Reply With Quote
Old 10-02-2012, 11:05 AM   #14
silver18
THE NOOB
silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.
 
silver18's Avatar
 
Posts: 701
Karma: 1545649
Join Date: Jan 2012
Location: Italy
Device: Kindle Touch 5.3.2
Ok, here I am...
The problem when "wgetting" a page from wikipedia is that you need to turn on recursive download as the normal "--page-requisites" option doesn't retrieve images (it gets only the html to be honest).
This is because images are hosted on a separate page!
BUT turning on "--recursive" gives you LOTS of things to download (as wiki pages are full of external & internal links).

So I ended up using --recursive + --span-hosts + --domains.
This is the complete command:
Code:
wget --recursive --span-hosts --domains=upload.wikimedia.org -e robots=off --random-wait --limit-rate=20K --page-requisites --no-parent --convert-links --adjust-extension --restrict-file-names=windows -U Mozilla -P "$folder" "$webpage"
Now images are retrived but I can't get the style sheet...
I'm going to investigate where the hell is it hosted...
silver18 is offline   Reply With Quote
Old 10-02-2012, 11:34 AM   #15
twobob
( ͡° ͜ʖ ͡°){ʇlnɐɟ ƃǝs}Týr
twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.
 
twobob's Avatar
 
Posts: 6,586
Karma: 6299991
Join Date: Jun 2012
Location: uti gratia usura (Yao ying da ying; Mo ying da yieng)
Device: PW-WIFI|K5-3G+WIFI| K4|K3-3G|DXG|K2| Rooted Nook Touch
Quote:
<link rel="stylesheet" href="//bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&amp;lang=en&amp;modules=ext.g adget.ReferenceTooltips%2Ccharinsert%2Cteahouse%7C ext.wikihiero%7Cmediawiki.legacy.commonPrint%2Csha red%7Cskins.vector&amp;only=styles&amp;skin=vector &amp;*" />
<meta name="ResourceLoaderDynamicStyles" content="" />
<link rel="stylesheet" href="//bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&amp;lang=en&amp;modules=site& amp;only=styles&amp;skin=vector&amp;*" />
<style>a:lang(ar),a:lang(ckb),a:lang(fa),a:lang( kk-arab),a:lang(mzn),a:lang(ps),a:lang(ur){text-decoration:none}

/* cache key: enwiki:resourceloader:filter:minify-css:7:8d95de22da3b74bdc8517ef8752d1bee */
</style>

<script src="//bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&amp;lang=en&amp;modules=start up&amp;only=scripts&amp;skin=vector&amp;*"></script>
Looks to be a vaguely protected module loader. good luck!
twobob is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Could anyone compile full-feature wget for Touch wisecrick Kindle Developer's Corner 15 09-09-2012 10:41 AM
wget an ftp folder mephis PocketBook 3 10-11-2011 05:34 PM
902 wget pdf newspaper from site? niroma PocketBook 23 07-10-2011 03:09 PM
How to use wget to download an online HTML book amoroso Lounge 11 04-25-2011 05:10 AM
Recursive conversion of Websites to Kindle oecherprinte Amazon Kindle 0 10-18-2010 07:22 AM


All times are GMT -4. The time now is 07:43 AM.


MobileRead.com is a privately owned, operated and funded community.