Wget & recursive download

[silver18]

Good afternoon!!!

I'm here to ask a noobish question (once again)...
I'm using

wget

to download a page but it seems our Kindle can't manage the

-r

option to retrieve all images (and links)...

Is there a workaround?
Thanks to everyone!!

[geekmaster]

You could use a standalone "full featured" wget. There may be one in twobob's collection...

[silver18]

I tried downloading this wget, I extracted the package and placed in /mnt/us.

Then I tried:

/mnt/us/wget http://www.google.com

And I got:

-sh: /mnt/us/wget: not found

I'm really a PITA with unix and packages...

[knc1]

Hmm...
/mnt/us is FAT32 - -
Doesn't FAT32 support execute permission?

Do an:
ls -l /mnt/us/wget
And see if it is marked executable.

Also do a:
mount
and see if the mount parameters given when mounting the FAT32 was given the "exec" permission or was given the "noexec" permission.

Note:
My fingers and my keyboard are not getting along today.
Always wait at least 10 minutes after this post time before quoting or replying - it can take me that long to spot errors and typos.

[geekmaster]

Originally Posted by knc1

Doesn't FAT32 support execute permission? ...

EVERYTHING on the USB drive "automagically" has execute permission (not a FAT attribute, so assumed).

Originally Posted by silver18

I tried downloading this wget, I extracted the package and placed in /mnt/us. ...

You probably need "armel" instead of "armhf". Here is the armel wget:
http://packages.debian.org/sid/armel/wget/download

It is the same URL that you provided (http://packages.debian.org/sid/armhf/wget/download), but with "armhf" changed to "armel". Try that one.

[baf]

You may use wget from here, which I built for KT. At the bottom of the webpage.

[geekmaster]

Originally Posted by baf

You may use wget from here, which I built for KT. At the bottom of the webpage.

Thanks! I know there were a few versions floating around.

[silver18]

Thanks Baf, that wget works really well!!!
It's a good starting point for my future WAF (if it will ever become real...)

[twobob]

yeah I was gonna point him at bafs. my work is already done.

There is another one linked to a static curl ligging about somewhere IIRC also.

Baf's will do the job fine (@ GM I didn't even bother building one, nor a locate, nor an updatedb, nor anything baf had already done. his versions are shiny and tidy, I use them)

[silver18]

Ok, here I am again...

Baf's wget works really fine (thanks a lot!!!) but I really don't know how to set it up nicely...
Here the situation:
I would like to download a web page (single web page) from Wikipedia, so I run this:

export PAGES="/mnt/us/WebPages"

webpage="http://it.wikipedia.org/wiki/Wget"

/mnt/us/extensions/offlinepages/bin/resource/Wget/wget -e robots=off --wait=20 --limit-rate=20K --quiet --no-parent --page-requisites --convert-links --adjust-extension -U Mozilla -P "$PAGES" "$webpage"

It goes past the robot.txt exclusions and, thanks to -U, --wait and --limit-rate it isn't detected as a site crawler.
BUT it doesn't download images...
It retrieves only Wget.html and favicon.ico!!
Isn't it supposed to download all the necessary to display the page?

If I turn on the --recursive, I get all the linked html to the wget page but no images again!!

Any help?
Thanks guys!!!

[knc1]

Images are probably protected from off-site references.

Try having wget pass that web-site's set of cookies so that it thinks the image references are coming from a browser viewing that site.

You might also have to fake up a "refer" header for wget.

Both described in the wget documentation available - I don't know the details off hand (I would have to look them up like any other mere moral.)

[silver18]

Thanks for the hint...

I tried this:

/mnt/us/extensions/offlinepages/bin/resource/Wget/wget --random-wait -e robots=off --limit-rate=20K --page-requisites --no-parent --convert-links --adjust-extension --restrict-file-names=windows --cookies=on --load-cookies=/var/local/waf/browser/cookieJar --keep-session-cookies -U mozilla -P "$PAGES" "$webpage"

to use the browser's cookies (I visited the webpage using the browser before doing it).
I even tried with:

--referer=http://it.wikipedia.org

But no luck....

[twobob]

hmm. possibly do a bit of playing around with fiddler then. in firefox.

[silver18]

Ok, here I am...
The problem when "wgetting" a page from wikipedia is that you need to turn on recursive download as the normal "--page-requisites" option doesn't retrieve images (it gets only the html to be honest).
This is because images are hosted on a separate page!
BUT turning on "--recursive" gives you LOTS of things to download (as wiki pages are full of external & internal links).

So I ended up using --recursive + --span-hosts + --domains.
This is the complete command:

wget --recursive --span-hosts --domains=upload.wikimedia.org -e robots=off --random-wait --limit-rate=20K --page-requisites --no-parent --convert-links --adjust-extension --restrict-file-names=windows -U Mozilla -P "$folder" "$webpage"

Now images are retrived but I can't get the style sheet...
I'm going to investigate where the hell is it hosted...

[twobob]

<link rel="stylesheet" href="//bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&amp;lang=en&amp;modules=ext.g adget.ReferenceTooltips%2Ccharinsert%2Cteahouse%7C ext.wikihiero%7Cmediawiki.legacy.commonPrint%2Csha red%7Cskins.vector&amp;only=styles&amp;skin=vector &amp;*" />
<meta name="ResourceLoaderDynamicStyles" content="" />
<link rel="stylesheet" href="//bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&amp;lang=en&amp;modules=site& amp;only=styles&amp;skin=vector&amp;*" />
<style>a:lang(ar),a:lang(ckb),a:lang(fa),a:lang( kk-arab),a:lang(mzn),a:lang(ps),a:lang(ur){text-decoration:none}

/* cache key: enwiki:resourceloader:filter:minify-css:7:8d95de22da3b74bdc8517ef8752d1bee */
</style>

<script src="//bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&amp;lang=en&amp;modules=start up&amp;only=scripts&amp;skin=vector&amp;*"></script>

Looks to be a vaguely protected module loader. good luck!