Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > Amazon Kindle > Kindle Developer's Corner

Notices

Reply
 
Thread Tools Search this Thread
Old 09-21-2012, 10:20 AM   #1
silver18
THE NOOB
silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.
 
silver18's Avatar
 
Posts: 698
Karma: 1545625
Join Date: Jan 2012
Location: Italy
Device: Kindle Touch 5.3.2
Wget & recursive download

Good afternoon!!!

I'm here to ask a noobish question (once again)...
I'm using
Code:
wget
to download a page but it seems our Kindle can't manage the
Code:
-r
option to retrieve all images (and links)...

Is there a workaround?
Thanks to everyone!!
silver18 is offline   Reply With Quote
Old 09-21-2012, 10:38 AM   #2
geekmaster
Всё гениальное просто.
geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.
 
geekmaster's Avatar
 
Posts: 5,070
Karma: 6789001
Join Date: Nov 2011
Location: Щедрость не имеет пределов.
Device: *.*
You could use a standalone "full featured" wget. There may be one in twobob's collection...
geekmaster is offline   Reply With Quote
Old 09-21-2012, 12:04 PM   #3
silver18
THE NOOB
silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.
 
silver18's Avatar
 
Posts: 698
Karma: 1545625
Join Date: Jan 2012
Location: Italy
Device: Kindle Touch 5.3.2
I tried downloading this wget, I extracted the package and placed in /mnt/us.

Then I tried:
Code:
/mnt/us/wget http://www.google.com
And I got:
Code:
-sh: /mnt/us/wget: not found
I'm really a PITA with unix and packages...
silver18 is offline   Reply With Quote
Old 09-21-2012, 12:23 PM   #4
knc1
Helpdesk Junkie
knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.
 
knc1's Avatar
 
Posts: 7,001
Karma: 6327868
Join Date: Feb 2012
Device: Too many.
Hmm...
/mnt/us is FAT32 - -
Doesn't FAT32 support execute permission?

Do an:
ls -l /mnt/us/wget
And see if it is marked executable.

Also do a:
mount
and see if the mount parameters given when mounting the FAT32 was given the "exec" permission or was given the "noexec" permission.

Note:
My fingers and my keyboard are not getting along today.
Always wait at least 10 minutes after this post time before quoting or replying - it can take me that long to spot errors and typos.
knc1 is offline   Reply With Quote
Old 09-21-2012, 12:33 PM   #5
geekmaster
Всё гениальное просто.
geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.
 
geekmaster's Avatar
 
Posts: 5,070
Karma: 6789001
Join Date: Nov 2011
Location: Щедрость не имеет пределов.
Device: *.*
Quote:
Originally Posted by knc1 View Post
Doesn't FAT32 support execute permission? ...
EVERYTHING on the USB drive "automagically" has execute permission (not a FAT attribute, so assumed).
Quote:
Originally Posted by silver18 View Post
I tried downloading this wget, I extracted the package and placed in /mnt/us. ...
You probably need "armel" instead of "armhf". Here is the armel wget:
http://packages.debian.org/sid/armel/wget/download

It is the same URL that you provided (http://packages.debian.org/sid/armhf/wget/download), but with "armhf" changed to "armel". Try that one.

Last edited by geekmaster; 09-21-2012 at 12:46 PM.
geekmaster is offline   Reply With Quote
Old 09-21-2012, 12:38 PM   #6
baf
Addict
baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.baf ought to be getting tired of karma fortunes by now.
 
Posts: 332
Karma: 1346560
Join Date: May 2012
Device: kt
You may use wget from here, which I built for KT. At the bottom of the webpage.
baf is offline   Reply With Quote
Old 09-21-2012, 01:00 PM   #7
geekmaster
Всё гениальное просто.
geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.geekmaster ought to be getting tired of karma fortunes by now.
 
geekmaster's Avatar
 
Posts: 5,070
Karma: 6789001
Join Date: Nov 2011
Location: Щедрость не имеет пределов.
Device: *.*
Quote:
Originally Posted by baf View Post
You may use wget from here, which I built for KT. At the bottom of the webpage.
Thanks! I know there were a few versions floating around.
geekmaster is offline   Reply With Quote
Old 09-23-2012, 08:56 AM   #8
silver18
THE NOOB
silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.
 
silver18's Avatar
 
Posts: 698
Karma: 1545625
Join Date: Jan 2012
Location: Italy
Device: Kindle Touch 5.3.2
Thanks Baf, that wget works really well!!!
It's a good starting point for my future WAF (if it will ever become real...)
silver18 is offline   Reply With Quote
Old 09-23-2012, 08:59 AM   #9
twobob
( ͡° ͜ʖ ͡°){ʇlnɐɟ ƃǝs}Tır
twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.
 
twobob's Avatar
 
Posts: 6,552
Karma: 6021146
Join Date: Jun 2012
Location: uti gratia usura (Yao ying da ying; Mo ying da yieng)
Device: PW-WIFI|K5-3G+WIFI| K4|K3-3G|DXG|K2| Rooted Nook Touch
yeah I was gonna point him at bafs. my work is already done.

There is another one linked to a static curl ligging about somewhere IIRC also.

Baf's will do the job fine (@ GM I didn't even bother building one, nor a locate, nor an updatedb, nor anything baf had already done. his versions are shiny and tidy, I use them)
twobob is offline   Reply With Quote
Old 09-28-2012, 01:00 PM   #10
silver18
THE NOOB
silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.
 
silver18's Avatar
 
Posts: 698
Karma: 1545625
Join Date: Jan 2012
Location: Italy
Device: Kindle Touch 5.3.2
Ok, here I am again...

Baf's wget works really fine (thanks a lot!!!) but I really don't know how to set it up nicely...
Here the situation:
I would like to download a web page (single web page) from Wikipedia, so I run this:
Code:
export PAGES="/mnt/us/WebPages"

webpage="http://it.wikipedia.org/wiki/Wget"

/mnt/us/extensions/offlinepages/bin/resource/Wget/wget -e robots=off --wait=20 --limit-rate=20K --quiet --no-parent --page-requisites --convert-links --adjust-extension -U Mozilla -P "$PAGES" "$webpage"
It goes past the robot.txt exclusions and, thanks to -U, --wait and --limit-rate it isn't detected as a site crawler.
BUT it doesn't download images...
It retrieves only Wget.html and favicon.ico!!
Isn't it supposed to download all the necessary to display the page?

If I turn on the --recursive, I get all the linked html to the wget page but no images again!!

Any help?
Thanks guys!!!
silver18 is offline   Reply With Quote
Old 09-28-2012, 01:37 PM   #11
knc1
Helpdesk Junkie
knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.knc1 ought to be getting tired of karma fortunes by now.
 
knc1's Avatar
 
Posts: 7,001
Karma: 6327868
Join Date: Feb 2012
Device: Too many.
Images are probably protected from off-site references.

Try having wget pass that web-site's set of cookies so that it thinks the image references are coming from a browser viewing that site.

You might also have to fake up a "refer" header for wget.

Both described in the wget documentation available - I don't know the details off hand (I would have to look them up like any other mere moral.)
knc1 is offline   Reply With Quote
Old 09-28-2012, 02:30 PM   #12
silver18
THE NOOB
silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.
 
silver18's Avatar
 
Posts: 698
Karma: 1545625
Join Date: Jan 2012
Location: Italy
Device: Kindle Touch 5.3.2
Thanks for the hint...

I tried this:

Code:
/mnt/us/extensions/offlinepages/bin/resource/Wget/wget --random-wait -e robots=off --limit-rate=20K --page-requisites --no-parent --convert-links --adjust-extension --restrict-file-names=windows --cookies=on --load-cookies=/var/local/waf/browser/cookieJar --keep-session-cookies -U mozilla -P "$PAGES" "$webpage"
to use the browser's cookies (I visited the webpage using the browser before doing it).
I even tried with:
Code:
--referer=http://it.wikipedia.org
But no luck....
silver18 is offline   Reply With Quote
Old 09-28-2012, 02:35 PM   #13
twobob
( ͡° ͜ʖ ͡°){ʇlnɐɟ ƃǝs}Tır
twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.
 
twobob's Avatar
 
Posts: 6,552
Karma: 6021146
Join Date: Jun 2012
Location: uti gratia usura (Yao ying da ying; Mo ying da yieng)
Device: PW-WIFI|K5-3G+WIFI| K4|K3-3G|DXG|K2| Rooted Nook Touch
hmm. possibly do a bit of playing around with fiddler then. in firefox.
twobob is offline   Reply With Quote
Old 10-02-2012, 12:05 PM   #14
silver18
THE NOOB
silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.silver18 ought to be getting tired of karma fortunes by now.
 
silver18's Avatar
 
Posts: 698
Karma: 1545625
Join Date: Jan 2012
Location: Italy
Device: Kindle Touch 5.3.2
Ok, here I am...
The problem when "wgetting" a page from wikipedia is that you need to turn on recursive download as the normal "--page-requisites" option doesn't retrieve images (it gets only the html to be honest).
This is because images are hosted on a separate page!
BUT turning on "--recursive" gives you LOTS of things to download (as wiki pages are full of external & internal links).

So I ended up using --recursive + --span-hosts + --domains.
This is the complete command:
Code:
wget --recursive --span-hosts --domains=upload.wikimedia.org -e robots=off --random-wait --limit-rate=20K --page-requisites --no-parent --convert-links --adjust-extension --restrict-file-names=windows -U Mozilla -P "$folder" "$webpage"
Now images are retrived but I can't get the style sheet...
I'm going to investigate where the hell is it hosted...
silver18 is offline   Reply With Quote
Old 10-02-2012, 12:34 PM   #15
twobob
( ͡° ͜ʖ ͡°){ʇlnɐɟ ƃǝs}Tır
twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.twobob ought to be getting tired of karma fortunes by now.
 
twobob's Avatar
 
Posts: 6,552
Karma: 6021146
Join Date: Jun 2012
Location: uti gratia usura (Yao ying da ying; Mo ying da yieng)
Device: PW-WIFI|K5-3G+WIFI| K4|K3-3G|DXG|K2| Rooted Nook Touch
Quote:
<link rel="stylesheet" href="//bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&amp;lang=en&amp;modules=ext.g adget.ReferenceTooltips%2Ccharinsert%2Cteahouse%7C ext.wikihiero%7Cmediawiki.legacy.commonPrint%2Csha red%7Cskins.vector&amp;only=styles&amp;skin=vector &amp;*" />
<meta name="ResourceLoaderDynamicStyles" content="" />
<link rel="stylesheet" href="//bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&amp;lang=en&amp;modules=site& amp;only=styles&amp;skin=vector&amp;*" />
<style>a:lang(ar),a:lang(ckb),a:lang(fa),a:lang( kk-arab),a:lang(mzn),a:lang(ps),a:lang(ur){text-decoration:none}

/* cache key: enwiki:resourceloader:filter:minify-css:7:8d95de22da3b74bdc8517ef8752d1bee */
</style>

<script src="//bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&amp;lang=en&amp;modules=start up&amp;only=scripts&amp;skin=vector&amp;*"></script>
Looks to be a vaguely protected module loader. good luck!
twobob is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
K5 Could anyone compile full-feature wget for Touch wisecrick Kindle Developer's Corner 15 09-09-2012 11:41 AM
wget an ftp folder mephis PocketBook 3 10-11-2011 06:34 PM
902 wget pdf newspaper from site? niroma PocketBook 23 07-10-2011 04:09 PM
How to use wget to download an online HTML book amoroso Lounge 11 04-25-2011 06:10 AM
Recursive conversion of Websites to Kindle oecherprinte Amazon Kindle 0 10-18-2010 08:22 AM


All times are GMT -4. The time now is 12:28 AM.


MobileRead.com is a privately owned, operated and funded community.