Wget & recursive download - Page 2

twobob · 10-02-2012, 11:43 AM

Hint. just write a replacement using firebug exports (or something) and hard-reference that.

silver18 · 10-02-2012, 12:26 PM

Ok, it was easier than I thought...I just added bits.wikimedia.org to the --domains option!!
Now I can retrieve wiki pages...

I wrote a script to download pages based on the browser's bookmarks named OFFLINE and a WAF to display them (landing page with dropdown menu+content window).

It's nice, but if a user must add domains to a whitelist if he wants to download a page similar to wikipedia (images hosted somewhere else).

I can provide a small whitelist based on the pages I find having problems with, but I can't provide a comprehensive list.
So, this needs a *bit* of user interaction.

twobob · 10-02-2012, 12:48 PM

hmm... yeah. possibly so. check out if wildcards work

or dynamically override the check perhaps (might be possible via a deep java hack / class override)

create a service that lives on one URL (localhost??) that can unroll the requests to other urls.

use a json solution to get other URL data and fiddle with the security of that.

offload the requests to other URLs to something else like curl? something external to whatever scope is limiting your request.

the list goes on...

You get the idea

(gimme a worked "for example" of an item you want to get, for better input)

knc1 · 10-02-2012, 02:02 PM

Quote:

Originally Posted by silver18

I can provide a small whitelist based on the pages I find having problems with, but I can't provide a comprehensive list.
So, this needs a *bit* of user interaction.

What? No comprehensive list?

http://www.isc.org/solutions/survey
Couldn't you at least include the oldest 900 million sites?

Oh, wait, there are at least three or four characters in the TLD (counting the period) plus the line ending character - -
5 * 900 million == more bytes than the Kindle has for storage, before adding the domain name to the fixed TLD strings.

twobob · 10-02-2012, 02:48 PM

Yup. I smell "workaround" written all over it.

ppoo · 10-03-2012, 12:44 AM

We can also use axel to, it's lighter and faster. Download here (tested on KT, but it should run well on K3...).

knc1 · 10-03-2012, 01:15 AM

Quote:

Originally Posted by ppoo

We can also use axel to, it's lighter and faster. Download here (tested on KT, but it should run well on K3...).

The link you posted is a direct link to an ELF binary.

Blindly "click and load" is not the way development is done (at least not here).

Is this the utility you are suggesting?
http://www.cyberciti.biz/tips/downlo...ine-tools.html

Of note is the option to limit the number of concurrent connections, something the end-user needs to adjust to match the site limits (even when doing http).

The http server almost always has a limiter on the number of concurrent connections.

10-02-2012, 12:48 PM	#18
twobob ( ͡° ͜ʖ ͡°){ʇlnɐɟ ƃǝs}Týr Posts: 6,586 Karma: 6299991 Join Date: Jun 2012 Location: uti gratia usura (Yao ying da ying; Mo ying da yieng) Device: PW-WIFI\|K5-3G+WIFI\| K4\|K3-3G\|DXG\|K2\| Rooted Nook Touch	hmm... yeah. possibly so. check out if wildcards work or dynamically override the check perhaps (might be possible via a deep java hack / class override) create a service that lives on one URL (localhost??) that can unroll the requests to other urls. use a json solution to get other URL data and fiddle with the security of that. offload the requests to other URLs to something else like curl? something external to whatever scope is limiting your request. the list goes on... You get the idea (gimme a worked "for example" of an item you want to get, for better input) Last edited by twobob; 10-02-2012 at 02:50 PM. Reason: ideas

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Could anyone compile full-feature wget for Touch	wisecrick	Kindle Developer's Corner	15	09-09-2012 10:41 AM
wget an ftp folder	mephis	PocketBook	3	10-11-2011 05:34 PM
902 wget pdf newspaper from site?	niroma	PocketBook	23	07-10-2011 03:09 PM
How to use wget to download an online HTML book	amoroso	Lounge	11	04-25-2011 05:10 AM
Recursive conversion of Websites to Kindle	oecherprinte	Amazon Kindle	0	10-18-2010 07:22 AM

10-02-2012, 11:43 AM	#16
twobob ( ͡° ͜ʖ ͡°){ʇlnɐɟ ƃǝs}Týr Posts: 6,586 Karma: 6299991 Join Date: Jun 2012 Location: uti gratia usura (Yao ying da ying; Mo ying da yieng) Device: PW-WIFI\|K5-3G+WIFI\| K4\|K3-3G\|DXG\|K2\| Rooted Nook Touch	Hint. just write a replacement using firebug exports (or something) and hard-reference that.

10-02-2012, 12:26 PM	#17
silver18 THE NOOB Posts: 701 Karma: 1545649 Join Date: Jan 2012 Location: Italy Device: Kindle Touch 5.3.2	Ok, it was easier than I thought...I just added bits.wikimedia.org to the --domains option!! Now I can retrieve wiki pages... I wrote a script to download pages based on the browser's bookmarks named OFFLINE and a WAF to display them (landing page with dropdown menu+content window). It's nice, but if a user must add domains to a whitelist if he wants to download a page similar to wikipedia (images hosted somewhere else). I can provide a small whitelist based on the pages I find having problems with, but I can't provide a comprehensive list. So, this needs a bit of user interaction.

10-02-2012, 02:48 PM	#20
twobob ( ͡° ͜ʖ ͡°){ʇlnɐɟ ƃǝs}Týr Posts: 6,586 Karma: 6299991 Join Date: Jun 2012 Location: uti gratia usura (Yao ying da ying; Mo ying da yieng) Device: PW-WIFI\|K5-3G+WIFI\| K4\|K3-3G\|DXG\|K2\| Rooted Nook Touch	Yup. I smell "workaround" written all over it.

10-03-2012, 12:44 AM	#21
ppoo Connoisseur Posts: 56 Karma: 40393 Join Date: May 2012 Device: Kindle Touch	We can also use axel to, it's lighter and faster. Download here (tested on KT, but it should run well on K3...).