Idea for a "Bookit" Plugin -- Maybe Kovid?

[beowulf573]

1) To be honest I'm not 100% clear either, it has to do with how deep and how many links are followed to create the final file.

2) I see the same thing when executing web2lrf from the command line. I didn't see anything obviously unusual about the links.

[kovidgoyal]

max-recursions controls the number of levels of recursion when downloading and link-levels controls the number of elvels of recursion when converting from HTML to LRF

There are two options because there are actually two separate components under the hood that do the downloading and converting respectively

The wikipedia links show up fine for me

[kovidgoyal]

Originally Posted by beowulf573

2) I see the same thing when executing web2lrf from the command line. I didn't see anything obviously unusual about the links.

Actually, there was a regression introduced in 0.4.61 causing this. Fixed in 0.4.63

[alexxxm]

Originally Posted by kovidgoyal

Actually, there was a regression introduced in 0.4.61 causing this. Fixed in 0.4.63

perfect, I just checked and now the link text appears - as it should be.

Unfortunately the test did not start correctly the 1st time, since I still had in the preferences the values "max recursions=1, link levels=1" I used yesterday for testing ... the result was an lrf file more than 6MB big!

Speaking about it - you said:

Originally Posted by kovidgoyal

max-recursions controls the number of levels of recursion when downloading and link-levels controls the number of elvels of recursion when converting from HTML to LRF

I'm still not so sure I understand it - so let's try a typical test case: if I wanted to get an HTML page, plus all the others just 1 link away and no more, which values should I set?

Last point: one option I always find useful in webscrapers programs is the possibility to ask them to follow just links local to the starting website. This is still not possible in web2lrf, correct?


Thanks for the help

alessandro

[kovidgoyal]

For links just one level away -r 1 should do the trick. You can easily ask the scraper to follow only links of a certain type using the --match-regexp option

[alexxxm]

Originally Posted by kovidgoyal

For links just one level away -r 1 should do the trick. You can easily ask the scraper to follow only links of a certain type using the --match-regexp option

I'm still asking you here even thou I just discovered the other thread on "content" - I'll move there once I'm clarified with this:

I'm trying what you said, put in the bookit options "Max recursions=1", but I'm having trouble with regexps:
from the site http://www.cityguide.travel-guides.c...rope/Lyon.html

I wanted to follow all the internal links having "72" in the address:

I tried putting Meta-data>Additional parameters
"--match-regexp 72", "--match-regexp=72", "--match-regexp *72*", "--match-regexp=*72*", but none worked: it just saves the original page and that's all

any hint?

thanks...

alessandro

[kovidgoyal]

--match-regexp ".*72.*"

[alexxxm]

Originally Posted by kovidgoyal

--match-regexp ".*72.*"

thanks but not, still not working.
The executed line is:

/usr/bin/python /usr/bin/web2lrf -u http://www.cityguide.travel-guides.c...rope/Lyon.html -o /usr/src/bookit/Lyon City Guide _ Lyon City Break.lrf -t Lyon City Guide | Lyon City Break -a Bookit -r 1 --link-levels=0 --left-margin=0 --right-margin=0 --top-margin=0 --bottom-margin=0 --match-regexp .*72.* default

and it still does not follow any link (all those visible on the right of the page)

alessandro

[igorsk]

I think you need to escape the asterisks on command line.

[kovidgoyal]

Try .+72.+ to avoid the asterisk problem

[dsuden]

Beowulf, thanks again for Bookit! I'm trying it in the newest Firefox, Beta 3.1b2 and it looks as though something has changed that is disallowing the plugin.

BTW, Beta 3.1b2 is very cool...it allows mousepad gestures for navigation on Macbooks!

Dane