Request for generic 'multiple pages' code

fw190a8 · 11-08-2011, 04:57 PM

Being a programmer but a bit new to python and the world of feed parsing, could anybody help out with feeds that lead to stories with multiple pages? I am looking at the PC Gamer site first of all, to give you some context, and this post in particular is an example of one that has multiple pages.

Not all posts have multiple pages, and those that do, simply have 'next' buttons, rather than a full pager, so it's impossible to say from the first page how many pages are in a given article. It would be a case of saying: while (there's a 'next' button on the page) keep appending the page content and getting the next page - unfortunately my python's not good enough.

I'd like to ask for some help for a generic solution so I can adapt this for other purposes. I'll post any recipes I create using this, in case others are interested. Thanks in advance!

kovidgoyal · 11-08-2011, 09:35 PM

You use match_regexps to match only the next page link and set recursions to some large number, say 10.

Starson17 · 11-09-2011, 09:22 AM

Quote:

Originally Posted by fw190a8

Being a programmer but a bit new to python and the world of feed parsing, could anybody help out with feeds that lead to stories with multiple pages? I am looking at the PC Gamer site first of all, to give you some context, and this post in particular is an example of one that has multiple pages.

Not all posts have multiple pages, and those that do, simply have 'next' buttons, rather than a full pager, so it's impossible to say from the first page how many pages are in a given article. It would be a case of saying: while (there's a 'next' button on the page) keep appending the page content and getting the next page - unfortunately my python's not good enough.

I'd like to ask for some help for a generic solution so I can adapt this for other purposes. I'll post any recipes I create using this, in case others are interested. Thanks in advance!

It's been written many times. Kovid's explained how it works. See the Adventure Gamers recipe that's builtin for a sample recipe. Search for "multipage" in this forum for comments on it and other examples.

Also, see here.

11-08-2011, 04:57 PM	#1
fw190a8 Junior Member Posts: 7 Karma: 10 Join Date: Aug 2011 Device: Kindle 3	Request for generic 'multiple pages' code Being a programmer but a bit new to python and the world of feed parsing, could anybody help out with feeds that lead to stories with multiple pages? I am looking at the PC Gamer site first of all, to give you some context, and this post in particular is an example of one that has multiple pages. Not all posts have multiple pages, and those that do, simply have 'next' buttons, rather than a full pager, so it's impossible to say from the first page how many pages are in a given article. It would be a case of saying: while (there's a 'next' button on the page) keep appending the page content and getting the next page - unfortunately my python's not good enough. I'd like to ask for some help for a generic solution so I can adapt this for other purposes. I'll post any recipes I create using this, in case others are interested. Thanks in advance!

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Sidebars Across Multiple Pages	Rand Brittain	Sigil	0	01-08-2011 12:11 AM
Skipping multiple pages on Kindle DX	brianpc	Amazon Kindle	8	12-13-2010 12:13 AM
Multiple Cover Pages	Desertway	Calibre	0	09-12-2010 03:16 PM
Creating eBooks from multiple web pages	tichkiir	Workshop	1	10-08-2007 06:16 PM

11-08-2011, 09:35 PM	#2
kovidgoyal creator of calibre Posts: 45,194 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You use match_regexps to match only the next page link and set recursions to some large number, say 10.

Advert