[XanthanGum]

Originally Posted by kiklop74

Updated recipe Ars technica with multipage news support

kiklop74,

Your latest revised Ars Technica recipe seems to be working fine. Thanks a million.

I guess this segment of your code is what fetches articles continued across multiple pages:

def append_page(self, soup, appendtag, position):
        pager = soup.find('div',attrs={'id':'pager'})
        if pager:           
           for atag in pager.findAll('a',href=True):
               str = self.tag_to_string(atag)
               if str.startswith('Next'):
                  soup2 = self.index_to_soup(atag['href'])
                  texttag = soup2.find('div', attrs={'class':'news-item-text'})
                  for it in texttag.findAll(style=True):
                      del it['style']
                  newpos = len(texttag.contents)          
                  self.append_page(soup2,texttag,newpos)
                  texttag.extract()
                  pager.extract()
                  appendtag.insert(position,texttag)

Again, thanks.

Xanthan Gum