More than one page of TOC?

[Steven630]

INDEX = 'http://....'
soup = self.index_to_soup(self.INDEX)

What if I have more than one page of index?

[kovidgoyal]

soup2 = self.index_to_soup(INDEX2)

[Steven630]

Originally Posted by kovidgoyal

soup2 = self.index_to_soup(INDEX2)

Thank you. Do I need another def parse_index(self)?

[kovidgoyal]

no .

[Steven630]

Originally Posted by kovidgoyal

no .

I'm still a bit confused.

    def parse_index(self):
        articles = []
        soup = self.index_to_soup(self.INDEX)
        feeds = []
        for section in soup.findAll('div'):
             ...
            for post in section.findAll('li'):
            ...
            if articles:
                feeds.append((section_title, articles))

        return feeds

Do I have to use two "for section in"? (for section in soup.findAll and for section in soup2.findAll?)

UPDATE: I tried two for loops and got all the articles. But articles under the same section name was split.

For example, if in both page 1 and page 2 there's a section called "ABC", then two sections with the name "ABC" appear in TOC.

How can I avoid that?

[kovidgoyal]

Append the articles from the second section into the first.