captureing all articles but all under the first section

[Steven630]

Here is how the webpage looks like

<div class='module'>

<h3>Section1</h3>

<ul>
<li>
<h4>articles links and article titles</h4>
</li>
……(other articles)
</ul>

<h3>Section2</h3>

<ul>
<li>
<h4>articles links and article titles</h4>
</li>
…… (other articles)
</ul>

So I tried to parse it, setting 'module' as the section.:

But it turned out that though all the articles were fetched correctly, they all end up in the first section.But I am at a loss what to do, since all section names are included in "h3", and unlike webpage of built-in recipes, <div class='module'> appears only before the first section, not every section (which I think explains the failure). Can anyone help me out? Just a quick answer is appreciated.

[Steven630]

I think it has something to do with section.find('h3') this only find the first one. But when I changed this to section.findAll('h3') , it failed to fetch anything (not even the first section).

[Steven630]

Just edited the thread starter to reflect a string I didn't mention.

[Steven630]

I changed the recipe and made every single "ul" as a section, leaving out section title. This time, Calibre correctly divided articles into several sections. But the drawback is that each section has no name. I'm thinking about grabbing h3 and add those to section names. Is that possible? Or can I just enter the section name in the recipe one by one?

[kovidgoyal]

When you find a ul use ul.findPreviousSibling('h3') to get the title. Read the beautifulsoup documentation, it will teach you how to do this, and many other things.

[Steven630]

Originally Posted by kovidgoyal

When you find a ul use ul.findPreviousSibling('h3') to get the title. Read the beautifulsoup documentation, it will teach you how to do this, and many other things.

Thank you so much! I'm been looking for something like that.