|
No, it has always been present. I didn't bother to remove the next page sections, you should do that via postprocess_html, something like this
def postprocess_html(self, soup, first_fetch):
for div in soup.findAll(id='next_page_whatever'):
div.extract()
return soup
|