help on Heise CT recipe

[MasterPlexus]

Hi There,

the Heise CT recipe works so far, after I modified slightly the login part, as it was outdated.
Anyway, I could see that all Articles are ending after one page, and the rest of the Article is no longer there.
I could see that if I use a normal Browser, there is no Paging and no JavaScript used for the article text itself.
When I comment out the remove tags part I could see that the pages really looks like not completely downloaded.

Not sure what I can do. I tried to include browser_type = 'webengine' to check if may the bot check is blocking it, but including this parameter I get the following error:
AttributeError: 'WebEngineBrowser' object has no attribute 'select_form'
Any help would be appreciated,
MP

[MasterPlexus]

OK, I could see now, that it looks like the DOM is may not ready in webengine or qt? did I miss here something for the usage when it is not mechanize?

def get_browser(self):
br = BasicNewsRecipe.get_browser(self)
if self.username is not None and self.password is not None:
loginURL = 'https://www.heise.de/sso/login?forward=%2Fselect'
br.open(loginURL)
br.select_form(action='/sso/login/login/nojs')
br['username'] = self.username
br['password'] = self.password
br.submit()

return br

[MasterPlexus]

OK, I made an update to 7.20 from 7.19, and now the pages are fully downloaded.

Anyway, I would like to understand still why other browser_types are not working. so any help on this would be appreciated.

[kovidgoyal]

That's because I never got around to implementing select_form() for the non mechanize browsers.

[MasterPlexus]

Is there a way, to stop deleting the created Temporary files? If so I could check what's the content and to check may the reason, or to build may a other DOM access for it?

[kovidgoyal]

Not sure what you are asking. If you want to read the html you canalways do so from the browser object.

[MasterPlexus]

During the process several Files will be created in Temp folder. But the files are deleted at the end of the process (only some final files will stay as epub). When there would be a switch not to delete them could help what is happening for all (also during the cleanup for example).

[kovidgoyal]

As I said read the html you want from the browser object and save it wherever you like on the filesystem.

[MasterPlexus]

Originally Posted by kovidgoyal

As I said read the html you want from the browser object and save it wherever you like on the filesystem.

ok, will give it a try when I will go deeper. thanks.

[TheGreen_1]

Originally Posted by MasterPlexus

ok, will give it a try when I will go deeper. thanks.

@MasterPlexus: Have you been able to get this recipe working? I'm currently struggling getting things running since I'm new with calibre...