Login subscription 'Aachener Zeitung'

[lucis_lupinum]

Hi!

I am currently trying to create a recipe for our local newspaper and I want to include a login to download all of the content (some of the articles are only fully viewable with a valid login.

So I tried to use the method for this login which I found in several forums, but I can't figure out how to use it properly

On the main page (Aachener Zeitung) there are two possibilities to log in.
You can either click the button on the top right or you can use the bottom bar.

I didn't find out how to adress one of these properly, so I tried another login page. On the mobile site AZ mobile the login fields are better visible and I tried to access them with the following code:

Spoiler:

def get_browser(self):
        br = BasicNewsRecipe.get_browser(self)
        if self.username is not None and self.password is not None:
            br.open('http://www.aachener-zeitung.de/mobile/extern/sso-templates/registrieren')
            br.select_form(nr=0)
            br['login_loginname'] = self.username
            br['login_password'] = self.password
            br.submit()
        return br

It seems to work, but it shows no effect. There are no errors, but the articles are still not visible in the downloaded content.

So maybe somebody can tell me what I have to correct in my code? Or what I have to do to use the 'normal' forms on the regular webpage?

[kovidgoyal]

It's quite possible that logging in with the mobile site will not allow you to fetch articles from the normal site, test that with a browser. And check the returned html after doing a submit, like this

html = br.submit().read()

The htmnl should contain something indicating you are succesfully looged in.

[lucis_lupinum]

Hi,

thanks a lot for your quick reply.

I already tried if the login on the mobile site works with the regular page. So I logged in and then just visited the main page and I can read the otherwise locked articles. The page also shows me that I am logged in with my account…
So I assumed this should work. This was on my normal browser (Chrome for desktop)

I tried to create a HTML as you told me, but I am pretty new to python and I am currently 'working' with it in the calibre interface for recipes. So I don't know where to find the HTML output (if its now even created within the news fetching process)

So what's the best way now for me to go on with this problem?

[kovidgoyal]

See http://manual.calibre-ebook.com/news...ng-new-recipes

and just do

print (html)

to get the html printed out on the console or if you want to save it to a file do

open('path_to_file.html', 'wb').write(html)

[lucis_lupinum]

Hmmmm okay, I did as you told me, but in the output file there is no hint that I am logged in.
The website looks the same like before and I can't find anything which gives me a clue that the login was succesful...

That makes sense of course since I don't get the articles... But I can't figure out where my mistake is. The selected form seems to be right, I don't get an error or something like this and there is also no form of an error message that the user is unknown or so.

I really have no idea now how to go on.

I already tried to use another login page which I found AZ mobile login 2 but this doesn't work either...

Any ideas left? :-S

[kovidgoyal]

The login probably requires javascript, in which case you have to use the javascript login technique, see for example the wall street journal recipe.

[lucis_lupinum]

I tried the javascript login, but it still doesn't seem to work

I put

use_javascript_to_login = True

in my code and also the procedure:

def javascript_login(self, browser, username, password):
	browser.visit('http://www.aachener-zeitung.de/')
	f = browser.select_form('form[name="login"]')
	f['login_loginname'] = username
	f['login_password'] = password
	browser.submit(timeout=120)

I thought I got it right, but it seems I am wrong.
I tried to adress the login box which appears when you click the button in the top-right corner (AZ)

I can't check if I am logged in because the submit().read() doesn't work here...

[kovidgoyal]

You use browser.html to get the html in this case.

[lucis_lupinum]

eeerh...
How should it look like then?

I just tried and can't figure out how to do it

[lucis_lupinum]

Okay, I got it.

Spoiler:

def javascript_login(self, browser, username, password):
	browser.visit('http://www.aachener-zeitung.de/')
	f = browser.select_form('form[class="small dark left"]')
	f['login_loginname'] = username
	f['login_password'] = password
	browser.submit(timeout=120)
	html = browser.html
	open(r'C:\test.html', 'wb').write(html)

And hooray - the output html-file indicates that I am successfully logged in. I got my username and there is also a logout-button.

But sadly in the finished e-book the articles still don't appear and there is also a message that i should login...
So it seems that the login works now, but the fetching browser still isn't logged in :-S

Do I have to manually pass the information (cookies?) to the other browser instance? I didn't think so till now, in the recipes I also didn't find a hint for that...

[kovidgoyal]

No, the cookies are copied automatically. If it isn't working it probably means the entire site relies on javascript, which means you should use a javascript recipe for it. See the builting recipe for time.com for an example.

[lucis_lupinum]

Oh nooo
It seems you are right. If I turn off javascript in my browser and visit content only available for subscribers the page is almost empty. All the other articles work properly...
So yeah - looks like this is gonna be a lot more work.
But thanks for the help already.
Sadly, I have actually no clue how and where to start now. I don't need all of the code in the Time recipe, right? For example since I am using feeds, I don't need the find_articles right? :-S


I am close to giving up now - this is pretty frustrating because I only have basic knowledge in python and I can't figure out now which parts I need to use and how my recipe should look like...

[kovidgoyal]

Javascript recipes are different than normal ones. You have to implement the get_publication_data() method. Which has to return an object like this:

{'index':[list of articles]
}


Each element of the list must be a 2-element tuple
of the form ``('feed title', list of articles)``.

Each list of articles must contain dictionaries of the form::

{
'title' : article title,
'url' : URL of print version,
'date' : The publication date of the article as a string,
'description' : A summary of the article
'content' : The full article (can be an empty string). Obsolete
do not use, instead save the content to a temporary
file and pass a file:///path/to/temp/file.html as
the URL.
}

Basically you have to get that data from the feed, the javascript recipe wont do that for you.

[lucis_lupinum]

pheew...

I don't know if I will be succcessful with that. I will in try in the next weeks, but I am afraid I will have to try out very much... :-S

Isn't there an easier way to handle this? Maybe by using another browser in the background or something like this?

I have to admit, I do not understand much of what you are posting :-/
So if anyone out there also has a login for the site and is willing to try out furthermore, I would appreciate it very much. But otherwise I don't see a realistic chance of getting it to work for me...