Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 10-22-2013, 04:12 PM   #1
lucis_lupinum
Member
lucis_lupinum began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Oct 2013
Device: Kindle
Login subscription 'Aachener Zeitung'

Hi!

I am currently trying to create a recipe for our local newspaper and I want to include a login to download all of the content (some of the articles are only fully viewable with a valid login.

So I tried to use the method for this login which I found in several forums, but I can't figure out how to use it properly

On the main page (Aachener Zeitung) there are two possibilities to log in.
You can either click the button on the top right or you can use the bottom bar.

I didn't find out how to adress one of these properly, so I tried another login page. On the mobile site AZ mobile the login fields are better visible and I tried to access them with the following code:

Spoiler:
Code:
def get_browser(self):
        br = BasicNewsRecipe.get_browser(self)
        if self.username is not None and self.password is not None:
            br.open('http://www.aachener-zeitung.de/mobile/extern/sso-templates/registrieren')
            br.select_form(nr=0)
            br['login_loginname'] = self.username
            br['login_password'] = self.password
            br.submit()
        return br


It seems to work, but it shows no effect. There are no errors, but the articles are still not visible in the downloaded content.

So maybe somebody can tell me what I have to correct in my code? Or what I have to do to use the 'normal' forms on the regular webpage?

lucis_lupinum is offline   Reply With Quote
Old 10-22-2013, 10:36 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,350
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
It's quite possible that logging in with the mobile site will not allow you to fetch articles from the normal site, test that with a browser. And check the returned html after doing a submit, like this

html = br.submit().read()

The htmnl should contain something indicating you are succesfully looged in.
kovidgoyal is offline   Reply With Quote
 
Advertisement
Old 10-23-2013, 03:33 AM   #3
lucis_lupinum
Member
lucis_lupinum began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Oct 2013
Device: Kindle
Hi,

thanks a lot for your quick reply.

I already tried if the login on the mobile site works with the regular page. So I logged in and then just visited the main page and I can read the otherwise locked articles. The page also shows me that I am logged in with my account…
So I assumed this should work. This was on my normal browser (Chrome for desktop)

I tried to create a HTML as you told me, but I am pretty new to python and I am currently 'working' with it in the calibre interface for recipes. So I don't know where to find the HTML output (if its now even created within the news fetching process)

So what's the best way now for me to go on with this problem?
lucis_lupinum is offline   Reply With Quote
Old 10-23-2013, 04:15 AM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,350
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
See http://manual.calibre-ebook.com/news...ng-new-recipes

and just do

print (html)

to get the html printed out on the console or if you want to save it to a file do

open('path_to_file.html', 'wb').write(html)
kovidgoyal is offline   Reply With Quote
Old 10-23-2013, 08:05 AM   #5
lucis_lupinum
Member
lucis_lupinum began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Oct 2013
Device: Kindle
Hmmmm okay, I did as you told me, but in the output file there is no hint that I am logged in.
The website looks the same like before and I can't find anything which gives me a clue that the login was succesful...

That makes sense of course since I don't get the articles... But I can't figure out where my mistake is. The selected form seems to be right, I don't get an error or something like this and there is also no form of an error message that the user is unknown or so.

I really have no idea now how to go on.

I already tried to use another login page which I found AZ mobile login 2 but this doesn't work either...

Any ideas left? :-S
lucis_lupinum is offline   Reply With Quote
Old 10-23-2013, 08:08 AM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,350
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The login probably requires javascript, in which case you have to use the javascript login technique, see for example the wall street journal recipe.
kovidgoyal is offline   Reply With Quote
Old 10-23-2013, 10:10 AM   #7
lucis_lupinum
Member
lucis_lupinum began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Oct 2013
Device: Kindle
I tried the javascript login, but it still doesn't seem to work

I put
Code:
use_javascript_to_login = True
in my code and also the procedure:

Code:
def javascript_login(self, browser, username, password):
	browser.visit('http://www.aachener-zeitung.de/')
	f = browser.select_form('form[name="login"]')
	f['login_loginname'] = username
	f['login_password'] = password
	browser.submit(timeout=120)
I thought I got it right, but it seems I am wrong.
I tried to adress the login box which appears when you click the button in the top-right corner (AZ)

I can't check if I am logged in because the submit().read() doesn't work here...
lucis_lupinum is offline   Reply With Quote
Old 10-23-2013, 10:18 AM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,350
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You use browser.html to get the html in this case.
kovidgoyal is offline   Reply With Quote
Old 10-23-2013, 10:33 AM   #9
lucis_lupinum
Member
lucis_lupinum began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Oct 2013
Device: Kindle
eeerh...
How should it look like then?

I just tried and can't figure out how to do it
lucis_lupinum is offline   Reply With Quote
Old 10-25-2013, 08:35 AM   #10
lucis_lupinum
Member
lucis_lupinum began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Oct 2013
Device: Kindle
Okay, I got it.

Spoiler:
Code:
def javascript_login(self, browser, username, password):
	browser.visit('http://www.aachener-zeitung.de/')
	f = browser.select_form('form[class="small dark left"]')
	f['login_loginname'] = username
	f['login_password'] = password
	browser.submit(timeout=120)
	html = browser.html
	open(r'C:\test.html', 'wb').write(html)


And hooray - the output html-file indicates that I am successfully logged in. I got my username and there is also a logout-button.

But sadly in the finished e-book the articles still don't appear and there is also a message that i should login...
So it seems that the login works now, but the fetching browser still isn't logged in :-S

Do I have to manually pass the information (cookies?) to the other browser instance? I didn't think so till now, in the recipes I also didn't find a hint for that...
lucis_lupinum is offline   Reply With Quote
Old 10-25-2013, 08:58 AM   #11
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,350
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No, the cookies are copied automatically. If it isn't working it probably means the entire site relies on javascript, which means you should use a javascript recipe for it. See the builting recipe for time.com for an example.
kovidgoyal is offline   Reply With Quote
Old 10-25-2013, 09:56 AM   #12
lucis_lupinum
Member
lucis_lupinum began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Oct 2013
Device: Kindle
Unhappy

Oh nooo
It seems you are right. If I turn off javascript in my browser and visit content only available for subscribers the page is almost empty. All the other articles work properly...
So yeah - looks like this is gonna be a lot more work.
But thanks for the help already.
Sadly, I have actually no clue how and where to start now. I don't need all of the code in the Time recipe, right? For example since I am using feeds, I don't need the find_articles right? :-S


I am close to giving up now - this is pretty frustrating because I only have basic knowledge in python and I can't figure out now which parts I need to use and how my recipe should look like...
lucis_lupinum is offline   Reply With Quote
Old 10-25-2013, 12:55 PM   #13
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,350
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Javascript recipes are different than normal ones. You have to implement the get_publication_data() method. Which has to return an object like this:

{'index':[list of articles]
}


Each element of the list must be a 2-element tuple
of the form ``('feed title', list of articles)``.

Each list of articles must contain dictionaries of the form::

{
'title' : article title,
'url' : URL of print version,
'date' : The publication date of the article as a string,
'description' : A summary of the article
'content' : The full article (can be an empty string). Obsolete
do not use, instead save the content to a temporary
file and pass a file:///path/to/temp/file.html as
the URL.
}

Basically you have to get that data from the feed, the javascript recipe wont do that for you.
kovidgoyal is offline   Reply With Quote
Old 10-28-2013, 07:12 AM   #14
lucis_lupinum
Member
lucis_lupinum began at the beginning.
 
Posts: 18
Karma: 10
Join Date: Oct 2013
Device: Kindle
pheew...

I don't know if I will be succcessful with that. I will in try in the next weeks, but I am afraid I will have to try out very much... :-S

Isn't there an easier way to handle this? Maybe by using another browser in the background or something like this?

I have to admit, I do not understand much of what you are posting :-/
So if anyone out there also has a login for the site and is willing to try out furthermore, I would appreciate it very much. But otherwise I don't see a realistic chance of getting it to work for me...

Last edited by lucis_lupinum; 10-28-2013 at 09:33 AM.
lucis_lupinum is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Neue Osnabrücker Zeitung Krittika Goyal Recipes 10 06-09-2013 11:31 AM
New recipe for "Süddeutsche Zeitung" using "E-Paper mobile" subscription Ernst Recipes 3 02-16-2013 08:37 AM
when need subscription, login success or fail? flyingfoxlee Recipes 5 01-03-2013 04:46 AM
recipe for Aachener Nachrichten - german schuster Recipes 4 11-30-2012 10:37 AM
German: Sueddeutsche Zeitung is broken kbaerwald Recipes 3 11-18-2010 06:57 AM


All times are GMT -4. The time now is 03:56 PM.


MobileRead.com is a privately owned, operated and funded community.