Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 06-23-2010, 01:24 PM   #16
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,775
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You'll have to track down the original author of the recipe and ask for a fix. (davec and rollercoaster)
kovidgoyal is offline   Reply With Quote
Old 06-24-2010, 11:05 AM   #17
meso123
Member
meso123 began at the beginning.
 
Posts: 13
Karma: 10
Join Date: Apr 2009
Device: kindle
Google reader does not seem to work for me now in calibre 7.4. Is this a problem for other people also? Thanks.
P.S. I love calibre. Thank you so much.
meso123 is offline   Reply With Quote
Advert
Old 07-03-2010, 02:07 PM   #18
depend
Junior Member
depend began at the beginning.
 
Posts: 1
Karma: 10
Join Date: Jul 2010
Device: SONY PRS-900BC
I start getting the same error too. It was good on 6/20. I don't remember exactly which version I was using since there have been quite a few new versions published.
depend is offline   Reply With Quote
Old 07-03-2010, 07:58 PM   #19
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by depend View Post
I start getting the same error too. It was good on 6/20. I don't remember exactly which version I was using since there have been quite a few new versions published.
Version does not matter since the recipe never changed. Apparently something in Google Reader changed and the recipe needs to be fixed to match.

This post entitled "Changes to sending authenticated requests to Google Reader" on the Google Reader Blog might hold the key but I don't have the skill set needed to correct the problem.
DoctorOhh is offline   Reply With Quote
Old 07-04-2010, 09:37 AM   #20
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by dwanthny View Post
This post entitled "Changes to sending authenticated requests to Google Reader" on the Google Reader Blog might hold the key but I don't have the skill set needed to correct the problem.
Looking at the code in the first post of this thread convinces me that your link does hold the key. The current recipe finds and uses the SID cookie. Google stopped using SID cookie authentication and now wants an AUTH header. I'd ask for help in the dedicated recipe thread and make sure you include the link above. I don't use GoogleReader and I've only played around with adding headers and mechanize once.
Starson17 is offline   Reply With Quote
Advert
Old 07-09-2010, 12:53 AM   #21
rollercoaster
Zealot
rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.
 
rollercoaster's Avatar
 
Posts: 126
Karma: 1826
Join Date: Jan 2010
Device: Kindle 2
hi all, sorry for replying late but I was on vacation.

I can figure out how to fix the recipe but my problem is I dont know python. The last change I did was a minor string.replace(...) that is usually same on all languages.

I could write Pseudocode if someone is willing to convert it to python. (Like in post 13 of this thread)
rollercoaster is offline   Reply With Quote
Old 07-09-2010, 09:35 AM   #22
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by rollercoaster View Post
hi all, sorry for replying late but I was on vacation.

I can figure out how to fix the recipe but my problem is I dont know python. The last change I did was a minor string.replace(...) that is usually same on all languages.

I could write Pseudocode if someone is willing to convert it to python. (Like in post 13 of this thread)
Why don't you post some pseudocode. I've offered to help over in the custom recipe thread, but it will be a while 'til I get to it. (my primary development computer is down)

This isn't a Python issue as much as it is a mechanize issue and knowing where to get the authorization data that Google needs for the header to be built in mechanize. I've used mechanize a few times and am comfortable in basic Python. I suggest we stay in this thread, and if anyone offers help over in the custom recipe thread, we can steer them here.

Last edited by Starson17; 07-12-2010 at 10:42 AM.
Starson17 is offline   Reply With Quote
Old 07-09-2010, 05:11 PM   #23
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Brief update -

I've got the recipe retrieving the Auth code needed, instead of the SID. I've got to head out, so it will still be a while, but if all goes well, it's just a matter of building the required header with the auth code. I'll keep you posted.
Starson17 is offline   Reply With Quote
Old 07-10-2010, 01:36 PM   #24
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Now I need some help from someone who knows something about Google Reader.

Step 1 was to retrieve the auth code. I needed to change the request a bit from the old recipe, but that was successful.

Step 2 was to use the auth code and add it in as a header to each request. That was a pain. I got the header in OK, but it wasn't authorizing. For one thing, the API specs as to the exact format weren't very clear and for another, the mechanize docs weren't very clear, and for a third, unless you ask the right way, the auth code you get in step 1 is wrong. Usually, I can watch the headers when I retrieve from a website, then just do the same thing, but this is a program interface, not a browser interface, and I don't have anything working in the browser that I can copy.

Nonetheless, I was able to get it authenticating by changing the original request and copying the header format I found in a blog.

At that point, I expected success, and I suppose I may have it, but there's no content

I can retrieve an authenticated page at:

http://www.google.com/reader/api/0/tag/list

That's where the recipe gets the feeds.

The recipe then builds URLs for a feed list. Here's some feeds that it built:

(u'National Review Blogs', 'http://www.google.com/reader/atom/user/18271557450980043868/label/National%20Review%20Blogs?n=50&xt=user/-/state/com.google/read'),
(u'North Carolina', 'http://www.google.com/reader/atom/user/18271557450980043868/label/North%20Carolina?n=50&xt=user/-/state/com.google/read'),
(u'Watts Up With That', 'http://www.google.com/reader/atom/user/18271557450980043868/label/Watts%20Up%20With%20That?n=50&xt=user/-/state/com.google/read'),

However, these don't seem to have any content. I have no idea why the recipe builds these particular URLs, and since I'm dealing with a programmatic access to the Google Reader, not a browser access, I can't copy a successful browser session. Perhaps the recipe should build some other URLs for its feeds from the contents of the page at http://www.google.com/reader/api/0/tag/list, but if so, I don't know what it should build? Maybe my test account needs to have articles tagged or marked in some special way to show up at these locations?

Alternatively, maybe I need some special cookies for the content to show up?

It would probably be a heck of a lot easier for me to just rewrite the whole dang thing from scratch trying to base it on FireFox browser access instead of this API.

If anyone knows the URL that the recipe should be building for each feed, let me know. If you want to test further, replace the get_browser in the recipe with this:

Code:
    def get_browser(self):
        br = BasicNewsRecipe.get_browser(self)
        # Print HTTP headers.
        br.set_debug_http(True)
        orig_open_novisit = br.open_novisit
        if self.username is not None and self.password is not None:
            request = urllib.urlencode([('Email', self.username), ('Passwd', self.password),
                                        ('service', 'reader'), ('accountType', 'HOSTED_OR_GOOGLE'), ('source', __appname__)])
            response = br.open('https://www.google.com/accounts/ClientLogin', request)
            auth = re.search('Auth=(\S*)', response.read()).group(1)
            def my_open_no_visit(url, **kwargs):
                req = mechanize.Request(
                    url,
                    headers = {
                        'Authorization':'GoogleLogin auth='+auth,
                        })
                return orig_open_novisit(req)
        br.open_novisit = my_open_no_visit
        return br
For all I know, it may start working, depending on how your account is set up. Alternatively, Google may have changed the way access is made, although the blog seemed to say it hasn't.
Starson17 is offline   Reply With Quote
Old 07-11-2010, 05:11 AM   #25
rollercoaster
Zealot
rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.
 
rollercoaster's Avatar
 
Posts: 126
Karma: 1826
Join Date: Jan 2010
Device: Kindle 2
@Starson17 Your work helped me a lot and I think I know how to fix this but I still need your help to write python code. You are on the spot with fetching the AUTH token.

The issue is after the " def get_feeds(self): " method. We get the various feeds here and then each feed is queried to get articles/items under them. The request to those feeds DOES NOT pass the AUTH token as a header. And so, if you look at the log in jobs list then you will see that, google redirects these requests with a "302 Moved Temporarily" and sends to the login page. So we never get to the list of items in the feed.

Code:
send: 'POST /accounts/ClientLogin HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 97\r\nHost: www.google.com\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection: close\r\nUser-Agent: Mozilla/5.0 (X11; U; i686 Linux; en_US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4\r\n\r\n'
send: 'Email=someone&Passwd=someone&service=reader&accountType=HOSTED_OR_GOOGLE&source=calibre'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: text/plain
header: Cache-control: no-cache, no-store
header: Pragma: no-cache
header: Expires: Mon, 01-Jan-1990 00:00:00 GMT
header: Date: Sun, 11 Jul 2010 08:50:30 GMT
header: X-Content-Type-Options: nosniff
header: X-XSS-Protection: 1; mode=block
header: Content-Length: 733
header: Server: GSE
header: Connection: close
send: 'GET /reader/api/0/tag/list HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.google.com\r\nConnection: close\r\nAuthorization: GoogleLogin auth=DQAAAKIAAABiWf2ZNf7xjpkr97aUJusKEEnI9ClihCCQMtpu3NrqbJRAOzU-ix4K42nnKcAjsQaREfV0XrS-ISsZRs6YrMoH_gBB3M0rUD2thscZHmkvBxZP0S07gvwthEaruqD-j2hAwIKTXWh4csgdmhzjjvgcP6dSlJPqnzMVuxgC2J-t-EyFNDY4W8Ihi-ybimjip_pITYMP7vSuPCyJWOFCrQzM1ES8Ewmp_nn1LU4-_uihxw\r\nUser-Agent: Mozilla/5.0 (X11; U; i686 Linux; en_US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: text/xml; charset=UTF-8
header: X-Reader-Google-Version: 506-000
header: X-Reader-User: 10347379608328268864
header: Set-Cookie: GRLD=UNSET:10347379608328268864;Path=/reader/;HttpOnly
header: Date: Sun, 11 Jul 2010 08:50:30 GMT
header: Expires: Sun, 11 Jul 2010 08:50:30 GMT
header: Cache-Control: private, max-age=0
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Connection: close
send: 'GET /reader/atom/user/10347379608328268864/state/com.google/starred?n=250&xt=user/-/state/com.google/read HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.google.com\r\nCookie: GRLD=UNSET:10347379608328268864\r\nConnection: close\r\nUser-Agent: Mozilla/5.0 (X11; U; i686 Linux; en_US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4\r\n\r\n'
reply: 'HTTP/1.1 302 Moved Temporarily\r\n'
header: Content-Type: text/html; charset=UTF-8
header: X-Reader-Google-Version: 506-000
header: Location: https://www.google.com/accounts/ServiceLogin?service=reader&passive=1209600&continue=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Fstarred%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread&followup=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Fstarred%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread
header: Date: Sun, 11 Jul 2010 08:50:31 GMT
header: Expires: Sun, 11 Jul 2010 08:50:31 GMT
header: Cache-Control: private, max-age=0
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: Content-Length: 1381
header: Server: GSE
header: Connection: close
send: 'GET /accounts/ServiceLogin?service=reader&passive=1209600&continue=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Fstarred%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread&followup=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Fstarred%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.google.com\r\nConnection: close\r\nUser-Agent: Mozilla/5.0 (X11; U; i686 Linux; en_US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: text/html; charset=UTF-8
header: Cache-control: no-cache, no-store
header: Pragma: no-cache
header: Expires: Mon, 01-Jan-1990 00:00:00 GMT
header: Set-Cookie: GALX=T4Ka0MjTUcE;Path=/accounts;Secure
header: Date: Sun, 11 Jul 2010 08:50:32 GMT
header: X-Content-Type-Options: nosniff
header: X-XSS-Protection: 1; mode=block
header: Content-Length: 13863
header: Server: GSE
header: Connection: close
send: 'GET /reader/atom/user/10347379608328268864/state/com.google/reading-list?n=250&xt=user/-/state/com.google/read HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.google.com\r\nCookie: GRLD=UNSET:10347379608328268864\r\nConnection: close\r\nUser-Agent: Mozilla/5.0 (X11; U; i686 Linux; en_US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4\r\n\r\n'
reply: 'HTTP/1.1 302 Moved Temporarily\r\n'
header: Content-Type: text/html; charset=UTF-8
header: X-Reader-Google-Version: 506-000
header: Location: https://www.google.com/accounts/ServiceLogin?service=reader&passive=1209600&continue=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Freading-list%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread&followup=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Freading-list%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread
header: Date: Sun, 11 Jul 2010 08:50:33 GMT
header: Expires: Sun, 11 Jul 2010 08:50:33 GMT
header: Cache-Control: private, max-age=0
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: Content-Length: 1381
header: Server: GSE
header: Connection: close
send: 'GET /accounts/ServiceLogin?service=reader&passive=1209600&continue=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Freading-list%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread&followup=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Freading-list%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.google.com\r\nCookie: GALX=T4Ka0MjTUcE\r\nConnection: close\r\nUser-Agent: Mozilla/5.0 (X11; U; i686 Linux; en_US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: text/html; charset=UTF-8
header: Cache-control: no-cache, no-store
header: Pragma: no-cache
header: Expires: Mon, 01-Jan-1990 00:00:00 GMT
header: Set-Cookie: GALX=T4Ka0MjTUcE;Path=/accounts;Secure
header: Date: Sun, 11 Jul 2010 08:50:34 GMT
header: X-Content-Type-Options: nosniff
header: X-XSS-Protection: 1; mode=block
header: Content-Length: 13901
header: Server: GSE
header: Connection: close
I was using trial n error to write the code but there is too little documentation for both mechanize and calibre recipes. It took me an hour to find out that I can write to the log by using " PRINT <value> "

Anyway, have a look at this. I tried to get it working but could not. Language barrier I guess
It this doesnt/cant work for some reason then, conceivably, we can fill the login form with the username / pass each time and then submit and read the list of items. but that is the long way around.

So in summary, we need get the individual feed requests after "get_feeds(self)" to include the AUTH token and the recipe should hum along beautifully.

Last edited by rollercoaster; 07-11-2010 at 05:26 AM.
rollercoaster is offline   Reply With Quote
Old 07-11-2010, 09:28 AM   #26
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by rollercoaster View Post
@Starson17 Your work helped me a lot and I think I know how to fix this but I still need your help to write python code. You are on the spot with fetching the AUTH token.
Yes, until the auth token was sent correctly, we couldn't get the list of feeds from http://www.google.com/reader/api/0/tag/list

Obviously, that link doesn't contain the user info. As soon as I fixed the auth header, that link worked, so I thought I was sending the header correctly for all links. Stupid me, I didn't check. (funny thing - I left in the print headers code so you could see I was sending the auth code and you used it to see that I wasn't sending it - at least not for the article fetch. )

Quote:
The issue is after the " def get_feeds(self): " method. We get the various feeds here and then each feed is queried to get articles/items under them. The request to those feeds DOES NOT pass the AUTH token as a header. And so, if you look at the log in jobs list then you will see that, google redirects these requests with a "302 Moved Temporarily" and sends to the login page. So we never get to the list of items in the feed.
Yes, I saw I was getting redirects, but didn't realize I wasn't sending the header. I used the same method I've used before, and it was working on the links page. It didn't work on the links page until I got the header format corrected, so I assumed it was working elsewhere to send that header.

Quote:
Anyway, have a look at this. I tried to get it working but could not. Language barrier I guess
It this doesnt/cant work for some reason then, conceivably, we can fill the login form with the username / pass each time and then submit and read the list of items. but that is the long way around.
I'd seen this link, but since I thought I was sending the header correctly, I never tried this method of adding headers. As soon as I did, it worked to pull all the articles. Here's the fixed recipe:

Spoiler:
Code:
import urllib, re, mechanize
from calibre.web.feeds.recipes import BasicNewsRecipe
from calibre import __appname__

class GoogleReader(BasicNewsRecipe):
    title   = 'Google Reader'
    description = 'Feeds from your Google Reader account.'
    needs_subscription = True
    __author__ = 'davec, rollercoaster, Starson17'
    base_url = 'http://www.google.com/reader/atom/'
    max_articles_per_feed = 50
    get_options = '?n=%d&xt=user/-/state/com.google/read' % max_articles_per_feed
    use_embedded_content = True

    def get_browser(self):
        br = BasicNewsRecipe.get_browser(self)
        if self.username is not None and self.password is not None:
            request = urllib.urlencode([('Email', self.username), ('Passwd', self.password),
                                        ('service', 'reader'), ('accountType', 'HOSTED_OR_GOOGLE'), ('source', __appname__)])
            response = br.open('https://www.google.com/accounts/ClientLogin', request)
            auth = re.search('Auth=(\S*)', response.read()).group(1)
            print 'Auth is: ', auth
            cookies = mechanize.CookieJar()
            br = mechanize.build_opener(mechanize.HTTPCookieProcessor(cookies))
            br.addheaders = [('Authorization', 'GoogleLogin auth='+auth)]
        return br

    def get_feeds(self):
        feeds = []
        soup = self.index_to_soup('http://www.google.com/reader/api/0/tag/list')
        for id in soup.findAll(True, attrs={'name':['id']}):
            url = id.contents[0]
            feeds.append((re.search('/([^/]*)$', url).group(1),
                          self.base_url + urllib.quote(url.encode('utf-8')) + self.get_options))
        return feeds

Test it, and if it also works for you, pass it to Kovid. You may also want to modify and test the Uber version. You shouldn't have any trouble, but if you do, let me know.

Last edited by Starson17; 07-12-2010 at 10:45 AM.
Starson17 is offline   Reply With Quote
Old 07-13-2010, 06:32 AM   #27
rollercoaster
Zealot
rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.
 
rollercoaster's Avatar
 
Posts: 126
Karma: 1826
Join Date: Jan 2010
Device: Kindle 2
It works great! Tested with various settings and about 110 articles.

Changed oldest_article = 365 and max_articles_per_feed = 250. Also, replaced broadcast feed with reading-list feed. reading-list is the one that contains articles not in other feeds i.e uncategorized.

Also I think we should remove the uber version and just update the 'Google reader' recipe.

Updated recipe: See first post.

Last edited by rollercoaster; 07-13-2010 at 06:44 AM.
rollercoaster is offline   Reply With Quote
Old 07-13-2010, 08:08 AM   #28
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by rollercoaster View Post
It works great! Tested with various settings and about 110 articles.

Changed oldest_article = 365 and max_articles_per_feed = 250. Also, replaced broadcast feed with reading-list feed. reading-list is the one that contains articles not in other feeds i.e uncategorized.

Also I think we should remove the uber version and just update the 'Google reader' recipe.

Updated recipe: See first post.
First a big two thumbs up for getting this working.

Now the reason the two recipes should not be combined.

The reason I used the Googlereader and not Googlereaderuber is because Googlereader recipe doesn't have the reading list. Read this post for details. The bottom line is because I have my subscriptions in folders this recipe (first post) now downloads all my articles twice and I end up with a book that has about twice as many pages as a result.

Update: The attached images are the exact same articles downloaded today with the two different recipes. Even though the second has about 33% more pages the actual size wasn't much larger (7.9M vs 8.5M) so I guess I'm not locked into having two recipes.

One recipe could be fine but if someone thinks they will be able to easily navigate through the mess that is the Reading List with "uncategorized" articles and all of the rest of the "categorized articles"... well it is practically useless.

Of course until they learn how to place their feeds in folders at least they will be reaching their goal of grabbing articles using Google Reader.
Attached Thumbnails
Click image for larger version

Name:	Googlereader_without_reading_list.jpg
Views:	243
Size:	74.2 KB
ID:	55114   Click image for larger version

Name:	Googlereader_with_reading_list.jpg
Views:	221
Size:	79.5 KB
ID:	55115  

Last edited by DoctorOhh; 07-13-2010 at 08:47 AM.
DoctorOhh is offline   Reply With Quote
Old 07-13-2010, 08:57 AM   #29
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,775
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Let me know when you've settled on the final recipe(s) and I'll update them in calibre.
kovidgoyal is offline   Reply With Quote
Old 07-13-2010, 10:19 AM   #30
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by kovidgoyal View Post
Let me know when you've settled on the final recipe(s) and I'll update them in calibre.
Since I don't use either of them, I'll rely on those who do to finalize them. Dwanthny's comments seem like a fine reason to keep both. I'll be happy to help move the new authentication code over to the uber version, but it's trivial to do that - just replace the old with the new and leave the other parts of the recipe untouched.

BTW, during the step of obtaining the auth code with the user/pass, I added the data: 'accountType', 'HOSTED_OR_GOOGLE'. That was recommended in the API and it worked. However, I did see a post from someone who had trouble with that and had to use 'accountType', 'GOOGLE' (he was not fetching from GoogleReader).

I didn't go deeply enough into the API to find out if it could cause a problem in the recipe. It may even work without that data (I believe it may default to 'HOSTED_OR_GOOGLE' if not specified). Apparently there are various kinds of accounts - but it wasn't something I wanted to delve into deeply. If someone else does, they may wish to change this to specify the accountType. I mention it only to make a record of it here in case someone pops up saying this recipe doesn't work for them. They may have multiple different types of accounts and this data field might fix it, if changed to one type or the other.
Starson17 is offline   Reply With Quote
Reply

Tags
google reader, google reader uber, hack, recpie

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Google Reader recipe not working :( techie_007 Calibre 1 01-26-2010 09:58 PM
Tagging and deleting for Google Reader Recipe jomaweb Calibre 14 01-26-2010 11:31 AM
Recipe Google Reader vs Google Reader Uber DoctorOhh Calibre 0 01-26-2010 04:37 AM
Google Uber Recipe takes so much time jomaweb Calibre 1 01-26-2010 03:21 AM
Read O'Reilly Hacks Series books using Google hack Brian Lounge 12 02-19-2009 03:17 PM


All times are GMT -4. The time now is 11:33 AM.


MobileRead.com is a privately owned, operated and funded community.