|
|
Thread Tools | Search this Thread |
06-23-2010, 01:24 PM | #16 |
creator of calibre
Posts: 44,351
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
You'll have to track down the original author of the recipe and ask for a fix. (davec and rollercoaster)
|
06-24-2010, 11:05 AM | #17 |
Member
Posts: 13
Karma: 10
Join Date: Apr 2009
Device: kindle
|
Google reader does not seem to work for me now in calibre 7.4. Is this a problem for other people also? Thanks.
P.S. I love calibre. Thank you so much. |
Advert | |
|
07-03-2010, 02:07 PM | #18 |
Junior Member
Posts: 1
Karma: 10
Join Date: Jul 2010
Device: SONY PRS-900BC
|
I start getting the same error too. It was good on 6/20. I don't remember exactly which version I was using since there have been quite a few new versions published.
|
07-03-2010, 07:58 PM | #19 | |
US Navy, Retired
Posts: 9,865
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
This post entitled "Changes to sending authenticated requests to Google Reader" on the Google Reader Blog might hold the key but I don't have the skill set needed to correct the problem. |
|
07-04-2010, 09:37 AM | #20 | |
Wizard
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
Quote:
|
|
Advert | |
|
07-09-2010, 12:53 AM | #21 |
Zealot
Posts: 126
Karma: 1826
Join Date: Jan 2010
Device: Kindle 2
|
hi all, sorry for replying late but I was on vacation.
I can figure out how to fix the recipe but my problem is I dont know python. The last change I did was a minor string.replace(...) that is usually same on all languages. I could write Pseudocode if someone is willing to convert it to python. (Like in post 13 of this thread) |
07-09-2010, 09:35 AM | #22 | |
Wizard
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
Quote:
This isn't a Python issue as much as it is a mechanize issue and knowing where to get the authorization data that Google needs for the header to be built in mechanize. I've used mechanize a few times and am comfortable in basic Python. I suggest we stay in this thread, and if anyone offers help over in the custom recipe thread, we can steer them here. Last edited by Starson17; 07-12-2010 at 10:42 AM. |
|
07-09-2010, 05:11 PM | #23 |
Wizard
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
Brief update -
I've got the recipe retrieving the Auth code needed, instead of the SID. I've got to head out, so it will still be a while, but if all goes well, it's just a matter of building the required header with the auth code. I'll keep you posted. |
07-10-2010, 01:36 PM | #24 |
Wizard
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
Now I need some help from someone who knows something about Google Reader.
Step 1 was to retrieve the auth code. I needed to change the request a bit from the old recipe, but that was successful. Step 2 was to use the auth code and add it in as a header to each request. That was a pain. I got the header in OK, but it wasn't authorizing. For one thing, the API specs as to the exact format weren't very clear and for another, the mechanize docs weren't very clear, and for a third, unless you ask the right way, the auth code you get in step 1 is wrong. Usually, I can watch the headers when I retrieve from a website, then just do the same thing, but this is a program interface, not a browser interface, and I don't have anything working in the browser that I can copy. Nonetheless, I was able to get it authenticating by changing the original request and copying the header format I found in a blog. At that point, I expected success, and I suppose I may have it, but there's no content I can retrieve an authenticated page at: http://www.google.com/reader/api/0/tag/list That's where the recipe gets the feeds. The recipe then builds URLs for a feed list. Here's some feeds that it built: (u'National Review Blogs', 'http://www.google.com/reader/atom/user/18271557450980043868/label/National%20Review%20Blogs?n=50&xt=user/-/state/com.google/read'), (u'North Carolina', 'http://www.google.com/reader/atom/user/18271557450980043868/label/North%20Carolina?n=50&xt=user/-/state/com.google/read'), (u'Watts Up With That', 'http://www.google.com/reader/atom/user/18271557450980043868/label/Watts%20Up%20With%20That?n=50&xt=user/-/state/com.google/read'), However, these don't seem to have any content. I have no idea why the recipe builds these particular URLs, and since I'm dealing with a programmatic access to the Google Reader, not a browser access, I can't copy a successful browser session. Perhaps the recipe should build some other URLs for its feeds from the contents of the page at http://www.google.com/reader/api/0/tag/list, but if so, I don't know what it should build? Maybe my test account needs to have articles tagged or marked in some special way to show up at these locations? Alternatively, maybe I need some special cookies for the content to show up? It would probably be a heck of a lot easier for me to just rewrite the whole dang thing from scratch trying to base it on FireFox browser access instead of this API. If anyone knows the URL that the recipe should be building for each feed, let me know. If you want to test further, replace the get_browser in the recipe with this: Code:
def get_browser(self): br = BasicNewsRecipe.get_browser(self) # Print HTTP headers. br.set_debug_http(True) orig_open_novisit = br.open_novisit if self.username is not None and self.password is not None: request = urllib.urlencode([('Email', self.username), ('Passwd', self.password), ('service', 'reader'), ('accountType', 'HOSTED_OR_GOOGLE'), ('source', __appname__)]) response = br.open('https://www.google.com/accounts/ClientLogin', request) auth = re.search('Auth=(\S*)', response.read()).group(1) def my_open_no_visit(url, **kwargs): req = mechanize.Request( url, headers = { 'Authorization':'GoogleLogin auth='+auth, }) return orig_open_novisit(req) br.open_novisit = my_open_no_visit return br |
07-11-2010, 05:11 AM | #25 |
Zealot
Posts: 126
Karma: 1826
Join Date: Jan 2010
Device: Kindle 2
|
@Starson17 Your work helped me a lot and I think I know how to fix this but I still need your help to write python code. You are on the spot with fetching the AUTH token.
The issue is after the " def get_feeds(self): " method. We get the various feeds here and then each feed is queried to get articles/items under them. The request to those feeds DOES NOT pass the AUTH token as a header. And so, if you look at the log in jobs list then you will see that, google redirects these requests with a "302 Moved Temporarily" and sends to the login page. So we never get to the list of items in the feed. Code:
send: 'POST /accounts/ClientLogin HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 97\r\nHost: www.google.com\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection: close\r\nUser-Agent: Mozilla/5.0 (X11; U; i686 Linux; en_US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4\r\n\r\n' send: 'Email=someone&Passwd=someone&service=reader&accountType=HOSTED_OR_GOOGLE&source=calibre' reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: text/plain header: Cache-control: no-cache, no-store header: Pragma: no-cache header: Expires: Mon, 01-Jan-1990 00:00:00 GMT header: Date: Sun, 11 Jul 2010 08:50:30 GMT header: X-Content-Type-Options: nosniff header: X-XSS-Protection: 1; mode=block header: Content-Length: 733 header: Server: GSE header: Connection: close send: 'GET /reader/api/0/tag/list HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.google.com\r\nConnection: close\r\nAuthorization: GoogleLogin auth=DQAAAKIAAABiWf2ZNf7xjpkr97aUJusKEEnI9ClihCCQMtpu3NrqbJRAOzU-ix4K42nnKcAjsQaREfV0XrS-ISsZRs6YrMoH_gBB3M0rUD2thscZHmkvBxZP0S07gvwthEaruqD-j2hAwIKTXWh4csgdmhzjjvgcP6dSlJPqnzMVuxgC2J-t-EyFNDY4W8Ihi-ybimjip_pITYMP7vSuPCyJWOFCrQzM1ES8Ewmp_nn1LU4-_uihxw\r\nUser-Agent: Mozilla/5.0 (X11; U; i686 Linux; en_US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: text/xml; charset=UTF-8 header: X-Reader-Google-Version: 506-000 header: X-Reader-User: 10347379608328268864 header: Set-Cookie: GRLD=UNSET:10347379608328268864;Path=/reader/;HttpOnly header: Date: Sun, 11 Jul 2010 08:50:30 GMT header: Expires: Sun, 11 Jul 2010 08:50:30 GMT header: Cache-Control: private, max-age=0 header: X-Content-Type-Options: nosniff header: X-Frame-Options: SAMEORIGIN header: X-XSS-Protection: 1; mode=block header: Server: GSE header: Connection: close send: 'GET /reader/atom/user/10347379608328268864/state/com.google/starred?n=250&xt=user/-/state/com.google/read HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.google.com\r\nCookie: GRLD=UNSET:10347379608328268864\r\nConnection: close\r\nUser-Agent: Mozilla/5.0 (X11; U; i686 Linux; en_US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4\r\n\r\n' reply: 'HTTP/1.1 302 Moved Temporarily\r\n' header: Content-Type: text/html; charset=UTF-8 header: X-Reader-Google-Version: 506-000 header: Location: https://www.google.com/accounts/ServiceLogin?service=reader&passive=1209600&continue=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Fstarred%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread&followup=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Fstarred%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread header: Date: Sun, 11 Jul 2010 08:50:31 GMT header: Expires: Sun, 11 Jul 2010 08:50:31 GMT header: Cache-Control: private, max-age=0 header: X-Content-Type-Options: nosniff header: X-Frame-Options: SAMEORIGIN header: X-XSS-Protection: 1; mode=block header: Content-Length: 1381 header: Server: GSE header: Connection: close send: 'GET /accounts/ServiceLogin?service=reader&passive=1209600&continue=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Fstarred%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread&followup=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Fstarred%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.google.com\r\nConnection: close\r\nUser-Agent: Mozilla/5.0 (X11; U; i686 Linux; en_US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: text/html; charset=UTF-8 header: Cache-control: no-cache, no-store header: Pragma: no-cache header: Expires: Mon, 01-Jan-1990 00:00:00 GMT header: Set-Cookie: GALX=T4Ka0MjTUcE;Path=/accounts;Secure header: Date: Sun, 11 Jul 2010 08:50:32 GMT header: X-Content-Type-Options: nosniff header: X-XSS-Protection: 1; mode=block header: Content-Length: 13863 header: Server: GSE header: Connection: close send: 'GET /reader/atom/user/10347379608328268864/state/com.google/reading-list?n=250&xt=user/-/state/com.google/read HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.google.com\r\nCookie: GRLD=UNSET:10347379608328268864\r\nConnection: close\r\nUser-Agent: Mozilla/5.0 (X11; U; i686 Linux; en_US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4\r\n\r\n' reply: 'HTTP/1.1 302 Moved Temporarily\r\n' header: Content-Type: text/html; charset=UTF-8 header: X-Reader-Google-Version: 506-000 header: Location: https://www.google.com/accounts/ServiceLogin?service=reader&passive=1209600&continue=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Freading-list%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread&followup=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Freading-list%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread header: Date: Sun, 11 Jul 2010 08:50:33 GMT header: Expires: Sun, 11 Jul 2010 08:50:33 GMT header: Cache-Control: private, max-age=0 header: X-Content-Type-Options: nosniff header: X-Frame-Options: SAMEORIGIN header: X-XSS-Protection: 1; mode=block header: Content-Length: 1381 header: Server: GSE header: Connection: close send: 'GET /accounts/ServiceLogin?service=reader&passive=1209600&continue=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Freading-list%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread&followup=http%3A%2F%2Fwww.google.com%2Freader%2Fatom%2Fuser%2F10347379608328268864%2Fstate%2Fcom.google%2Freading-list%3Fn%3D250%26xt%3Duser%252F-%252Fstate%252Fcom.google%252Fread HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: www.google.com\r\nCookie: GALX=T4Ka0MjTUcE\r\nConnection: close\r\nUser-Agent: Mozilla/5.0 (X11; U; i686 Linux; en_US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: text/html; charset=UTF-8 header: Cache-control: no-cache, no-store header: Pragma: no-cache header: Expires: Mon, 01-Jan-1990 00:00:00 GMT header: Set-Cookie: GALX=T4Ka0MjTUcE;Path=/accounts;Secure header: Date: Sun, 11 Jul 2010 08:50:34 GMT header: X-Content-Type-Options: nosniff header: X-XSS-Protection: 1; mode=block header: Content-Length: 13901 header: Server: GSE header: Connection: close Anyway, have a look at this. I tried to get it working but could not. Language barrier I guess It this doesnt/cant work for some reason then, conceivably, we can fill the login form with the username / pass each time and then submit and read the list of items. but that is the long way around. So in summary, we need get the individual feed requests after "get_feeds(self)" to include the AUTH token and the recipe should hum along beautifully. Last edited by rollercoaster; 07-11-2010 at 05:26 AM. |
07-11-2010, 09:28 AM | #26 | |||
Wizard
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
Quote:
Obviously, that link doesn't contain the user info. As soon as I fixed the auth header, that link worked, so I thought I was sending the header correctly for all links. Stupid me, I didn't check. (funny thing - I left in the print headers code so you could see I was sending the auth code and you used it to see that I wasn't sending it - at least not for the article fetch. ) Quote:
Quote:
Spoiler:
Test it, and if it also works for you, pass it to Kovid. You may also want to modify and test the Uber version. You shouldn't have any trouble, but if you do, let me know. Last edited by Starson17; 07-12-2010 at 10:45 AM. |
|||
07-13-2010, 06:32 AM | #27 |
Zealot
Posts: 126
Karma: 1826
Join Date: Jan 2010
Device: Kindle 2
|
It works great! Tested with various settings and about 110 articles.
Changed oldest_article = 365 and max_articles_per_feed = 250. Also, replaced broadcast feed with reading-list feed. reading-list is the one that contains articles not in other feeds i.e uncategorized. Also I think we should remove the uber version and just update the 'Google reader' recipe. Updated recipe: See first post. Last edited by rollercoaster; 07-13-2010 at 06:44 AM. |
07-13-2010, 08:08 AM | #28 | |
US Navy, Retired
Posts: 9,865
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
Now the reason the two recipes should not be combined. The reason I used the Googlereader and not Googlereaderuber is because Googlereader recipe doesn't have the reading list. Read this post for details. The bottom line is because I have my subscriptions in folders this recipe (first post) now downloads all my articles twice and I end up with a book that has about twice as many pages as a result. Update: The attached images are the exact same articles downloaded today with the two different recipes. Even though the second has about 33% more pages the actual size wasn't much larger (7.9M vs 8.5M) so I guess I'm not locked into having two recipes. One recipe could be fine but if someone thinks they will be able to easily navigate through the mess that is the Reading List with "uncategorized" articles and all of the rest of the "categorized articles"... well it is practically useless. Of course until they learn how to place their feeds in folders at least they will be reaching their goal of grabbing articles using Google Reader. Last edited by DoctorOhh; 07-13-2010 at 08:47 AM. |
|
07-13-2010, 08:57 AM | #29 |
creator of calibre
Posts: 44,351
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Let me know when you've settled on the final recipe(s) and I'll update them in calibre.
|
07-13-2010, 10:19 AM | #30 | |
Wizard
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
Quote:
BTW, during the step of obtaining the auth code with the user/pass, I added the data: 'accountType', 'HOSTED_OR_GOOGLE'. That was recommended in the API and it worked. However, I did see a post from someone who had trouble with that and had to use 'accountType', 'GOOGLE' (he was not fetching from GoogleReader). I didn't go deeply enough into the API to find out if it could cause a problem in the recipe. It may even work without that data (I believe it may default to 'HOSTED_OR_GOOGLE' if not specified). Apparently there are various kinds of accounts - but it wasn't something I wanted to delve into deeply. If someone else does, they may wish to change this to specify the accountType. I mention it only to make a record of it here in case someone pops up saying this recipe doesn't work for them. They may have multiple different types of accounts and this data field might fix it, if changed to one type or the other. |
|
Tags |
google reader, google reader uber, hack, recpie |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Google Reader recipe not working :( | techie_007 | Calibre | 1 | 01-26-2010 09:58 PM |
Tagging and deleting for Google Reader Recipe | jomaweb | Calibre | 14 | 01-26-2010 11:31 AM |
Recipe Google Reader vs Google Reader Uber | DoctorOhh | Calibre | 0 | 01-26-2010 04:37 AM |
Google Uber Recipe takes so much time | jomaweb | Calibre | 1 | 01-26-2010 03:21 AM |
Read O'Reilly Hacks Series books using Google hack | Brian | Lounge | 12 | 02-19-2009 03:17 PM |