Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Recipes

Notices

Reply
 
Thread Tools Search this Thread
Old 01-09-2010, 12:59 PM   #1
rollercoaster
Zealot
rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.
 
rollercoaster's Avatar
 
Posts: 126
Karma: 1826
Join Date: Jan 2010
Device: Kindle 2
Thumbs up Google Reader Recipe hack - Download all unread insted of just starred

Hi everyone, I have been searching for a few hours and couldnt find a fix for the 'google reader shows only starred'

Finally I decided to get my hands dirty and read the Google Reader API documentation however I am new to caliber and dont know python at all but (being a .net dev) was able to make a simple modification that helped some what. here is the recipe without further ado

Google Reader: This recipe fetches from your Google Reader account unread Starred items and unread Feeds you have placed in a folder via the manage subscriptions feature.
Spoiler:

PHP Code:
import urllibremechanize
from calibre
.web.feeds.recipes import BasicNewsRecipe
from calibre import __appname__

class GoogleReader(BasicNewsRecipe):
    
title   'Google Reader'
    
description 'This recipe fetches from your Google Reader account unread Starred items and unread Feeds you have placed in a folder via the manage subscriptions feature.'
    
needs_subscription True
    __author__ 
'davec, rollercoaster, Starson17'
    
base_url 'http://www.google.com/reader/atom/'
    
oldest_article 365
    max_articles_per_feed 
50
    get_options 
'?n=%d&xt=user/-/state/com.google/read' max_articles_per_feed
    use_embedded_content 
True

    def get_browser
(self):
        
br BasicNewsRecipe.get_browser(self)
        if 
self.username is not None and self.password is not None:
            
request urllib.urlencode([('Email'self.username), ('Passwd'self.password),
                                        (
'service''reader'), ('accountType''HOSTED_OR_GOOGLE'), ('source'__appname__)])
            
response br.open('https://www.google.com/accounts/ClientLogin'request)
            
auth re.search('Auth=(\S*)'response.read()).group(1)
            
cookies mechanize.CookieJar()
            
br mechanize.build_opener(mechanize.HTTPCookieProcessor(cookies))
            
br.addheaders = [('Authorization''GoogleLogin auth='+auth)]
        return 
br

    def get_feeds
(self):
        
feeds = []
        
soup self.index_to_soup('http://www.google.com/reader/api/0/tag/list')
        for 
id in soup.findAll(Trueattrs={'name':['id']}):
            
url id.contents[0]
            
feeds.append((re.search('/([^/]*)$'url).group(1),
                          
self.base_url urllib.quote(url.encode('utf-8')) + self.get_options))
        return 
feeds 

Google Reader uber: Fetches all feeds from your Google Reader account including the uncategorized items.
Spoiler:

PHP Code:
import urllibremechanize
from calibre
.web.feeds.recipes import BasicNewsRecipe
from calibre import __appname__

class GoogleReaderUber(BasicNewsRecipe):
    
title   'Google Reader uber'
    
description 'Fetches all feeds from your Google Reader account including the uncategorized items.'
    
needs_subscription True
    __author__ 
'davec, rollercoaster, Starson17'
    
base_url 'http://www.google.com/reader/atom/'
    
oldest_article 365
    max_articles_per_feed 
250
    get_options 
'?n=%d&xt=user/-/state/com.google/read' max_articles_per_feed
    use_embedded_content 
True

    def get_browser
(self):
        
br BasicNewsRecipe.get_browser(self)
        if 
self.username is not None and self.password is not None:
            
request urllib.urlencode([('Email'self.username), ('Passwd'self.password),
                                        (
'service''reader'), ('accountType''HOSTED_OR_GOOGLE'), ('source'__appname__)])
            
response br.open('https://www.google.com/accounts/ClientLogin'request)
            
auth re.search('Auth=(\S*)'response.read()).group(1)
            
cookies mechanize.CookieJar()
            
br mechanize.build_opener(mechanize.HTTPCookieProcessor(cookies))
            
br.addheaders = [('Authorization''GoogleLogin auth='+auth)]
        return 
br

    def get_feeds
(self):
        
feeds = []
        
soup self.index_to_soup('http://www.google.com/reader/api/0/tag/list')
        for 
id in soup.findAll(Trueattrs={'name':['id']}):
            
url id.contents[0].replace('broadcast','reading-list')
            
feeds.append((re.search('/([^/]*)$'url).group(1),
                          
self.base_url urllib.quote(url.encode('utf-8')) + self.get_options))
        return 
feeds 

For a more detailed comparison see this thread. Thanks @dwanthny.

Notes:
Spoiler:

The only difference is in the max count and the line
url = id.contents[0] -->
url = id.contents[0].replace('broadcast','reading-list')

For now this seems to work by replacing the starred tag listing with the wanted 'reading-list' which is all the unread posts. Technically, 'http://www.google.com/reader/api/0/tag/list' does not return all the possible tag/state values.

I am sure other recipe super heroes here can make this better.

There is one thing though. I set the max count to 250 but it only fetched about 57, may be a limit by google as in my google reader account there were 121 items unread. (update: fixed bu adding 'oldest_article' value)

If I missed a already posted and better recipe for this please point me to it so I can benifit from it as well and dont hold back on your comments

-----------------------------------------------------------------------

Update Recipe (13/07/10): The recipe has been updated to work with the changed authentication method in Google Reader API. Thanks to Starson17.

@Kovid The name was changed from 'Google Reader Uber' to just 'Google reader' to merge the two similar recipes. The Uber verson can now be removed from the recipe set.

-----------------------------------------------------------------------
Update Recipe (14/07/10): Both recipes have been updated to work with the changed authentication method in Google Reader API. Thanks to Starson17.

.

Last edited by rollercoaster; 07-15-2010 at 12:45 AM. Reason: The recipe has been updated
rollercoaster is offline   Reply With Quote
Old 01-09-2010, 01:13 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,338
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Looks fine. I don;t use greader myself, so I can't test it. But if it seems stable to you for a few days, let me know and I'll add it to calibre.
kovidgoyal is offline   Reply With Quote
 
Advertisement
Old 01-12-2010, 08:09 AM   #3
rollercoaster
Zealot
rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.
 
rollercoaster's Avatar
 
Posts: 126
Karma: 1826
Join Date: Jan 2010
Device: Kindle 2
thanks for replying.. I will use it for few days and give feedback
rollercoaster is offline   Reply With Quote
Old 01-14-2010, 08:12 AM   #4
rollercoaster
Zealot
rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.
 
rollercoaster's Avatar
 
Posts: 126
Karma: 1826
Join Date: Jan 2010
Device: Kindle 2
The recipe is not downloading entries older then a week. Is that a forced limitation in caliber? The fetching url itself does download those entries when used directly in the browser.

If yes can it be overridden via the recipe as unread posts from google reader can be any amount of old. specially for 'starred' etc items
rollercoaster is offline   Reply With Quote
Old 01-14-2010, 12:14 PM   #5
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,338
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Use the oldest_article setting in the recipe
kovidgoyal is offline   Reply With Quote
Old 01-15-2010, 08:49 AM   #6
rollercoaster
Zealot
rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.
 
rollercoaster's Avatar
 
Posts: 126
Karma: 1826
Join Date: Jan 2010
Device: Kindle 2
great! that works perfectly. Kovid you have made a great thing this caliber

Now just sync once a day and have all my fav blogs on kindle!
now only if wireless delivery was free in india... one can dream right?

----

Next modification - I was thinking to mark downloaded posts as read so that one doesnt have to go to google reader and mark as read manually. otherwise they will get download each time. Any Ideas how to plug into a 'feed item downloaded' kind of event so that I can make the right API call to google reader to mark the said item as read?
rollercoaster is offline   Reply With Quote
Old 01-15-2010, 12:01 PM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,338
Karma: 5382313
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Override the article_downloaded method. Just call the base class method first and then do watever is needed.
kovidgoyal is offline   Reply With Quote
Old 01-17-2010, 02:33 AM   #8
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,888
Karma: 12755553
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by rollercoaster View Post
great! that works perfectly. Kovid you have made a great thing this caliber

Now just sync once a day and have all my fav blogs on kindle!


The last time I tried the Google reader recipe I was only able to get the last 10 posts in one blog. Even though I tagged 30 only 10 would come through. I finally gave up.

This is working great!

**You just need to change the description from tagged to all unread.


Kovid you should include this in the standard set a recipes along side the other Google Reader recipe.
DoctorOhh is offline   Reply With Quote
Old 01-21-2010, 12:14 AM   #9
rollercoaster
Zealot
rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.
 
rollercoaster's Avatar
 
Posts: 126
Karma: 1826
Join Date: Jan 2010
Device: Kindle 2
Quote:
Originally Posted by kovidgoyal View Post
Override the article_downloaded method. Just call the base class method first and then do watever is needed.
sadly I dont know python syntax. will have to learn that first then be able to get all OOPS on this recipe's a$$. I want to but lets see if I get time for it.. Help is welcome

@dwanthny you are welcome and I fixed the description.. I always seem to miss the comments on what ever code i write :P

can anyone link me to a *very* short python tutorial/book please?

Last edited by rollercoaster; 01-21-2010 at 12:24 AM.
rollercoaster is offline   Reply With Quote
Old 02-09-2010, 09:27 AM   #10
thread314
Junior Member
thread314 began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Feb 2010
Device: Kindle 2
Hi,

I've been looking for something just like this for a while, and am so glad to find.

However i can't get it to work, is there some simple mistake I'm making? I've tried copying and pasting the code into a blank recipe, but i get an error (I'll happily post the specific error if people like). Is there somewhere I should be enterring my username and password? (I tried a couple of spots that looked logical to no avail) or is there something else altogether that I'm missing?

Thanks very much for a great recipe!
thread314 is offline   Reply With Quote
Old 02-09-2010, 09:47 AM   #11
thread314
Junior Member
thread314 began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Feb 2010
Device: Kindle 2
ignore that, I've figured it out.

My apologies.
thread314 is offline   Reply With Quote
Old 02-10-2010, 12:48 AM   #12
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,888
Karma: 12755553
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by thread314 View Post
ignore that, I've figured it out.

My apologies.

For others out there this recipe comes with Calibre under unknown.
DoctorOhh is offline   Reply With Quote
Old 03-17-2010, 06:05 AM   #13
rollercoaster
Zealot
rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.rollercoaster once ate a cherry pie in a record 7 seconds.
 
rollercoaster's Avatar
 
Posts: 126
Karma: 1826
Join Date: Jan 2010
Device: Kindle 2
Question

Quote:
Originally Posted by kovidgoyal View Post
Override the article_downloaded method. Just call the base class method first and then do watever is needed.
I am returning to this after a bit of time. had a look at the documentation but couldn't find the signature etc for the article_downloaded method.

What i need to do is: Override article_downloaded(<params???>) and
  • call the base article_downloaded()
  • For each entry in each feed (//feed/entry) -
    • Get the ID of the entry from the XML (soup?) (//feed/entry/id)
    • POST to an API url to mark it as read with the cookie and token fetched in get_browser() [with the same browser instance?]

is article_downloaded() called once or once per feed or once per entry?
also how to I write to the log so I can do some debugging etc..

sorry if I am frustrating u. I dont really know python and just read an overview of the syntax. thanks in advance

Last edited by rollercoaster; 03-17-2010 at 06:47 AM.
rollercoaster is offline   Reply With Quote
Old 06-23-2010, 06:28 AM   #14
cclambie
Junior Member
cclambie began at the beginning.
 
Posts: 1
Karma: 10
Join Date: Jun 2010
Device: BeBook Neo
Error :(


Hi there community.

Just got my BeBook Neo yesterday and came across this calibre software today... fantastic idea.
Have sort out most of the news feeds I want and read regularily, which is great!

Now I just want my blogs on there.....
Tried these two Google Reader ones, this and a number of combinations, but I am getting an error

Was hoping someone could help me out on here??

Thanks

C

Error
----

InputFormatPlugin: Recipe Input running
Python function terminated unexpectedly
(Error Code: 1)
Traceback (most recent call last):
File "site.py", line 103, in main
File "site.py", line 85, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 99, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 24, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 815, in run
File "site-packages\calibre\customize\conversion.py", line 211, in __call__
File "site-packages\calibre\web\feeds\input.py", line 104, in convert
File "site-packages\calibre\web\feeds\news.py", line 702, in download
File "site-packages\calibre\web\feeds\news.py", line 856, in build_index
File "site-packages\calibre\web\feeds\news.py", line 1301, in parse_feeds
File "c:\users\user\appdata\local\temp\calibre_0.7.4_kp y1dw_recipes\recipe0.py", line 35, in get_feeds
soup = self.index_to_soup('http://www.google.com/reader/api/0/tag/list')
File "site-packages\calibre\web\feeds\news.py", line 474, in index_to_soup
File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_opener.py", line 202, in open
File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_http.py", line 612, in http_response
File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_opener.py", line 225, in error
File "urllib2.py", line 367, in _call_chain
File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_http.py", line 633, in http_error_default
urllib2.HTTPError: HTTP Error 401: Unauthorized
cclambie is offline   Reply With Quote
Old 06-23-2010, 02:23 PM   #15
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,888
Karma: 12755553
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by cclambie View Post
urllib2.HTTPError: HTTP Error 401: Unauthorized
I haven't used this recipe in a while but I get this error now too.
DoctorOhh is offline   Reply With Quote
Reply

Tags
google reader, google reader uber, hack, recpie

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Google Reader recipe not working :( techie_007 Calibre 1 01-26-2010 10:58 PM
Tagging and deleting for Google Reader Recipe jomaweb Calibre 14 01-26-2010 12:31 PM
Recipe Google Reader vs Google Reader Uber DoctorOhh Calibre 0 01-26-2010 05:37 AM
Google Uber Recipe takes so much time jomaweb Calibre 1 01-26-2010 04:21 AM
Read O'Reilly Hacks Series books using Google hack Brian Lounge 12 02-19-2009 04:17 PM


All times are GMT -4. The time now is 08:41 PM.


MobileRead.com is a privately owned, operated and funded community.