Google Reader Recipe hack - Download all unread insted of just starred

[rollercoaster]

Originally Posted by dwanthny

The reason I used the Googlereader and not Googlereaderuber is because Googlereader recipe doesn't have the reading list. Read this post for details. The bottom line is because I have my subscriptions in folders this recipe (first post) now downloads all my articles twice and I end up with a book that has about twice as many pages as a result.

Of course until they learn how to place their feeds in folders at least they will be reaching their goal of grabbing articles using Google Reader.

I understand your point and it is fine by me to keep both the versions.

I personally am more of a uncategorized person as i add and remove feeds to my google reader all the time and mostly that leaves things in eternal chaos.

Since the original recipe left out uncategorized items, I took it to be an obvious flaw. I thought just duplicating the posts was not enough of a point to keep separate recipes as there is not that much of a size issue and this way the classification work more like 'tag's instead of 'folders' i.e one item can have more then one tag.

In any case it doesn't hurt to keep both the recipes. I will update my first post to include both updated versions as well as link to your excellent post so people can choose the right one and learn to be better organized.

Thinking about it, for users like me who add/remove feeds like mad and have 200 unread items every day, this also opens the door to fetch only selected feeds. Just have the ones you want to read tagged and leave the others untagged and use the 'Google reader' recipe.

-EDIT-------------------------------------------------------------------

On second thought, is it possible to give the user a checkbox from a recipe, just like we allow them to enter username/password? For e.g a checkbox saying "Download all articles including uncategorized".

This would also be helpful for the feature i want to add at some point, i.e a checkbox to enable marking downloaded articles as read so that they are not re-downloaded in the next news fetch.

Originally Posted by Starson17

BTW, during the step of obtaining the auth code with the user/pass, I added the data: 'accountType', 'HOSTED_OR_GOOGLE'.

I think it is for those sevices like Google's Corporate mail service and website hosting service by Google that dont end in @google.com

It doesnt hurt by being present and it already the border value so lets leave it in.

[Starson17]

Originally Posted by rollercoaster

In any case it doesn't hurt to keep both the recipes. I will update my first post to include both updated versions as well as link to your excellent post so people can choose the right one and learn to be better organized.

If you or dwanthny will test the second version and confirm it works, I'd say it's ready to go to Kovid for inclusion in the next update.

[kovidgoyal]

At the moment its not possible for users to do GUI based customization of recipes, other than for username password and a couple of items in advanced.

[DoctorOhh]

Originally Posted by Starson17

If you or dwanthny will test the second version and confirm it works, I'd say it's ready to go to Kovid for inclusion in the next update.

I tested both recipes and they work fine. I clarified the description in the Googlereader recipe.

Thanks to Starson17 and rollercoaster, nice team effort.

[kovidgoyal]

Updated, thanks.

[Starson17]

Originally Posted by dwanthny

I tested both recipes and they work fine. I clarified the description in the Googlereader recipe.

Excellent.

Thanks to Starson17 and rollercoaster, nice team effort.

I think I would have given up if not for rollercoaster spotting that I wasn't adding the header to article requests.

[rollercoaster]

Originally Posted by kovidgoyal

At the moment its not possible for users to do GUI based customization of recipes, other than for username password and a couple of items in advanced.

I imaging the day that happens, it will bring a whole new era of cooking with the recipes.

Originally Posted by dwanthny

I tested both recipes and they work fine. I clarified the description in the Googlereader recipe.

I have also tested and they work well. I will add your description to the first post as well.

Originally Posted by dwanthny

Thanks to Starson17 and rollercoaster, nice team effort.

Originally Posted by Starson17

I think I would have given up if not for rollercoaster spotting that I wasn't adding the header to article requests.

I am just an extra pair of eyes on this one. You did great. Do you think you are up for adding another feature to this recipe i.e to mark downloaded posts as read? Assuming more people then just me would like to see that happen.

[Starson17]

Originally Posted by rollercoaster

Do you think you are up for adding another feature to this recipe i.e to mark downloaded posts as read? Assuming more people then just me would like to see that happen.

I'm willing to put it on my todo list, if you'll read the Google API and figure out how to mark downloaded posts as read. Ideally, some pseudocode would be great. This recipe is tough for me to work on as I don't use GR. I looked at the recipe results that are supposed to be "working" with the new code, and to be honest, I couldn't even tell if they were what you were supposed to get.

I should warn that my todo list has a few things on it and I'm really slow. I only tackled this because it looked easy (Ha!), and I was one of the few people who'd already written a recipe that changed headers. (I'm still trying to figure out why my first header control method failed. I went back and tested my earlier recipe with both header control methods, and both worked. I don't yet know why only the second one worked for Google Reader.)

[rollercoaster]

Originally Posted by Starson17

I'm willing to put it on my todo list, if you'll read the Google API and figure out how to mark downloaded posts as read. Ideally, some pseudocode would be great.

Fair enough! I will expand on my previous pseudocode and try to be as detailed as I can be. There is no hurry for this anyway.

Originally Posted by Starson17

I'm still trying to figure out why my first header control method failed.

Could it be because it does not navigate to the feed urls with the same method that you have overridden and used some other method. That way the override code to insert the header would not be executed.

May be we should ask the other devs to add support for checkboxes or a dropdown of options in recipes as it is possible that some users will not like the idea of marking the posts read. It would be great to have that option under user control but I dont know how hard it will be from a code point of view. We only need to store and read the state of the checkbox/dropdown/other controls. May be you can contact someone who can give an informed answer.

[rollercoaster]

Well it appears there are two methods to do this. The first is straight forward. Lets try it...

After all feeds have been processed, make a GET request to "http://www.google.com/reader/atom/user/-/state/com.google/reading-list?n=1" and from the received XML(see Reference Feed structure) extract the feed id and timestamp values.

Then POST to "http://www.google.com/reader/api/0/mark-all-as-read" with the data "s=<Feed ID>&ts=<TimeStamp in microseconds>&T=<token>" (may need to URI encode values)

Note that this must be done AFTER the articles have been processed/saved as they cannot be downloaded again after successfully marking all as read. If possible, do it after the ebook as been generated and saved successfully so that conversion or other external errors done mess this up.

Reference Feed structure:

<feed>
  ...
  <id>
    tag:google.com,2005:reader/user/10347379608328268864/state/com.google/reading-list
  </id>
  ...
  <entry gr:crawl-timestamp-msec="1279242802553">
    ...
  </entry>
</feed>

To get the Feed ID: read value at <feed><id>[VALUE]<id>....</feed>
XPATH = //feed/id


To get the Time Stamp: read value at
<feed><entry gr:crawl-timestamp-msec="[VALUE]"></entry>....</feed>
XPATH=//feed/entry[1]/[crawl-timestamp-msec]
Note 1: we only need the time stamp of the FIRST <entry>
Note 2: The value in the feeds is in millisecond but for the "mark-all-as-read" we need microsecond. So to get the right value do this: Microsecond_Time_Stamp = ((MiliSecond_Time_Stamp + 1) * 1000) -1


To get Token value
Execute a GET request for "http://www.google.com/reader/api/0/token" and read the response. it is a 57 chars string.

Final Note 1: All and any requests must include the "Authorization=GoogleLogin auth=<AUTH TOKEN>" header.

------------------------

if this doesnt work then we will have to try the other method that will be more complicated as low performing. I.e. mark individual items as read.

[Starson17]

Originally Posted by rollercoaster

Fair enough! I will expand on my previous pseudocode and try to be as detailed as I can be. There is no hurry for this anyway.

The pseudocode looks good. It will sit in this thread until I pick it up. I've got main code work (merge custom fields), rtf conversion code, and this on my list, plus problems with the memory in my development machine - this weekend I'll tackle at least one or two of those, and get back to this when I have a chance. I learned several useful new things doing this recipe. It's a bit beyond most recipe work, so it's interesting, even though I don't use it.

Could it be because it does not navigate to the feed urls with the same method that you have overridden and used some other method. That way the override code to insert the header would not be executed.

Possible, but I think it's unlikely. The overridden method is used in essentially the same way in my GoComics code. I'll admit, I don't know for sure - it's based on Kovid's pseudocode. I haven't had a chance to fully track down how it functions.

May be we should ask the other devs to add support for checkboxes or a dropdown of options in recipes as it is possible that some users will not like the idea of marking the posts read. It would be great to have that option under user control but I dont know how hard it will be from a code point of view. We only need to store and read the state of the checkbox/dropdown/other controls. May be you can contact someone who can give an informed answer.

I've previously posted on the desirability of providing additional user controls in recipes. I've written code that stores and retrieves options in the main GUI, but never looked at whether this could be done in a recipe. Until then, we can just use the multiple recipe method.

[Starson17]

Originally Posted by rollercoaster

Well it appears there are two methods to do this. The first is straight forward. Lets try it...

Looking at this, I don't see any straightforward way to do this. Calibre's recipe system isn't set up at this low level of sending GET/POST. It operates at a higher level. Yes, we could start digging down into the code, to override the methods, but that's deeper than I want to go for something I don't use.

For example:

After all feeds have been processed, make a GET request to "http://www.google.com/reader/atom/user/-/state/com.google/reading-list?n=1" and from the received XML(see Reference Feed structure) extract the feed id and timestamp values.

There's no problem reading and extracting the feed id and timestamp values with def_browser, but def_browser operates before the articles are pulled. That's what we want to do for authorization and login, but here it needs to wait until "After all feeds have been processed." I don't know of any way to do this after the articles have been fetched.

Here's another problem:

If possible, do it after the ebook as been generated and saved successfully so that conversion or other external errors done mess this up.

AFAICT, the recipe runs first to produce HTML, then the conversion runs to build the EPUB. The recipe has stopped before the EPUB is started. Nothing in the recipe can tell if the EPUB was successful.

Sorry to disappoint you, but I'm going to focus on a couple of other things of interest to me in Calibre code.

[nandocuci]

I need to download all starred feeds (read and unread), not only the last 20 entries.
Is it possible?

[DoctorOhh]

Originally Posted by nandocuci

I need to download all starred feeds (read and unread), not only the last 20 entries.
Is it possible?

It may or may not be possible, but as it stands now the recipes do not accomplish your stated goal. Both existing recipes only read unread entries, up to 250 unread articles per feed. Read this post for more details.

[rollercoaster]

Originally Posted by Starson17

Calibre's recipe system isn't set up at this low level of sending GET/POST. It operates at a higher level.

I dont understand. GET/POST are not low lever operations. A quick search for "python POST request" shows me many examples of people doing this. e.g -

pass a data keyword argument to urllib.urlopen:

u = urllib.urlopen('http://www.domain.com/cgi-bin/cgi.py',
data=urllib.urlencode({'name': 'pythonguy'}))

POST is the way to submit form data. For e.g. Username password forms. The main difference with GET (any normal read url request) is that the parameters are submitted as multi-part form-data instead of in the query string and are not limited in size as URLs usually are.

Originally Posted by Starson17

Sorry to disappoint you, but I'm going to focus on a couple of other things of interest to me in Calibre code.

That is too bad but I can understand the limitations.

Thanks very much for you time though. May be we can pick this up later sometime. Recipes could do with more advanced features like additional UI elements and more control over the whole fetch process.