Sometimes the browser seems to temporarily lose its session

[RobFreundlich]

I'm working on a recipe to get at the subscriber-only content for the Boston Globe (the existing recipes all seem to use the fairly limited RSS feeds the Globe offers for non-subscribers, and as a subscriber I'd like everything). It logs in, then scrapes the "Today's Paper" page for sections and articles, sets them up as feeds, and then does some processing on the actual articles to clean them up.

Seems like pretty standard stuff, from what I've seen in the existing recipes and here on the forum.

However, I've run into a very bizarre situation. For some articles, instead of getting the subscriber content for the article, calibre ends up with the non-subscriber content (that is, the soup passed into post_process_html contains the non-subscriber content). What's weird is that if I open my browser, log in, and go to the article's URL (the one that calibre is using), I get the subscriber content.

So the URL itself isn't the problem; rather, it looks like in these cases, calibre's python browser has lost its session or something like that.

I've managed to hack around the problem by having post_process_html recognize the two flavors of page and do the right thing. In the non-subscriber page case, that involves finding a "Next" link and doing another page fetch, since one thing they do is split the articles across multiple pages.

However, this is not a very palatable solution. And in at least one case, it fails - if the editorial cartoon ends up in non-subscriber mode, the cartoon itself isn't fetched because they don't let non-subscribers see it.

What I'd really like to do is to figure out why this is happening and put in some magic to avoid it. Has anyone seen anything like this?

oh, and by the way, if there are other Boston Globe subscribers out there, I'll be happy to share the recipe once it's ready.

[mijairaf]

Hello Rob.. Is your recipe still working ? I had a problem with it recently. It appears to think I am not a subscriber and puts out a messages about only 10 fee article per day. Have you run into this problem ?

[RobFreundlich]

Originally Posted by mijairaf

Hello Rob.. Is your recipe still working ? I had a problem with it recently. It appears to think I am not a subscriber and puts out a messages about only 10 fee article per day. Have you run into this problem ?

I'm having the same problem. I took a quick look, and it looks like they're including all of the "you aren't a subscriber" parts and just hiding them if you really are a subscriber. Either they weren't including those parts before, or the hiding method changed and my recipe isn't removing them.

I'm up to my ears in other things right now (new house, upcoming Bar Mitzvah, you know, tiny little things that don't take up any attention ), so I've been living with it until I have a chance to fix it.

Things should settle down in about three weeks. Can you live with it for a while longer, or should I see whether I can find a couple of hours to beat it into submission?

[mijairaf]

Hi Rob,
Well thanks for quick response. I can wait... I don't have much of a choice... Thanks for all your great work to date.

[RobFreundlich]

Thanks for the understanding. Can you do me a favor and ping me in about a month if you haven't heard anything from me by then?

Thanks!

[mijairaf]

Hi Rob,
It has not been quite a month yet.... But if you have any spare bandwidth can you take a look at the recipe. I going abroad next weekend and I was hoping to use calibre to deliver the globe to me via email.
Thanks

[RobFreundlich]

You bet! We've got the house unpacked, and the Bar Mitzvah was this weekend, so I've got a little time on my hands.

I'll see what I can do in the next day or two.

[RobFreundlich]

Well, that turned out to be much easier than anticipated. The attached zipfile contains the updated recipe.

Enjoy, and have a great trip!

[mijairaf]

u da man! Thanks!!!

[RobFreundlich]

You're welcome!

[mijairaf]

Hello Rob,

The Boston Globe recipe is not working again. It seems to think I am not a subscriber. Have you run into this issue ? This is similar to the last time. Any chance you can tell me how you do your debugging ? Perhaps I can learn how to fix it myself so you can enjoy your new house and family without my interruptions.

Thanks

[RobFreundlich]

I haven't looked at the paper in a while, so I haven't noticed. But if you want to take on debugging it, power to you! Here's what I do.

First, from my development directory (basically a dir that contains the recipe and some scripts), I run the following command:

ebook-convert "Boston Globe Subscription_1005.recipe" .EPUB --test -vv -debug-pipeline debug --username my_username --password my_password

ebook-convert lives in the Calibre installation directory, so you'll need that on your PATH (it's probably already there). This command tells it to run the recipe in test mode (so it only fetches a couple of sections, and only a couple of articles per section), save all of the steps along the way into a subdirectory called "debug", and create an EPUB file with the results.

The debug subdir will contain subdirs of its own with all of the interim results, with a README.txt describing them.

I then open http://www.bostonglobe.com/todayspaper in Chrome and use Chrome's Developer Tools (Ctrl+Shift+I) to look at the structure of the page and compare that to what the Python code in the recipe is doing, to try to figure out what went wrong.

I'll add self.log() statements to the code to add debugging messages to the console output, or pdb.set_trace() statements to enter the Python debugger where necessary. Some of those are already in the code, commented out, at strategic points.

I also recommend reading the docs on how to write custom recipes (referenced elsewhere in this forum, I think). And feel free to PM me if you'd like my email address or IM info to get more direct help.

[mijairaf]

Thanks.. I'll let you know how I make out.