Instapaper recipe not working as of this week

[largeboulder]

Is anyone else having this problem? I notice that Instapaper changed their web interface, so maybe they changed other things and broke the recipe?

I don't get an error message when I download, but I just get an empty file.

Any help would be appreciated. (FYI, I've updated to the most recent version of Calibre and this still happens)

[Krittika Goyal]

needs a username and password. Put in dummy values for the moment and pm them to me and i will give it a shot on monday or teusday. Once it is fixed or I fail you can change password again.

[Action Superhero]

Basically, in parse_index, loop over divs with class="title_row" instead of class="cornerControls".

I long ago customized my recipe a bit more than that, but this checks out against instapaper.recipe in the source code I downloaded two minutes ago.

[EliezerYudkowsky]

I didn't see a part of my Instapaper-downloading recipe that talked about cornercontrols. I tried substituting 'title_row' for 'secondaryControls' here:

                      # The 'secondaryControls' div (in the grandparent of the
                        # 'Text' button) contains a bunch of class="host" spans:
                        # source site, date added, Share, Edit, [Move,] Delete
                        secondary = item.parent.parent.find('div', attrs={'class':'secondaryControls'})
                        if secondary:
                            spans = secondary('span', attrs={'class':'host'})
                            if len(spans) >= 2:
                                source = spans[0].string.strip()
                                article['description'] = source
                                article['date'] = self.parse_instapaper_date(spans[1].string.strip())

But this gave me the same error as previously:

Python function terminated unexpectedly
  'NoneType' object is not iterable (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 132, in main
  File "site.py", line 109, in run_entry_point
  File "site-packages\calibre\utils\ipc\worker.py", line 186, in main
  File "site-packages\calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 989, in run
  File "site-packages\calibre\customize\conversion.py", line 239, in __call__
  File "site-packages\calibre\ebooks\conversion\plugins\recipe_input.py", line 109, in convert
  File "site-packages\calibre\web\feeds\news.py", line 891, in download
  File "site-packages\calibre\web\feeds\news.py", line 1060, in build_index
  File "site-packages\calibre\web\feeds\__init__.py", line 346, in feeds_from_index
TypeError: 'NoneType' object is not iterable

[Action Superhero]

I was working off the stable recipe included in the Calibre source code, i.e. https://bitbucket.org/khromov/calibr...cipe?at=master. In the middle of the parse_index method there's

for item in soup.findAll('div', attrs={'class':'cornerControls'}):

and that's where you can simply plug in 'title_row' instead.

I see the code you posted in the dev version. I have not tried running this recipe, so this suggestion is going off a desk read only. I hope this helps, but it's an idea for further development rather than something I expect to actually work. If you fix it up, you can share the results.

In parse_index (as are all the changes), change

items = soup.findAll('a', attrs={'class':'actionButton textButton'})

to

article_inner_items = soup.findAll('div', attrs={'class': 'article_inner_item'})
items = [title_row.find('a') for title_row in article_inner_items.find('div', attrs={'class': 'title_row'})]

Change

archive = item.parent.find('a', attrs={'class':'actionButton archiveButton'})

to

archive = item.parent.parent.find('a', attrs={'class': 'action_link archive_button'})

The description and date aren't in the same place now, so replace the whole block that started with

secondary = ...

with:

article['description'] = item.parent.parent.find('span', attrs={'class': 'host'}).string.strip()
article['date'] = self.parse_instapaper_date(item.parent.parent.find('span', attrs={'class': 'meta_date'}).string.strip())

The summary:

summary = item.parent.parent.find('div', attrs={'class':'summary'})

merely changed to a "preview":

summary = item.parent.parent.find('div', attrs={'class':'article_preview'})

That's all I see. If you're comfortable debugging recipes, see how far that gets you.

For Krittika or anyone else who doesn't use Instapaper, here's what an article on the new feed page looks like:

<div class="article_item" data-article-id="409202621">
<div class="article_inner_item">
<div class="title_row">
<a href="/read/409202621"
>Why Draper University Won’t Work (But Could)</a>
</div>
<div class="title_meta">
<span class="host">
<a href="http://calnewport.com/blog/2013/08/28/why-draper-university-wont-work-but-could/">
<span>

calnewport.com

</span>
<i class="externalLink icon-circle-arrow-right"></i>
</a>
</span>

&middot; by Study Hacks

</div>
<div class="primary_actions">

<a href="/star_toggle/409202621" class="action_link star_toggle ">
<span class="icon icon-like"></span>
</a>


<a title="Archive" class="action_link archive_button" id="skip409202621" href="/skip/409202621" title="Archive">
<span class="icon icon-drawer"></span>
</a>



</div>
<div class="article_preview">
School for Heroes
Every morning, the students at the Draper University School for Heroes recite an oath:
“I will promote freedom at all costs.”
“I will do…
</div>
<div class="article_item_footer group">
<div class="mobile_options">
<span>hide </span></a>options
</div>
<div class="meta">
<span class="meta_date">8/29</span>

&middot; <span class="meta_read_time">2 min</span>

</div>
<div class="secondary_actions group">
<div class="menu_container share_container">
<a class="action_link" href="#"><i class="icon-share"></i><span>share</span></a>
<div class="action_menu share_menu">
<div class="menu_options">
<div class="menu_inner">
<a class="menu_item emailShare shareOut" id="shareToEmail" href="mailto:?subject=Why Draper University Won’t Work (But Could)&body=Why+Draper+University+Won%E2%80%99t+Work+%28But+Could%29%0Ahttp%3A%2F%2Fcalnewport.com%2Fblog%2F2013%2F08%2F28%2Fwhy-draper-university-wont-work-but-could%2F%0A%0A(via%20Instapaper)">
<i class="icon-envelope"></i> <span>Email</span>
</a>






</div>
</div>

</div>
</div>

[EliezerYudkowsky]

Shifting to the stable recipe does let me download again (yay!) but the archive = True setting no longer causes items to be archived after they are downloaded. Since Instapaper apparently no longer has an 'Archive All' button this is a substantial inconvenience.

[Action Superhero]

Originally Posted by EliezerYudkowsky

Shifting to the stable recipe does let me download again (yay!) but the archive = True setting no longer causes items to be archived after they are downloaded. Since Instapaper apparently no longer has an 'Archive All' button this is a substantial inconvenience.

Add this to the end of the stable recipe to virtually click all the archive buttons.

    def postprocess_book(self, oeb, opts, log):
        br = self.get_browser()
        for feedobj in self.get_feeds():
            feedtitle, feedurl = feedobj
            self.report_progress(0, 'Archiving feed'+' %s...'%(feedtitle if feedtitle else feedurl))
            soup = self.index_to_soup(feedurl)
            for item in soup.findAll('div', attrs={'class':'title_row'}):
                archive = item.parent.parent.find('a', attrs={'class': 'action_link archive_button'})
                br.open(archive['href'])

This is right out of my current recipe, so this bit is tested. It's unconditional, so comment out the whole thing if you have to turn it off. And don't add anything to Instapaper while it's running: the postprocessing will happily archive things that the main processing never downloaded.

[EliezerYudkowsky]

All is right with the world, my old workflow is restored. Thank you!