Calibre + Instapaper not downloading all articles!

[khromov]

I did make some small improvements to Dereks code above.

Changes:
- Removed "Loved/Starred" category (Because it fills up over time)
- Remove the "<< Back to Instapaper" text at the end of every article.

import urllib
from calibre import strftime
from calibre.web.feeds.news import BasicNewsRecipe

class AdvancedUserRecipe1299694372(BasicNewsRecipe):
    title                             = u'Instapaper'
    __author__                  = 'Darko Miletic'
    publisher                     = 'Instapaper.com'
    category                      = 'info, custom, Instapaper'
    oldest_article               = 365
    max_articles_per_feed = 100
    no_stylesheets        = True
    remove_javascript     = True
    remove_tags              = [
	dict(name='div', attrs={'id':'text_controls_toggle'})
	,dict(name='script')
	,dict(name='div', attrs={'id':'text_controls'})
	,dict(name='div', attrs={'id':'editing_controls'})
	,dict(name='div', attrs={'class':'bar bottom'})
	 ]
    use_embedded_content  = False
    needs_subscription    = True
    INDEX                 = u'http://www.instapaper.com'
    LOGIN                 = INDEX + u'/user/login'


    feeds          = [(u'Instapaper Unread', u'http://www.instapaper.com/u')]

    def get_browser(self):
        br = BasicNewsRecipe.get_browser()
        if self.username is not None:
            br.open(self.LOGIN)
            br.select_form(nr=0)
            br['username'] = self.username
            if self.password is not None:
               br['password'] = self.password
            br.submit()
        return br

    def parse_index(self):
        totalfeeds = []
        lfeeds = self.get_feeds()
        for feedobj in lfeeds:
            feedtitle, feedurl = feedobj
            self.report_progress(0, _('Fetching feed')+' %s...'%(feedtitle if feedtitle else feedurl))
            articles = []
            soup = self.index_to_soup(feedurl)
            for item in soup.findAll('div', attrs={'class':'cornerControls'}):
                description = self.tag_to_string(item.div)
                atag = item.a
                if atag and atag.has_key('href'):
                    url         = atag['href']
                    articles.append({
                                     'url'        :url
                                    })
            totalfeeds.append((feedtitle, articles))
        return totalfeeds

    def print_version(self, url): 
        return 'http://www.instapaper.com' + url

    def populate_article_metadata(self, article, soup, first):
        article.title  = soup.find('h1').contents[0].strip()

[Dereks]

Yes, if you are using your own folders in instapapers, they won't show up in this recipe. But you can add them easily in the same way, as "Loved Articles" was added.

Just find this line in the recipe:

    feeds          = [(u'Instapaper Unread', u'http://www.instapaper.com/u')]

and then add something like this inside the square brackets:

(u'Folder Name', u'http://www.instapaper.com/u/folder/numericalidentificator/folder name')

you can find the exact link by simply hovering over the folder in instapaper sidebar. Just copy it and insert as explain above and articles from this folder will appear in instapaper recipe feed as a separate section.
Also, note that the numerical indentificator in the url between /folder/ and /folder name is always unique for each folder - it's not linked to users account or anything, be sure to copy-paste it all correctly.

[khromov]

Another update incoming. I noticed that the text-only view on some articles did not have a <h1> tag for me to reflect the title, so I changed the script to use the <title> tag instead, which always reflects the article title.

The code in its entirety can be found below: (Remember to use Dereks post above if you want to add your Archive and other custom folders.)

import urllib
from calibre import strftime
from calibre.web.feeds.news import BasicNewsRecipe

class AdvancedUserRecipe1299694372(BasicNewsRecipe):
    title                             = u'Instapaper'
    __author__                  = 'Darko Miletic'
    publisher                     = 'Instapaper.com'
    category                      = 'info, custom, Instapaper'
    oldest_article               = 365
    max_articles_per_feed = 100
    no_stylesheets        = True
    remove_javascript     = True
    remove_tags              = [
	dict(name='div', attrs={'id':'text_controls_toggle'})
	,dict(name='script')
	,dict(name='div', attrs={'id':'text_controls'})
	,dict(name='div', attrs={'id':'editing_controls'})
	,dict(name='div', attrs={'class':'bar bottom'})
	 ]
    use_embedded_content  = False
    needs_subscription    = True
    INDEX                 = u'http://www.instapaper.com'
    LOGIN                 = INDEX + u'/user/login'


    feeds          = [(u'Instapaper Unread', u'http://www.instapaper.com/u')]

    def get_browser(self):
        br = BasicNewsRecipe.get_browser()
        if self.username is not None:
            br.open(self.LOGIN)
            br.select_form(nr=0)
            br['username'] = self.username
            if self.password is not None:
               br['password'] = self.password
            br.submit()
        return br

    def parse_index(self):
        totalfeeds = []
        lfeeds = self.get_feeds()
        for feedobj in lfeeds:
            feedtitle, feedurl = feedobj
            self.report_progress(0, _('Fetching feed')+' %s...'%(feedtitle if feedtitle else feedurl))
            articles = []
            soup = self.index_to_soup(feedurl)
            for item in soup.findAll('div', attrs={'class':'cornerControls'}):
                description = self.tag_to_string(item.div)
                atag = item.a
                if atag and atag.has_key('href'):
                    url         = atag['href']
                    articles.append({
                                     'url'        :url
                                    })
            totalfeeds.append((feedtitle, articles))
        return totalfeeds

    def print_version(self, url): 
        return 'http://www.instapaper.com' + url

    def populate_article_metadata(self, article, soup, first):
        article.title  = soup.find('title').contents[0].strip()

[brendanl79]

I have a couple dozen unread articles on my Instapaper account but my fetch yields only an empty EPUB (cover page and Instapaper TOC are there, but no articles).

Calibre 0.8.12 on Fedora 14-x64. :-(
UPDATE: same result with Calibre 0.8.12 on Win7-x64.

I'm not too familiar with Python - any hints as to how I might begin to diagnose this?

[khromov]

Originally Posted by brendanl79

I have a couple dozen unread articles on my Instapaper account but my fetch yields only an empty EPUB (cover page and Instapaper TOC are there, but no articles).

Calibre 0.8.12 on Fedora 14-x64. :-(
UPDATE: same result with Calibre 0.8.12 on Win7-x64.

I'm not too familiar with Python - any hints as to how I might begin to diagnose this?

I assume you are using the built-in Instapaper recipe in Calibre 0.8.12. That recipe has a thread here:
https://www.mobileread.com/forums/sho...d.php?t=143343

The easiest way to diagnose this would be if you sent me the log file to a failed run of your recipe via PM. (Don't post the log on the forum as it contains your Instapaper password, or remove the password before posting the log.)