Calibre + Instapaper not downloading all articles!

[Maxiboost]

Hi!

(duplicate of thread posted in wrong forum: https://www.mobileread.com/forums/sho...d.php?t=102841, dwanthny recommended I re-posted here)

Firstly can I say what a fabulous bit of software Calibre is! Well done to Kovid for what has made life so much easier..

I'm in the same boat as the OP when trying to download articles from Instapaper. It varies how many articles are actually downloaded. For instance, the first time I downloaded a bunch of articles the filesize was 4.3MB - a few minutes later when I tried again the size was 0.8MB and only a handful! I tried this a few days later and the same 0.8mb file was downloaded.

Do you think this is a Calibre issue or an Instapaper one? The recipe is as follows:

__license__   = 'GPL v3'
__copyright__ = '2009-2010, Darko Miletic <darko.miletic at gmail.com>'
'''
www.instapaper.com
'''

import urllib
from calibre import strftime
from calibre.web.feeds.news import BasicNewsRecipe

class Instapaper(BasicNewsRecipe):
    title                 = 'The KindleMagic Daily'
    __author__            = 'Unknown'
    description           = '''Personalized news feeds. Go to instapaper.com to
                               setup up your news. Fill in your instapaper
                               username, and leave the password field
                               below blank.'''
    publisher             = 'Instapaper.com'
    category              = 'news, custom'
    oldest_article        = 50
    max_articles_per_feed = 100
    no_stylesheets        = True
    use_embedded_content  = False
    needs_subscription    = True
    INDEX                 = u'http://www.instapaper.com'
    LOGIN                 = INDEX + u'/user/login'

    conversion_options = {
                          'comment'   : description
                        , 'tags'      : category
                        , 'publisher' : publisher
                        }

    feeds = [
              (u'Uncategorised articles' , INDEX + u'/u')
             ,(u'Starred articles', INDEX + u'/starred')
	   ,(u'News', INDEX + u'/u/folder/number_removed/news')
	   ,(u'Sport', INDEX + u'/u/folder/number_removed/sport') 
	   ,(u'Technology', INDEX + u'/u/folder/number_removed/technology')
	   ,(u'Gaming', INDEX + u'/u/folder/number_removed/gaming')
	   ,(u'Comment', INDEX + u'/u/folder/number_removed/comment') 
	   ,(u'Gossip/Rubbish', INDEX + u'/u/folder/number_removed/gossip-rubbish')

]

    def get_browser(self):
        br = BasicNewsRecipe.get_browser()
        if self.username is not None:
            br.open(self.LOGIN)
            br.select_form(nr=0)
            br['username'] = self.username
            if self.password is not None:
               br['password'] = self.password
            br.submit()
        return br

    def parse_index(self):
        totalfeeds = []
        lfeeds = self.get_feeds()
        for feedobj in lfeeds:
            feedtitle, feedurl = feedobj
            self.report_progress(0, _('Fetching feed')+' %s...'%(feedtitle if feedtitle else feedurl))
            articles = []
            soup = self.index_to_soup(feedurl)
            for item in soup.findAll('div', attrs={'class':'titleRow'}):
                description = self.tag_to_string(item.div)
                atag = item.a
                if atag and atag.has_key('href'):
                    url         = atag['href']
                    title       = self.tag_to_string(atag)
                    date        = strftime(self.timefmt)
                    articles.append({
                                      'title'      :title
                                     ,'date'       :date
                                     ,'url'        :url
                                     ,'description':description
                                    })
            totalfeeds.append((feedtitle, articles))
        return totalfeeds

    def print_version(self, url):
        return self.INDEX + '/text?u=' + urllib.quote(url)

This appears to also be the same for the standard built-in recipe for Instapaper - so is it recipe related?

For info, I am running Calibre 0.7.30 on Linux Ubuntu 10.10.

Really hoping someone can help here folks!

Thanks in advance for your time

Cheers
Maxi (UK)

PS, not sure if this is relevant, but on my work's box I am running Ubuntu 10.04 - every time I download I get all the articles (i.e. 4.3mb) so it's working fine there. An Ubuntu 10.10 thing??

PPS, still have a problem on 10.04 which is on my laptop.





-----------

[Robot Monkey]

I'm having the same problem. I get the 10 most recently added articles. The recipe says "max 100 articles" so I have no idea what to try next.

[Maxiboost]

Sorry to hear that... same boat as me then

I raised a ticket with the developer and a day later the ticket was closed, saying:

Changes (by darkom):

* status: assigned => closed
* resolution: => worksforme


Comment:

I'm unable to reproduce this on Windows. Apparently it is something related
to your Linux distribution. You are on your own here.

Little disappointed with that if I am honest, but I understand the error is difficult to re-create. That said, it seems a few of us are experiencing these strangeties on Linux so maybe it's time for the dev to take a serious look..

[kovidgoyal]

make sure you are running the latest calibre and install the official calibre binary from http://calibre-ebook.com not the ubuntu package.

[Robot Monkey]

I should've said before, but I have the problem with the latest Calibre on Windows 7 64-bit. For reference, Instafetch on my Android phone grabs all my articles.

Calibre is so awesome I feel like a dick even bringing it up, but there you have it. My other complaint is that I pronounce it cuh-LEE-bray, but I think most people pronounce it CAL-i-burr. My way is fancier.

[kovidgoyal]

I'm afraid I don't help with recipe problems for recipes I haven't written. If I did that, then I would have no time to do actual development, so you are on your own, unless the recipe author happens across this thread.

[Robot Monkey]

That's cool; thanks for the wonderful application anyway!

[kiklop74]

Originally Posted by Maxiboost

Sorry to hear that... same boat as me then

I raised a ticket with the developer and a day later the ticket was closed, saying:



Little disappointed with that if I am honest, but I understand the error is difficult to re-create. That said, it seems a few of us are experiencing these strangeties on Linux so maybe it's time for the dev to take a serious look..

And why do you assume a developer should do anything? This is open-source project where all of the developers participate on pro-bono basis. If you think there is a problem and the original developer (in this case me) decides not to do anything than you can :

1. whine that the world is unjust
2. do something about that issue yourself

[mornington]

There seems to be a small problem with the Instapaper recipe when used with the Kindle 3 at any rate.

The basic issue seems to be that it's displayed with one line too many per page, such that the final line is overwritten by the kindle's "menu bar" at the bottom of each page. I've no idea if this problem is to do with the recipe or calibre's feed-to-mobi code, or even if it's kindle 3-specific.

I do know that other news feeds, such as the Guardian newspaper one, don't have this problem when viewed on the Kindle. And if I get instapaper to send a mobi of articles directly to the kindle (via email and wifi) then those mobis display fine.

Anybody any idea what might be causing this extra-line issue?

[zach382]

I am having this problem as well. I've tried changing the oldest_article and max_articles_per_feed number. Currently it's only grabbing the first 10 in unread. Here is the error it gives me when it tries to grab any of the other articles beyond the first 10.

Why Is Illinois So Corrupt? from Unread articles
http://www.chicagomag.com/Chicago-Magazine/December-2010/Why-Is-Illinois-So-Corrupt-Local-Government-Experts-Explain/index.php?cparticle=1&siarticle=0#artanc
Traceback (most recent call last):
  File "site-packages\calibre\utils\threadpool.py", line 95, in run
  File "site-packages\calibre\web\feeds\news.py", line 839, in fetch_article
  File "site-packages\calibre\web\feeds\news.py", line 835, in _fetch_article
Exception: Could not fetch article. Run with -vv to see the reason

I get this error dozens as times as it goes through all the articles.

[zach382]

I think I know what might be the problem. It's not using the Inspapaper text converted URL but trying to grab it from the original sites URL. This is causing the fetch to fail. In the source for Instapaper there is a div with a direct link to the text only version so I am trying to use the URL in the controlCorners div instead of the url this script is creating. My python is not very good though so I am having some difficulty modifying the script.

[kiklop74]

The instapaper uses text converted url only. If you take a look at print_version method this is exactly what I'm doing.

[kiklop74]

Originally Posted by kiklop74

The instapaper uses text converted url only. If you take a look at print_version method this is exactly what I'm doing.

An I just tried downloading article you meantion and it got downloaded just fine. The error is not in recipe but somewhere else.

[matznet]

I am a bit puzzled now!

I have a prs-650 and I am using Calibre with instapaper, standard built-in recipe.
If I am using it (version 0.7.38) on my iMac with Snow Leopard, I get the known problem: only 10 articles downloaded.
If I am using it (version 0.7.35) on my Macbook pro with Leopard, I do not have that problem: all the articles are downloaded!

Any idea how to have it working on Snow Leopard too? :/

UPDATE: I tried version 0.7.35 on Snow Leopard but the problem remains.

[Gomez]

Strange, i have the exact same problem. Only 10 Article ;(

My System is a Mac with SL.

Maybe its a problem with the OS?