Instapaper - Updated recipe

[ubieubie]

went ahead and posted the whole recipe maybe you can show me where i am supposed to be adding it. Im sure that is what the issue is. Bold is the line I added I have other folders to add but thats the only one ive tried. Thanks and sorry for the numerous posts.



import urllib, sys
from calibre import strftime
from calibre.web.feeds.news import BasicNewsRecipe

class AdvancedUserRecipe1299694372(BasicNewsRecipe):
title = u'Instapaper'
__author__ = 'Darko Miletic, Stanislav Khromov'
publisher = 'Instapaper.com'
category = 'info, custom, Instapaper'
feeds = []

#if all feeds are empty, no periodical will be generated
remove_empty_feeds = True

# - Recipe configuration - #

#Set archive to True if you want to auto-archive items
archive = False
#Set include_liked to true if you want to include liked items in your feed.
#Warning: Please avoid enabling this in conjunction with auto-archive.
include_liked = False
#Set to true if you want to reverse article order
reverse_article_order = False
#Set the maximum amount of articles you'd like to receive here
max_articles_per_feed = 30

#Set your own custom folders here, simply re-paste the example line below for each new folder you need
#Example:
#feeds+=[(u'My custom folder', u'http://www.instapaper.com/u/folder/1321400/my-custom-folder')]
#feeds+=[(u'Long-term reading', u'http://www.instapaper.com/u/folder/1232089/long-term-reading')]
feeds+=[(u'Genesis Reviews', u'http://www.instapaper.com/u/folder/1538365/genesis-reviews')]
# - End of recipe configuration - #

oldest_article = 0
no_stylesheets = True
remove_javascript = True
remove_tags = [
dict(name='div', attrs={'id':'text_controls_toggle'})
,dict(name='script')
,dict(name='div', attrs={'id':'text_controls'})
,dict(name='div', attrs={'id':'editing_controls'})
,dict(name='div', attrs={'class':'bar bottom'})
,dict(name='div', attrs={'id':'controlbar_container'})
,dict(name='div', attrs={'id':'footer'})
]
use_embedded_content = False
needs_subscription = True
INDEX = u'http://www.instapaper.com'
LOGIN = INDEX + u'/user/login'

if include_liked:
feeds = feeds+[(u'Instapaper Liked', u'http://www.instapaper.com/starred')]
feeds = feeds+[(u'Instapaper Unread', u'http://www.instapaper.com/u')]
feeds.reverse()

cleanup_items = []

def get_browser(self):
br = BasicNewsRecipe.get_browser()
if self.username is not None:
br.open(self.LOGIN)
br.select_form(nr=0)
br['username'] = self.username
if self.password is not None:
br['password'] = self.password
br.submit()
return br

def parse_index(self):
totalfeeds = []
lfeeds = self.get_feeds()

for feedobj in lfeeds:
feedtitle, feedurl = feedobj
self.report_progress(0, _('Fetching feed')+' %s...'%(feedtitle if feedtitle else feedurl))
articles = []

#folder page
current_page = 1

#so we run the first time
articles_on_page = 1

while articles_on_page>0:
soup = self.index_to_soup(feedurl+u'/'+str(current_page))

#get and count number of items on current page
items = soup.findAll('a', attrs={'class':'actionButton textButton'})
cleanup = soup.findAll('a', attrs={'class':'actionButton archiveButton'})

#Add cleanup items
for clean_item in cleanup:
self.cleanup_items.append(clean_item);

articles_on_page = len(items)

# Add each item
for item in items:
description = self.tag_to_string(item.div)
#print atag
#sys.exit()
if item and item.has_key('href'):
url = item['href']
articles.append({
'url' :url
})

current_page+=1

#Don't append empty feeds
if len(articles)!=0:
totalfeeds.append((feedtitle, articles))

if len(totalfeeds)==0:
return None
else:
return totalfeeds

def print_version(self, url):
return 'http://www.instapaper.com' + url

def populate_article_metadata(self, article, soup, first):
#adds the title to the metadata
article.title = soup.find('title').contents[0].strip()

def postprocess_html(self, soup, first_fetch):
#adds the title to each story, as it is not always included
for link_tag in soup.findAll(attrs={"id" : "story"}):
link_tag.insert(0,'<h1>'+soup.find('title').conten ts[0].strip()+'</h1>')

#print repr(soup)
return soup

def cleanup(self):
if self.archive:
self.report_progress(0, "Archiving articles...")
# Clean up each item
for cleanup_item in self.cleanup_items:
self.report_progress(0, "Cleaning up... : " + self.INDEX+cleanup_item['href'])
self.browser.open(self.INDEX+cleanup_item['href']);

[khromov]

Originally Posted by ubieubie

went ahead and posted the whole recipe

The folder looks correct and is in the proper position.

Python is a language that requires indentation of the code. Indentation is done by pressing the Tab key on your keyboard.

Can you please try opening the recipe file in a good text editor, like Notepad2:
http://www.flos-freeware.ch/notepad2.html

When you add the line for your folder, make sure to press the Tab button on your keyboard once before the line, like in this example:


Let me know if this works for you!

[ubieubie]

ok got it to work
tried it it notepad 2 with the tab still failed. Added a few spaces in calibre and it worked fine. Guess it has be in an exact spot. Im going to try and add a few more folders hopefully it wont break.

Thanks for all your help

[PeterT]

Originally Posted by ubieubie

went ahead and posted the whole recipe maybe you can show me where i am supposed to be adding it. Im sure that is what the issue is. Bold is the line I added I have other folders to add but thats the only one ive tried. Thanks and sorry for the numerous posts.

Spoiler:

import urllib, sys
from calibre import strftime
from calibre.web.feeds.news import BasicNewsRecipe

class AdvancedUserRecipe1299694372(BasicNewsRecipe):
    title                             = u'Instapaper'
    __author__                  = 'Darko Miletic, Stanislav Khromov'
    publisher                     = 'Instapaper.com'
    category                      = 'info, custom, Instapaper'
    feeds = []

    #if all feeds are empty, no periodical will be generated
    remove_empty_feeds = True

    # - Recipe configuration - #

    #Set archive to True if you want to auto-archive items
    archive    = False
    #Set include_liked to true if you want to include liked items in your feed.
    #Warning: Please avoid enabling this in conjunction with auto-archive.
    include_liked    = False
    #Set to true if you want to reverse article order
    reverse_article_order = False
    #Set the maximum amount of articles you'd like to receive here
    max_articles_per_feed = 30

    #Set your own custom folders here, simply re-paste the example line below for each new folder you need
    #Example:
    #feeds+=[(u'My custom folder', u'http://www.instapaper.com/u/folder/1321400/my-custom-folder')]
    #feeds+=[(u'Long-term reading', u'http://www.instapaper.com/u/folder/1232089/long-term-reading')]
      feeds+=[(u'Genesis Reviews', u'http://www.instapaper.com/u/folder/1538365/genesis-reviews')]
    # - End of recipe configuration - #

    oldest_article        = 0
    no_stylesheets        = True
    remove_javascript     = True
    remove_tags              = [
        dict(name='div', attrs={'id':'text_controls_toggle'})
        ,dict(name='script')
        ,dict(name='div', attrs={'id':'text_controls'})
        ,dict(name='div', attrs={'id':'editing_controls'})
        ,dict(name='div', attrs={'class':'bar bottom'})
        ,dict(name='div', attrs={'id':'controlbar_container'})
        ,dict(name='div', attrs={'id':'footer'})
         ]
    use_embedded_content  = False
    needs_subscription    = True
    INDEX                 = u'http://www.instapaper.com'
    LOGIN                 = INDEX + u'/user/login'
    
    if include_liked:
        feeds = feeds+[(u'Instapaper Liked', u'http://www.instapaper.com/starred')]
    feeds = feeds+[(u'Instapaper Unread', u'http://www.instapaper.com/u')]
    feeds.reverse()
    
    cleanup_items = []

    def get_browser(self):
        br = BasicNewsRecipe.get_browser()
        if self.username is not None:
            br.open(self.LOGIN)
            br.select_form(nr=0)
            br['username'] = self.username
            if self.password is not None:
                br['password'] = self.password
            br.submit()
        return br

    def parse_index(self):
        totalfeeds = []
        lfeeds = self.get_feeds()

        for feedobj in lfeeds:
            feedtitle, feedurl = feedobj
            self.report_progress(0, _('Fetching feed')+' %s...'%(feedtitle if feedtitle else feedurl))
            articles = []

            #folder page
            current_page = 1
            
            #so we run the first time
            articles_on_page = 1
			
            while articles_on_page>0:
                soup = self.index_to_soup(feedurl+u'/'+str(current_page))

                #get and count number of items on current page
                items = soup.findAll('a', attrs={'class':'actionButton textButton'})
                cleanup = soup.findAll('a', attrs={'class':'actionButton archiveButton'})
                
                #Add cleanup items
                for clean_item in cleanup:
                    self.cleanup_items.append(clean_item);
                
                articles_on_page = len(items)
                
                # Add each item
                for item in items:
                    description = self.tag_to_string(item.div)
                    #print atag
                    #sys.exit()
                    if item and item.has_key('href'):
                        url         = item['href']
                        articles.append({
                                         'url'        :url
                                        })                    
                
                current_page+=1
                
            #Don't append empty feeds
            if len(articles)!=0:
                totalfeeds.append((feedtitle, articles))
        
        if len(totalfeeds)==0:
            return None
        else:
            return totalfeeds

    def print_version(self, url):
        return 'http://www.instapaper.com' + url

    def populate_article_metadata(self, article, soup, first):
        #adds the title to the metadata
        article.title  = soup.find('title').contents[0].strip()

    def postprocess_html(self, soup, first_fetch):
        #adds the title to each story, as it is not always included
        for link_tag in soup.findAll(attrs={"id" : "story"}):
            link_tag.insert(0,'<h1>'+soup.find('title').contents[0].strip()+'</h1>')

        #print repr(soup)
        return soup

    def cleanup(self):
        if self.archive:
            self.report_progress(0, "Archiving articles...")
            # Clean up each item
            for cleanup_item in self.cleanup_items:
                self.report_progress(0, "Cleaning up... : " + self.INDEX+cleanup_item['href']) 
                self.browser.open(self.INDEX+cleanup_item['href']);

For future ease of posting, wrap recipes in a [ CODE ] .... [ /CODE ] block (without the spaces); this will preserve the formatting. Optionally, add in a [ SPOILER ] ... [ /SPOILER ] block around it as well so people dont have to see the whole block...

[dahjelle]

Is it possible to have the recipe optionally return Instapaper items as separate .epubs, rather than as a single collated .epub? I've started looking through the recipe code and whatnot, but nothing is jumping out at me…

[khromov]

Originally Posted by dahjelle

Is it possible to have the recipe optionally return Instapaper items as separate .epubs, rather than as a single collated .epub? I've started looking through the recipe code and whatnot, but nothing is jumping out at me…

I don't think it's possible to generate multiple output files from one recipe.

However, may I suggest the "Send to Kindle" bookmarklet that Readability provides:
http://www.readability.com/bookmarklets

[dahjelle]

Originally Posted by khromov

However, may I suggest the "Send to Kindle" bookmarklet that Readability provides:
http://www.readability.com/bookmarklets

That would probably be perfect…except I have a Kobo Touch, not a Kindle. ;-) Unless Calibre has some sort of automatic download mechanism from my Amazon account that I've not yet seen? Hmm…I'll have to dig around, because that would work just fine. All I'm looking to do is click a bookmarklet in my browser and have articles appear later in Calibre so that I can sync them to my Kobo. :-D

Thanks for the suggestion!

[khromov]

New stable and development versions out with fixes for some annoying issues!

Fixes (all versions):
- Added links to homepage and bitbucket in header.
- Disabled inserting title tag into article body (fixed on Instapapers end)
- Added config switch for aforementioned functionality.
- Added Jim to stable recipe authors

Fixes (development version):
- Multi-page article fetching: only download as many pages as needed to satisfy max_articles_per_feed, instead of all pages.
- Added "(devel)" to recipe name.

Repo:
https://bitbucket.org/khromov/calibre-instapaper

Download:
https://bitbucket.org/khromov/calibr...get/master.zip

[slex]

Where do I put in the recipe username and password?

[khromov]

Originally Posted by slex

Where do I put in the recipe username and password?

After pressing "Fetch news", when you have the recipe selected. See screenshot:

[slex]

Thanks, it worked!

[bigfatfrog67]

Hi,

I'm trying to get this recipe going, the stable version runs but just gives me an empty epub and the devel version gives the following error message, any ideas?

Thanks

Jonathan


calibre, version 0.8.64 (win32, isfrozen: True)
Conversion Error: Failed: Fetch news from Instapaper (devel)

Fetch news from Instapaper (devel)
Resolved conversion options
calibre version: 0.8.64
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_split_on_page_breaks': True,
'duplicate_links_in_toc': False,
'enable_heuristics': False,
'epub_flatten': False,
'extra_css': None,
'extract_to': None,
'filter_css': None,
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x0436F990>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.SonyReader300Output object at 0x0436FE50>,
'page_breaks_before': None,
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': None,
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': '',
'sr1_search': '',
'sr2_replace': '',
'sr2_search': '',
'sr3_replace': '',
'sr3_search': '',
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
InputFormatPlugin: Recipe Input running
Python function terminated unexpectedly
'NoneType' object is not iterable (Error Code: 1)
Traceback (most recent call last):
File "site.py", line 132, in main
File "site.py", line 109, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 192, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 979, in run
File "site-packages\calibre\customize\conversion.py", line 239, in __call__
File "site-packages\calibre\ebooks\conversion\plugins\recipe_ input.py", line 105, in convert
File "site-packages\calibre\web\feeds\news.py", line 881, in download
File "site-packages\calibre\web\feeds\news.py", line 1028, in build_index
File "site-packages\calibre\web\feeds\__init__.py", line 340, in feeds_from_index
TypeError: 'NoneType' object is not iterable

[khromov]

Originally Posted by bigfatfrog67

Hi,
I'm trying to get this recipe going, the stable version runs but just gives me an empty epub and the devel version gives the following error message, any ideas?

Are you getting this consistently? Try removing your reading list (or get a new account and add just a couple items to it) and see if that helps. If you are still getting errors/empty output, please private message me your Instapaper account credentials so I can test the recipe on your account.

[bigfatfrog67]

Originally Posted by khromov

Are you getting this consistently? Try removing your reading list (or get a new account and add just a couple items to it) and see if that helps. If you are still getting errors/empty output, please private message me your Instapaper account credentials so I can test the recipe on your account.

Hi,

Thanks for getting back to me, I was using my userid not my email as login credentials, as soon as I changed that it worked.

However, should it not give an error such as invalid username/password if it can't login to Instapaper, also it still errors if there are no items to downlaod. I don't want an error message in that instance it should just silently exit. Any thoughts?

Thanks

Jonathan

[khromov]

Originally Posted by bigfatfrog67

should it not give an error such as invalid username/password if it can't login to Instapaper,

You are right, there should be a fatal error on invalid credentials. I'll see about getting it fixed.

Originally Posted by bigfatfrog67

also it still errors if there are no items to downlaod. I don't want an error message in that instance it should just silently exit. Any thoughts?

Thanks

Jonathan

You can't silently exit from a recipe without causing a fatal error. If you don't want errors shown you can use the Calibre command line tools to schedule the periodicals instead.
http://manual.calibre-ebook.com/cli/ebook-convert.html