skepticblog.org - Recipe not working

[BuzzKill]

http://skepticblog.org

Ok, I tried to get this one to work, but it doesn't seem to work. Somewhere in the details during processing a HTTP 403 error seems to pop up, but I will appreciate if someone can check it.

#!/usr/bin/env  python

import re
from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import Tag

class SkepticBlog(BasicNewsRecipe):
    title                 = 'Skepticblog'
    __author__            = 'Multiple Authors'
    description           = 'A collaboration among some of the most recognized names in promoting science, critical thinking, and skepticism'
    oldest_article        = 5
    max_articles_per_feed = 15
    no_stylesheets        = True
    use_embedded_content  = False
    encoding              = 'utf-8'
    publisher             = 'Skeptic Magazine'
    category              = 'science, pseudoscience'
    language              = 'en'

    lang                  = 'en-US'

    conversion_options = {
                          'comment'          : description
                        , 'tags'             : category
                        , 'publisher'        : publisher
                        , 'language'         : lang
                        , 'pretty_print'     : True
                        }

    keep_only_tags = [dict(name='div', attrs={'class':'post'})]
    remove_tags_after  = [dict(name='div', attrs={'class':'ratingblock'})]

    feeds = [(u'SkepticBlog', u'http://skepticblog.org/feed')]

    def preprocess_html(self, soup):
        mtag = Tag(soup,'meta',[('http-equiv','Content-Type'),('context','text/html; charset=utf-8')])
        soup.head.insert(0,mtag)
        soup.html['lang'] = self.lang
        return self.adeify_images(soup)

[Starson17]

Originally Posted by BuzzKill

http://skepticblog.org

Ok, I tried to get this one to work, but it doesn't seem to work. Somewhere in the details during processing a HTTP 403 error seems to pop up, but I will appreciate if someone can check it.

That site is running the Bad Behavior module to prevent spam from being posted to the blog. It triggers when Calibre tries to read it. The header code I wrote for The Skeptic recipe will fix it:

    def get_browser(self):
        br = BasicNewsRecipe.get_browser(self)
        br.addheaders = [('Accept', 'text/html')]
        return br

BTW, why aren't you using that recipe? It's built in and includes the http://skepticblog.org/feed feed.

[BuzzKill]

Oh, I apologize, It didn't occur to me that there would be one already built in. Thank you.

[Starson17]

Originally Posted by BuzzKill

Oh, I apologize, It didn't occur to me that there would be one already built in. Thank you.

That's OK. I had to check myself to make sure it was included. It's hard to remember which feeds are in which recipes.

[steven.davis]

Not sure if anyone is still tracking this but I updated the feeds and it seems to be working.

from calibre.web.feeds.news import BasicNewsRecipe
import re

class Skeptic(BasicNewsRecipe):
    title          = u'The Skeptic'
    description         = 'Discussions with leading experts and investigation of fringe science and paranormal claims.'
    language       = 'en'
    __author__     = 'Starson17'
    oldest_article = 31
    cover_url           = 'http://www.skeptricks.com/images/Skeptic_Magazine.jpg'
    remove_empty_feeds    = True
    remove_javascript   = True
    max_articles_per_feed = 50
    no_stylesheets = True

    remove_tags = [dict(name='div', attrs={'class':['Introduction','divider']}),
                  dict(name='div', attrs={'id':['feature', 'podcast']}),
                  dict(name='div', attrs={'id':re.compile(r'follow.*', re.DOTALL|re.IGNORECASE)}),
                  dict(name='hr'),
                  ]


    feeds = [
            ('The Skeptic', 'http://feeds.feedburner.com/Skepticcom'),
            ('E-Skeptic', 'http://www.skeptic.com/eskeptic')
            ]

    def get_browser(self):
        br = BasicNewsRecipe.get_browser(self)
        br.addheaders = [('Accept', 'text/html')]
        return br

    extra_css = '''
                    h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;}
                    h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal;font-size:small;}
                    p{font-family:Arial,Helvetica,sans-serif;font-size:small;}
                    body{font-family:Helvetica,Arial,sans-serif;font-size:small;}
		'''