Straits Times (Singapore)

[rty]

The current recipe by Mr Darko Miletic for the Singapore Straits Times is no longer working. I made some changes here to make it work again.

I don't why most of the images won't load. Is it because they are enclosed within the <IMAGES> tags or they use "IMG SRCSET" instead of "IMG SRC"?

Spoiler:

__license__   = 'GPL v3'
__copyright__ = '2009, Darko Miletic <darko.miletic at gmail.com>'
'''
www.straitstimes.com
'''

import re
from calibre.web.feeds.recipes import BasicNewsRecipe

class StraitsTimes(BasicNewsRecipe):
    title                  = 'The Straits Times'
    __author__             = 'Darko Miletic'
    description            = 'Singapore newspaper'
    oldest_article         = 2
    max_articles_per_feed  = 100
    no_stylesheets         = True
    use_embedded_content   = False
    encoding               = 'utf-8'
    publisher              = 'Singapore Press Holdings Ltd.'
    category               = 'news, politics, singapore, asia'
    language               = 'en_SG'

    conversion_options = {
                             'comments'  : description
                            ,'tags'      : category
                            ,'language'  : language
                            ,'publisher' : publisher
                         }

    preprocess_regexps = [
                           (re.compile(
                            r'<meta name="description" content="[^"]+"\s*/?>',
                            re.IGNORECASE|re.DOTALL),
                            lambda m:''),
                           (re.compile(r'<!--.+?-->', re.IGNORECASE|re.DOTALL),
                               lambda m: ''),
                         ]
    remove_tags = [
        dict(name=['object','link','map', 'style']),
        dict(attrs={'class':'dropdown-menu'}),
    ]

    keep_only_tags = [
                        dict(name='h1', attrs={'class':'headline node-title'}),
                        dict(name='div', attrs={'class':'media-group'}),
                        dict(name='div', attrs={'itemprop':'articleBody'})
                    ]
    remove_tags_after=dict(name='div',attrs={'class':'story-keywords hidden-print '})

    feeds = [
               (u'Singapore'       , u'http://www.straitstimes.com/news/singapore/rss.xml' )
              ,(u'Asia'            , u'http://www.straitstimes.com/news/asia/rss.xml'       )
              ,(u'Business'        , u'http://www.straitstimes.com/news/business/rss.xml'     )
              ,(u'Sport'           , u'http://www.straitstimes.com/news/sport/rss.xml'     )
              ,(u'World'           , u'http://www.straitstimes.com/news/world/rss.xml'     )
              ,(u'Lifestyle'       , u'http://www.straitstimes.com/news/lifestyle/rss.xml' )
            ]

    def preprocess_html(self, soup):
        for a in soup.findAll('a', attrs={'class':'thumb'}):
            img = a.find('img')
            if img is not None:
                img['src'] = a['href']
        return soup

[kovidgoyal]

Many sites nowadays use dynamic img loading. Usually the correct img url is in some data attribute and is added dynamically after the page is loaded. You can emulate that in preprocess_html by copy it from the data- attribute to the src attribute.

[edwardwong]

Hi,

I just updated the recipe to include RSS feeds for the print edition as well as the images fix. Just replace the portions below.

Some articles are showing up blank (Header, but no body) eg. the "Opinion" section . Still trying to figure that one out.

    feeds = [
               (u'Top of the News' , u'http://www.straitstimes.com/print/top-of-the-news/rss.xml' )
              ,(u'World'           , u'http://www.straitstimes.com/print/world/rss.xml'       )
              ,(u'Home'            , u'http://www.straitstimes.com/print/home/rss.xml'     )
              ,(u'Business'        , u'http://www.straitstimes.com/print/business/rss.xml'     )
              ,(u'Life'            , u'http://www.straitstimes.com/print/life/rss.xml'     )
              ,(u'Science'         , u'http://www.straitstimes.com/print/science/rss.xml' )
              ,(u'Digital'         , u'http://www.straitstimes.com/print/digital/rss.xml'     )
              ,(u'Insight'         , u'http://www.straitstimes.com/print/insight/rss.xml'     )
              ,(u'Opinion'         , u'http://www.straitstimes.com/print/opinion/rss.xml'     )
              ,(u'Forum'           , u'http://www.straitstimes.com/print/forum/rss.xml' )
              ,(u'Big Picture'     , u'http://www.straitstimes.com/print/big-picture/rss.xml' )
              ,(u'Community'       , u'http://www.straitstimes.com/print/community/rss.xml' )
              ,(u'Education'       , u'http://www.straitstimes.com/print/education/rss.xml' )
]

    def preprocess_html(self, soup):
        for img in soup.findAll('img', srcset=True):
            img['src'] = img['srcset'].partition(' ')[0]
            img['srcset'] = ''
        return soup

[mrshister]

I happened to work on this separately, and here's code that worked for me.

__license__ = 'GPL v3'
__copyright__ = '2017, mrshister'
'''
www.straitstimes.com
'''

import re
from calibre.web.feeds.recipes import BasicNewsRecipe


class StraitsTimes(BasicNewsRecipe):
    title = 'The Straits Times'
    __author__ = 'mrshister'
    description = 'Singapore newspaper'
    oldest_article = 2
    max_articles_per_feed = 100
    no_stylesheets = True
    use_embedded_content = False
    encoding = 'utf-8'
    publisher = 'Singapore Press Holdings Ltd.'
    category = 'news, politics, singapore, asia'
    language = 'en_SG'

    conversion_options = {
        'comments': description, 'tags': category, 'language': language, 'publisher': publisher
    }

    preprocess_regexps = [
        (re.compile(
            r'<meta name="description" content="[^"]+"\s*/?>',
            re.IGNORECASE | re.DOTALL),
         lambda m:''),
        (re.compile(r'<!--.+?-->', re.IGNORECASE | re.DOTALL),
         lambda m: ''),
    ]
  
    headline_reg_exp = 'headline node-title'         # Headline
    img_reg_exp = 'media-entity'                     # Main Image
    body_reg_exp = 'odd\sfield-item'                 # Article Body
    subheadline_reg_exp = 'node-subheadline'         # Sub-headline
    related_reg_exp = '^.*related_story.*$'          # Related Stories

    keep_only_tags = [
               dict(name='h1', attrs={'class': re.compile(headline_reg_exp, re.IGNORECASE)}) 
               ,dict(name='figure', attrs={'itemprop': re.compile(img_reg_exp, re.IGNORECASE)})
               ,dict(name='div', attrs={'class': 'story-postdate'})    # Publish time
               ,dict(name='h2', attrs={'class': re.compile(subheadline_reg_exp, re.IGNORECASE)})
               ,dict(name='div', attrs={'class': re.compile(body_reg_exp, re.IGNORECASE)})    # Article Body
    
    ]
    
    remove_tags = [
               dict(name='div', attrs={'class': re.compile(related_reg_exp, re.IGNORECASE)})
    ]
    
    remove_tags_after = dict(name='div', attrs={'class': 'hr_thin'})
    

    feeds = [
              (u'Top of the News' , u'http://www.straitstimes.com/print/top-of-the-news/rss.xml')
              ,(u'World'           , u'http://www.straitstimes.com/print/world/rss.xml')
              ,(u'Home'            , u'http://www.straitstimes.com/print/home/rss.xml')
              ,(u'Business'        , u'http://www.straitstimes.com/print/business/rss.xml')
              ,(u'Life'            , u'http://www.straitstimes.com/print/life/rss.xml')
              ,(u'Science'         , u'http://www.straitstimes.com/print/science/rss.xml')
              ,(u'Digital'         , u'http://www.straitstimes.com/print/digital/rss.xml')
              ,(u'Insight'         , u'http://www.straitstimes.com/print/insight/rss.xml')
              ,(u'Opinion'         , u'http://www.straitstimes.com/print/opinion/rss.xml')
              ,(u'Forum'           , u'http://www.straitstimes.com/print/forum/rss.xml')
              ,(u'Big Picture'     , u'http://www.straitstimes.com/print/big-picture/rss.xml')
              ,(u'Community'       , u'http://www.straitstimes.com/print/community/rss.xml')
              ,(u'Education'       , u'http://www.straitstimes.com/print/education/rss.xml')
    ]

    def preprocess_html(self, soup):
        for img in soup.findAll('img', srcset=True):
            img['src'] = img['srcset'].partition(' ')[0]
            img['srcset'] = ''
        return soup