Quote:
Originally Posted by timoy_kindle
dear all,
i am trying to write a recipe for this
http://www.scmp.com/frontpage/hk
the text is fine to be retrieved.
however, the image is added later after the page is loaded. can we use recipe to get this?
thanks!
timoy |
below is my recipe.
For example, I can get text and one inside image in the url:
http://www.scmp.com/news/world/unite...ions-customers
however, the large "YAHOO" picture cannot be grabbed.
please kindly advise.
__license__ = 'GPL v3'
__copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>'
'''
scmp.com
'''
from calibre.web.feeds.news import BasicNewsRecipe
class SCMP(BasicNewsRecipe):
title = 'South China Morning Post'
__author__ = 'llam'
description = "SCMP.com, Hong Kong's premier online English daily provides exclusive up-to-date news, audio video news, podcasts, RSS Feeds, Blogs, breaking news, top stories, award winning news and analysis on Hong Kong and China."
publisher = 'South China Morning Post Publishers Ltd.'
category = 'SCMP, Online news, Hong Kong News, China news, Business news, English newspaper, daily newspaper, Lifestyle news, Sport news, Audio Video news, Asia news, World news, economy news, investor relations news, RSS Feeds'
oldest_article = 2
delay = 1
max_articles_per_feed = 200
no_stylesheets = True
encoding = 'utf-8'
auto_cleanup = True
use_embedded_content = False
language = 'en_CN'
remove_empty_feeds = True
needs_subscription = 'optional'
publication_type = 'newspaper'
masthead_url = 'http://www.scmp.com/images/logo_scmp_home.gif'
extra_css = ' body{font-family: Arial,Helvetica,sans-serif } '
conversion_options = {
'comment' : description
, 'tags' : category
, 'publisher' : publisher
, 'language' : language
}
feeds = [
(u'Business' , u'http://www.scmp.com/rss/business.xml' )
#,(u'Hong Kong' , u'http://www.scmp.com/rss/hong_kong.xml' )
#,(u'China' , u'http://www.scmp.com/rss/china.xml' )
#,(u'Asia & World' , u'http://www.scmp.com/rss/news_asia_world.xml')
#,(u'Opinion' , u'http://www.scmp.com/rss/opinion.xml' )
#,(u'LifeSTYLE' , u'http://www.scmp.com/rss/lifestyle.xml' )
#,(u'Sport' , u'http://www.scmp.com/rss/sport.xml' )
]
def preprocess_html(self, soup):
for item in soup.findAll("img"):
item.name = 'img' # converts the links to img
item['src']= item['data-original']
return soup