10-04-2016, 11:01 PM | #1 |
Junior Member
Posts: 7
Karma: 10
Join Date: Aug 2013
Device: kindle paperwhite
|
parse image url added later in the webpage
dear all,
i am trying to write a recipe for this http://www.scmp.com/frontpage/hk the text is fine to be retrieved. however, the image is added later after the page is loaded. can we use recipe to get this? thanks! timoy |
10-04-2016, 11:11 PM | #2 | |
Junior Member
Posts: 7
Karma: 10
Join Date: Aug 2013
Device: kindle paperwhite
|
More info to this
Quote:
For example, I can get text and one inside image in the url: http://www.scmp.com/news/world/unite...ions-customers however, the large "YAHOO" picture cannot be grabbed. please kindly advise. __license__ = 'GPL v3' __copyright__ = '2010, Darko Miletic <darko.miletic at gmail.com>' ''' scmp.com ''' from calibre.web.feeds.news import BasicNewsRecipe class SCMP(BasicNewsRecipe): title = 'South China Morning Post' __author__ = 'llam' description = "SCMP.com, Hong Kong's premier online English daily provides exclusive up-to-date news, audio video news, podcasts, RSS Feeds, Blogs, breaking news, top stories, award winning news and analysis on Hong Kong and China." publisher = 'South China Morning Post Publishers Ltd.' category = 'SCMP, Online news, Hong Kong News, China news, Business news, English newspaper, daily newspaper, Lifestyle news, Sport news, Audio Video news, Asia news, World news, economy news, investor relations news, RSS Feeds' oldest_article = 2 delay = 1 max_articles_per_feed = 200 no_stylesheets = True encoding = 'utf-8' auto_cleanup = True use_embedded_content = False language = 'en_CN' remove_empty_feeds = True needs_subscription = 'optional' publication_type = 'newspaper' masthead_url = 'http://www.scmp.com/images/logo_scmp_home.gif' extra_css = ' body{font-family: Arial,Helvetica,sans-serif } ' conversion_options = { 'comment' : description , 'tags' : category , 'publisher' : publisher , 'language' : language } feeds = [ (u'Business' , u'http://www.scmp.com/rss/business.xml' ) #,(u'Hong Kong' , u'http://www.scmp.com/rss/hong_kong.xml' ) #,(u'China' , u'http://www.scmp.com/rss/china.xml' ) #,(u'Asia & World' , u'http://www.scmp.com/rss/news_asia_world.xml') #,(u'Opinion' , u'http://www.scmp.com/rss/opinion.xml' ) #,(u'LifeSTYLE' , u'http://www.scmp.com/rss/lifestyle.xml' ) #,(u'Sport' , u'http://www.scmp.com/rss/sport.xml' ) ] def preprocess_html(self, soup): for item in soup.findAll("img"): item.name = 'img' # converts the links to img item['src']= item['data-original'] return soup |
|
Advert | |
|
10-05-2016, 12:56 AM | #3 |
creator of calibre
Posts: 43,892
Karma: 22666668
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Turn off auto_cleanup, like this: https://github.com/kovidgoyal/calibr...f4f0d20015bd08
|
10-05-2016, 04:10 AM | #4 | ||
Junior Member
Posts: 7
Karma: 10
Join Date: Aug 2013
Device: kindle paperwhite
|
Quote:
Quote:
It is because I have not got the latest version? Why "auto_cleanup" is such a magic?! calibre is great! |
||
10-05-2016, 05:18 AM | #5 |
creator of calibre
Posts: 43,892
Karma: 22666668
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
auto_cleanup is a shortcut that tries to extract the actual content from the page using heuristics. It makes it easier to develop recipes, but for best results, you should turn it off and hand-tune the recipe, see the user manual section on creating your own news sources for more details.
|
Advert | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
image url looking in temp folder for file | david_f1 | Sigil | 23 | 08-01-2016 03:40 PM |
PRS-350 Shut Down Image - Missing from PRS+ or Added by Another Dev? | Prestidigitweeze | Sony Reader Dev Corner | 2 | 01-19-2013 09:03 PM |
HTML Meta tag for cover image URL? | MrPLD | Conversion | 5 | 01-29-2012 12:15 AM |
Grabbing and including image from another url | Rasmus | Recipes | 4 | 07-08-2011 12:06 PM |