Grabbing and including image from another url

[Rasmus]

Hi,
I am trying to improve the Spiegel Int'l receipt.

So far I got sections going and removed a lot of noise by using the print version of articles.

However, printed articles does not include images. I like these and would prefer to include them.

I have written the following simple script which grabs an image from the non-printed version of an article:

    def get_img(self, url):
        txt = BeautifulSoup(urllib2.urlopen(url))
        img = soup.find('div', {'class' : 'spGalleryBigPic'}).find('img')['src']

I assume that using some magic trick ebook-convert knows url (it does in the print function).

My questions are

1. How do I include this picture in the actual ebook?
2. Am I 'allowed' to use urllib2 in receipts or is some other method preferred? (Most likely it is)
3. Should I add some kind of robustness to img or will Calibre handle it? (Probably not; I guess I could just try img to avoid exceptions)

These questions are probably obvious, but I did not seem to be able to find all of the documentation that I wanted in on receipt...

Cheers,
Rasmus

PS: I wrote a receipt for Economist's Daily Chart. Should I share it or what is the custom for these things? Should I share my improved Spiegel receipt?

[kovidgoyal]

Implement preprocess_html in your recipe and inset the img elements there, calibre will do the downloading for you. Sharing your recipes is always a good thing.

[Rasmus]

Thanks for the quick reply. I wasn't specific enough.

I do

import urllib2

in order to download the html page, which in turn is turned into soup. How could I internalize this?

I'll get the new Spiegel int'l receipt to you once I have gotten images integrated. Thanks for the tip on preprocess_html.

And thanks for Calibre! I run the cli tools from a server every morning and enjoy all of my favorite news when I get up. It is great!

--Rasmus

[Rasmus]

I will need a bit more help, it seems.

This is a clean Python example:

>>> u = 'http://www.spiegel.de/international/europe/0,1518,773071,00.html'
>>> v = urllib2.urlopen(u).read()
>>> soup = BeautifulSoup(v) # this should be identical to Calibre's Soup
>>> img = soup.find('div', {'class' : 'spGalleryBigPic'}).find('img')
>>> type(img)
     <class 'BeautifulSoup.Tag'>

So basically, I want to insert this into the article (below the heading, but for now I just want to get it into the epub article).

So I wrote the following preprocess function:

    def preprocess_html(self, soup):
        soup = soup.find('div', {'class' : 'spGalleryBigPic'}).find('img')
        return soup

which returns what I called img above.

I get the article using:

    def print_version(self, url): 
        'from Spigelde.receipt'
        rmt = url.rpartition('#')[0]
        main, sep, rest = rmt.rpartition(',')
        rmain, rsep, rrest = main.rpartition(',')
        purl = rmain + ',druck-' + rrest + ',' + rest
        return purl

But currently it only works when I do not use preprocess_html.

[kovidgoyal]

Use the index_to_soup() method instead of urllib2. WHat you want to do in preprocess_html is

run index_to_soup, find the img element you want in the soup returned by index_to_soup and insert it into the soup passed into preprocess_html. See the BeautifulSoup docs for how to do that.