Script to scrape page for a cover image for recipe?

[adoucette]

I'm making a recipe to capture an online periodical. The cover image is always on the front page with a file name like "cover0212_227033.jpg" where the "0212" portion is the month and year and the "_227033" is random. So if it had to use a regular expression, I think the script would need to look for

cover\d\d\d\d_.*\.jpg

(if that helps).
Is there a way to put this into the Calibre recipe to grab the cover?
Thanks,
Ari

[kovidgoyal]

def get_cover_url(self):
    soup = self.index_to_soup(url_of_cover_page)
    cov = soup.find('img', src=re.compile(r'.*cover\d{4}_\d{6}.jpg$'))
    if cov is not None:
         self.cover_url = cov['src']

[adoucette]

Thanks for the reply Kovid - you have a great product and I've certainly donated.
The regexp works properly, but I get a syntax error when I try to add/update the recipe in Calibre. Here's the syntax I'm using:

    def get_cover_url(self):
        cover_url = None
        soup = self.index_to_soup('http://www.abc.com')
        cov = soup.find('img', src=re.compile(r'\w*?cover\w{1,22}\.jpg')
        if is not None:
            self.cover_url = cov['src']

and the error I get is

calibre, version 0.8.41
ERROR: Invalid input: <p>Could not create recipe. Error:<br>invalid syntax (<string>, line 13)

[kovidgoyal]

if cov is not None

[adoucette]

No, same error - I neglected 'cov' just here on the forum - my bad.
Here's the text of the recipe with the syntax error:

class AdvancedUserRecipe1330393641(BasicNewsRecipe):
    title          = u'abc'
    oldest_article = 30
    max_articles_per_feed = 100
    auto_cleanup = True
    feeds          = [I took these out for the forum post here to save space]
    def print_version(self, url):
       return url.replace('/article/', '/printarticle/')
    def get_cover_url(self):
        cover_url = None
        soup = self.index_to_soup('http://www.abc.com')
        cov = soup.find('img', src=re.compile(r'\w*?cover\w{1,22}\.jpg')
        if cov is not None:
            self.cover_url = cov['src']

and the error I get is

calibre, version 0.8.41
ERROR: Invalid input: <p>Could not create recipe. Error:<br>invalid syntax (<string>, line 13)

[kovidgoyal]

Missing closing bracket after the soup.find

cov = soup.find('img', src=re.compile(r'\w*?cover\w{1,22}\.jpg'))

[adoucette]

That fixes the syntax error, thanks. But it does not download a cover... I still get the default Calibre cover.

    def get_cover_url(self):
        cover_url = None
        soup = self.index_to_soup('http://www.abc.com')
        cov = soup.find('img', src=re.compile(r'\w*?cover\w{1,22}\.jpg'))
        if cov is not None:
            self.cover_url = cov['src']

Any thoughts on that? I've tried it with the full url also, as

cov = soup.find('img', src=re.compile(r'http\S+?cover\w{1,22}\.jpg'))

and still get the same effect

[kovidgoyal]

print out the value of self.cover_url check that it is correct, for example, it might be relative, or the regex might need to be adjusted.

[adoucette]

I get this:

1% Trying to download cover...
At this point in the code the value of variable cov is:  <img alt="" src="http://media.abc.com/images/2012/01/30/cover0212_227033.jpg" />
At this point in the code the value of variable self.cover_url is:  http://media.abc.com/images/2012/01/30/cover0212_227033.jpg

which is a good image.
So is there something I'm missing as to how to actually assign the image as the cover? Is there some obvious thing I've left out of the end of the recipe?
Again, the whole recipe I have so far is: (with feeds list truncated to save space)

class AdvancedUserRecipe1330393641(BasicNewsRecipe):
    title          = u'abc'
    oldest_article = 30
    max_articles_per_feed = 100
    auto_cleanup = True
    def get_cover_url(self):
        cover_url = None
        soup = self.index_to_soup('http://www.abc.com')
        cov = soup.find('img', src=re.compile(r'\w*?cover\w{1,22}\.jpg'))
        if cov is not None:
            self.cover_url = cov['src']
        print 'At this point in the code the value of variable cov is: ', cov
        print 'At this point in the code the value of variable self.cover_url is: ', self.cover_url
    feeds          = [(...)]
    def print_version(self, url):
       return url.replace('/article/', '/printarticle/')

[kovidgoyal]

Sorry, rather than doing self.url = .. you need return ...

[adoucette]

Thanks Kovid, that did solve the issue. The final code for getting the cover is:

    def get_cover_url(self):
        cover_url = None
        soup = self.index_to_soup('http://www.abc.com')
        cover_item = soup.find('img', src=re.compile(r'\w*?cover\w{1,22}\.jpg'))
        if cover_item:
            cover_url = cover_item['src']
        return cover_url

I'll PM you the final recipe if you want it for Calibre.

[kovidgoyal]

If you think the recipe is of value to others, post it here, I will add it to calibre automatically.

[adoucette]

Originally Posted by kovidgoyal

If you think the recipe is of value to others, post it here, I will add it to calibre automatically.

OK, thanks, I posted it here https://www.mobileread.com/forums/sho...d.php?t=170512 and have edited it with the recent update.