View Single Post
Old 12-09-2012, 01:23 PM   #24
kiklop74
Guru
kiklop74 can program the VCR without an owner's manual.kiklop74 can program the VCR without an owner's manual.kiklop74 can program the VCR without an owner's manual.kiklop74 can program the VCR without an owner's manual.kiklop74 can program the VCR without an owner's manual.kiklop74 can program the VCR without an owner's manual.kiklop74 can program the VCR without an owner's manual.kiklop74 can program the VCR without an owner's manual.kiklop74 can program the VCR without an owner's manual.kiklop74 can program the VCR without an owner's manual.kiklop74 can program the VCR without an owner's manual.
 
kiklop74's Avatar
 
Posts: 780
Karma: 194642
Join Date: Dec 2007
Location: Argentina
Device: Kindle PaperWhite, Motorola Xoom
Sometimes sites can be badly implemented or overloaded so that first fetch of an article fails but second or third passes OK. To add that functionality to the calibre recipe you can use this approach:

Code:
# In the include section add this
from calibre.ptempfile import PersistentTemporaryFile

#later in the recipe class add this
class MyRecipeclass(BasicNewsRecipe):
# ...
    temp_files              = []
    articles_are_obfuscated = True  

# and than somewhere in the class add this method

    def get_obfuscated_article(self, url):
        count = 0
        attempts = 4
        html = None
        while (count < attempts):
            try:
                response = self.browser.open(url)
                html = response.read()
                count = attempts
            except:
                print "Retrying download..."
            count += 1
            
        if html is None:
           pass
           
        tfile = PersistentTemporaryFile('_fa.html')
        tfile.write(html)
        tfile.close()
        self.temp_files.append(tfile)
        
        return tfile.name
Replaces the value of variable attempts to change number of download attempts. This works just fine. That approach was used in Financial Times UK recipe.
kiklop74 is offline   Reply With Quote