Custom recipes (archive, read-only)

[PaxtonReader]

GRiker - yes, this is what I have done - one day after another, not the same day. And you are correct - doing it multiple times with the same timestamp will do a replace. What has been happening to me with the USB transfer (Send to device, main memory) is that yesterday's issue gets overlaid with today's, and nothing gets put in the archive. If I select each previous day's issue and say send it via email, the archiving takes place fine, even if I do them one after the other on the same day. I won't bother you further on this, but I am trying to determine what is different between the email transfer vs. the USB transfer (at least for me - I guess it works for others?).

[GRiker]

PaxtonReader: I do exactly what you describe every day with multiple recipes, downloading via USB, and archiving works as expected for me. I will try your recipe again tomorrow to see what happens in my environment.

G

[square4761]

I am using the wsj (not the free one) here is the error that I got:
(username removed) any help would be greatly appreciated

ERROR: Conversion Error: <b>Failed</b>: Fetch news from The Wall Street Journal (US)

Fetch news from The Wall Street Journal (US)
Resolved conversion options
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'dont_justify': False,
'dont_split_on_page_breaks': True,
'extra_css': None,
'extract_to': None,
'flow_size': 260,
'font_size_mapping': None,
'footer_regex': '(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s* <a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)' ,
'header_regex': '(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s* <a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)' ,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x639bad0>,
'insert_blank_line': False,
'insert_metadata': False,
'isbn': None,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'max_toc_links': 50,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'output_profile': <calibre.customize.profiles.SonyReaderOutput object at 0x639bcd0>,
'page_breaks_before': None,
'password': 'b1i2t3e4m5e6',
'prefer_metadata_cover': False,
'preprocess_html': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_first_image': False,
'remove_footer': False,
'remove_header': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'series': None,
'series_index': None,
'tags': None,
'test': False,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'use_auto_toc': False,
'username': '',
'verbose': 2}
Python function terminated unexpectedly:
InputFormatPlugin: Recipe Input running
Traceback (most recent call last):
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.6/site.py", line 147, in main
return run_entry_point()
File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.6/site.py", line 116, in run_entry_point
return getattr(pmod, func)()
File "site-packages/calibre/utils/ipc/worker.py", line 99, in main
File "site-packages/calibre/gui2/convert/gui_conversion.py", line 24, in gui_convert
File "site-packages/calibre/ebooks/conversion/plumber.py", line 787, in run
File "site-packages/calibre/customize/conversion.py", line 211, in __call__
File "site-packages/calibre/web/feeds/input.py", line 100, in convert
File "site-packages/calibre/web/feeds/news.py", line 567, in __init__
File "/var/folders/zp/zpekbvZJErKlkluKwdjwyE+++TI/-Tmp-/calibre_0.6.47_RDtr6v_recipes/recipe0.py", line 56, in get_browser
res = br.submit()
File "site-packages/mechanize/_mechanize.py", line 547, in submit
File "site-packages/mechanize/_mechanize.py", line 209, in open
File "site-packages/mechanize/_mechanize.py", line 261, in _mech_open
mechanize._response.httperror_seek_wrapper: HTTP Error 400: Bad Request

[kovidgoyal]

do your username or password have special characters?

[jpalmado]

Hi! I would like to have this rss feeds in spanish, but i get an error in the download and i don't know why. Could somebody help me? These are the channels I use in the basic recipe. thank you:

http://www.nationalgeographic.com.es/feeds/rss.html

http://feeds.feedburner.com/ElLabora...win?format=xml

[square4761]

no just letters and numbers

[square4761]

also no changes of username or password within the last month and it worked until 2 days ago

[kovidgoyal]

It may be a temporary error with their website. Can someone else who uses WSJ comment if it is working for them?

[GRiker]

Originally Posted by PaxtonReader

I have a recipe which I use to download a daily paper (code for the main options shown below). I've been told that version 6.46 will archive older issues as new ones are delivered, whether or not they are emailed to the device or transferred via USB cable. I am not finding that to be the case. It will archive the issues if I email them to my Kindle, but if I transfer via the cable, the current issue just overlays yesterday's, and nothing goes to the archive. What am I doing wrong?

I think I may have a lead on your problem with your periodicals not being archived properly. I'm discussing it with Kovid and will report back with a solution.

G

[mhdamoody]

Hi,

I would love to have the below Online Newspapers added to the Calibre Fetch News feature.

http://www.nation.co.ke/
http://www.standardmedia.co.ke/

Plus: Any link on how to add that to my Calibre application. Thanks.

[RafaelBastos]

Hello everyone,

I would like to know if it is possible to create a recipe for another brazilian newspaper called Folha de Sao Paulo: http://www.folha.com.br/fsp

I'm missing the most read newspaper in Brazil.

I can provide login and password to the website if needed.

Many thanks from a great fan of you work

[hansd]

My first published custom recipe.

Upd: Better to use the provided zip file with an updated version (see later)

__license__   = 'GPL v3'
__copyright__ = '2010, Hans Donner <hans.donner at pobox.com>'
'''
www.standardmedia.co.ke
'''

import os
from contextlib import nested, closing
from calibre import strftime, __appname__, __version__
import calibre.utils.PythonMagickWand as pw
from ctypes import byref
from calibre import fit_image

class StandardMediaKeRecipe(BasicNewsRecipe):
    
    __author__ = 'Hans Donner'

    title = u'The Standard'
    description = 'News from Kenia'
    language = 'en'
    country = 'KE'
    publication_type = 'newspaper' 
    publisher = 'standardmedia.co.ke'
    category = 'news, politics, Kenia'

    cover_img_url = 'http://www.standardmedia.co.ke/images/easLogoOther.gif'
    masthead_url = cover_img_url

    max_articles_per_feed = 200
    oldest_article = 3

    use_embedded_content = False
    remove_empty_feeds = True

    no_stylesheets = False

    feeds = [(u'Headlines', u'http://www.standardmedia.co.ke/rss/headlines.php'),
             (u'Business', u'http://www.standardmedia.co.ke/rss/business.php'),
             (u'Politics', u'http://www.standardmedia.co.ke/rss/politics.php'),
             (u'Editorial', u'http://www.standardmedia.co.ke/rss/editorial.php'),
             (u'Columnists', u'http://www.standardmedia.co.ke/rss/columnists.php'),
             (u'Sports', u'http://www.standardmedia.co.ke/rss/sports.php'),
             (u'Entertainment', u'http://www.standardmedia.co.ke/rss/entertain.php')]

    conversion_options = {
                          'comment'   : description
                        , 'tags'      : category
                        , 'publisher' : publisher
                        , 'language'  : language
                        }


    def print_version(self, url):
        import re
        p = re.compile('http://www.standardmedia.co.ke/.*InsidePage.php')
        return p.sub('http://www.standardmedia.co.ke/print.php', url)

    def preprocess_html(self, soup):
        return self.adeify_images(soup)

    def get_cover_img_url(self):
        return getattr(self, 'cover_img_url', None)

    def _download_cover_img(self):
        self.report_progress(1, _('Try downloading cover img %s') % self.get_cover_img_url())

        self.cover_img_path = None
        try:
            cu = self.get_cover_img_url()
        except Exception, err:
            cu = None
            self.log.error(_('Could not download cover: %s')%str(err))
            self.log.debug(traceback.format_exc())
        if cu is not None:
            ext = cu.split('/')[-1].rpartition('.')[-1]
            if '?' in ext:
                ext = ''
            ext = ext.lower() if ext and '/' not in ext else 'jpg'
            cpath = os.path.join(self.output_dir, 'cover_img_tmp.'+ext)
            if os.access(cu, os.R_OK):
                with open(cpath, 'wb') as cfile:
                    cfile.write(open(cu, 'rb').read())
            else:
                self.report_progress(1, _('Downloading cover from %s')%cu)
                with nested(open(cpath, 'wb'), closing(self.browser.open(cu))) as (cfile, r):
                    cfile.write(r.read())
            if ext.lower() == 'pdf':
                from calibre.ebooks.metadata.pdf import get_metadata
                stream = open(cpath, 'rb')
                mi = get_metadata(stream)
                cpath = None
                if mi.cover_data and mi.cover_data[1]:
                    cpath = os.path.join(self.output_dir,
                            'cover_img_tmp.'+mi.cover_data[0])
                    with open(cpath, 'wb') as f:
                        f.write(mi.cover_data[1])
            outfile = os.path.join(self.output_dir, 'cover_img.jpg')
            self.prepare_masthead_image(cpath, outfile) 
            self.cover_img_path = cpath

    def download_cover_img(self):
        try:
            self._download_cover_img()
            self.report_progress(1, _('Downloaded cover to %s') % self.cover_img_path)
        except:
            self.log.exception('Failed to download cover img')
            self.cover_path = None

    
    def prepare_cover_image(self, path_to_image, out_path):
        with pw.ImageMagick():
            img = pw.NewMagickWand()
            if img < 0:
                raise RuntimeError('Out of memory')
            if not pw.MagickReadImage(img, path_to_image):
                severity = pw.ExceptionType(0)
                msg = pw.MagickGetException(img, byref(severity))
                raise IOError('Failed to read image from: %s: %s'
                        %(path_to_image, msg))
            if not pw.MagickWriteImage(img, out_path):
                raise RuntimeError('Failed to save image to %s'%out_path)
            pw.DestroyMagickWand(img)


    def default_cover(self, cover_file):
        '''
        Create a generic cover for recipes that have a special cover img
        '''
        try:
            try:
                from PIL import Image, ImageDraw, ImageFont
                Image, ImageDraw, ImageFont
            except ImportError:
                import Image, ImageDraw, ImageFont
            font_path = P('fonts/liberation/LiberationSerif-Bold.ttf')
            title = self.title if isinstance(self.title, unicode) else \
                    self.title.decode(preferred_encoding, 'replace')
            date = strftime(self.timefmt)
            app = '['+__appname__ +' '+__version__+']'

            COVER_WIDTH, COVER_HEIGHT = 590, 750
            img = Image.new('RGB', (COVER_WIDTH, COVER_HEIGHT), 'white')
            draw = ImageDraw.Draw(img)
            # Title
            font = ImageFont.truetype(font_path, 44)
            width, height = draw.textsize(title, font=font)
            left = max(int((COVER_WIDTH - width)/2.), 0)
            top = 15
            draw.text((left, top), title, fill=(0,0,0), font=font)
            bottom = top + height
            # Date
            font = ImageFont.truetype(font_path, 32)
            width, height = draw.textsize(date, font=font)
            left = max(int((COVER_WIDTH - width)/2.), 0)
            draw.text((left, bottom+15), date, fill=(0,0,0), font=font)
            # Vanity
            font = ImageFont.truetype(font_path, 28)
            width, height = draw.textsize(app, font=font)
            left = max(int((COVER_WIDTH - width)/2.), 0)
            top = COVER_HEIGHT - height - 15
            draw.text((left, top), app, fill=(0,0,0), font=font)

            # Logo
            logo_file = I('library.png')
            self.download_cover_img()
            if getattr(self, 'cover_img_path', None) is not None:
                logo_file = self.cover_img_path
            self.report_progress(1, _('using cover img from %s') % logo_file)
            logo = Image.open(logo_file, 'r')
            width, height = logo.size
            left = max(int((COVER_WIDTH - width)/2.), 0)
            top = max(int((COVER_HEIGHT - height)/2.), 0)
            img.paste(logo, (left, top))
            img = img.convert('RGB').convert('P', palette=Image.ADAPTIVE)
            img.convert('RGB').save(cover_file, 'JPEG')
            cover_file.flush()
        except Exception, e:
            self.log.exception('Failed to generate default cover ', e)
            return False
        return True

[PaxtonReader]

Originally Posted by GRiker

I think I may have a lead on your problem with your periodicals not being archived properly. I'm discussing it with Kovid and will report back with a solution.

G

G and Kovid - I appreciate the time you take with people's small problems. You have written a great program and offer terrific support.

[kiklop74]

Revista El Pais (weekly suplement for El pais):

[hansd]

standard is the same as the previous, but now as a more convenient zip file.