Custom recipes (archive, read-only)

[plaid]

Is it possible for Calibre to pull down a mobile site rather than rss feeds? For local Winnipeg Canada news all the rss feeds I can find have only one line of text, hardly worth reading. The Winnipeg Sun has a mobile version that will let me see the whole articles. I did some experimenting and looked at the recipe guides but I've not had any luck. The ultimate destination of the news is my Kobo

http://m.winnipegsun.com/

[rty]

Originally Posted by plaid

Is it possible for Calibre to pull down a mobile site rather than rss feeds? For local Winnipeg Canada news all the rss feeds I can find have only one line of text, hardly worth reading. The Winnipeg Sun has a mobile version that will let me see the whole articles. I did some experimenting and looked at the recipe guides but I've not had any luck. The ultimate destination of the news is my Kobo

http://m.winnipegsun.com/

I took a glance at it and it looks OK. Give me a week.

[plaid]

Originally Posted by rty

I took a glance at it and it looks OK. Give me a week.

Ok, that would be great. I'll try to keep an eye on this thread, thanks!

[rty]

To all the good folks in Winnipeg, here is the recipe for Winnipeg Sun.


@plaid: The RSS feeds are used only to pull down the full artilces.

[plaid]

Originally Posted by rty

To all the good folks in Winnipeg, here is the recipe for Winnipeg Sun.


@plaid: The RSS feeds are used only to pull down the full artilces.

THANKS! it works great.

[nook.life]

Originally Posted by Starson17

I was probably in a grumpy mood that day

Whatever I posted, it wasn't the final recipe, as what you were working with still had lots of junk in it. This is closer to the final I came up with, but my earlier version had some text that identified the comic. You want it rotated, so I removed the text above to give more room for the comic.

Try this:

Spoiler:

from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup
import re
import calibre.utils.PythonMagickWand as pw
import calibre.utils.PythonMagickWand

class Explosm(BasicNewsRecipe):
    title               = 'Explosm'
    __author__          = 'Starson17'
    description         = 'Explosm'
    language            = 'en'
    use_embedded_content= False
    no_stylesheets      = True
    oldest_article      = 24
    remove_javascript   = True
    remove_empty_feeds    = True
    max_articles_per_feed = 10

    feeds = [
             (u'Explosm Feed', u'http://feeds.feedburner.com/Explosm')
             ]

    keep_only_tags     = [dict(name='div', attrs={'align':'center'})]
    remove_tags = [dict(name='span'),
                   dict(name='table')]

    def postprocess_html(self, soup, first):
        #process all the images. assumes that the new html has the correct path
        for tag in soup.findAll(lambda tag: tag.name.lower()=='img' and tag.has_key('src')):
            iurl = tag['src']
            print 'resizing image' + iurl
            with pw.ImageMagick():
                img = pw.NewMagickWand()
                p = pw.NewPixelWand()
                if img < 0:
                    raise RuntimeError('Out of memory')
                if not pw.MagickReadImage(img, iurl):
                    severity = pw.ExceptionType(0)
                    msg = pw.MagickGetException(img, byref(severity))
                    raise IOError('Failed to read image from: %s: %s'
                        %(iurl, msg))
                width = pw.MagickGetImageWidth(img)
                height = pw.MagickGetImageHeight(img)
                if( width > height ) :
                    print 'Rotate image'
                    pw.MagickRotateImage(img, p, 90)
                if not pw.MagickWriteImage(img, iurl):
                    raise RuntimeError('Failed to save image to %s'%iurl)
                pw.DestroyMagickWand(img)
        return soup

    extra_css = '''
                    h1{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;}
                    h2{font-family:Arial,Helvetica,sans-serif; font-weight:normal;font-size:small;}
                    p{font-family:Arial,Helvetica,sans-serif;font-size:small;}
                    body{font-family:Helvetica,Arial,sans-serif;font-size:small;}
		'''

Hey Starson, thanks so much for posting the code. It definitely looks a lot cleaner without the text above and makes it more readable. Unfortunately, it is still clipping it at the top for some of the comics. I looked at the html test version and it still seems to be outputting some sort of table (you can see the outline) even though you removed it. Starting to wonder if my output settings are messed up in calibre. I uninstalled and reinstalled it, but it seems like it keeps the settings (my custom recipes and scheduled were still there after the new install)

here is what it looks like
http://picturepush.com/public/3708883

[Starson17]

Originally Posted by nook.life

Unfortunately, it is still clipping it at the top for some of the comics.

That seems to be the right, not the top. I don't have a Nook, so can't do much testing.

I looked at the html test version and it still seems to be outputting some sort of table (you can see the outline) even though you removed it.

I checked my output - there aren't any tables there. Did you see a table tag in your output?

[nook.life]

Originally Posted by Starson17

That seems to be the right, not the top. I don't have a Nook, so can't do much testing.


I checked my output - there aren't any tables there. Did you see a table tag in your output?

My apologies, I was referring to the top of the cartoon (since it is rotated).

No there are no table tags since you already took it out with the code, but for some reason it is still doing some sort of formatting where it draws a frame and places the image inside it. When I open the html in mozialla you can see the frame.

In the epub viewer within calibre it shows the cartoon clipped on the right like you say, but because it has a scroll bar at the bottom you can scroll and see the rest of the cartoon. The nook does not have such scroll bar so it crops the image as shown by the pic i posted (and deletes the rest?). I just dont know how to get rid of that extra space on the left.

I do not experience this problem with other cartoon feeds like Dilbert or comics.com even though I pasted in the rotate code to rotate them.

[Dereks]

For some reason i can no longer download google reader recipe. Starting from 0.7.x version I get this error:

ERROR: Conversion Error: <b>Failed</b>: Fetch news from Google Reader

Fetch news from Google Reader
Resolved conversion options
calibre version: 0.7.6
{'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0,
'book_producer': None,
'change_justification': 'original',
'chapter': None,
'chapter_mark': 'pagebreak',
'comments': None,
'cover': None,
'debug_pipeline': None,
'disable_font_rescaling': False,
'dont_download_recipe': False,
'enable_autorotation': False,
'extra_css': None,
'font_size_mapping': None,
'footer_regex': '(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s* <a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)' ,
'header': False,
'header_format': '%t by %a',
'header_regex': '(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s* <a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)' ,
'header_separation': 0,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x05189A50>,
'insert_blank_line': False,
'insert_metadata': False,
'isbn': None,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0,
'linearize_tables': False,
'lrf': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'max_toc_links': 50,
'minimum_indent': 0,
'mono_family': None,
'no_chapters_in_toc': False,
'no_inline_navbars': False,
'output_profile': <calibre.customize.profiles.SonyReaderOutput object at 0x05189DF0>,
'page_breaks_before': None,
'password': 'xxxxxxxxxxx',
'prefer_metadata_cover': False,
'preprocess_html': False,
'pretty_print': False,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': None,
'remove_first_image': False,
'remove_footer': False,
'remove_header': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'render_tables_as_images': False,
'sans_family': None,
'series': None,
'series_index': None,
'serif_family': None,
'tags': None,
'test': False,
'text_size_multiplier_for_rendered_tables': 1.0,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'use_auto_toc': False,
'username': 'dereksts',
'verbose': 2,
'wordspace': 2.5}
InputFormatPlugin: Recipe Input running
Python function terminated unexpectedly
(Error Code: 1)
Traceback (most recent call last):
File "site.py", line 103, in main
File "site.py", line 85, in run_entry_point
File "site-packages\calibre\utils\ipc\worker.py", line 99, in main
File "site-packages\calibre\gui2\convert\gui_conversion.py", line 24, in gui_convert
File "site-packages\calibre\ebooks\conversion\plumber.py", line 815, in run
File "site-packages\calibre\customize\conversion.py", line 211, in __call__
File "site-packages\calibre\web\feeds\input.py", line 104, in convert
File "site-packages\calibre\web\feeds\news.py", line 705, in download
File "site-packages\calibre\web\feeds\news.py", line 835, in build_index
File "site-packages\calibre\web\feeds\news.py", line 1280, in parse_feeds
File "c:\users\dereks\appdata\local\temp\calibre_0.7.6_ 1vhut0_recipes\recipe0.py", line 35, in get_feeds
soup = self.index_to_soup('http://www.google.com/reader/api/0/tag/list')
File "site-packages\calibre\web\feeds\news.py", line 474, in index_to_soup
File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_opener.py", line 202, in open
File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_http.py", line 612, in http_response
File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_opener.py", line 225, in error
File "urllib2.py", line 367, in _call_chain
File "site-packages\mechanize-0.1.11-py2.6.egg\mechanize\_http.py", line 633, in http_error_default
urllib2.HTTPError: HTTP Error 401: Unauthorized

[ehsahog]

Hi all,

I just recently discovered Calibre and the feature to get news.
Unfortunately, I'm not very good at Phyton or HTML/CSS so I haven't been successful in my attempts to customize the recipes to get the news I want.

Would it be possible for someone with knowledge to write a recipe for akihabara news (http://en.akihabaranews.com/feed)

Lots of thanks in advance!
/Anders

[Starson17]

Originally Posted by nook.life

No there are no table tags since you already took it out with the code, but for some reason it is still doing some sort of formatting where it draws a frame and places the image inside it. When I open the html in mozialla you can see the frame.... I just dont know how to get rid of that extra space on the left.

I do not experience this problem with other cartoon feeds like Dilbert or comics.com even though I pasted in the rotate code to rotate them.

There is a <div> tag in the original that has a style with a defined width that matches the width of the image before rotation, and a text-align attribute that centers the image in that width.

I don't have a nook, but try this: Paste this into the recipe after the rotate code (after pw.DestroyMagickWand(img)), and inside the postprocess_html code:

                for divtag in soup.findAll('div'):
                   del(divtag['style'])

This deletes the style and should let it put the image to the left.

[sibermage]

I did a search but couldn't find anything regarding the Sing Tao news site.
Is it possible to obtain the news articles for http://news.singtao.ca/toronto/

Thanks.

[elsuave]

Did a recipe for Foreign Policy (http://www.foreignpolicy.com/) ever come out? It's been mentioned a couple of times in this thread, with an unsuccessful attempt here: https://www.mobileread.com/forums/sho...&postcount=616.

If not, would anybody like to try their hand at it? RSS feed is available here: http://www.foreignpolicy.com/node/feed

[Daffy6964]

A recipe request for The Statesman. News for Austin, TX.
http://www.statesman.com/

Thank you!

[rty]

Recipe for the STATESMAN for folks in Austin, Texas!

Note: Articles in the feeds linked from blogs and Austin360.com are ignored.