Custom recipes (archive, read-only) - Page 187

kiklop74 · 09-21-2010, 04:35 PM

New recipe for Slo-Tech:

krunk · 09-21-2010, 05:11 PM

Ok, now I'm attempting to remove duplicate urls that might appear in multiple feeds that I'm aggregating.

I created a list called 'added_links' then overloaded the is_link_wanted method like so:

Code:

    def is_link_wanted(self, url, tag):
        wanted = False

        if url not in self.added_links:
            self.added_links.append(url)
            wanted = True

        return wanted

This seems to accurately catch duplicate urls. If I print out the "added_links" array, each url is only listed once. However, the duplicate articles/urls still appear in final ebook.

Starson17 · 09-21-2010, 05:37 PM

Quote:

Originally Posted by krunk

@Starson17
Many thanks, that's a far more thorough recipe than I was working on.

It shouldn't be hard to pull out the Accept header and see if it fixes your Bad Behavior problem.

Code:

    def get_browser(self):
        br = BasicNewsRecipe.get_browser(self)
        br.addheaders = [('Accept', 'text/html')]
        return br

When I first saw your post, I didn't realize you were working on the Skeptic Blog. That's where I found the Bad Behavior problem, too, so that code should probably fix it.

greenapple · 09-21-2010, 07:12 PM

For teleread blog (one of the built-in recipes), is it possible to incorporate 'comments' made by visitors as part of the download? They are often sharp, witty and fun to read.

bhandarisaurabh · 09-21-2010, 08:12 PM

Quote:

Originally Posted by TonytheBookworm

If your actually trying to modify the built in recipe. I do not see why. I testing it on my end and do not see after running it where any of the articles were not in print version. Also, I ran a test with print statements included and I do not see anywhere where the original url is what you stated of being changed. It appears to follow the flow that the original author of the recipe expected and looked for. In other words, kinda hard to fix something that isn't broken. <shrug>

As far as the indents you have to make sure they are spaced out correctly.

Spoiler:

****notice the return statement is directly under the print_url statement.

it is giving this error
ERROR: Invalid input: <p>Could not create recipe. Error:<br>unindent does not match any outer indentation level (recipe46.py, line 51)

Code:

#!/usr/bin/env  python

__license__   = 'GPL v3'
__copyright__ = '2009, Darko Miletic <darko.miletic at gmail.com>'
'''
www.business-standard.com
'''

from calibre.web.feeds.recipes import BasicNewsRecipe

class BusinessStandard(BasicNewsRecipe):
    title                  = 'Business Standard'
    __author__             = 'Darko Miletic'
    description            = "India's most respected business daily"
    oldest_article         = 7
    max_articles_per_feed  = 100
    no_stylesheets         = True
    use_embedded_content   = False
    encoding               = 'cp1252'
    publisher              = 'Business Standard Limited'
    category               = 'news, business, money, india, world'
    language               = 'en_IN'

    conversion_options = {
                             'comments'        : description
                            ,'tags'            : category
                            ,'language'        : language
                            ,'publisher'       : publisher
                            ,'linearize_tables': True
                         }

    remove_attributes=['style']
    remove_tags = [dict(name=['object','link','script','iframe'])]

    feeds = [
                (u'Todays Newspaper'            , u'http://feeds.business-standard.com/rss/paper.xml'   )
              ,(u'Banking & finance'   , u'http://feeds.business-standard.com/rss/1.xml'   )
              ,(u'Companies & Industry', u'http://feeds.business-standard.com/rss/2.xml')
              ,(u'Economy & Policy'    , u'http://feeds.business-standard.com/rss/3.xml'    )
              ,(u'Opinion and analysis', u'http://feeds.business-standard.com/rss/5_0.xml')
              ,(u'Life & Leisure'      , u'http://feeds.business-standard.com/rss/6_0.xml'      )
              ,(u'Markets & Investing' , u'http://feeds.business-standard.com/rss/12.xml' )
              ,(u'Management & Mktg'   , u'http://feeds.business-standard.com/rss/7_0.xml'   )
              ,(u'Tech World',u'http://feeds.business-standard.com/rss/8_0.xml')
            
            ]

    def print_version(self, url):
        print 'ORG URL IS: ', url
        split1 = url.split("/")
        print 'THE SPLIT IS: ', split1 
        id = len(split1)
        # we want to find the size of the array split 
        # because we know the id will always be in the last index
        


        print_url = ‘http://www.business-standard.com/india/printpage.php?autono=’ + split1[id]+ ‘&tp=’
        return print_url
       
        
      
    def get_article_url(self, article):
        return article.get('guid',  None)

actually I am using different feeds as compared to inbuilt recipe
the rss feeds page is
http://feeds.business-standard.com/

TonytheBookworm · 09-21-2010, 09:37 PM

Quote:

Originally Posted by bhandarisaurabh

it is giving this error
ERROR: Invalid input: <p>Could not create recipe. Error:<br>unindent does not match any outer indentation level (recipe46.py, line 51)

actually I am using different feeds as compared to inbuilt recipe
the rss feeds page is
http://feeds.business-standard.com/

alright, first lets not piggyback but yet make our own version since the feeds are different and all. With that being said, I had to test the code to get it correct because the index on the split is 0 based. and also the very last index was blank so even though lets say the length of the split array was 8 then the id would be in the 6th position. so i just idnum = len(split1) -2
anyway this code works.

Spoiler:

Code:

from calibre.web.feeds.news import BasicNewsRecipe
from calibre.ebooks.BeautifulSoup import BeautifulSoup, re
class AdvancedUserRecipe1282101454(BasicNewsRecipe):
    title = 'Business Standard modified'
    language = 'en'
    __author__ = 'TonytheBookworm'
    description = 'Business Standard modified'
    publisher = 'Business Standard'
    category = ''
    oldest_article = 5
    max_articles_per_feed = 100
    no_stylesheets = True
    #extra_css = '.headline {font-size: x-large;} \n .fact { padding-top: 10pt }'
    #masthead_url = 'http://gawand.org/wp-content/uploads/2010/06/ajc-logo.gif'
    #keep_only_tags    = [
     #                    dict(name='div', attrs={'class':['blogEntryHeader','blogEntryContent']})
      #                 ,dict(attrs={'id':['cxArticleText','cxArticleBodyText']})
      #                  ]
    feeds = [
             (u'Todays Newspaper',u'http://feeds.business-standard.com/rss/paper.xml'),
             (u'Banking & finance',u'http://feeds.business-standard.com/rss/1.xml'),
             (u'Companies & Industry', u'http://feeds.business-standard.com/rss/2.xml'),
             (u'Economy & Policy'    , u'http://feeds.business-standard.com/rss/3.xml'),
             (u'Opinion and analysis', u'http://feeds.business-standard.com/rss/5_0.xml'),
             (u'Life & Leisure'      , u'http://feeds.business-standard.com/rss/6_0.xml'),
             (u'Markets & Investing' , u'http://feeds.business-standard.com/rss/12.xml'),
             (u'Management & Mktg'   , u'http://feeds.business-standard.com/rss/7_0.xml'),
             (u'Tech World',u'http://feeds.business-standard.com/rss/8_0.xml'),
            ]
    def print_version(self, url):
        split1 = url.split("/")
        print 'ORG URL IS: ', url
        id = len(split1)-2 # had to offset it by 2 because it is 0 based and also the last index is blank 
        idnum = split1[id] # get the actual value of the id article
        print 'the idnum is: ', idnum
        print_url = 'http://www.business-standard.com/india/printpage.php?autono=' + idnum + '&tp='
        print 'PRINT URL IS: ', print_url
        return print_url

N13L5 · 09-22-2010, 03:00 AM

would love the RSS feed from

http://www.globalpropertyguide.com

http://www.Wired.com

holy cow! 187 pages to this thread, did not see that!!

god knows the wired recipe is probably somewhere in there already

t3d · 09-22-2010, 06:20 AM

Here are 3 new recipes for polish feeds:
http://github.com/t3d/kalibrator/raw...24_ESKN.recipe
http://github.com/t3d/kalibrator/raw...4_fakty.recipe
http://github.com/t3d/kalibrator/raw...omorska.recipe

The favicon for the first two is here:
http://www.rmf24.pl/favicon.ico

And for the last one:
http://www.pomorska.pl/favicon.ico

I'd also like to see my recipe nczas.recipe removed from calibre, since its feed is broken for monhts.

Moreover here are some favicon that weren't added to calibre yet:
fronda.recipe:
http://fronda.pl/favicon.ico
legeartis.recipe:
http://olgierd.bblog.pl/s/favicon.ico
michalkiewicz.recipe:
http://michalkiewicz.pl/favicon.ico

TonytheBookworm · 09-22-2010, 06:22 AM

Quote:

Originally Posted by N13L5

would love the RSS feed from

http://www.globalpropertyguide.com

http://www.Wired.com

holy cow! 187 pages to this thread, did not see that!!

god knows the wired recipe is probably somewhere in there already

wired is in the built in recipes in calibre already both the daily edition, the magazine and the uk version

and here is globalproperty

Spoiler:

eldan · 09-22-2010, 08:15 AM

Hi,

first of all I have to thank you guys for your awesome work. Calibre, a Kindle and your recipes are giving me a real living the future feeling.

There is one thing I would like to have that is not working for me: It's the Google Reader recipe. As soon as I start to fetch the news from this recipe I'll get this error:

Spoiler:

I'm using my personal domain to login to google reader. e.g. myname@mydomain.com. Could that be the problem for the 403 error? I would like to debug the recipe but I could not find it. Maybe someone could tell me the name of the recipe or give me some hints how to solve the problem. Is there any public repository and/or bug tracking system for all the recipe?

Thanks a lot.

Starson17 · 09-22-2010, 08:54 AM

Quote:

Originally Posted by eldan

Hi,

first of all I have to thank you guys for your awesome work. Calibre, a Kindle and your recipes are giving me a real living the future feeling.

There is one thing I would like to have that is not working for me: It's the Google Reader recipe. As soon as I start to fetch the news from this recipe I'll get this error:

Code:

    response = br.open('https://www.google.com/accounts/ClientLogin', request)
 mechanize._response.httperror_seek_wrapper: HTTP Error 403: Forbidden

I'm using my personal domain to login to google reader. e.g. myname@mydomain.com. Could that be the problem for the 403 error? I would like to debug the recipe but I could not find it. Maybe someone could tell me the name of the recipe or give me some hints how to solve the problem. Is there any public repository and/or bug tracking system for all the recipe?

Thanks a lot.

1) That error indicates your login credentials are not being sent correctly. Can you login with those credentials manually?

Go here and put in your email and password to test:
https://www.google.com/accounts/ServiceLogin

2) The name of the recipe is the same as the name you used to select it for normal use/download. Choose the pull down arrow next to News and select "Add a custom news source" then "Customize builtin recipe" then choose the name to add it to the custom list, then select it in the list to see the code.

3) Yes, you can report bugs, (http://bugs.calibre-ebook.com) but don't do it until you're sure you have the right login credentials (I wrote the current login/authentication for the Google Reader recipe, but no other parts) If it's broken, I'll fix it. I don't use that recipe normally.

N13L5 · 09-22-2010, 09:17 AM

Quote:

Originally Posted by TonytheBookworm

wired is in the built in recipes in calibre already both the daily edition, the magazine and the uk version

and here is globalproperty

TonytheBookworm · 09-22-2010, 11:02 AM

Could someone do me a favor and shed some light my way in regard to the extra_css ? Basically what is the difference between #bd and .bd and so forth?
I took and looked at some of the recipes and used firebug to try and find in the html where #bd is or whatever and i can't seem to find it.

Let say for instance i had
<div class='helloworld'>What a beautiful day it is. </div>
and i wanted it to be bold or what have you.
why doesn't this work ?

Spoiler:

thanks in advance

Starson17 · 09-22-2010, 11:42 AM

Quote:

Originally Posted by TonytheBookworm

Could someone do me a favor and shed some light my way in regard to the extra_css ? Basically what is the difference between #bd and .bd and so forth?
I took and looked at some of the recipes and used firebug to try and find in the html where #bd is or whatever and i can't seem to find it.

Let say for instance i had
<div class='helloworld'>What a beautiful day it is. </div>
and i wanted it to be bold or what have you.
why doesn't this work ?

Spoiler:

thanks in advance

Possibly:
<div class='helloworld'>
vs spelling of:
extra_css = ''' .helloword

??????

You probably know this, but just in case, the "difference between #bd and .bd" is that the first is for id="bd" and the second for class="bd". Usually, a class is for multiple elements on a page, while id is for individual elements, although that general rule is observed mostly in the breach.

TonytheBookworm · 09-22-2010, 11:51 AM

Quote:

Originally Posted by Starson17

Possibly:
<div class='helloworld'>
vs spelling of:
extra_css = ''' .helloword

??????

Yeah, sorry about the typo but i tried that and still did not work.
I have noticed that in some cases the author will use a .headline
or simply type h1 or even #div
I was just hoping there was some documentation on it somewhere other than the general overview in the calibre docs. I flat out stink with css but I feel if i could get the syntax of the extra_css then I should be able to format some of the things more to my liking.

And a side note. Man I have noticed after learning from you. I have been making recipes left and right and all for myself and for others. Never thought I would be doing that but none the less thanks for teaching me.

09-21-2010, 05:11 PM	#2792
krunk Member Posts: 19 Karma: 10 Join Date: Feb 2010 Location: Los Angeles, CA Device: Kindle 3	Ok, now I'm attempting to remove duplicate urls that might appear in multiple feeds that I'm aggregating. I created a list called 'added_links' then overloaded the is_link_wanted method like so: Code: def is_link_wanted(self, url, tag): wanted = False if url not in self.added_links: self.added_links.append(url) wanted = True return wanted This seems to accurately catch duplicate urls. If I print out the "added_links" array, each url is only listed once. However, the duplicate articles/urls still appear in final ebook.

09-22-2010, 03:00 AM	#2797
N13L5 tenjooberrymuds Posts: 58 Karma: 12 Join Date: Sep 2010 Device: Android	would love the RSS feed from http://www.globalpropertyguide.com http://www.Wired.com holy cow! 187 pages to this thread, did not see that!! god knows the wired recipe is probably somewhere in there already Last edited by N13L5; 09-22-2010 at 03:07 AM. Reason: 187!

09-22-2010, 08:15 AM	#2800
eldan Junior Member Posts: 2 Karma: 10 Join Date: Sep 2010 Device: Kindle 3G	Hi, first of all I have to thank you guys for your awesome work. Calibre, a Kindle and your recipes are giving me a real living the future feeling. There is one thing I would like to have that is not working for me: It's the Google Reader recipe. As soon as I start to fetch the news from this recipe I'll get this error: Spoiler: Fetch news from Google Reader Resolved conversion options calibre version: 0.7.19 {'asciiize': False, 'author_sort': None, 'authors': None, 'base_font_size': 0, 'book_producer': None, 'change_justification': 'original', 'chapter': None, 'chapter_mark': 'pagebreak', 'comments': None, 'cover': None, 'debug_pipeline': None, 'disable_font_rescaling': False, 'dont_compress': False, 'dont_download_recipe': False, 'extra_css': None, 'font_size_mapping': None, 'footer_regex': '(?i)(?<=<hr>)((\\s<a name=\\d+></a>((<img.+?>)<br>\\s)?\\d+<br>\\s.?\\s)\|(\\s* <a name=\\d+></a>((<img.+?>)<br>\\s)?.?<br>\\s\\d+))(?=<br>)' , 'header_regex': '(?i)(?<=<hr>)((\\s<a name=\\d+></a>((<img.+?>)<br>\\s)?\\d+<br>\\s.?\\s)\|(\\s* <a name=\\d+></a>((<img.+?>)<br>\\s)?.?<br>\\s\\d+))(?=<br>)' , 'html_unwrap_factor': 0.40000000000000002, 'input_encoding': None, 'input_profile': <calibre.customize.profiles.InputProfile object at 0x67b1650>, 'insert_blank_line': False, 'insert_metadata': False, 'isbn': None, 'keep_ligatures': False, 'language': None, 'level1_toc': None, 'level2_toc': None, 'level3_toc': None, 'line_height': 0, 'linearize_tables': False, 'lrf': False, 'margin_bottom': 5.0, 'margin_left': 5.0, 'margin_right': 5.0, 'margin_top': 5.0, 'max_toc_links': 50, 'no_chapters_in_toc': False, 'no_inline_navbars': False, 'no_inline_toc': False, 'output_profile': <calibre.customize.profiles.SonyReader300Output object at 0x67b1a10>, 'page_breaks_before': None, 'password': '@google895@', 'personal_doc': '[PDOC]', 'prefer_author_sort': False, 'prefer_metadata_cover': False, 'preprocess_html': False, 'pretty_print': False, 'pubdate': None, 'publisher': None, 'rating': None, 'read_metadata_from_opf': None, 'remove_first_image': False, 'remove_footer': False, 'remove_header': False, 'remove_paragraph_spacing': False, 'remove_paragraph_spacing_indent_size': 1.5, 'rescale_images': False, 'series': None, 'series_index': None, 'smarten_punctuation': False, 'tags': None, 'test': False, 'timestamp': None, 'title': None, 'title_sort': None, 'toc_filter': None, 'toc_threshold': 6, 'toc_title': None, 'use_auto_toc': False, 'username': 'dan@imojo.de', 'verbose': 2} Python function terminated unexpectedly: InputFormatPlugin: Recipe Input running Traceback (most recent call last): File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.6/site.py", line 147, in main return run_entry_point() File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.6/site.py", line 116, in run_entry_point return getattr(pmod, func)() File "site-packages/calibre/utils/ipc/worker.py", line 99, in main File "site-packages/calibre/gui2/convert/gui_conversion.py", line 24, in gui_convert File "site-packages/calibre/ebooks/conversion/plumber.py", line 832, in run File "site-packages/calibre/customize/conversion.py", line 211, in __call__ File "site-packages/calibre/web/feeds/input.py", line 101, in convert File "site-packages/calibre/web/feeds/news.py", line 613, in __init__ File "/var/folders/64/64aSvmYRGCK0iuJhF0Ppyk+++TI/-Tmp-/calibre_0.7.19_tmp_lvdacv/calibre_0.7.19_vhAJsX_recipes/recipe0.py", line 24, in get_browser response = br.open('https://www.google.com/accounts/ClientLogin', request) File "site-packages/mechanize/_mechanize.py", line 209, in open File "site-packages/mechanize/_mechanize.py", line 261, in _mech_open mechanize._response.httperror_seek_wrapper: HTTP Error 403: Forbidden I'm using my personal domain to login to google reader. e.g. myname@mydomain.com. Could that be the problem for the 403 error? I would like to debug the recipe but I could not find it. Maybe someone could tell me the name of the recipe or give me some hints how to solve the problem. Is there any public repository and/or bug tracking system for all the recipe? Thanks a lot. Last edited by eldan; 09-22-2010 at 08:18 AM.

09-22-2010, 11:02 AM	#2803
TonytheBookworm Addict Posts: 264 Karma: 62 Join Date: May 2010 Device: kindle 2, kindle 3, Kindle fire	Could someone do me a favor and shed some light my way in regard to the extra_css ? Basically what is the difference between #bd and .bd and so forth? I took and looked at some of the recipes and used firebug to try and find in the html where #bd is or whatever and i can't seem to find it. Let say for instance i had <div class='helloworld'>What a beautiful day it is. </div> and i wanted it to be bold or what have you. why doesn't this work ? Spoiler: Code: extra_css = ''' .helloword{font-family:Arial,Helvetica,sans-serif; font-weight:bold;font-size:large;} ''' thanks in advance

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Custom column read ?	pchrist7	Calibre	2	10-04-2010 02:52 AM
Archive for custom screensavers	sleeplessdave	Amazon Kindle	1	07-07-2010 12:33 PM
How to back up preferences and custom recipes?	greenapple	Calibre	3	03-29-2010 05:08 AM
Donations for Custom Recipes	ddavtian	Calibre	5	01-23-2010 04:54 PM
Help understanding custom recipes	andersent	Calibre	0	12-17-2009 02:37 PM

09-21-2010, 07:12 PM	#2794
greenapple Evangelist Posts: 404 Karma: 664 Join Date: Dec 2009 Device: Kindle Paperwhite, Kindle DX, Kobo Aura HD	For teleread blog (one of the built-in recipes), is it possible to incorporate 'comments' made by visitors as part of the download? They are often sharp, witty and fun to read.

09-22-2010, 06:20 AM	#2798
t3d Enthusiast Posts: 38 Karma: 10 Join Date: Nov 2009 Location: Poland Device: kindle 1st gen, kindle dxg, kindle paperwhite2	Here are 3 new recipes for polish feeds: http://github.com/t3d/kalibrator/raw...24_ESKN.recipe http://github.com/t3d/kalibrator/raw...4_fakty.recipe http://github.com/t3d/kalibrator/raw...omorska.recipe The favicon for the first two is here: http://www.rmf24.pl/favicon.ico And for the last one: http://www.pomorska.pl/favicon.ico I'd also like to see my recipe nczas.recipe removed from calibre, since its feed is broken for monhts. Moreover here are some favicon that weren't added to calibre yet: fronda.recipe: http://fronda.pl/favicon.ico legeartis.recipe: http://olgierd.bblog.pl/s/favicon.ico michalkiewicz.recipe: http://michalkiewicz.pl/favicon.ico

Advert

Advert