Custom recipes (archive, read-only)

[TonytheBookworm]

Originally Posted by Starson17

Exactly correct.
So you want all the secrets eh? I quote: "first_fetch – True if this is the first page of an article."

Ha Knowing a lot of the secrets prevent me from posting a gazillion questions on here. But yeah I like knowing the secrets and thanks again.

[cynvision]

Everyone here has been so helpful with getting one of my recipes to work. However, I've taken a try at using the command line tester and in Windows I cannot get it to process.
The command line also isn't placing the output in the folder I request, it's rolling that out into my Windows docs on F:, but that could just be my inexpert use of the cmd window. What is bothering me is I sort of get the command line to run in cmd but it stops and says it's looking for the in-built recipe and fails to find it. I thought the point was there didn't have to be one there yet if I was using Notepad++ to edit??? So then the test quits.

ebook-convert F:\Program_Files\Calibre2\Bluffton_Today.recipe F:\Program_Files\Calibre2\blufftonoutput\ --test -vv > bluffton.txt

The recipe is over in the Calibre2 folder on F: but the output is creating a bluffton.txt over in F:\Documents_and_Settings\Cynthia_Clavey
And I've tried a few different variations. Maybe this is a Window's question more than a recipe question, but I have included the errors below from the txt file it generates. Maybe a Windows pro can set me straight? Thanks!

Spoiler:

Resolved conversion options
calibre version: 0.7.17
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'disable_font_rescaling': False,
 'dont_download_recipe': False,
 'extra_css': None,
 'font_size_mapping': None,
 'footer_regex': '(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)',
 'header_regex': '(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)',
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x03D42170>,
 'insert_blank_line': False,
 'insert_metadata': False,
 'isbn': None,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'max_toc_links': 50,
 'no_chapters_in_toc': False,
 'no_inline_navbars': False,
 'output_profile': <calibre.customize.profiles.OutputProfile object at 0x03D42350>,
 'page_breaks_before': None,
 'password': None,
 'prefer_metadata_cover': False,
 'preprocess_html': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_first_image': False,
 'remove_footer': False,
 'remove_header': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'series': None,
 'series_index': None,
 'tags': None,
 'test': True,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'use_auto_toc': False,
 'username': None,
 'verbose': 2}
1% Converting input to HTML...
InputFormatPlugin: Recipe Input running
Failed to compile downloaded recipe. Falling back to builtin one
Traceback (most recent call last):
  File "site-packages\calibre\web\feeds\input.py", line 77, in convert
  File "site-packages\calibre\web\feeds\recipes\__init__.py", line 33, in compile_recipe
TypeError: 'NoneType' object is unsubscriptable

Python function terminated unexpectedly
  Failed to find builtin recipe: Bluffton_Today (Error Code: 1)

And this is the last spit out of the CMD window:

Traceback (most recent call last):
  File "site.py", line 103, in main
  File "site.py", line 85, in run_entry_point
  File "site-packages\calibre\ebooks\conversion\cli.py", line 254, in main
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 815, in run
  File "site-packages\calibre\customize\conversion.py", line 211, in __call__
  File "site-packages\calibre\web\feeds\input.py", line 92, in convert
ValueError: Failed to find builtin recipe: Bluffton_Today

[TonytheBookworm]

Originally Posted by cynvision

Everyone here has been so helpful with getting one of my recipes to work. However, I've taken a try at using the command line tester and in Windows I cannot get it to process.
The command line also isn't placing the output in the folder I request, it's rolling that out into my Windows docs on F:, but that could just be my inexpert use of the cmd window. What is bothering me is I sort of get the command line to run in cmd but it stops and says it's looking for the in-built recipe and fails to find it. I thought the point was there didn't have to be one there yet if I was using Notepad++ to edit??? So then the test quits.

ebook-convert F:\Program_Files\Calibre2\Bluffton_Today.recipe F:\Program_Files\Calibre2\blufftonoutput\ --test -vv > bluffton.txt

The recipe is over in the Calibre2 folder on F: but the output is creating a bluffton.txt over in F:\Documents_and_Settings\Cynthia_Clavey
And I've tried a few different variations. Maybe this is a Window's question more than a recipe question, but I have included the errors below from the txt file it generates. Maybe a Windows pro can set me straight? Thanks!

Spoiler:

Resolved conversion options
calibre version: 0.7.17
{'asciiize': False,
 'author_sort': None,
 'authors': None,
 'base_font_size': 0,
 'book_producer': None,
 'change_justification': 'original',
 'chapter': None,
 'chapter_mark': 'pagebreak',
 'comments': None,
 'cover': None,
 'debug_pipeline': None,
 'disable_font_rescaling': False,
 'dont_download_recipe': False,
 'extra_css': None,
 'font_size_mapping': None,
 'footer_regex': '(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)',
 'header_regex': '(?i)(?<=<hr>)((\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?\\d+<br>\\s*.*?\\s*)|(\\s*<a name=\\d+></a>((<img.+?>)*<br>\\s*)?.*?<br>\\s*\\d+))(?=<br>)',
 'input_encoding': None,
 'input_profile': <calibre.customize.profiles.InputProfile object at 0x03D42170>,
 'insert_blank_line': False,
 'insert_metadata': False,
 'isbn': None,
 'keep_ligatures': False,
 'language': None,
 'level1_toc': None,
 'level2_toc': None,
 'level3_toc': None,
 'line_height': 0,
 'linearize_tables': False,
 'lrf': False,
 'margin_bottom': 5.0,
 'margin_left': 5.0,
 'margin_right': 5.0,
 'margin_top': 5.0,
 'max_toc_links': 50,
 'no_chapters_in_toc': False,
 'no_inline_navbars': False,
 'output_profile': <calibre.customize.profiles.OutputProfile object at 0x03D42350>,
 'page_breaks_before': None,
 'password': None,
 'prefer_metadata_cover': False,
 'preprocess_html': False,
 'pretty_print': True,
 'pubdate': None,
 'publisher': None,
 'rating': None,
 'read_metadata_from_opf': None,
 'remove_first_image': False,
 'remove_footer': False,
 'remove_header': False,
 'remove_paragraph_spacing': False,
 'remove_paragraph_spacing_indent_size': 1.5,
 'series': None,
 'series_index': None,
 'tags': None,
 'test': True,
 'timestamp': None,
 'title': None,
 'title_sort': None,
 'toc_filter': None,
 'toc_threshold': 6,
 'use_auto_toc': False,
 'username': None,
 'verbose': 2}
1% Converting input to HTML...
InputFormatPlugin: Recipe Input running
Failed to compile downloaded recipe. Falling back to builtin one
Traceback (most recent call last):
  File "site-packages\calibre\web\feeds\input.py", line 77, in convert
  File "site-packages\calibre\web\feeds\recipes\__init__.py", line 33, in compile_recipe
TypeError: 'NoneType' object is unsubscriptable

Python function terminated unexpectedly
  Failed to find builtin recipe: Bluffton_Today (Error Code: 1)

And this is the last spit out of the CMD window:

Traceback (most recent call last):
  File "site.py", line 103, in main
  File "site.py", line 85, in run_entry_point
  File "site-packages\calibre\ebooks\conversion\cli.py", line 254, in main
  File "site-packages\calibre\ebooks\conversion\plumber.py", line 815, in run
  File "site-packages\calibre\customize\conversion.py", line 211, in __call__
  File "site-packages\calibre\web\feeds\input.py", line 92, in convert
ValueError: Failed to find builtin recipe: Bluffton_Today

My first guess would be your not running cmd as administrator.. In windows you typically can no write to anything within the program files directory because you only have read access not write. So if you take and search for cmd and right click on it and hit run as administrator this should solve your issue.

p.s.
Also you might wanna run notepad++ as administrator. I had this issue when I was using geany. I would write a recipe and save it and all looked fine but then it never saved. And the reason being is you don't have access to that folder. Again you should be able to solve the issue like i stated. or you could simply move the files outside of the program files directory.

[cynvision]

I will give the admin parts a try. All I know is the more I want to try to make it sophisticated, the faster I break my code. The site I'm working on is a newspaper that wants to be very visual and have a scan of each page to click to navigate. Which makes it harder to pull the articles on the RSS feed front.
I feel a bit bad for the IT guy. They're going to have a bunch of hits from me pressing "download" a bunch of times trying to get this thing to pull a pretty e-book down.

[mcphail]

Can anyone get help me get a recipe for JCO? In the table of contents the sections are enclosed in <h2> tags, subsections in <h3> and articles in <h4>. If someone could help with a recipe to parse and download the free abstracts I'm sure I could tweak it for the subscribed full text...

Thanks in anticipation

[Corpbob]

My wife is from Kansas City and I'm look for a Recipe for Kansas City Star...

[TonytheBookworm]

Originally Posted by Corpbob

My wife is from Kansas City and I'm look for a Recipe for Kansas City Star...

Here you go. Kansas City Star

[somedayson]

First time poster here...read through most of the pages and trying to make sense of all this.

Using all this data, I made a couple of recipes from things I'm interested in, but they are easier ones as they have rss feeds. Still have lots of extra stuff, I'd appreciate any help anyone could send my way:

This gets the basics from hockey news, but lots of extra stuff in each one:

class AdvancedUserRecipe1283662436(BasicNewsRecipe):
title = u'Hockey News'
oldest_article = 7
max_articles_per_feed = 100

feeds = [(u'Hockey News', u'http://www.thehockeynews.com/rss/9-Headlines.xml')]



I'd also really appreciate any help (or quite honestly one of you experts stirring up a recipe for me for these local newspapers, listed in order of preference:

http://www.journalgazette.net/

http://www.news-sentinel.com/



Further down the list would be trying to find out how to get this column every week:


http://www.aspentimes.com/article/20...ntprofile=1061


thanks for any help anyone can provide....just an awesome program!

Grateful,
Matt

[TonytheBookworm]

Anyone know if when you connect to the calibre library (server) through the kindle and download one or more of your books if amazon is charging you a download fee? Can anyone confirm this of yes/no ?

[Corpbob]

Originally Posted by TonytheBookworm

Here you go. Kansas City Star

Thanks!

[somedayson]

With huge help from buyo and his template, I made this and I'm really close to getting my local paper. I have all the categories just the way I want them, and the sections of the paper look awesome, but I can't fully read the articles.

Anyone willing to have mercy on a guy who doesn't know what he's doing but it sure trying? I'm guessing that I'm limiting something I shouldn't be. I'd really appreciate the help!

class WinnipegFreePress(BasicNewsRecipe):
title = u'Journal Gazette'
__author__ = 'somedayson'
description = 'News from Fort Wayne Indiana'
oldest_article = 1
max_articles_per_feed = 30
category = 'News, Fort Wayne, Indiana'
cover_url = 'http://media.winnipegfreepress.com/designimages/winnipegfreepress_WFP.gif'
no_stylesheets = True
encoding = 'UTF-8'
remove_javascript = True
use_embedded_content = False
language = 'en_CA'

feeds = [(u'Opinion', u'http://journalgazette.net/apps/pbcs.dll/section?Category=EDIT&template=blogrss&mime=xml'),
(u'Local News',u'http://journalgazette.net/apps/pbcs.dll/section?Category=LOCAL&template=blogrss&mime=xml') ,
(u'Sports',u'http://journalgazette.net/apps/pbcs.dll/section?Category=SPORTS&template=blogrss&mime=xml' ),
(u'Features',u'http://journalgazette.net/apps/pbcs.dll/section?Category=FEAT&template=blogrss&mime=xml'),
(u'Business',u'http://journalgazette.net/apps/pbcs.dll/section?Category=BIZ&template=blogrss&mime=xml'),
(u'Ice Chips',u'http://journalgazette.net/apps/pbcs.dll/section?Category=BLOGS11&template=blogrss&mime=xml '),
(u'Entertainment',u'http://journalgazette.net/apps/pbcs.dll/section?Category=ENT&template=blogrss&mime=xml'),
(u'Food',u'http://journalgazette.net/apps/pbcs.dll/section?Category=FOOD&template=blogrss&mime=xml')
]

keep_only_tags = [
dict(name='div', attrs={'id':'article_header'}),
dict(name='div', attrs={'class':'article'}),
]

[somedayson]

Tony...if you use the [user]@free.kindle.com address, it's sent wi-fi and no charge

If you use 3G and send to [user]@kindle.com there is a charge. (.15 per MB maybe)

I've gotten lots of newspapers for free. Just awesome!


Hope this helps,
Matt

[somedayson]

Getting even closer.

I can read all the articles now, but there's stuff before and after them that I'm picking up off the web site. I can't figure out how to

1. Get it to the print only page
2. Get the stuff at the beginning (really disruptive for reading) and the end (not as bad but would love to remove it)


Thanks for any assistance anyone can provide. I certainly wouldn't mind a little .rar pack with the answer in it either!

Grateful either way,
Matt


class WinnipegFreePress(BasicNewsRecipe):
title = u'Journal Gazette'
__author__ = 'somedayson'
description = 'News from Fort Wayne Indiana'
oldest_article = 1
max_articles_per_feed = 30
category = 'News, Fort Wayne, Indiana'
cover_url = 'http://media.winnipegfreepress.com/designimages/winnipegfreepress_WFP.gif'
no_stylesheets = True
encoding = 'UTF-8'
remove_javascript = True
use_embedded_content = False
language = 'en_CA'

feeds = [(u'Opinion', u'http://journalgazette.net/apps/pbcs.dll/section?Category=EDIT&template=blogrss&mime=xml'),
(u'Local News',u'http://journalgazette.net/apps/pbcs.dll/section?Category=LOCAL&template=blogrss&mime=xml') ,
(u'Sports',u'http://journalgazette.net/apps/pbcs.dll/section?Category=SPORTS&template=blogrss&mime=xml' ),
(u'Features',u'http://journalgazette.net/apps/pbcs.dll/section?Category=FEAT&template=blogrss&mime=xml'),
(u'Business',u'http://journalgazette.net/apps/pbcs.dll/section?Category=BIZ&template=blogrss&mime=xml'),
(u'Ice Chips',u'http://journalgazette.net/apps/pbcs.dll/section?Category=BLOGS11&template=blogrss&mime=xml '),
(u'Entertainment',u'http://journalgazette.net/apps/pbcs.dll/section?Category=ENT&template=blogrss&mime=xml'),
(u'Food',u'http://journalgazette.net/apps/pbcs.dll/section?Category=FOOD&template=blogrss&mime=xml')
]

[cynvision]

Well, I tried the Ft. Wayne site but it's choking on the URL variable. Maybe someone can run with this.

Spoiler:

class AdvancedUserRecipe1283666183(BasicNewsRecipe):
    title          = u'Journal Gazette Ft. Wayne IN'
    oldest_article = 4
    max_articles_per_feed = 4
    index                 = 'http://www.journalgazette.net/section/LOCAL'
    keep_only_tags = [dict(name='div', attrs={'id':'coll'}), 
                               dict(name='div', attrs={'id':'main'})
                                ]
    def parse_index(self):
        totalfeeds = []
        soup   = self.index_to_soup(self.index)
        features = soup.findAll('li')
        if features:
           farticles = []
           for li in features:
               divurl = li.find('a', href=True)
               if divurl:
                      divdesc = li.find('div',attrs={'class':'postinfo'})
                      url = 'http://www.journalgazette.net' + divurl.a['href']
                      title = self.tag_to_string(divurl.a)
                      description = self.tag_to_string(divdesc)
                      date  = strftime(self.timefmt)
                      divurl= []
                      if '/article/' in url:
                           farticles.append({
                                      'title'      :title
                                     ,'date'       :date
                                     ,'url'        :url
                                     ,'description':description
                                    })

           totalfeeds.append(('Local', farticles))
        return totalfeeds

[TonytheBookworm]

Originally Posted by somedayson

Tony...if you use the [user]@free.kindle.com address, it's sent wi-fi and no charge

If you use 3G and send to [user]@kindle.com there is a charge. (.15 per MB maybe)

I've gotten lots of newspapers for free. Just awesome!


Hope this helps,
Matt

I'm not talking about the email thing. I'm talking about if you make calibre run as a webserver and then you go to the webbrowsers on the kindle and then click on the book you wish to read from your server and it automatically download it for you. Was wondering if amazon charges you a data transfer fee.