Adding Newsfeeds to Calibre

[p3aul]

Hi!
I'm having trouble adding a newsfeed to Calbre. Here's what I have done and the resukts I got.

[1] Browsed to the NOAA website, clicked on Red/White XML button.

[2] Copied the URL to the clipboard.

[3] In Calibre, I clicked on "Add A Custom News Source"

[4] In the dialog I chose the Feed Title and the Recipe Title as "NOAA"

[5] Pasted the Feed URL into the box.

[6] Clicked on "Add Feed" and "Add/Update Recipe"

[7] Closed the dialog and chose Schedule news Download"

[8] In the dialog, Clicked on Custom->NOAA Download Now.

[9] Selected NOAA to view.

[10] result 3 pages page 3 just said Failed.

Tried again, same steps, but used the URL of the website.

Result: 21 pages all HTML.


Paul

RESULT:

[kiklop74]

This site requires more advanced recipe. Like this one:

[p3aul]

Thanks, that works but it's a bit of overkill. How can I modify it so it only gets my local weather? This is the main website:
http://forecast.weather.gov/MapClick...2=-83.6313&e=0

This the website that comes up when I click on XML:
http://forecast.weather.gov/MapClick...&FcstType=dwml

If you could give me this code it sure would be much appreciated!
Thanks,
Paul

[kiklop74]

Originally Posted by p3aul

Thanks, that works but it's a bit of overkill. How can I modify it so it only gets my local weather? This is the main website:
http://forecast.weather.gov/MapClick...2=-83.6313&e=0

This the website that comes up when I click on XML:
http://forecast.weather.gov/MapClick...&FcstType=dwml

If you could give me this code it sure would be much appreciated!
Thanks,
Paul

That is completely different kind of content and much more complex recipe is needed. Right now I do not have time to do that nor can I tell you when will I have such time.

[p3aul]

Why did Kovid even include something in the application that the average person can't comprehend? It makes no sense to me to have a feature that doesn't work without extensive programming experience.
Paul

[kiklop74]

Originally Posted by p3aul

Why did Kovid even include something in the application that the average person can't comprehend? It makes no sense to me to have a feature that doesn't work without extensive programming experience.
Paul

Web scraping is a complex topic and calibre's framework by default covers only basic cases. It would be fairly difficult and time consuming to create something more user friendly. It can be done but since this is non-profit project it will be done (or not) when Kovid or somebody else decides to work on that.

What you can do in this particular case is following:
1. Calibre has a recipe for instapaper.com which is a great tool for scraping web pages.
2. Go to instapaper.com and open an account, make sure to set password (it is required by calibre)
2a. Make sure to uncheck "Automatically move articles to the Archive when I read them" on instapaper.com
3. Once logged in your instapaper.com account look at the "read later" button on the frontpage. Drag that button to the favorites in your browser.




4. Open now in the same browser your weather forecast and click on the read later link in your favorites.
5. You just added the weather forecast page to your instapaper feed.
6. Now open calibre and locate the instapaper.com recipe and enter your credentials
7. Get the news and you will have as a result your weather forecast page in calibre.

With this approach you can download with calibre any static page without any coding.

[p3aul]

That worked great!

Paul

[seanrose]

Thanks! This is just what I was looking for.

[moosejons_dad]

Can the news feed for the Wall Street Journal be checked as it seems to hang up when it is 47% downloaded. I am using 6.19 on Calibre.

Thanks in advance.

[kovidgoyal]

I don't subscribe to WSJ so I can't check it.

[olaf]

My local newspaper just went to a pay-for-online-content model which requires paper subscribers to enter a userid/password to view local articles. I am a subscriber, and have the userid/password (which I have entered on the 'Fetch News' screen), but I still get errors when running the recipe (to which I've added "needs_subscription=True"). Do I need to add other specialized code to the recipe for this to work? I get a lot of the paper, but links to the local section don't go to the articles - they go to random other spots.
------------------------

import string, re

class Telegram(BasicNewsRecipe):
title = 'Telegram'
oldest_article = 2
timefmt = ''
max_articles_per_feed = 100
no_stylesheets = False
use_embedded_content = False
encoding = 'cp1252'
needs_subscription = True
publication_type = 'newspaper'
remove_empty_feeds = True
extra_css = ' body{font-family: Verdana,sans-serif} .headline{font-size: xx-large; font-weight: bold} .mainPhotoCaption{font-size: x-small} '

keep_only_tags = [dict(name='div', attrs={'id':'articleWell'})]
remove_tags_before = dict(attrs={'class':'headline'})
remove_tags_after = dict(attrs={'id':'zoom1'})
remove_tags = [
dict(name='div', attrs={'class':'relatedContent'})
,dict(name=['object','link','iframe'])
]

feeds = [(u'Front Page News', u'http://www.telegram.com/apps/pbcs.dll/section?Category=RSS03&MIME=xml'),
(u'World & Regional', u'http://www.telegram.com/apps/pbcs.dll/section?Category=rss01&MIME=xml&profile=1052'),
(u'Living', u'http://www.telegram.com/apps/pbcs.dll/section?Category=rss01&MIME=xml&profile=1011'),
(u'Local News', u'http://www.telegram.com/apps/pbcs.dll/section?Category=rss01&MIME=xml&profile=1101'),
(u'Business', u'http://www.telegram.com/apps/pbcs.dll/section?Category=rss01&MIME=xml&profile=1002'),
(u'Opinion', u'http://www.telegram.com/apps/pbcs.dll/section?Category=rss01&MIME=xml&profile=1017'),
(u'Deaths', u'http://www.telegram.com/apps/pbcs.dll/section?Category=rss01&MIME=xml&profile=1001'),
(u'Sports', u'http://telegram.com/apps/pbcs.dll/section?Category=rss01&MIME=xml&profile=1009'),
(u'Court Records', u'http://www.telegram.com/apps/pbcs.dll/section?Category=rss01&MIME=xml&profile=1056'),
(u'As I See It', u'http://www.telegram.com/apps/pbcs.dll/section?Category=rss01&MIME=xml&profile=1054')]

[kovidgoyal]

yes, you need to write the code to login. Look in the user manual it has examples of this.