Track Yahoo Finance message boards via RSS

[Alexander Turcic]

I just wrote YahooMsg2RSS, a new tool which allows you to track Yahoo Finance message boards via RSS feeds. You can use it for instance to track everything that is going on in the Palm message board.

Your suggestions are always welcomed.

very nice! i was wondering when someone would do this (like yahoo).

would something like this be adaptable to, say, treocentral boards, or others very generically?

thanks.

[Alexander Turcic]

Originally Posted by Unregistered

would something like this be adaptable to, say, treocentral boards, or others very generically?

That wouldn't be a trivial task because boards differ in layout and there is no way to predict what data to extract.

yeah, i realized that, then thought it could work like a plugin to a reader and each board could be user configurable, since all boards have certain things in common despite different layout. anyway, just smells like a useful bigger thing to me!

i'm checking palm now, seems updating is sporadic. do you have the bandwidth to run this through mobileread (assuming that's how you do it)?
i use greatnews (and thunderbird and feedreader depending) and the validate feed tool came back with an error (for palm) but a few recent posts updated though not the latest.

[Alexander Turcic]

Originally Posted by alta88

yeah, i realized that, then thought it could work like a plugin to a reader...

Yeah that would be possible In fact, for all my custom feeds I am already using the same underlying engine.

i'm checking palm now, seems updating is sporadic. do you have the bandwidth to run this through mobileread (assuming that's how you do it)?

The tool receives the latest 40 posts from Yahoo every time you fetch the feed with your RSS client, but only once every one hour. So if you don't use your RSS client for some time, it's possible that you'll miss some older messages (if there were more than 40 messages posted within this time). But that's the nature of RSS - you only get back as far as the numbers of last items that are returned by the feed. You can easily see if you missed messages by the running ID in front of each item title.

i use greatnews

Great choice! I switched from FeedDemon to GN a few months ago.

the validate feed tool came back with an error (for palm) but a few recent posts updated though not the latest.

Thanks for the hint! It should valdidate fine now. I switched from using the <author> tag to <dc:creator> since <author> would requires an complete e-mail address.

you and jack should get together on this plugin idea

so, you mean regardless of the interval i have set in greatnews or manual force update, the tool pops max once an hour and then only gets last 40? i mean, i can see why, but..

[Alexander Turcic]

Originally Posted by alta88

you and jack should get together on this plugin idea

so, you mean regardless of the interval i have set in greatnews or manual force update, the tool pops max once an hour and then only gets last 40? i mean, i can see why, but..

I checked some of the most busy Y! Finance boards and once an hour seems to be a good interval to not miss any messages. More advanced clients like Greatnews will notice that the feed hasn't changed (they get a 304 header) and won't try to reload it every time you click on manual update.

The problem is that there are some smart people out there who update their feeds every 10 seconds (!) and that would definitely be a killer for a project like this one (not to mention Y! probably banning our server).

how about once every half hour?

much of yahoo is entertainment only, but some boards are actually amazingly useful and knowledgable..

funny that you're in switz; i'm in rome these days.

btw, you've posted good stuff over on GN.

best.

[Alexander Turcic]

Lemme explore the logs in the next few days and depending on traffic I'll reduce it to every 30 mins.

alexander,

i've been using your tool for a few days now, and it's changed forever how i read the Y! message boards. really terrific. in GN i can sort by author and get rid of known bad ones/keep known good ones, and save useful stuff. for multiples faster than before.

i don't know your plans for this, but hope you turn it into a generic tool. someday phpBB and the others might have this sort of feature built in, but.. it'll be a one by one progression etc.

msg boards are a different beast, but i wish more sites did what PubMed does, ie allow sending a user customized keyword search as an RSS feed.

some feedback. i know it's only a yahoo message board, and you don't need two-phased commit or checksum architecture to guarantee message integrity. but, you sort of have to guarantee no message gaps. palm eg. is in a very interesting period, and heavy msg volume friday led to hourly gaps. it's the ancient problem of tradeoff between getting, in realtime, both very high volume and very low volume messages.

conversely, on one very lightly used board, i got duplicates after i deleted posts i didn't want. but not all, just a few reappeared. i will watch this carefully again and report a better description.

thanks!

[Alexander Turcic]

alta, I am happy you like the tool!

Regarding 'generic': It's often up to the webmaster whether to offer RSS feeds for his forum. For instance, Mobileread has RSS feeds for every forum section. Returning RSS feeds based on customized keyword searches sounds useful, but causes some serious scaling issues. As long as you restrict the number of feeds, you can easily cache them for use by multiple people. But if you offer keyword searches, it's possible that your server has to great hundreds if not thousands of individual RSS feeds, definitely putting a lot of stress on any server (bandwidth and CPU).

I am still not sure about refreshing the feeds more than every one hour. So far word hasn't spread much of this tool - but if more people start using it, one hour might be the best frame to limit stress on the servers. Regarding duplicate posts - that issue is most likely related to your feed client (GN) which doesn't keep track of what items you've deleted before and once you refetch the feed, the previously deleted items come back.

update: I've changed the update frequency to every 30 minutes. Let's see how it goes...

Best
Alex

yes, the best way is for all boards to offer a feed. but they don't. surprising about yahoo, they have almost everything else.

as an fyi, check out
http://www.ncbi.nlm.nih.gov/entrez/q...arch&DB=pubmed
where a p53 gene search is returned. Send to dropdown, RSS feed. i'm guessing they have a lot of resources, every reasearcher in the world can do a custom search(es) on a very large database..

yes, it's GN reposting deletes, i recall the bug mentioned now.

i guess the refresh rate is of secondary importance to message gaps.

thanks.

forgot to mention, feed for EXAS doesn't validate..

thanks.

[Alexander Turcic]

Originally Posted by alta88

forgot to mention, feed for EXAS doesn't validate..

thanks.

Yahoo is having issues with the forum for this stock. See attached screenshot of http://messages.yahoo.com/?action=q&board=EXAS.

yes, the best way is for all boards to offer a feed. but they don't.

Best would be to ask the administrator of the board you are interested in to offer RSS.

i'm guessing they have a lot of resources, every reasearcher in the world can do a custom search(es) on a very large database..

You're probably right.

i guess the refresh rate is of secondary importance to message gaps.

The refresh rate is the only reason for having message gaps since only the latest 40 posts are returned each time you query the updated feed.

[Alexander Turcic]

OK, I added some extra sanity checks in 1.01 to remove illegal content from broken boards such as EXAS.