Rss2Book

[fritz_the_blank]

Originally Posted by geekraver

The basic requirement of a publish/subscribe server is WebDAV filesystem support; I believe IIS has this and Apache certainly does via the DAV module. If anyone has a reliable server with that then I can make the switch.

My server runs windows with IIS. If you would like to try using that, and it works, you are more than welcome to use me as a mirror or as your primary hosting. It is the least that I can do after all of your work.

FtB

[geekraver]

It seems that the problem may have been with the router I use (WRT54G) and the way it was configured (long idle timeouts and small connection table size). I have updated to the latest DD-WRT firmware and reconfigured it, so I'd be interested to hear if people start finding the connectivity issues improve.

[geekraver]

I have just finished migrating my server at home from Linux to FreeBSD. I switched from about 12 years of using FreeBSD to (Gentoo) Linux about 4 years ago, and the server worked well, until a couple of months back when I did my 6-monthly system update. Those of you who use web2book have probably found the publish/subscribe feature almost totally unusable in this period. I never managed to figure out why the machine had become so unreliable, and decided to flatten it, and also go back to FreeBSD, as I have many years experience with the latter and think it will cause me less work. I'd be interested to hear if people find the server more reliable now (if not then I have router trouble I guess).

[Hadrien]

Originally Posted by geekraver

I have just finished migrating my server at home from Linux to FreeBSD. I switched from about 12 years of using FreeBSD to (Gentoo) Linux about 4 years ago, and the server worked well, until a couple of months back when I did my 6-monthly system update. Those of you who use web2book have probably found the publish/subscribe feature almost totally unusable in this period. I never managed to figure out why the machine had become so unreliable, and decided to flatten it, and also go back to FreeBSD, as I have many years experience with the latter and think it will cause me less work. I'd be interested to hear if people find the server more reliable now (if not then I have router trouble I guess).

FreeBSD is a kickass OS. We're running Feedbooks on FreeBSD+Lighttpd. Better and faster than the Linux+Apache combo.

[squeezebag]

All:

My Wife and I decided to sell the house, the car, and move to Mexico. One of the things that I would really miss is my subscription to the NewYorker (the surface mail there is useless). Someone told me that I could get a Sony Reader and pick up the weekly subscription via RSS. Imagine my dissapointment when I brought it home and discovered the out-of-the-box RSS support - weak.

Then I stumbled upon GeekRavers excellent software - and thought that my problems were solved. I'm trying to work my way through the examples given in the post but without a lot of luck. I'm wondering if someone could show me what the feed settings should be to achieve my goal.

What I'd like is to retrieve, on a weekly basis, is the full printable versions of the content at this URL: http://www.newyorker.com/services/rss/summary.

If I could end up with a weekly, sony-reader-friendly, version of the NewYorker, I'd be a happy, happy, camper. Any assistance with the settings would be greatly appreciated.

[geekraver]

I published an entry for the New Yorker; you should be able to use Subscribe to add it.

It doesn't include the leading pictures; if you want those modify the content extraction pattern to say "start article rail" instead of "start article body".

BTW it doesn't work well with the built-in PDF and RTF converters (due to unprocessed 16-bit characters); seems to work fine with LRF though. I haven't tried it with the htmldoc PDF conversion option.

[geekraver]

BTW you may have trouble hitting the server. I'm now convinced the issue is with my router (Linksys WRT54G with DD-WRT firmware), and not the server. I can access the server just fine from behind the router, and from outside I have no problem hitting the server on IMAP and SMTP ports, but for some reason HTTP isn't being forwarded even though it is configured just the same.

I think I'll try different firmware on the router tonight.

[adinb]

I published a feed for the New Yorker as well.

The issue seems to be that the New Yorker won't allow the robot fetch to go directly to the printable version of the page/article. (cookies, spoofing referring pages, and/or spoofing the user agent string might fix that..I hope to see something like that in future versions of web2book)

One thing that I can't seem to remember how to do--how to get the link reformatter to reference the original link element *and* the regex string fetched via the link extractactor pattern when the "apply extractor to linked content instead of link text" option is selected. From my dim memory, I don't remember being able to really use the link reformatter if the follow option is checked, but I could be *totally* wrong.

Oh, BTW, publish appeared to work for me tonight.

-adin

[squeezebag]

Regarding the NewYorker feeds,

Thanks a ton. I'm now able to pick up the full articles from the print links (including the pictures and captions). I used the following settings:

URL: http://feeds.newyorker.com/services/...everything.xml
Link Element: Link
Apply extractor to linked contentchecked)
Link Reformatter: {0}?printable=true
Content Extraction pattern: <!-- start article rail -->(.*) <!-- end article body -->

Converts to LRF perfectly. I have two remaining questions.

-I've been able to filter out most of the garbage with the Content Extraction Pattern but I'm still picking up a "keywords" section that I'd like to exclude. Does the Content Extraction thing allow me to extract from A to B, and then from C to D? In other words, there is stuff in the beginning and stuff on the end that I'd like to exclude. There is also a block of stuff in the middle that I'd like to filter out. What's the format for this?

-Also, is there any way to build a table of contents? I can pick up the section summaries from: http://feeds.newyorker.com/services/...everything.xml but is there any way that I can prepend the full extraction with this file? A perfect world would allow me to link from the TOC to the full articles but I'll live with whatever I can get.

Thanks again for your help.

Also, the subscribe function works flawlessly now!

[geekraver]

For TOC, you have a couple of options: using htmldoc for PDF, or writing your own output plugin that pre-massages the HTML. I may add this as a feature later.

For content extraction, in the regular expression pattern you need to group the various parts you want in parentheses; you then use {0}, {1}, {2}, etc in the formatter to represent the matched blocks. So you might use a pattern like:

<!-- start article rail -->(.*)<foo>.*<bar>(.*)<!-- end article body -->

assuming <foo> started the tag section you wanted to skip and <bar> ended it (".*" represent any sequence of characters, in case you don't know that already) .

[ddavtian]

Asked a question that was answered right in front of my post.
Sorry for stupid post.

Waiting for GeekRaver to add a ToC feature.

[_underzcore_]

geekraver, your app is tantalizingly good . . . but I'm getting killed on the feed I'm trying to save (The Economist print edition). There seem to be two big hurdles the way they have the feed set up:

1) It's hosted by a second party, so there's an intermediary link through pheedo.com that then points the browser back to the article at economist.com.

2) The articles at economist.com are stored in different directories (e.g. ".../opinion/," ".../world/la/"). This seems to be fouling up my efforts to change a "displaystory.cfm?story_id=" into a "PrinterFriendly.cfm?story_id=" with just one set of regular expressions. And it's hard to tell if I'm properly sidestepping the pheedo.com blind alley.

Help?

[flamaest]

Does someone have a list of RSS feeds which are FULL article feeds..?

from any news sources, I don't care...

Most of the one's i find are intro-snippet only..

POST YOUR FULL RSS URLs.. please..?

Help??
F.

[geekraver]

Originally Posted by flamaest

Does someone have a list of RSS feeds which are FULL article feeds..?

from any news sources, I don't care...

Most of the one's i find are intro-snippet only..

POST YOUR FULL RSS URLs.. please..?

Help??
F.

Much of the usefulness of web2book is that it turns partial feeds into full feeds. Just try use the subscribe feature (on the file menu). My server might be a bit slow as it is doing a on-line backup but be patient and you should get a list.

[geekraver]

I'm heading out on vacation; I'll respond when I get back.

Originally Posted by _underzcore_

geekraver, your app is tantalizingly good . . . but I'm getting killed on the feed I'm trying to save (The Economist print edition). There seem to be two big hurdles the way they have the feed set up:

1) It's hosted by a second party, so there's an intermediary link through pheedo.com that then points the browser back to the article at economist.com.

2) The articles at economist.com are stored in different directories (e.g. ".../opinion/," ".../world/la/"). This seems to be fouling up my efforts to change a "displaystory.cfm?story_id=" into a "PrinterFriendly.cfm?story_id=" with just one set of regular expressions. And it's hard to tell if I'm properly sidestepping the pheedo.com blind alley.

Help?