vBulletin

[wintoid]

Forums such as this one often use vBulletin as their software. vBulletin can make RSS feeds, so for example, you can read the Mobile Read forums at https://www.mobileread.com/forums/external.php?type=rss2 as an RSS feed.

Now, I can't find a recipe for a Calibre vBulletin forum anywhere, and when I've tried just the most basic steps, the results have been unsatisfactory.

Anyone got any ideas?

[Starson17]

Originally Posted by wintoid

Anyone got any ideas?

Add

    use_embedded_content= True

I tried it, and it works fine for me.

[wintoid]

Brilliant, thank you. That solves the first problem, and now I'm getting a decent looking RSS feed on my Kindle. The next problem is that I only get the first post to each thread. I have oldest_article=7 and max_articles_per_feed=100 which seem to be the defaults for a recipe, but I am not getting 7 days of feeds, or anything like 100 articles.

Sorry, I feel like such a leech!

[Starson17]

Originally Posted by wintoid

Brilliant, thank you. That solves the first problem, and now I'm getting a decent looking RSS feed on my Kindle. The next problem is that I only get the first post to each thread. I have oldest_article=7 and max_articles_per_feed=100 which seem to be the defaults for a recipe, but I am not getting 7 days of feeds, or anything like 100 articles.

Sorry, I feel like such a leech!

You're not a leech. It's OK to ask questions.

" use_embedded_content= True" Simply means to show you the feed page, not the linked articles. The feed page has only one post from each thread.

"oldest_article=" controls how old the article can be on the RSS feed page. Since all links are to recent posts, I suspect that all will be less than 7 days old.

"max_articles_per_feed=100" simply means if there are more than 100, don't show them. The feed appears to have only 15 articles.

If you want the linked pages (the threads) you need to write a recipe for them. You need to decide how far back you want to go and you'd need to write a multipage recipe, if you want multiple pages. Personally, I don't like forum recipes because if I'm interested enough to read it, I want to be able to post into the forum.

I assumed you just wanted to see the articles in the feed, then if you were interested, you'd click on the link within the recipe-ebook to open the forum in your browser, where you could read the entire thread.

[wintoid]

Thanks again. Yes, what I'm hoping for is to get all "posts" within the "thread". I've programmed C before but not Python, and am completely new to Calibre. Is there an example somewhere I can see that iterates through the "posts"?

Also, does "article" mean the actual main "threads" or the "posts" within the "threads"?

Thanks for all your help

[Starson17]

Originally Posted by wintoid

Thanks again. Yes, what I'm hoping for is to get all "posts" within the "thread". I've programmed C before but not Python, and am completely new to Calibre. Is there an example somewhere I can see that iterates through the "posts"?

Also, does "article" mean the actual main "threads" or the "posts" within the "threads"?

Thanks for all your help

Calibre terminology is that the "feed" is a link to a page that has links to "articles." I checked the links, and in this case the "article" is the first page in a new thread. Depending on whether there are any replies to the first post, there may be additional pages. You might find something useful in the builtin slashdot recipe. I know there are some other forum recipes, but I don't recall any names.

You don't need to iterate through posts. They will all be on the linked "article" page, although in a very active thread, if you run the recipe late, there might be additional pages you would need to follow to get everything. You will need to strip the stuff you don't want. Read the "Using News Recipes" sticky here, particularly at the end for some useful starting links.

[wintoid]

OK, just so we're talking about the same thing, the feed I really want to read on my Kindle is http://www.rangefinderforum.com/foru....php?type=rss2 which is a vBulletin-based forum, like mobileread itself.

My recipe currently looks like this:

class AdvancedUserRecipe1305962894(BasicNewsRecipe):
title = u'Rangefinder Forum'
oldest_article = 7
max_articles_per_feed = 100
use_embedded_content = True
feeds = [(u'RFF', u'http://www.rangefinderforum.com/forums/external.php?type=rss2')]

This works beautifully except that it only ever gets the first post in a forum thread. Are you saying that every response post will be treated as a new "page"? To date, Calibre has never fetched ANY response post.

Again, my thanks for your help.

[Starson17]

Originally Posted by wintoid

OK, just so we're talking about the same thing, the feed I really want to read on my Kindle is http://www.rangefinderforum.com/foru....php?type=rss2 which is a vBulletin-based forum, like mobileread itself.

My recipe currently looks like this:

class AdvancedUserRecipe1305962894(BasicNewsRecipe):
title = u'Rangefinder Forum'
oldest_article = 7
max_articles_per_feed = 100
use_embedded_content = True
feeds = [(u'RFF', u'http://www.rangefinderforum.com/forums/external.php?type=rss2')]

This works beautifully except that it only ever gets the first post in a forum thread. Are you saying that every response post will be treated as a new "page"? To date, Calibre has never fetched ANY response post.

Again, my thanks for your help.

I'm saying that this:
use_embedded_content = True
only shows what's on the feed page., and that page never has response posts. You need to change that line to False (which is where you started) and then write the recipe to deal with images and all the junk it pulls down. If you need it, you'll also need to write it to grab the second and third pages, etc.

[wintoid]

Great! Now my recipe looks like this:

class AdvancedUserRecipe1305902518(BasicNewsRecipe):
title = u'Rangefinder Forum'
oldest_article = 7
max_articles_per_feed = 100

feeds = [(u'RFF', u'http://www.rangefinderforum.com/forums/external.php?type=rss2')]
use_embedded_content= False
keep_only_tags = [dict(name='div', attrs={'id':'posts'})]

However, the problem now is that the RSS is only returning 15 articles, but how has it determined that 15 is "since I last read it"?

[Starson17]

Originally Posted by wintoid

Great! Now my recipe looks like this:

class AdvancedUserRecipe1305902518(BasicNewsRecipe):
title = u'Rangefinder Forum'
oldest_article = 7
max_articles_per_feed = 100

feeds = [(u'RFF', u'http://www.rangefinderforum.com/forums/external.php?type=rss2')]
use_embedded_content= False
keep_only_tags = [dict(name='div', attrs={'id':'posts'})]

However, the problem now is that the RSS is only returning 15 articles, but how has it determined that 15 is "since I last read it"?

An RSS feed page has links to web pages called "articles." In your case the article is the forum page with one or more posts. Your recipe gets everything on that article page. It gets all the articles on the feed. So, if there were 15 links on your feed page, you'd get 15 articles.

Trying to write a recipe to only retrieve unread articles will be much more complex. I don't even know if the RSS page has that info. I suspect it doesn't. Have you looked into the RSS aggregator recipes like Google Reader?