Is it possible to provide external URL for recipe processing?

Hi folks,
I've been trying to convert a bunch of articles from online journal (namely LWN.net). Yet recipe only fetches current issue if you're member, or previous one if you're not. I couldn't find a way to provide a specific article URL to fetch & process in API documentation neither to ebook-convert utility nor to BasicNewsRecipe class (I wanted to convert some of old issues).

So question is: are there any ways to use news recipe for specific article or issue, which URL can be passed as parameters for ebook-convert or some other way for recipe methods?

Thank you!

[kovidgoyal]

Simply implement parse_index in your custom recipe class, it can return any set of urls you like. See the recipe api documentation for examples.

Originally Posted by kovidgoyal

Simply implement parse_index in your custom recipe class, it can return any set of urls you like. See the recipe api documentation for examples.

I'm doing the similar way, only take actual article URL from shell environment variable and invoke ebook-convert in loop each time with next article from list. What I was actually interested in is more clean & generic way to deal with arbitrary URLs, but it seems that recipes API lacks this kind of feature

[kovidgoyal]

Whydont you simply have your shell script generate a custom recipe, filling in the feed URLs from the environment/stdin/whatever. You can put all URLS into one custom recipe.

what I did is mangle original recipe a bit:

diff --git a/recipes/lwn_weekly.recipe b/recipes/lwn_weekly.recipe
index 6392772..b4055ef 100644
--- a/recipes/lwn_weekly.recipe
+++ b/recipes/lwn_weekly.recipe
@@ -9,6 +9,7 @@
 from calibre.web.feeds.news import BasicNewsRecipe
 import re
 import sys
+import os
 
 class WeeklyLWN(BasicNewsRecipe):
     title = 'LWN.net Weekly Edition'
@@ -72,7 +73,9 @@ def parse_index(self):
         if self.username is not None and self.password is not None:
             index_url = self.print_version('/current/bigpage')
         else:
-            index_url = self.print_version('/free/bigpage')
+            issue = os.environ.get('OVERRIDE_LWN_ISSUE')
+            index_url = self.print_version("/Articles/%s/bigpage" % issue)
+            print >>sys.stderr, "overriden issue %s" % index_url
         soup = self.index_to_soup(index_url)
         curr = soup.body
         if curr.h1.string is not None:

and use it together with small shell script, plus list of issues grabbed from my RSS reader:

#!/bin/bash
counter=0
for OVERRIDE_LWN_ISSUE in 515278 514343 510440...; do
	if [ -f ./out/lwn_$counter.epub ]; then
		((counter++));
		continue
	fi
	echo process $OVERRIDE_LWN_ISSUE $counter;
	export OVERRIDE_LWN_ISSUE
	ebook-convert ./lwn_weekly.recipe ./out/lwn_$counter.epub ;
	if [ $? != 0 ]; then
		echo "processing issue $OVERRIDE_LWN_ISSUE $counter failed"
		exit
	fi
	((counter++));
done

Converted this way all issues for last 2 years.
Not sure that somebody needs this, but I just leave it here anyway.