Custom recipes (archive, read-only)

[kiklop74]

You are on your own here. I'm not interested in doing this recipe. Ask Kovid.

[kovidgoyal]

I believe there is already a recipe posted for fanfiction.net somewhere on these forums by moggie, I don't recall if it supports multiple chapters.

[ilovejedd]

Originally Posted by Hypernova

I tried to make a recipe for FanFiction.net. Well, unless it is a multiple chapters story, it is easy enough.

class FanFiction(BasicNewsRecipe):
    title          = u'FanFiction'
    oldest_article = 7
    max_articles_per_feed = 10
    use_embedded_content  = False
    remove_javascript     = True
    keep_only_tags     = [dict(name='div', attrs={'id':'storytext'})]
    
    feeds          = [(u'Just In', u'http://www.fanfiction.net/atom/j/0/0/0/')]

But I don't know what to do with multiple chapters story (example: http://www.fanfiction.net/s/4952058/1/Moonlight_Spell). I tried looking at Ars Technica recipe, but have no idea what's going on. Can I request a recipe that works for multiple chapters story as well?

I have a working recipe but it involves running an HTTP server w/PHP on your PC. How comfortable are you with that?

Originally Posted by kovidgoyal

I believe there is already a recipe posted for fanfiction.net somewhere on these forums by moggie, I don't recall if it supports multiple chapters.

Nah, that's for TheSugarQuill. Not very easy adapting it for FanFiction.Net. I tried and failed. Although, that probably has to do with my unfamiliarity with Python instead of any actual programming difficulty.

Recipe for those running a local server:

#!/usr/bin/env  python

__license__   = 'GPL v3'
__copyright__ = '2009, tifa1984'
'''
www.fanfiction.net
'''

import string, re
from calibre.web.feeds.news import BasicNewsRecipe

class FFNetFeed(BasicNewsRecipe):
    title          = u'FFNetFeed'
    __author__     = 'FanFiction.Net'
    description    = 'unleash your imagination'
    
    oldest_article = 1
    
    feeds          = [
                      ('Inuyasha - Inuyasha/Kagome', 'http://www.fanfiction.net/atom/l/436/10/0/1/1/0/646/647/0/0/'),
                      ('Inuyasha - Sesshoumaru/Kagome', 'http://www.fanfiction.net/atom/l/436/10/0/1/1/0/650/647/0/0/'),
                      ('Rurouni Kenshin - Kaoru/Kenshin', 'http://www.fanfiction.net/atom/l/355/10/0/1/1/0/410/411/0/0/')
                     ]
    
    def get_cover_url(self):
        self.cover_url = u'http://localhost/flag/ffnetlogo.png'
        cover = self.cover_url
        return cover
    
    def print_version(self, url):
        main, sep, rest = url.rpartition('/1/')
        rmain, rsep, storyid = main.rpartition('/s/')
        return 'http://localhost/flag/getstory.php?source=ffnet&format=news&storyid=' + storyid

Attached are the PHP scripts to create a printable version of a story. Easiest way to get it working would be to install XAMPP. You need to enable the php curl and php tidy extensions. In XAMPP for Windows, I just had to uncomment the following lines from \xampp\apache\bin\php.ini:

extension=php_curl.dll
extension=php_tidy.dll

[kovidgoyal]

I am going to hold off on looking at this as I plan to eventually implement a general famework in calibre for getting non-periodical online content, like the recipe system, but one that supports getting lists of available content, searching, etc. When that happens, it should be relatively easy to add support for ff.net

[Hypernova]

Thanks for all the help, especially ilovejedd. I'll use your method if I get too lazy to do Copy & Paste to Word.

I am going to hold off on looking at this as I plan to eventually implement a general famework in calibre for getting non-periodical online content, like the recipe system, but one that supports getting lists of available content, searching, etc. When that happens, it should be relatively easy to add support for ff.net

I'm looking forward to that!

[ilovejedd]

FanFiction.Net is actually pretty adaptible to the current recipe system assuming you only want to download collections/omnibus of newly updated fanfics. It already has RSS feeds so there's no need to manually parse for article/story links. The above recipe can also be used to retrieve all stories by an author or in a C2 Community.

I don't think a Python-only recipe would be too difficult to accomplish for someone who's already familiar with the language. It's just hard for me since I'm not familiar with the syntax and built-in functions.

Originally Posted by Hypernova

Thanks for all the help, especially ilovejedd. I'll use your method if I get too lazy to do Copy & Paste to Word.

You're welcome. What end format do you need to convert to by the way? html, epub, or something else? The script I attached can be used in commandline mode to download individual stories. If you have Calibre installed, you can even have it converted to epub or lrf. erayd (the original author of the script) had a web service for downloading fanfics from FF.Net but it's been down for the past few days.

Addendum:
I fixed the PHP script (found a bunch of character conversion errors) and uploaded it to my webhost. Well, whaddya know, it works. The following recipe should be a good enough stop-gap until someone can convert the chapter retrieval logic into pure Python. The zip file in the original message message has also been replaced with the fixed PHP scripts.

Recipe for online server:

#!/usr/bin/env  python

__license__   = 'GPL v3'
__copyright__ = '2009, tifa1984'
'''
www.fanfiction.net
'''

import string, re
from calibre.web.feeds.news import BasicNewsRecipe

class FFNetFeed(BasicNewsRecipe):
    title          = u'FFNetFeed'
    __author__     = 'FanFiction.Net'
    description    = 'unleash your imagination'
    
    oldest_article = 1
    
    feeds          = [
                      ('Just In', 'http://www.fanfiction.net/atom/j/0/0/0/'),
                      ('Inuyasha - Inuyasha/Kagome', 'http://www.fanfiction.net/atom/l/436/10/0/1/1/0/646/647/0/0/'),
                      ('Inuyasha - Sesshoumaru/Kagome', 'http://www.fanfiction.net/atom/l/436/10/0/1/1/0/650/647/0/0/'),
                      ('Rurouni Kenshin - Kaoru/Kenshin', 'http://www.fanfiction.net/atom/l/355/10/0/1/1/0/410/411/0/0/')
                     ]
    
    def get_cover_url(self):
        self.cover_url = u'http://utterlyinsane.org/fflag/ffnetlogo.png'
        cover = self.cover_url
        return cover
    
    def print_version(self, url):
        main, sep, rest = url.rpartition('/1/')
        rmain, rsep, storyid = main.rpartition('/s/')
        return 'http://utterlyinsane.org/fflag/print.php?storyid=' + storyid

Notes:

• users need to change the feeds to whatever categories they follow
• I have the recipe scheduled to download everyday in Calibreso the default is to download all stories updated within 1 day, just modify oldest_article = 1 to reflect your downloading schedule
• if downloading all stories by an author, you might want to change oldest_article to a really high value, e.g. 32767
• downloading a whole bunch of stories might take a while so if you're downloading from a very active category (e.g. Harry Potter), you might want to limit the number of downloads or filter the RSS feed as much as possible (FanFiction.Net RSS feeds change based on the selected filters: genre, fiction rating, language, length, character, status)

[Hypernova]

Wow. I just try your script ilovejedd. Working great. Now my PRS-505 is even more useful. Thank you again.

So this recipe is using your server right? Is it alright for me to keep using it? I think I can set up a php server if I really try, but I am no expert in the topic.

[iNdioNicarao]

I would appreciate if someone can come up with a recipe for La Prensa -- www.laprensa.com.ni/archivo/noticias/rss/

Thank you

[donCalpe]

Originally Posted by ligos

I have been using Calibre for some time and made some of my own recipes. Over a long period I have been using Calibre 0.4.67. and those recipes were working on this Calibre version. I have tried few next Calibre versions but those custom recipes were not working on those trials.

Now I have 0.5.2. Calibre version installed and those recipes do not work either. The following Conversion Error shows up:


The error is the same in every recipe except for the name of the recipe (in the example above DiePresseWirtschaft).
The code of each recipe is a little bit different and it generally presents itself in this fashion:

from libprs500.ebooks.lrf.web.profiles import DefaultProfile 

import re 

class DiePresseWirtschaft(DefaultProfile): 
    title = 'DiePresseWirtschaft' 
    timefmt = ' [%d %b %Y]' 
    summary_length = 1000
    oldest_article = 1
    max_articles_per_feed = 100
    max_recursions = 2 
    html_description = True 
    no_stylesheets = True 

    def get_feeds(self):  
        return [ ('Die Presse Wirtschaft', 'http://www.diepresse.com/rss/Wirtschaft') ]  

    def print_version(self,url): 
        return url.replace('index.do?from=rss', 'print.do') 

    preprocess_regexps = [
        (re.compile(r'<script>.*?</script>', re.IGNORECASE | re.DOTALL), lambda match : ''),
        (re.compile(r'<H4>.*?</H4>', re.IGNORECASE | re.DOTALL), lambda match : ''),
        ]

Does anyone know what to do to to start my recipes working again?

the only rss feed which works is " http://diepresse.com/rss/home" for all other feeds I get the following error:
---------
Job: **Fetch news from DiePresse**
**tuple**: ('OEBError', u"File 'feed_0/article_4/index.html' does not appear to be (X)HTML")
**Traceback**:
Traceback (most recent call last):
File "/Applications/calibre.app/Contents/console.app/Contents/Resources/lib/python2.6/site-packages.zip/calibre/parallel.py", line 958, in worker
result = work(client_socket, func, args, kwdargs)
File "/Applications/calibre.app/Contents/console.app/Contents/Resources/lib/python2.6/site-packages.zip/calibre/parallel.py", line 916, in work
etc....
--------
the recipe is as simple as:
[CODE]class AdvancedUserRecipe1238335231(BasicNewsRecipe):
title = u'DiePresse'
oldest_article = 2
max_articles_per_feed = 10

feeds = [(u'home', u'http://diepresse.com/rss/home'),
]
def print_version(self,url):

return url.replace('index.do?from=rss', 'print.do')
[CODE]

this recipe works but if you replace the feed "home" with any other like "sport" or "kultur" it stops working with the above mentioned error.

[ilovejedd]

Originally Posted by Hypernova

Wow. I just try your script ilovejedd. Working great. Now my PRS-505 is even more useful. Thank you again.

So this recipe is using your server right? Is it alright for me to keep using it? I think I can set up a php server if I really try, but I am no expert in the topic.

Yep, that's on my server. It's fine if you keep using it. The server doesn't really get that much use anyway and I have a lot of unused bandwidth every month. Plus, there's more people who can test the script for bugs. Just yesterday, I had to do a couple of fixes because of the way FanFiction.Net changed handling of crossovers. I like the new system, I do, but it was a pain trying to figure out why some fanfics won't download but others do. A few days before that, I was fixing another script (this one is for dynamic browsing of selected categories on FF.Net from Stanza iPhone) because they added pairing information.

[kiklop74]

Originally Posted by iNdioNicarao

I would appreciate if someone can come up with a recipe for La Prensa -- www.laprensa.com.ni/archivo/noticias/rss/

Thank you

That rss feed gives links to non-existent pages. That means a custom scraping is needed. I'll see if I can find the time to do this in the following days.

[kovidgoyal]

@donCalpe

That's because those feeds produce pages that look like this

<!-- Bottom build 2.30 -->
<!-- Setup build 3.03 -->
<!-- Kontingent hashmap build 2.52 -->
<!-- Regroup hashmap build 2.45 -->
<!-- Level2 hashmap build 1.18 -->
<!-- Sitestat build 2.39 -->


<!-- End Sitestat4 code -->
<!-- Ohters build 2.21 -->
<!-- FragebogenOEWA -->

<!-- /FragebogenOEWA -->



<!-- Begin Sitestat4 Technical code -->

<!-- End Sitestat4 Technical code -->
java.lang.NullPointerException
	at java.util.regex.Matcher.getTextLength(Matcher.java:1140)
	at java.util.regex.Matcher.reset(Matcher.java:291)
	at java.util.regex.Matcher.<init>(Matcher.java:211)
	at java.util.regex.Pattern.matcher(Pattern.java:888)
	at org.apache.jsp.portal.services.article.print_jsp._jspService(print_jsp.java:125)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
	at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)
	at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
	at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672)
	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463)
	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398)
	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
	at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)
	at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)
	at org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:398)
	at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:318)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:241)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
	at at.beit.portal.tools.urlredirect.URLRedirectFilter.doFilter(URLRedirectFilter.java:120)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
	at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:425)
	at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:452)
	at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1285)
	at java.lang.Thread.run(Thread.java:619)
</init>

Try setting your output format to epub and then convert the epub to mobi, that will strip these kinds of pages.

[kiklop74]

Actually it was not that hard as I thought:

La Prensa - Nicaragua

[Mnementh]

Originally Posted by ilovejedd

I have a working recipe but it involves running an HTTP server w/PHP on your PC. How comfortable are you with that?

Hi ilovejedd,

I'm trying to run your script as I've been looking for something to do this with FFNet for a long time (it's a real pain having to cut and paste each page manually!) but every time I run it I get this error:

Warning: require_once(source/.source.php) [function.require-once]: failed to open stream: No such file or directory in C:\xampp\htdocs\flag\getstory.php on line 55

Fatal error: require_once() [function.require]: Failed opening required 'source/.source.php' (include_path='.;C:\xampp\php\pear\') in C:\xampp\htdocs\flag\getstory.php on line 55

I know I'm probably doing something really stupid and I have tried searching on the net to no avail, so any insight you could give me would be much appreciated. I have to admit right from the start I know nothing about php and have only ever run basic web servers (though I do know php is running and is version 5.2.8) so I'm really clueless as to what to do here.

TIA for any help you can give.

Mnementh

[ilovejedd]

Originally Posted by Mnementh

Warning: require_once(source/.source.php) [function.require-once]: failed to open stream: No such file or directory in C:\xampp\htdocs\flag\getstory.php on line 55

Fatal error: require_once() [function.require]: Failed opening required 'source/.source.php' (include_path='.;C:\xampp\php\pear\') in C:\xampp\htdocs\flag\getstory.php on line 55

Looks like you're not specifying the source. Which recipe are you using? The first one in the post with the zip or the last one I posted which links to my own webserver?

def print_version(self, url):
    main, sep, rest = url.rpartition('/1/')
    rmain, rsep, storyid = main.rpartition('/s/')
    return 'http://localhost/flag/getstory.php?source=ffnet&format=news&storyid=' + storyid

Note source=ffnet is specifically stated in the url. This script can also be used for FictionPress, you just change source=ffnet to source=fpress.

The zip file should contain a print.php file which you can also use. That one automatically uses ffnet as source. Just change the recipe to the following:

def print_version(self, url):
    main, sep, rest = url.rpartition('/1/')
    rmain, rsep, storyid = main.rpartition('/s/')
    return 'http://localhost/flag/print.php?storyid=' + storyid

However, I'm not sure what version of the script is still attached in the zip file. I think that one still hasn't been fixed to reflect the crossover changes FanFiction.Net implemented. I suggest using the online version of the recipe in the meantime.