Custom recipes (archive, read-only)

[kiklop74]

Judging you? Why would we do such thing?

[moggie]

I am trying to work on a recipe, is there a documented way of getting at the resizing functionality of comic2* from a feeds2* recipe ?

[moggie]

This recipe can be used to make lrf from pages at the sugarquill. The sugarquill is a website dedicated to relatively high quality Harry Potter fan fiction ( http://www.sugarquill.net/index.php?action=faq ).

This recipe takes an author id as the username. And searches for all stories by that author and makes a book up with each story appearing in the index. If a story has multiple chapters it pulls down each chapter.

feeds2lrf TheSugarQuill.py --username 310
..
Output written to /Users/jb23/tmp/quill/code/The Stories of Arabella and Zsenya at The sugar quill [Tue, 23 Dec 2008].lrf

It does it's best with the fan art section but I think that still needs some work, I would be interested in trying to reuse some comic2lrf code here.

This is my first bit of python programming but it seems to work quiet well.

mpb19815i:code jb23$ cat !$
#!/usr/bin/env  python

__license__   = 'GPL v3'
__copyright__ = '2008, James Beal <james_@catbus.co.uk>'
'''
www.sugarquill.net
'''
import re
from calibre.web.feeds.news import BasicNewsRecipe
from BeautifulSoup import BeautifulSoup 

class TheSugarQuill(BasicNewsRecipe):
    
    description = 'Harry Potter Stories'
    match_regexps = [r'read.php']
    no_stylesheets     = True


    def parse_index(self):
	list = []
	articles = []
       
        url = "http://www.sugarquill.net/index.php?action=profile&id=%s" % (self.username) 
        soup = self.index_to_soup(url)
	for table in soup.findAll('table', attrs={'width':'80%'}):
		td=table.find('td',attrs={'class':'highlightcolor2'})
		td1=td.findNext('td',attrs={'class':'highlightcolor2'})
                self.__author__ = td1.next
		self.title = "The Stories of %s at The sugar quill" % (self.__author__)
		self.html2lrf_options = [ 
                         '--author', td1.next,
                         '--publisher', 'The Sugar Quill'
                       ]
        
	for table in soup.findAll('table', attrs={'width':'90%'}):
            a = table.find('a')
            if a and a.has_key('href'):
		url = "http://www.sugarquill.net/%s" % (a['href'])
                title = self.tag_to_string(a)
		td=table.find('td',attrs={'class':'highlightcolor2'})
                description = td.next
		td1=table.find('td',attrs={'class':'highlightcolor1'})
		date=''
		list.append((title,[{
                                 'title':title, 'date':date,
                                 'url':url,
                                 'description':description
                                }]))
	return list


    def process_chapter(self,  page):
        head = page.head
	if head != None:
        	head.extract()
	td=page.find('td',attrs={'class':'top_pane'})
	if td != None:
		if td.parent != None: 
        		if td.parent.parent != None:
            	            td.parent.parent.extract()
        td=page.find('td',attrs={'class':'bottom_pane'})
        if td != None:
                if td.parent != None:
                        if td.parent.parent != None:
                            td.parent.parent.extract()
        td=page.find('td',attrs={'class':'info_pane'})
        if td != None:
                if td.parent != None:
                     td.parent.extract()
        td=page.find('td',attrs={'class':'info2_pane'})
        if td != None:
                if td.parent != None:
                     td.parent.extract()
	td=page.find("form", attrs={'name':'SQ3'})
        if td != None:
                if td.parent != None:
                        if td.parent.parent != None:
                            td.parent.parent.extract()
        td=page.find('td',attrs={'class':'highlightcolor1'})
        if td != None:
                if td.parent != None:
                     td.parent.extract()
        form=page.find("form", attrs={'action':'read.php'})
	if form != None:
           table = form.parent
           if table != None:
               table.extract()
        sugar_signoff = page.find("div" , { "style" : "font-family: Verdana; font-size: 10px;"} )
        if sugar_signoff != None:
            sugar_signoff.extract()
        return page

    def preprocess_html(self, soup):
	from urllib2 import urlopen 
	try:
		form = soup.find("form", attrs={'action':'read.php'})
	except:
		return self.process_chapter(soup)
	if form == None:
		return self.process_chapter(soup)
	process =  ''
	storyid= form.input['value']
	story_url= "http://www.sugarquill.net/read.php?storyid=%s" % ( storyid ) 
	num_chapters = int(len(form('option')))
	current_chapter = 1
	while current_chapter <= num_chapters :
		url = "%s&chapno=%d" % (story_url,current_chapter )
		process += '<span style="page-break-after: always"></span>' 
		process += str (self.process_chapter(BeautifulSoup(urlopen(url).read())))
		current_chapter += 1
	process += '<span style="page-break-after: always"></span>'
        return BeautifulSoup(process)

[kovidgoyal]

An interesting use of the recipe system One suggestion. Expand your description system to include instructions about the use of the username as an id, so that people who encounter the recipe in the GUI will know how to use it.

Using the image transformations of comic2lrf is overkill in IMO. The SONY Reader does an adequate job for pictures that don't contain much text.

[Amalthia]

But would this receipe work with other Harry Potter archives that use the same archiving software?

[moggie]

Originally Posted by Amalthia

But would this receipe work with other Harry Potter archives that use the same archiving software?

There are other Harry Potter sites ?, That use the same software.

If you or email/PM me some urls If it doesn't work I will do my best to make it work.

Although the website may end up having to go into the password field......

[moggie]

Originally Posted by kovidgoyal

An interesting use of the recipe system One suggestion. Expand your description system to include instructions about the use of the username as an id, so that people who encounter the recipe in the GUI will know how to use it.

Ok I will do that, is appropriate to bundle it in with the other recipes ? If so if I want to make changes in the future should I send in the same kind of patch as any other developer would eg use bzr ?

Originally Posted by kovidgoyal

Using the image transformations of comic2lrf is overkill in IMO. The SONY Reader does an adequate job for pictures that don't contain much text.

Remember I don't get a real reader until christmas day so I have only tested it with calibre's viewer. I would like to know as I could then start automating creating books from webcomics.

[kovidgoyal]

Originally Posted by moggie

Ok I will do that, is appropriate to bundle it in with the other recipes ? If so if I want to make changes in the future should I send in the same kind of patch as any other developer would eg use bzr ?



Remember I don't get a real reader until christmas day so I have only tested it with calibre's viewer. I would like to know as I could then start automating creating books from webcomics.

Bundling it would make it accessible to a lot of people with very little effort on your part. Sure you can just mail me patches. (or just a new version of the recipe file).

At some point I will have to create a framework in calibre for fetching more general online content (like books form feedboks/mobileread, content from fanfic sites, etc), but that's a big job, so for the time being, just add it as a recipe

[moggie]

Originally Posted by kovidgoyal

Bundling it would make it accessible to a lot of people with very little effort on your part. Sure you can just mail me patches. (or just a new version of the recipe file).

I would love you to bundle it It's already got the GPL V3 in there in the same way you have for your recipes. And if you are happy to take new versions from me or others I would be pleased.

Originally Posted by kovidgoyal

At some point I will have to create a framework in calibre for fetching more general online content (like books form feedboks/mobileread, content from fanfic sites, etc), but that's a big job, so for the time being, just add it as a recipe

I will happily add recipes as I can. Would it be possible to have a more flexible way of passing arguments over as username and password are a bit restricting. And remember that its my first bit of real python code.

[kovidgoyal]

Sure, just test the recipe with both epub and LRF output and when you're satisfied, either attach it here or open a ticket and attach it to that.

I have a very liberal policy when it comes to accepting patches

[Amalthia]

Originally Posted by moggie

There are other Harry Potter sites ?, That use the same software.

If you or email/PM me some urls If it doesn't work I will do my best to make it work.

Although the website may end up having to go into the password field......

I'm rather sure there are other archives that run off similar archiving software but I'll have to double check. I have most of my bookmarks on my home computer so it may take a few hours or a day before I can get back to you with links. (which I'll private message you with)

Amalthia

[orkan39]

Hi guys. Thanks for all the awesome recipes. Would it be possible for you to create feeds for:
Engadget - http://www.engadget.com/
Fudzilla - http://www.fudzilla.com/
Tom's Hardware - http://www.tomshardware.com/us/ (Articles and News are separate)
Sarcastic Gamer - http://sarcasticgamer.com/wp/

I realise that this is asking a lot but I would really appreciate it and it would make my PRS-700 so much more useful and valuable.
Thanks in advance

[macthekitten]

recipe huh...i make a mean potato & leek soup...

sorry I couldn't reist

I know I know, don't interrupt when grown ups are talking LOL

[kovidgoyal]

Feel free to create a custom potato and leek recipe for calibre

[macthekitten]

I'm on it!!!