FLAG (Fanfiction.net Lightweight Automated Grabber)

[erayd]

Whoa.... this thread seems to have taken on a life of its own! Apologies for not having consolidated everyone's changes yet into the main build - (paid) work has been getting in the way a bit of late. Rest assured I haven't forgotten about it, although with the number of changes that seem to have taken place since I last looked it seems there may be a bit of work involved! Thank you to everyone who's added their own bits and pieces - it sounds like it's a bit of a frankenstein at the moment, but a frankenstein that seems to be filling people's needs :-).

If anyone would prefer that I *not* integrate their changes into my main build, please sing out - the next release of this is intended to be GPLv2, but I'm happy to change that if there are objectors.

[araevin]

Ok does anybody know how to get this working on OSX 10.5.6 I run mamp pro and I've cant tell this script from the top to bottom anybody tell how to get started?

[ilovejedd]

erayd! Yay! You're back!

By the way, flag.erayd.net seems to be down the past few weeks.

FanFiction.Net changed their pages quite a bit (added graphics, changed handling of crossover fics, etc). Below are the changes I made to get FF.Net working again, albeit there's probably a few changes in there for additional information (I was working on an already modified version ).

function ffnet_get_meta($page) {
	$matches = array();

	//category and title
	if (preg_match('/<td>.*<a href.+>.+Crossover<\/a> »/Ui', $page, $matches)) {
		preg_match('/<td>.*<a href.+>(.+) Crossover<\/a> » <b>(.+)<\/b><\/td>/Ui', $page, $matches);
		$matches[1] = str_ireplace(',', '', $matches[1]);
		$tag = str_ireplace(' and ', ',', $matches[1]);
	}
	else preg_match('/<td>.+ » <a href=.+>(.+)<\/a> » <b>(.+)<\/b><\/td>/Ui', $page, $matches);

	//author
	preg_match("/<td>Author: <a href='\/u\/[0-9]+\/.+'>(.+)<\/a><\/td>/Ui", $page, $author);
	$meta['title'] = $matches[2];
	$meta['author'] = $author[1];
	$meta['category'] = $matches[1];
	$meta['crosscat'] = $tag;

	//rating, lang, category
	preg_match("/<TR><TD><a href='http:\/\/www.fictionratings.com\/guide.php' target='rating'>Fiction Rated: (.+)<\/a> - (.+) - +(.+) - Reviews: <a href='.+'>[0-9]+<\/a> - Published: ([0-9-]+) - Updated: ([0-9-]+) (- (Complete)+ )?- id:([0-9]+) <\/td><\/TD><\/TR>/Ui", $page, $matches);
	$meta['rating'] = $matches[1];
	$meta['language'] = $matches[2];
	$meta['genre'] = $matches[3];
	$meta['date_pub'] = $matches[4];
	$meta['date_update'] = $matches[5];
	$meta['complete'] = $matches[7] == 'Complete' ? true : false;
	$meta['ficstatus'] = $matches[7] == 'Complete' ? 'Complete' : 'In-Progress';
	$meta['id'] = $matches[8];

	//description
	preg_match('/<meta name="description" content="(.+), (.+),  (.+)">/Ui', $page, $matches);
	$meta['summary'] = $matches[3];
	$meta['summary'] = iconv('UTF-8', 'ISO-8859-15//TRANSLIT//IGNORE', $meta['summary']);

	//chapters
	if(preg_match("/<SELECT title='chapter navigation' Name=chapter onChange=\".+\">(<option.+)<\/select>/Ui", $page, $matches)) {
		preg_match_all("/<option  value=[0-9]+ (selected)?>([0-9]+). (.+)(?=<option|$)/Ui", $matches[1], $matches);
		foreach($matches[2] as $key => $chapnum) $chaparr = $chapnum;
		$meta['chapters'] = $chaparr;
	}

	return $meta;
}

Here are some changes I made to epub.codec.php so it will display additional meta information in Calibre. This works with the Windows CLI utilities. I think it should be the same for Linux. As far as I'm aware, the only changes rgibbo made in the Windows port is changing the handling of temp folder and removal of posix functions.

	//Fix summary
	$story['meta']['summary'] = str_replace('"', '\"', $story['meta']['summary']);

	//XPath for chapter detection
	$cli_chapter = "";
	$cli_chapter .= " --chapter-mark \"pagebreak\"";

	//metadata
	$cli_metadata = "";
	$cli_metadata .= " -t \"{$story['meta']['title']}\"";
	$cli_metadata .= " -a \"{$story['meta']['author']}\"";
	if(!($story['meta']['crosscat'] == '')) $cli_metadata .= " --subjects \"{$story['meta']['crosscat']}\"";
	else if(!($story['meta']['category'] == '')) $cli_metadata .= " --subjects \"{$story['meta']['category']}\"";
	if(!($story['source'] == '')) $cli_metadata .= " --publisher \"{$story['source']}\"";
	$cli_metadata .= " --comment \"<b>Story id:</b> {$config['storyid']}";
	if(!($story['meta']['category'] == '')) $cli_metadata .= "<br><b>Category:</b> {$story['meta']['category']}";
	if(!($story['meta']['genre'] == '')) $cli_metadata .= "<br><b>Genre:</b> {$story['meta']['genre']}";
	if(!($story['meta']['rating'] == '')) $cli_metadata .= "<br><b>Rating:</b> {$story['meta']['rating']}";
	if(!($story['meta']['ficstatus'] == '')) $cli_metadata .= "<br><b>Status:</b> {$story['meta']['ficstatus']}";
	if(!($story['meta']['summary'] == '')) $cli_metadata .= "<br><b>Summary:</b> {$story['meta']['summary']}";
	$cli_metadata .= '"';

	//Create the epub content and store it in a temp file
	exec("html2epub $cli_chapter $cli_metadata \"$temp_fname.html\"");
	$result = file_get_contents("$temp_fname.epub");

Anyway, it's good to have you back!

[EowynCarter]

Whoo that rocks !

I was thinking of making someting like this.
Calibre does fine in reading mobi files. as well as converting them from epub

[Mnementh]

ilovejedd,

Any chance of posting the whole code listing or maybe the updated files? When i try making the changes you've posted to the two versions of ffnet.source.php and epub.codec.php I have I get errors when trying to use them.

TIA

Mnementh

[ilovejedd]

Unfortunately, the ffnet.source.php that code snippet was taken from is in shambles and contains numerous errors. I just posted the fix relevant to the current "trunk" (basically rgibbo's Windows port of erayd's latest script). Even then, I had to fix up the snippet because it had bits and pieces of extraneous code.

By the way, mobileread seems to have changed & # 1 8 7 ; (remove spaces) to ». Try changing it back and see if that fixes things.

Okay, I re-downloaded and reverted back to rgibbo's version and implemented the changes I made. Here's to hoping it works.
http://www.utterlyinsane.org/fflag/ffnet.source.zip

The epub.codec.php snippet probably works, but if you can't study the source code and see where the problem is, I'd just leave the old version be. It's quite possible that the snippet won't work since it was tested with different versions of the scripts compared to what's been uploaded. Besides, the only relevant fix made between this and the version I uploaded a few days ago (in the custom recipe thread) is how it handles crossover fanfics.

Previous behavior for crossover fanfics is to create a new category/subject/tag Category 1 and Category 2 as its own separate category. With the fix, it will tag the fanfics as being part of each individual category, e.g. it will be tagged as Category 1 and Category 2 separately.

[Mnementh]

Looks like the problem was just with the ffnet.source.php, with the new version everything is working nicely again, thank you.

I really appreciate the time and effort everybody has put into this, just makes me wish I had something of my own to contribute.

/me goes to read up on php as there are quite a few sites that I would like to see a script for to do the same job as this one.

[ilovejedd]

I actually have working sources for TheSugarQuill and AdultFanfiction.net in case you need any for those sites. They should be attached somewhere in this thread.

[erayd]

Originally Posted by ilovejedd

erayd! Yay! You're back!

By the way, flag.erayd.net seems to be down the past few weeks.

Oops - moved servers a little while ago and forgot about flag - the web service is now back up and running where it was before. DNS has a 20min TTL, so shouldn't take too long to update. Webservice version is based on r29 svn, so it probably needs an upgrade. Doesn't seem to be broken though - whatever changes ffnet have made to their site don't seem to have troubled it.

Edit: although it does seem to be a tad slow.... I'll need to figure out why.

[araevin]

nah Im getting some error with the meta data the story authors names come out proper but the story names come out as flag and a bunch of numbers anybody else having this problem?

Also does anybody have any instructions as to get this working localy on a mac using mamp and OS X 10.5. I'm really unsure where to go from here with my host as Im unsure what its using but I bleve its linux. I also dont know about the dependancies (tidy & cli)

[erayd]

Hmm, I'll have another look - can you give me a failing storyid?

[araevin]

3928537

[ilovejedd]

If you only need the html version, you can use this temporarily:
http://localhost/fflag/getstory.php?...toryid=3928537

At least until erayd gets his server all fixed up (it's down again for me). It's easy enough to convert from html to epub using Calibre.

Unfortunately, can't help you with the Mac problem as I'm unfamiliar with both Mac and Linux.

@erayd
FanFiction.Net changed handling of crossover fanfics. The code snippet above (for the ffnet_get_meta function) should fix it, except, mobileread automatically substituted the >> character instead of showing the html entity. Are you sure svn r29 still works for you? Because right now, there's an extra <img src='/static/ficons/script.png' width=16 height=16 border=0 align=absmiddle> in between <td> and <a href.

[ruskie]

Hmm note to self: next time check first to see before writing your own.... Anyway my own is shell(bash) script based and outputs xhtml only. I'm not posting it anywhere yet as I'm abstracting some stuff as I want to cover other sites that might be unrelated to fanfic... And reading their ToS(yes I did this) I have it preserve the original link, copyright, author etc and as much data as I could possibly grab. I also limit requests to one per N seconds as their ToS states(paraphrased) you are allowed to use automatic tools as long as those tools do not exceed what a normal human being would do.

Name of my tool fanfic-dl

[Mnementh]

Originally Posted by ilovejedd

I actually have working sources for TheSugarQuill and AdultFanfiction.net in case you need any for those sites. They should be attached somewhere in this thread.

Yeah I've got those thanks. Not interested in TheSugarQuill but I've just had a look at adultfanfiction, I can now honestly say I've seen my first warning for Rimming <shudder> , the format of the site is pretty poor but looks like there might be some good stuff on there.