FLAG (Fanfiction.net Lightweight Automated Grabber)

[erayd]

Originally Posted by Amalthia

Thanks for getting back to me so quickly.

No problem :-). Of course, those instructions are for integrating that recipe into fflag, which I'm not sure is exactly what you wanted...

[Iteria]

Is there anyway that you could adapt this for Fictionpress.com? It's the sister-site to Fanfiction.net, so I don't think it would be too much trouble. I like to read there too and it would be cool to carry those stories with me also.

[ilovejedd]

Just discovered FLAG today and it's been pretty awesome so far (at least the online version is). Thank you very much for sharing this with us.

I used to do something similar semi-manually back in 2001~02 (I've got a working mirror of my favorite authors and fanfics on my hard drive) but the new system made it hard for me to use my semi-automated workflow. I was looking for an easier way to archive fanfics to my PC and read them on my iPhone (hooray for Stanza; Safari crashes way too often) and I'm happy to have found this.

Now, I'm going to have to find some way to install Apache/PHP on my PC as having command-line access to this program for batch processing will be absolutely invaluable.

Thanks again for this wonderful program!

[ilovejedd]

Originally Posted by rgibbo

Here's my version of flag for windows, it is now similar to the original so it *might* run on linux (i dont have linux so i don't know if it works on there or not)

Thanks for this. I have it installed on a virtual machine (VPC 2007: XP Pro SP2, Apache 2.2.11 and PHP 5.2.8) and it seems to be working quite well. I just copied the code from lrf.codec.php to get epub.codec.php working. Lol, I was too impatient to wait for an "official" fix. I had to copy all the contents of the calibre (v0.4.128) install directory to lib\calibre because of all the python dependencies of html2epub, though.

I appreciate how modular the code for this program is. It makes it very easy to add new sources and output file types. I'm currently attempting to make source files for other sites. I've also customized ffnet.source.php, epub.codec.php and lrf.codec.php to include the fanfic summary/description.

I'm attaching a source file for Fictionpress.com as well as codec and Fanfiction.net source files modified from rgibbo's Windows version. I reckon the source files should work for both Linux and Windows. I checked ffnet.source.php from the tarball and rgibbo's zip and there didn't seem to be any difference between the two.

[malkie13]

Love this app, though thanks to it, I'll be lucky to see my PRS505 anytime soon (the missus reads heaps off of fanfiction.net). That said, is there anyway to incorporate this functionality into Calibre directly (I'm runnin OSX, Calibre is how I dump all my books onto the reader)?

Edit: I'm using the web version to do the converting, not the actual app on a machine.

[erayd]

Yes, but not by me - you'd need to ask kovidgoyal for this functionality, as it would need to be rewritten as a calibre grabber.

[malkie13]

*nod* I've posted on the forum for Calibre as well, figured I'd hit both and see what people have done.

*chuckle* My wife absently asked me about reading FF on the Reader and I stumbled on your app totally on accident. It may well lead to me buying another Reader just for her because of this apps functionality.

[ilovejedd]

Help! I want to create a PHP script that returns just the cleaned-up html output of the story. How do I go about doing that?

For example:

URL

http://localhost/getstory.php?source=ffnet&storyid=1234567

I assume I'll have to create a getstory.php based on a modified fflag? Not exactly sure how I go about modifying fflag, though. Also, some modules (e.g. ffnet.source.php) display status messages. I assume I just need to comment out all printMsg lines from those modules?

EDIT:
Nevermind, got it working. Just had to edit two sections in the main fflag program. Didn't even need to touch any of the other modules. Additions in green, deletions in red.

	//Config
	require_once('inc/args.inc.php');
	//$config = getCliArgs();
	$config['source'] = $_GET['source'];
	$config['format'] = 'html';
	$config['storyid'] = $_GET['storyid'];

	//Output the compiled file if it the story is valid
	if ($story['meta']['title'] != "") {

		//Output file
		echo $output;

		/*
		//Create the file name
		if (!isset($config['outfile']) || $config['outfile'] == '-') $config['outfile'] = getBaseFilename($story) . ".{$config['format']}";
		
		//Output the file
		printMsg("\t* Writing output to {$config['outfile']}...\n");
		file_put_contents($config['outfile'], $output);
		*/

Oh yeah, just had a root canal so I had a bit of free time today. Just putting the finishing touches on a SugarQuill source.php and it'll be ready for posting.

Alas, porting the ffnet source to a Calibre/Python recipe is taking much longer. I have a working recipe used in conjuction with a local Apache+PHP server (that's the reason why I needed variable passing via URL) but converting the chapter retrieval and page cleaning logic to Python escapes me.

[ilovejedd]

Here are some source files, including a fixed version of fpress.source.php. The one I posted earlier had a bug in it, I think. Also included are sources for The Sugar Quill, A Single Spark (http://sesskag.com) and there's a whole folder for AdultFanFiction.net (darn their multiple subdomains). For the AFF sources, I couldn't figure out a way to pass the subdomain without creating additional command-line arguments, hence, I created a source for each archive/subdomain. I've only tested the source file for the InuYasha archive but I think the logic should work for all the other subdomains.

For the The Sugar Quill source.php, the original plan was to adapt moggie's TheSugarQuill Calibre recipe. I eventually scrapped it since I can't make sense of the Python syntax. I'm going to need to do some heavy reading on Python before I can program anything more complex than Hello World. Thankfully, PHP is remarkably similar to C/C++ which is what I'm familiar with. All source files are based heavily on erayd's ffnet.source.php.

Again, a really big erayd for releasing this program and making the source code available and rgibbo for porting it to Windows. It's become one of my most used programs as of late.

[ilovejedd]

Doesn't fetch story id: 4880173. Meta information is detected correctly but the chapter content is blank.

[erayd]

Originally Posted by ilovejedd

Doesn't fetch story id: 4880173. Meta information is detected correctly but the chapter content is blank.

Thanks - will investigate and try to figure out why this is.

[erayd]

Originally Posted by ilovejedd

Help! I want to create a PHP script that returns just the cleaned-up html output of the story. How do I go about doing that?

What exactly did you mean here? Are you aware that fflag can output cleaned HTML already as a standard output format?

Currently supported outputs are HTML, RTF, BBeB (LRF) and ePub.

To everyone submitting modified source:Would it be useful if I gave you guys commit access to the subversion repo for this? Alternatively, is there any chance you'd be willing to submit patches to the original source? That way we can keep everything in sync and stop it diverging too much - the way things are, there's a chance we could end up with way too many branches spread all over the thread in various zipfiles.

And finally, thanks heaps to all the users and modifiers in this thread :-). It's great to see people taking an interest, I must admit I thought for a while there that there were only a couple of people interested.

Second final note: To all who have contributed code, are you happy with your changes being integrated into trunk and released under GPLv2?

[Valloric]

Originally Posted by erayd

Would it be useful if I gave you guys commit access to the subversion repo for this?

Okay idea.

Originally Posted by erayd

Alternatively, is there any chance you'd be willing to submit patches to the original source?

Much better idea.

Originally Posted by erayd

Second final note: To all who have contributed code, are you happy with your changes being integrated into trunk and released under GPLv2?

If your original source code was release under the GPL, while it's polite to ask if you could integrate their code, they don't really have a say in this. They are bound to publicly release their changes to the code under a GPL-compatible license. Legally, you don't have to ask their permission, and they don't have to give you one. You can integrate their code either way.

I'm saying all this to make sure you are aware of your rights. I applaud the intent to ask first though.

[erayd]

Originally Posted by Valloric

If your original source code was release under the GPL, while it's polite to ask if you could integrate their code, they don't really have a say in this. They are bound to publicly release their changes to the code under a GPL-compatible license. Legally, you don't have to ask their permission, and they don't have to give you one. You can integrate their code either way.

I'm saying all this to make sure you are aware of your rights. I applaud the intent to ask first though.

I am aware of my rights... that said, my original code wasn't actually licensed to anyone - it was made available, but there was no provision for modification or distribution (and as such, all mods in this thread are technically illegal). I would like to rectify this and place it all under a license everyone is happy with, rather than simply catering to my own whims. I prefer GPLv2 for a number of reasons, but if others would prefer something else I'd be quite happy to consider it.

I wrote this because I wanted the functionality, and I released it because I thought others may also want it - the contributions were an unexpected surprise, but I'd like to tidy this up now while it's still easy, before there are so many changes it all gets a bit out of hand.

[ilovejedd]

Originally Posted by erayd

Thanks - will investigate and try to figure out why this is.

Figured it out. Lol, I didn't sleep until it was fixed. The fanfic was fairly long and the pcre.backtrack_limit was too restrictive (100,000). Changed the value to 1,000,000 and it works now.

Originally Posted by erayd

What exactly did you mean here? Are you aware that fflag can output cleaned HTML already as a standard output format?

Yep. I was just trying to figure out a way so it returns the HTML output directly/in-line. e.g. http://localhost/getstory.php?source=ffnet&storyid=1234567 would contain the cleaned HTML source instead of creating a separate cleaned HTML file. Needed this for def print_version() in a Calibre news feed while I'm still learning Python and unable to convert the chapter detection and page cleaning logic.

Originally Posted by erayd

To everyone submitting modified source:Would it be useful if I gave you guys commit access to the subversion repo for this? Alternatively, is there any chance you'd be willing to submit patches to the original source? That way we can keep everything in sync and stop it diverging too much - the way things are, there's a chance we could end up with way too many branches spread all over the thread in various zipfiles.

Would be great except I don't know how to use subversion, although I guess I could learn. Also, I'm not sure if my submitted code is Linux compatible.

Originally Posted by erayd

And finally, thanks heaps to all the users and modifiers in this thread :-). It's great to see people taking an interest, I must admit I thought for a while there that there were only a couple of people interested.

I think one of the reasons you're not getting more hits is because people aren't aware your program/service exists. I used to archive fanfics the hard way and I eventually gave up doing that since it just takes too much effort. Alas, short of posting a link on the FanFiction.Net forums (which might result in them asking you to take down the web service), I can't think of a way for people to learn about FLAG.

Originally Posted by erayd

Second final note: To all who have contributed code, are you happy with your changes being integrated into trunk and released under GPLv2?

Sure, no problem.

Addendum:
I just read a bit about subversion. I think patches would be better so we don't accidentally break something.