FLAG (Fanfiction.net Lightweight Automated Grabber)

[aleyx]

Originally Posted by AtomicDryad

This will only work for .html files written with r29mod2, which include a 'TotalPages' meta tag.

I'm note sure about <meta> tags. We can as easily (well, using XPath) count the number of <h3> elements, or more ideally, <h3 class="chapter_title">. As for the story ID, it's actually a useful info, which should be added to the 'front page'. Maybe between the summary and the dates, something like:

PHP Code:

<p>{$source->name} story <a id="story_id" href="{$story->url}">{$story->id}</a></p> 


This would allow us to easily retrieve the story ID, and give the reader the ability to directly go to the original.

Actually, id'ing the other meta fields would be a good idea. It would also allow the import of an .html, then convert to another format without having to reparse the story on the source site.

Originally Posted by AtomicDryad

The story summary is now included at the top of the output file, under title and author.

You should enclose it with a <p>, and use <em> instead of <i> (deprecated). The dates, as well (<p> instead of <br/>).

Originally Posted by AtomicDryad

Now searches for the story text via XPath, which should be more reliable. Requires php with dom and simplexml, change USEDOM=>1 to USEDOM=>0 if you get a php error.

I don't know... I think the 'dual mode' adds unneeded complexity. Users of FLAG most likely installed their own PHP stack, and so can install the needed extensions if they're not already bundled. A simple error message with the name of the missing dependencies, as with the codecs, should suffice here.

Originally Posted by AtomicDryad

Debug option for coders: -D will output debug text, and write all web GETS to debug-hostname-filename. If debug-hostname-filename exists, it will load that, instead of downloading. Good if you need to test without hammering fanfiction.net's server.

That's good indeed. We'll have to start thinking about a more robust logging facility though; we're having a lot of different functions logging in a lot of places, centralizing them would be good. Too bad PHP doesn't have something like Python's logging module...

Originally Posted by AtomicDryad

Alot of configuration options stored in $CONFIG hasharray at the top of the 'fflag' file, runtime options in $opt hasharray.

There's a good settings class here: http://www.php.net/manual/en/functio...file.php#92085

That would allow the configuration options to be removed from the code and put in a .conf.

[erayd]

OK, I've created a repo on github (http://github.com/erayd/flag), and am currently writing a basic OO skeleton for it - this will be uploaded either today or tomorrow, and then provided you're all happy with it we can start merging stuff.

Aleyx - you wanted to refactor the existing code into OO - feel free - but please hold fire until I've uploaded the skeleton; that way we can make sure nobody's duplicating work.

Re logging - I'll add this as part of the base, it makes more sense to be using the same thing from the start rather than trying to achieve the same thing in several different ways. Ditto for config.

[AtomicDryad]

Originally Posted by aleyx

I'm note sure about <meta> tags. We can as easily (well, using XPath) count the number of <h3> elements, or more ideally, <h3 class="chapter_title">.

You make a good point, though something can be said for the meta tag; one only need load the first 1024 bytes of a file. That would only matter for updates on -big- collections though.

Actually, id'ing the other meta fields would be a good idea. It would also allow the import of an .html, then convert to another format without having to reparse the story on the source site.

For now ff.net seems to have a nice list of js variables that I've been plundering, instead of parsing the HTML directly. If this is to work on multiple sites it might be best to standardize the meta fields for fflag, not the site(s).

You should enclose it with a <p>, and use <em> instead of <i> (deprecated). The dates, as well (<p> instead of <br/>).

I've already changed it to a <div> since I last updated. That seems more flexible.

I don't know... I think the 'dual mode' adds unneeded complexity. Users of FLAG most likely installed their own PHP stack, and so can install the needed extensions if they're not already bundled. A simple error message with the name of the missing dependencies, as with the codecs, should suffice here.

The added complexity adds a single boolean check at runtime. The regexp itself is rather simple thing that might serve use as a fallback instead. I'd really like to make sure site parsing is either really robust or easy to adjust, for the inevitable breakage caused by bored fanfiction.net webmasters.

That's good indeed. We'll have to start thinking about a more robust logging facility though; we're having a lot of different functions logging in a lot of places, centralizing them would be good. Too bad PHP doesn't have something like Python's logging module...

perhaps:
log($level,$str): level would be an int with 0=normal output 1=warn 2=error 3=debugging. This could be configured via
$CONFIG['loglevel']=array(0=>'php://stderr',1=>array('php://stderr','/var/log/whatever'),2=>'/var/log/whatever',3=>NULL); (syntax error in that, likely).
--loglevel 3 on commandline would add php://stderr to everything. One could also add functionality to adjust the level per namespace but that's probably going overboard.

[aleyx]

Originally Posted by erayd

OK, I've created a repo on github (http://github.com/erayd/flag), and am currently writing a basic OO skeleton for it - this will be uploaded either today or tomorrow, and then provided you're all happy with it we can start merging stuff.

Aleyx - you wanted to refactor the existing code into OO - feel free - but please hold fire until I've uploaded the skeleton; that way we can make sure nobody's duplicating work.

That's okay, I've only began small stuff. Codec and Source base classes, Codecs and Sources factories, a Story class. Did those more as a proof of concept (and my own amusement... mainly that, actually) than real work, really.

Originally Posted by erayd

Re logging - I'll add this as part of the base, it makes more sense to be using the same thing from the start rather than trying to achieve the same thing in several different ways. Ditto for config.

Makes sense.

N.

[AtomicDryad]

You should enclose it with a <p>, and use <em> instead of <i> (deprecated). The dates, as well (<p> instead of <br/>).

Just tested a new bookreader that doesn't deal nicely with CSS at all, nor does it properly handle <em>, so I'll have to disagree now :P '<i>' may be deprecated, but that also means more likely to be supported.

I'm eying the htmlout module and am considering a few things; the css file should be selectable and optional (like --css /home/user/book.css or --nocss). If the latter, it would use 'oldschool' html.

This also makes me wonder about commandline (and www) option selection in general. --css and --nocss would only apply for -f html and output modules that use htmlout as a base. Should module-specific options like that be a suboption, so there isn't a flood of options to maintain? Instead of:
fflag -i 65535 -f html --css /tmp/test.css --noindex
you would have:
fflag -i 65535 -f html --formatopt css=/tmp/test.css,noindex
or
fflag -i 65535 -F nocss,noindex,simple,onepage
or
fflag -i 65535 -f pdf -F font=/path/to/font.ttf,keephtml,reversecolor
etc.
The modules themselves can handle their own options and provide a means for --help to list them.

[aleyx]

Originally Posted by AtomicDryad

You make a good point, though something can be said for the meta tag; one only need load the first 1024 bytes of a file. That would only matter for updates on -big- collections though.

That's a good point.

Originally Posted by AtomicDryad

I've already changed it to a <div> since I last updated. That seems more flexible.

That works for HTML-only output, but converters for other formats may be confused by a paragraph that doesn't have a <p> tag.

Originally Posted by AtomicDryad

The added complexity adds a single boolean check at runtime. The regexp itself is rather simple thing that might serve use as a fallback instead. I'd really like to make sure site parsing is either really robust or easy to adjust, for the inevitable breakage caused by bored fanfiction.net webmasters.

Point. It'll make sense later to write a wrapper function around the two cases, or we'll have to rewrite the USEDOM test at each XPath use.

Originally Posted by AtomicDryad

perhaps:
log($level,$str): level would be an int with 0=normal output 1=warn 2=error 3=debugging. This could be configured via
$CONFIG['loglevel']=array(0=>'php://stderr',1=>array('php://stderr','/var/log/whatever'),2=>'/var/log/whatever',3=>NULL); (syntax error in that, likely).
--loglevel 3 on commandline would add php://stderr to everything. One could also add functionality to adjust the level per namespace but that's probably going overboard.

I think we should rather wait for erayd's implementation first, but I expect it'll look about the same.

N.

[aleyx]

Originally Posted by AtomicDryad

Just tested a new bookreader that doesn't deal nicely with CSS at all, nor does it properly handle <em>, so I'll have to disagree now :P '<i>' may be deprecated, but that also means more likely to be supported.

Currrrrrses!

Was the bookreader using straight HTML, or an ePub?

Originally Posted by AtomicDryad

I'm eying the htmlout module and am considering a few things; the css file should be selectable and optional (like --css /home/user/book.css or --nocss). If the latter, it would use 'oldschool' html.

If there's a --css option, the --nocss should be implicit, and so unneeded; unless I'm missing something.

Originally Posted by AtomicDryad

This also makes me wonder about commandline (and www) option selection in general. --css and --nocss would only apply for -f html and output modules that use htmlout as a base. Should module-specific options like that be a suboption, so there isn't a flood of options to maintain? Instead of:
fflag -i 65535 -f html --css /tmp/test.css --noindex
you would have:
fflag -i 65535 -f html --formatopt css=/tmp/test.css,noindex
or
fflag -i 65535 -F nocss,noindex,simple,onepage
or
fflag -i 65535 -f pdf -F font=/path/to/font.ttf,keephtml,reversecolor
etc.
The modules themselves can handle their own options and provide a means for --help to list them.

That's a very interesting idea! It would also allow for multiple output at once. Maybe something simplier, like:

fflag -i 65535 -f pdf:font=/path/to/font.ttf,keephtml,reversecolor -f html:css=stuff.css,noindex

And now that I think about it, the nocss option would be useful in such a scheme:

fflag -i 65535 --css stuff.css -f epub -f rtf -f html:nocss,noindex

The epub and rtf outputs would use the 'global' css option, but not the html output.

Which... doesn't really make sense in a real-world use case, but the concept is sound, I think.

Of course, the generation itself would be a little awkward as the code stands now, what with having to reset the config arrays for each format, but once we have individual instances for formats it'll be easy as pie.

N.

[AtomicDryad]

Regarding modules and commandline, there's also the question about what to do with url arguments. As is, for a new source plugin one must edit arg.inc.php and add a regexp. The following way of doing it would remove that need...the idea is real modularity; being able to drop in any valid .php in the source directory without editing the base program.
args.inc would process things like this:
parse $argc/$argv/parameters, setting non module specific options (which can include disabling a module etc)
stuff everything else into $arr['extra'] or ['moduleopts']
determine what modules to load, do so, and call the modules 'getinfo' function. One of the things that would do would be to add to an $argspec (or somesuch) array, which is a nested hash of regex strings and other info like this:

Spoiler:

$argspec=array(
array(regex=>"!(fanfiction\.net/s/([0-9]+)|^([0-9]+)$)!i",
site=>"ffnet",
type=>"story"),
array(regex=>"!(fanfiction\.net/u/([0-9]+))!i",
site=>"ffnet",
type=>"list"),
array(regex=>"!(fanfiction\.net/community/[^/]+/([0-9]+))!i",
site=>"ffnet",
type=>"list"),
array(regex=>"!archiveofourown\.org/works/([0-9]+)!i",
site=>"aooo",
type=>"story"),
);

Next, any ambiguous non-options like urls or files are processed to populate a 'downloads' nested hash. I've got a working example of this.
fflag -D http://archiveofourown.org/works/00000000 http://www.fanfiction.net/s/3748479/1/A_Very_Big_Weapon imjustaguy_-_A_Very_Big_Weapon.html http://fanfiction.net/u/12345
results in:

Spoiler:

[downloads] => Array
(
[story] => Array
(
[1] => Array
(
[site] => aooo
[id] => 00000000
[arg] => http://archiveofourown.org/works/00000000
)

[2] => Array
(
[site] => ffnet
[id] => fanfiction.net/s/3748479
[arg] => http://www.fanfiction.net/s/3748479/1/A_Very_Big_Weapon
)

)

[file] => Array
(
[3] => Array
(
[arg] => imjustaguy_-_A_Very_Big_Weapon.html
[site] => ffnet
[filename] => imjustaguy_-_A_Very_Big_Weapon.html
)

)
[list] => Array
(
[4] => Array
(
[site] => ffnet
[id] => fanfiction.net/u/12345
[arg] => http://fanfiction.net/u/12345
)

)

)

When fflag starts to process it's downloads:
* It first checks $opt['downloads']['file'] to determine what the file's site/story is, then:
* $opt['downloads']['list'], calling the appropriate source module's _get_storylist function, adding what it finds to:
* $opt['downloads']['story'], which is processed last, fflag selecting sites on a per entry basis, and output filenames where specified.

[AtomicDryad]

On a slightly related note, what -is- a good alternative to fanfiction.net? I'm discovering the site I used for testing is mostly pr0ns.
...stargate atlantis featuring rodney mckay pr0ns x.x

[aleyx]

Originally Posted by AtomicDryad

Regarding modules and commandline, there's also the question about what to do with url arguments. As is, for a new source plugin one must edit arg.inc.php and add a regexp. The following way of doing it would remove that need...the idea is real modularity; being able to drop in any valid .php in the source directory without editing the base program.

Migrating to OO will ease much of it. We'd start by including everything we need, from codecs to sources to global classes and basic includes:

PHP Code:

    $includes_paths = INCLUDE_DIR."/{inc,codec,source}/*.{class,inc,codec,source}.php";
    foreach (glob($includes_paths, GLOB_BRACE ) as $include_file) {
        require_once($include_file);
    } 


All source-specific configuration would be contained in one object, like FfmlSource, defined in "ffml.source.php", and be accessible from a Sources factory class.

If we have an URL but no source in the arguments, we'll be able to invoke Sources::getSource($url); and get back an object Source that will tell us everything we need to know about the URL: how to extract the story, its meta, whether the URL contains a story or a list of stories, etc.

However, not all inputs will be URLs, so I think it'd be better to allow the use of the same argument format for sources as for codecs:

./fflag --some-option --other-option=/some/file -o /some/dir -i ffml:id1,id2 -i siye:id3,id4 -i ficpress:id5,id6 -f html:option1,option2 -f epub:option3,option4 url1 url2

Processing this, we'd have your arrays:

Spoiler:

array(
    'ffml' => array( 'id1', 'id2', 'url1' ),
    'siye' => array( 'id3', 'id4', 'url2' ),
    'ficpress' => array ( 'id5', 'id6' )
)

Processing the array:

Spoiler:

PHP Code:

$stories = array();
foreach ( $processing_list as $source => $ids_list ) {
    Log::debug("Processing {$source->name} stories and/or lists.");
    $source = Source::getSource($source);
    foreach ( $ids_list as $id ) {
        Log::debug("Now processing $id");
        if ( $source->isList($id) ) {
            array_merge($stories, $source->getList($url);
        }
        else {
            $story = $source->getStory( $id );
            $stories[] = $story;
        }
        Log::debug("$id: done");
    }
    Log::debug($source->name.": done.");
}
foreach ( $stories as $story ) {
    Log::info("We've found and are now exporting: ".$story->__toString());
    $story->export();
    Log::info($story->__toString()." has been exported to ".$story->filename);
} 


The getStory() function will know the difference between an ID and an URL, and go from there. Idem for the export() function, which will be able to get from the opts what are the output formats and options, and call Codec instances accordingly.

Erayd, would it be possible to set up a bugzilla or something? We're getting technical on many different issues, and I don't know if a forum thread is the best place?

We'll need to document the code, too. How does everyone feel about Doxygen?

N.

[aleyx]

Originally Posted by AtomicDryad

On a slightly related note, what -is- a good alternative to fanfiction.net? I'm discovering the site I used for testing is mostly pr0ns.
...stargate atlantis featuring rodney mckay pr0ns x.x

I've set my FFNet filters to >20,000 words. I've also limited myself to a handful of C2s and author alerts and seldom, if ever, browse unfiltered lists. It cuts back the not-nice a little.

Aside from that, it really depends on your fandoms, I guess. I'm partial to SIYE, but that's because I'm a sucker for romance. YMMV.

There's a couple sites cited back in this thread, IIRC. We'll have to go back and check any source plugin to update them, anyway.

Oh, and McKay pr0n? Ew.

N.

[erayd]

Have just uploaded the skeleton so far to github - still working on it, but would be good to have some feedback on what's there so far - anything you guys want changed?

Edit: Commits are announced via flag_dev on twitter, for those who are interested in following the project, but don't have a github account.

[aleyx]

Originally Posted by erayd

Have just uploaded the skeleton so far to github - still working on it, but would be good to have some feedback on what's there so far - anything you guys want changed?

Thanks! I'll send you my comments through github (user nbriche).

N.

[aleyx]

Just realized that AtomicDryad wouldn't see my messages to you from github, so here are some thoughts:

Spoiler:

- Do you have a specific reason to use YAML as a conf language? If not, we can use a basic .ini or .json, which are much more common and would allow us to make away with the spyc dependency.

- I don't really understand flagFactory. Most of it seems to deal with sources, but every class inherit it. Why not make those classes call a static flagSources class as needed? Something like:

PHP Code:

class flagSources {

    /**
     * Array of flagSource objects
     */
    static private $sources = array();

    /**
     * Add a new source
     */    
    static function add($source) {
        if 
        self::$sources[$source->tag] = $source;
    }

    /**
     * Do we have that source?
     */
    static function has($source) {
        return in_array($source, array_keys(self::$sources));
    }

    /**
     * Return the source object
     */
    static function getSource($sources) {
        return self::$sources[$sources];
    }

    /**
     * Get all sources
     */
    static function getList() {
        return self::$sources;
    }
    
} 


Some autoadd:

flagSource.class.php:

PHP Code:

abstract class flagSource {
    private $tag = "";
    private $name = "";

    public function __construct() {
        Sources::add($this);
    }
} 


ffnet.source.php:

PHP Code:

<?php

class FfnetSource extends flagSource {
    private $tag = "ffnet";
    private $name="Fanfiction.net";
}
new FfmlSource();

?>

- In the misc.class.php:
Those two functions are only ever called by the config class; so maybe move them in that class? Actually, maybe create a flagCLI class and move everything CLI-related in it, in preparation of a flagWEB class.

- in config.class.php:
Depending on the PHP configuration, $argc and $argv may not be available (it depends on register_argc_argv). Using $_SERVER['argc'] and $_SERVER['argv'] would ensure we have them, and as a bonus, we have less global variables.


- If we can bump the PHP minimal requirement to 5.3, we can also namespace the project, and do away with all the 'flag' prefixes. That would clear up the code somewhat.

Thoughts?

N.

[erayd]

Originally Posted by aleyx

Just realized that AtomicDryad wouldn't see my messages to you from github...

Nah, he can see them - he's on github too, and also following the flag repo.

Do you have a specific reason to use YAML as a conf language?

Yes. JSON doesn't allow comments (which is a total pain; it's almost perfect otherwise), and INI doesn't support subsections or list of values without a named key (i.e. a numerically keyed array). YAML on the other hand supports both of those, plus it's also really nice to work with if hand-editing it.

Spyc is basically there in case the yaml extension is missing - the yaml extension is better, but depends on libyaml, and isn't bundled with PHP (it's in pear), so I figured having a fallback was a wise idea - hence that check at the beginning of flagFactory::init().

I don't really understand flagFactory. Most of it seems to deal with sources...

I think we covered that in the github discussion, but re your sources comment - that's there because it has to do with file loading and class instantiation, so flagFactory seemed like the logical place to put it. The other reason it's there is because that code needs to be run prior to config loading, and flagFactory::init() is run *very* early in the process.

I've got no problem with changing things around though - if you guys would rather do things a different way, I'm quite happy to change my approach.

...are only ever called by the config class; so maybe move them in that class?

They actually started out in the config class, but I moved them into misc because they didn't really have anything to do with config .

Actually, maybe create a flagCLI class and move everything CLI-related in it, in preparation of a flagWEB class.

I like that idea - will do.

Depending on the PHP configuration, $argc and $argv may not be available (it depends on register_argc_argv).

My understanding is that $argc and $argv are always available when using the CLI SAPI, regardless of the register_argc_argv setting. When not using CLI, we don't care about them anyway... although it would probably be a good idea to stick a php_sapi_name() call in there somewhere to figure out what kind of startup handling is required.

...as a bonus, we have less global variables.

$argc and $argv are always global, predefined by the interpreter - nothing we can do about that unfortunately.

If we can bump the PHP minimal requirement to 5.3, we can also namespace the project, and do away with all the 'flag' prefixes. That would clear up the code somewhat.

I *really* don't like that idea. While 5.3 would be very, very nice to have, it's not deployed anywhere near widely enough to count on. I'd rather not make life difficult for people trying to set this up, and a large number of distros don't even ship 5.3 yet (or at least not in an easily-obtainable way).

If we are going to go down that route though, I'd rather go the whole hog and jump to PHP 6 - but I still think that requiring anything higher than 5.2 is a bad idea.