FLAG (Fanfiction.net Lightweight Automated Grabber)

[aleyx]

Originally Posted by erayd

Yes. JSON doesn't allow comments (which is a total pain; it's almost perfect otherwise), and INI doesn't support subsections or list of values without a named key (i.e. a numerically keyed array). YAML on the other hand supports both of those, plus it's also really nice to work with if hand-editing it.

Fair points. It's true that JSON is pretty annoying that way. About .ini, it does allow numerically keyed arrays:

Spoiler:

[section]
option[] = value1
option[] = value2

returns, depending on parse_ini_file's options, either:

array
(
    [option] => Array
    (
        [0] => value1
        [1] => value2
    )
)

or:

array
(
    [section] => Array
    (
        [option] => Array
        (
            [0] => value1
            [1] => value2
        )
    )
)

but I have no idea if it works in more that one dimension.

Well, YAML works, and that all that matters.

Originally Posted by erayd

My understanding is that $argc and $argv are always available when using the CLI SAPI, regardless of the register_argc_argv setting. When not using CLI, we don't care about them anyway... although it would probably be a good idea to stick a php_sapi_name() call in there somewhere to figure out what kind of startup handling is required.

Indeed; if we implement flagCLI and flagWEB classes, we'll have to abstract much of that part anyway. But that's not for a little while yet ^_^

Originally Posted by erayd

I *really* don't like that idea. While 5.3 would be very, very nice to have, it's not deployed anywhere near widely enough to count on. I'd rather not make life difficult for people trying to set this up, and a large number of distros don't even ship 5.3 yet (or at least not in an easily-obtainable way).

If we are going to go down that route though, I'd rather go the whole hog and jump to PHP 6 - but I still think that requiring anything higher than 5.2 is a bad idea.

Fair enough; it's true that PHP5.3 is only mostly available in this year's releases. I still think that FLAG's user base would probably keep current though; and those that don't probably just use the online version anyway. It would be interesting to have a poll about this here ("What's your PHP version?"), if only to know exactly our minimal version target. It'd be too bad if we deny ourselves the niceties of PHP5.3 (namespaces and get_called_class() saved my sanity) for just a couple of <=5.2 installs.

But in the end, you're the boss.

N.

[erayd]

Originally Posted by aleyx

About .ini, it does allow numerically keyed arrays...

I never knew you could do that - that's awesome . Is that in the ini spec, or is that a PHP-ism?

...I still think that FLAG's user base would probably keep current though; and those that don't probably just use the online version anyway. It would be interesting to have a poll about this here...

OK - let's do that. It may be that I'm wrong about the current state of PHP deployment and the savvyness of flag's users / potential users. Watch this space...

Edit: Have started a poll here.

[AtomicDryad]

Personally I don't think eliminating function prefixes is worth additional requirements.
I also fail to see the problem with -limited- global variables such as $CONFIG and $opt nested hashes (in runtime, cfg loading is another matter). KISS FTW.
I'll take a look at the github discussion when I get home tuesday. I'll also have a r29mod ready which will support drop-in source modules, with one that grabs stories from forum.spacebattles.com :P

[aleyx]

Originally Posted by AtomicDryad

Personally I don't think eliminating function prefixes is worth additional requirements.

It's really a very difficult problem: finding the right balance between the added deployment charge of upped requirements and the reduced coding charge these new versions allow. With my own work projects, I'm very lucky in that I can control everything from server installations to client deployment, so I don't have to worry about that balance. But you're right; in this project, coding comfort lose charge of against additional requirements. This is unfortunate for the programmers, but good news for users who can't or don't want to upgrade.

Originally Posted by AtomicDryad

I also fail to see the problem with -limited- global variables such as $CONFIG and $opt nested hashes (in runtime, cfg loading is another matter). KISS FTW.

Well, there's nothing wrong per se, it's really just a matter of code structure and maintainability. We could, in a functionality standpoint, stay with global variables; the script would technically still work just as well. But since we're going the OO route, it makes sense, structurally, to also make those into objects.
It also has relatively bad effects (http://en.wikipedia.org/wiki/Global_variable) in certain situations, but this project is still small enough that we don't really have to care about that. Unless FLAG continues to grow in functionality and complexity, in which case globals can become a hindrance rather than a help (that's my real-life experience about such things).

Originally Posted by AtomicDryad

I'll take a look at the github discussion when I get home tuesday. I'll also have a r29mod ready which will support drop-in source modules, with one that grabs stories from forum.spacebattles.com :P

Spacebattles.com? The name itself is made of win. I'm gonna have to check this one out.

N.

[ilovejedd]

Really cool to see all the current activity here. While my sole interaction with the FLAG source code nowadays are quickfixes whenever an FF.Net update breaks something, I'm looking forward to see innovations from the community.

[AtomicDryad]

Originally Posted by aleyx

Unless FLAG continues to grow in functionality and complexity, in which case globals can become a hindrance rather than a help (that's my real-life experience about such things).

I'm mostly trying to avoid overhead and complexity with variable lookups. If the structure of the variable is planned out right, I think it's unlikely to become an issue, though I won't raise a stink if you guys decide otherwise.

I tend to try to minimize complexity as much as possible in some areas as it usually ends up a headache. For example, I'm discovering what a fracking PITA php's DOM library is, because someone decided it would be clever to convert everything it touches to UTF8. All I want is xpath matching, nstead I'm having to deal with spacebattle's iso8859 txt getting mangled.

Spacebattles.com? The name itself is made of win. I'm gonna have to check this one out.

N.

It's famous for it's vs debates, there are threads here people discuss stuff like Darth Vader fighting Kai from Lexx, or if Culture GSV could defeat the Reapers from Mass Effect. There's also a very active creative writing forum with stuff not found on fanfiction (3 Iain Banks fanfics for example). Pulling stories from threads where comments - and author replies to comments - are mixed in with story posts is a big pain. At least it was; sbattles.source.php is mostly functional now. ^^

The module should be easy to adapt to most vbulletin forums, I think.

[AtomicDryad]

r29 mod3:
The sbattles source module parses threads on http://forum.spacebattles.com and saves a merged collection of individual posts based on adjustable criteria.

Most fanfiction on the creative writing board (http://forums.spacebattles.com/forumdisplay.php?f=18) is broken up by people commenting on the story, and the OP replying to these comments. By default, it will filter these threads in the following way:
Reject all posts not started by the OP.
Reject all OP posts with less than 256 bytes of text. (likely a short comment)
Reject all OP posts in which 40% of the text is in a quote. (likely a reply to a comment)
Allow all OP posts that have more than 2048 bytes of text regardless. (just to be -safe-)

These parameters can be adjusted with the -F flag as so:
fflag 'http://forums.spacebattles.com/showthread.php?t=172856' -F sbattles:oponly,allowsize=256,quoteratio=40,quotea llowsize=2048
-or- (save everything)
fflag 'http://forums.spacebattles.com/showthread.php?t=172856' -F sbattles:nooponly -F ffnet:url=http://multiple.module.options.can/be/adjusted
-or- (adjust parameters and save a full copy to full-autoname.html)
fflag 'http://forums.spacebattles.com/showthread.php?t=172856' -F sbattles:fullthread=AUTO,allowsize=1024,quoteratio =10,quoteallowsize=65535
-or - (default should be ok for most cases)
fflag 'http://forums.spacebattles.com/showthread.php?t=172856'

fflag -F module:help will show a module's options.
fflag --fullhelp will show everything.

My todo list is now:
Feature freeze to merge with trunk on github.

I've never used git, can someone copypasta commands I can use to upload a branch?

[aleyx]

Originally Posted by AtomicDryad

r29 mod3:
My todo list is now:
Feature freeze to merge with trunk on github.

I don't think you'll be able to just merge with trunk. You'll have to pull the trunk (thus creating your own local repository), adapt your code to the new structure, commit to a local branch, then push that local branch to github.

Originally Posted by AtomicDryad

I've never used git, can someone copypasta commands I can use to upload a branch?

Try: http://www.kernel.org/pub/software/s...er-manual.html

To get a local copy:
git clone git://github.com/erayd/flag.git

More than that, I've never done yet...

N.

[AtomicDryad]

Yah I randomly decided to fix an opera api module and was forced to learn me some githubs >.>

I've not noticed anything going on with fflag's repo, which seems to be a rewrite. Some of what's already up looks useful. You guys still active?

I'm rather disgusted with php_dom's encoding mangling, and what seems to be a truncation bug. Is anyone aware of an alternative, for xpath selection with less bloat?

[erayd]

Originally Posted by AtomicDryad

I've not noticed anything going on with fflag's repo, which seems to be a rewrite. Some of what's already up looks useful. You guys still active?

Yep, although it's slid down my priority list somewhat - lots of talk, but only one person was prepared to have any meaningful interaction with the source, and only three people responded to the PHP poll. I've also had almost no feedback on the new code structure - if people like that approach, we can go with that, but if people would really prefer something else I'd rather know that up front so I can write something to fit rather than wasting a whole lot of time on something that will simply end up being replaced.

If people would rather I simply make unilateral decisions on the design of the project, I'm quite happy to just write whatever I prefer and upload that - but the impression I got was that you guys wanted to be involved throughout this process.

My current task on this is to add additional documentation to the code I've uploaded to github, as nbriche (aleyx on here) seemed to have a bit of trouble understanding it all based on what's there at the moment.

I'm rather disgusted with php_dom's encoding mangling, and what seems to be a truncation bug. Is anyone aware of an alternative, for xpath selection with less bloat?

Are you referring to DOMXPath, or something else?

[AtomicDryad]

Originally Posted by erayd

Yep, although it's slid down my priority list somewhat - lots of talk, but only one person was prepared to have any meaningful interaction with the source, and only three people responded to the PHP poll. I've also had almost no feedback on the new code structure - if people like that approach, we can go with that, but if people would really prefer something else I'd rather know that up front so I can write something to fit rather than wasting a whole lot of time on something that will simply end up being replaced.

If people would rather I simply make unilateral decisions on the design of the project, I'm quite happy to just write whatever I prefer and upload that - but the impression I got was that you guys wanted to be involved throughout this process.

Hmm I wasn't aware of discussions not on this board, and sortof assumed proposal via code, like I did with the structure and functionality I added in the last r29mod. I poke at what is on github thusfar, and try to add r29 functionality, but it didn't seem like a completed skel. Admittedly I only skimmed it.

I did notice that the url matching seemed limited, with only 1 entry and no passing of match returns. The hash based stuff I coded allows for multiple patterns, detects ambiguity, and provides a means to call different functions based on which pattern got a hit. I'll try to make it more OO.

As far as decision making, I don't want to say the creator of fflag shouldn't be making them. But I also see myself proposing changes to things if it would result in more versatility or effeciency. If that would be more frustrating to you, I'll fork and annotate what's on git, instead of fork/alter/pull request.

Are you referring to DOMXPath, or something else?

Yeah, that. All I need for vbulletin parsing is xpath matching. Not the extra troublesome baggage. Unfortunately, regex is not the best; matching the beginning is no problem, but finding the end where there are nested tags tends to be troublesome.

Perl's HTML::Treebuilder doesn't frak with encoding for instance.

[erayd]

Originally Posted by AtomicDryad

Hmm I wasn't aware of discussions not on this board...

There have been limited discussions on github, although only between myself and nbriche. I assumed you'd seen them, as you are following the repo.

...but it didn't seem like a completed skel.

It's not - it's the start of a skeleton; I've been waiting for feedback on the design before I sink some serious time into it - I'd rather get it right the first time than have to go back and redo it because people don't like the structure.

I did notice that the url matching seemed limited...

It's more or less a placeholder at this point - like I said, I've been waiting for feedback.

As far as decision making, I don't want to say the creator of fflag shouldn't be making them. But I also see myself proposing changes to things if it would result in more versatility or effeciency. If that would be more frustrating to you, I'll fork and annotate what's on git, instead of fork/alter/pull request.

Please feel more than free to suggest whatever comes to mind. I'm not tied to any particular approach, so if you don't like what's there, or you want to change it, please sing out.

I would like to request that we have a complete (or at least mostly complete) framework first though, before adding lots of special cases.

Yeah, that. All I need for vbulletin parsing is xpath matching. Not the extra troublesome baggage. Unfortunately, regex is not the best; matching the beginning is no problem, but finding the end where there are nested tags tends to be troublesome.

Agreed - regex is largely useless for that.

Perl's HTML::Treebuilder doesn't frak with encoding for instance.

Unfortunately this isn't written in perl . PHP's handling of various character encodings isn't great in places, I've found that it sometimes pays to do conversions first to avoid munging the data.

[aleyx]

Originally Posted by erayd

It's not - it's the start of a skeleton; I've been waiting for feedback on the design before I sink some serious time into it - I'd rather get it right the first time than have to go back and redo it because people don't like the structure.

For myself, I think that the current design, as I understand it, is on the right track:

A main Flag object, which houses:

• A global configuration object, accessible from anywhere,
• An array of sources, loaded on-demand,
  • each source could have its own customized configuration object, maybe derived from the global. The global would just pass the raw source options to the derived for final parsing, along with pre-computed parameters (if the global found some URLs matching that source, for exemple).
• An array of codecs, also loaded on-demand
  • as for the sources, a codec could have its own configuration object.
• An access to an object which'll handle the updating introduced in AtomicDryad's mods

The only point I'm dubious about is the one we've already touched on github, that is, making everything a factory singleton. I have nearly the same approach in my coding, and with the advent of PHP5.3 it also looks a lot like yours: a main SingletonFactory class, which stores factories derived from itself (before 5.3, each factory stored its instance within its own static member). But I tend to limit the factories to the fewest possible, and my SingletonFactory base class only stores and retrieves instances, no init() method. Validation of classes is managed by __autoload() as needed, and the rest would be done by the main class (since it's only ever called by the main class anyway).

So I have mainly three types of classes:

• Factories derived from SingletonFactory (flagFactory here), which have a simple get() static method to retrieve the instance (or create it): $people=People::get().
  Those do the heavy lifting: database access, loadings, savings, etc. They are the only classes allowed to instantiate anything (except for the bootstrap, obviously), and they do it through methods like (Person)$person = $people->GetPerson($id) or $list_people = People::get()->GetEveryone(), or any other Get method I need (GetDeparmtent($id_department), GetBySecurityLevel($level), etc.). It never does any manipulation of data, except those necessary for storing or retrieving it.
  
  For exemple, I'd have a Codecs class, which could allow me to get an existing codec instance, a list of all instances, the names of all codecs, etc.
  
• Data manipulation classes, which describe a data model and manipulates them.
  Those are not factories, they don't instantiate anything. However, they are instantiated with everything they need to know about themselves, or can compute what they don't know. From there, they can return info about themselves, or computed info based on parameters: a Person class, instantiated by $person=People::get()->GetPerson(12) could have a $person->GetFullName($title=false, $initials=false) method which would return Mr John Doe, or J. Doe. Also methods like $person->SetSecurityLevel($level), which would check the validity of the parameter before calling People::SetSecurityLevel($id, $level) for the database update.
  
  For our project, I'd have a Source class, like we have, but we wouldn't have to make it a factory, because any instantiation management we'd need would be taken care of by a Sources class (and ultimately by flagFactory, its parent class). Same for a Codecs (extends FlagFactory) class and a Codec class.
• And finally Controllers, which parse page and session parameters, call the necessary factory methods according to those parameters, and pass all that to a template engine (PHPTAL for my projects) for presentation.
  Here we'd have two controllers: CLI and web, but all the parameter parsing part is so complex that it's offloaded to the Config object(s). The controllers will merely have to call the Sources methods to retrieve the data, then the Codecs methods to save that data as files, all the while outputting a summary of the process.
  
  The template engine would also be way overkill, so the web controller would also need to take care of all the HTML generation.
  

Now that I think of it, it also means that flagFactory::$sourceList and flagFactory::$codecList would actually belong to Sources and Codecs respectively, and the main Flag and Config class would have two more members: $sources which would store the Sources instance, and $codecs for the Codecs instance. I can't think of anyone else that would need to keep both of them as members.

For the class autoloading, I myself like to use a separate autoloader.inc.php file, with only an __autoload() function, and called either in the php.ini (not really doable here), or in the 'flag' bootstrap.

I'm sorry that I hadn't had much time 'till now to work on this (work and all), but now that I have a week off I'll be able to have some fun with this!

N.

[chyron8472]

erayd,

First, I must say that I use the FLAG Webservice all the time, and I far prefer its formatting over and above the FanFictionDownloader. Good Job.


Now, I do have a minor issue that I would like to address.

When I download a story as ePub and edit it in Sigil, every story I download has 3 identical anchors per TOC entry. When I "validate epub" the majority of complaints I get are "ID value X is not unique" and have to run a Search&Replace to remove the 2 extraneous ones per chapter.



Can you fix this?




Again, for anyone who downloads stories form Fanfiction.net, FLAG is the best. Fanfiction Downloader, on the other hand, formats ebooks to look like mere .txt files.

[erayd]

Hi chyron8472,

The issue you're referring to is caused by the version of Calibre used to do the HTML -> ePub conversion after FLAG finishes doing its thing.

Assuming that this has also been fixed in Calibre, this should be sorted next time I update the backend - it won't be immediate, but it is on my list.