FLAG (Fanfiction.net Lightweight Automated Grabber)

[Mnementh]

Hmmmm I might be wrong here (probably am to be honest) but I was under the impression that pcre.backtrack_limit would be in php.ini?

I couldn't find it in my php.ini so double checked via google that that is where it should be and according to http://www.phpro.org/manual/ref.pcre.html I assumed correctly.

I added "pcre.backtrack_limit 1000000" to php.ini and restarted apache but when checking phpinfo() via the XAMPP control panel pcre.backtrack_limit was still showing as 100,000.

Am I missing something obvious here or am I looking in the wrong place?

[ilovejedd]

Originally Posted by Mnementh

You know just as you were posting that I re-read my post and saw the length and thought I'd read something about possible story length issues I was about to go re-reading the whole thread lol.

I'll try that and see if it works thanks.

Lol, kinda helps that I know around what pages I should look.

Originally Posted by Mnementh

Hmmmm I might be wrong here (probably am to be honest) but I was under the impression that pcre.backtrack_limit would be in php.ini?

I couldn't find it in my php.ini so double checked via google that that is where it should be and according to http://www.phpro.org/manual/ref.pcre.html I assumed correctly.

I added "pcre.backtrack_limit 1000000" to php.ini and restarted apache but when checking phpinfo() via the XAMPP control panel pcre.backtrack_limit was still showing as 100,000.

Am I missing something obvious here or am I looking in the wrong place?

Yes, it's in php.ini. Depending on your set-up, though, you might have multiple php.ini configuration files. On XAMPP 1.7.0, the php.ini that controls the webserver is in \xampp\apache\bin\ while the one that controls the command-line is in \xampp\php\. No idea where else it could be hiding if you used a different method to install.

*sigh* I really should finish that update I promised erayd. I've also got to work on that pre-made FLAG-compatible XAMPP install. I'm working less hours now so I should have a bit more time to devote to FLAG. I just need to stop getting distracted by random stuff.

[Mnementh]

Ahhh I didn't realise there were seperate php.ini files for the webserver and cli, I was changing the cli version and checking the webserver DOH...

Changed it in both locations and now everything seems to be working fine, again many thanks for your help ilovejedd.

Edit: Just tried to give you some K+ but apparently I need to "spread my loving around" before giving you any more, sorry. Sounds vaguely wrong

[ilovejedd]

Thank you for trying to give me karma. Just a thought, though, have you given karma to erayd, rgibbo and kovid? Without them, this wouldn't be possible.

[Mnementh]

I'm actually quite embarrassed that the thought never occurred to me to do that, thank you for pointing it out.

K+ has been sent their way.

[ilovejedd]

Originally Posted by Mnementh

I'm actually quite embarrassed that the thought never occurred to me to do that, thank you for pointing it out.

K+ has been sent their way.

Cool.

Okay, just as an update since I think it's been 4 months since I promised erayd the cleaned-up scripts with my changes. I've been working on them here and there, however, everytime I actually accomplish stuff, there's always some other thing that crops up that I have to change.

Here's a laundry-list of things I'm trying to do related to FanFiction.Net:

• dynamic catalog creation - For those people with iPhones and iPod Touches, no need to manually check story ids and convert to epub. You can browse new and updated fanfics from favorite categories within Stanza directly. Requires Apache, PHP and Calibre (for command-line conversion functions only).
• automated cover creation - Currently using PHP GD2. If someone can help me with Imagick on PHP, it would be much appreciated. For the artistically-inclined, cover design ideas would be great, too.
• switch to utf-8 encoding - The current html.codec.php uses Windows-1252 and apparently, Stanza doesn't like it all that much.
• revised epub creation - I was getting all these weird characters in the title and summary/description when they contain multi-byte characters. Silly me for not thinking that dos only supports ascii, not utf-8. I'm adding very simple opf creation so Calibre can pull the metadata from opf and not command-line.
• oebps zip file output format - Similar to the one Calibre creates when you add an html file to the library. epub is pretty okay as a final format, but it's a pain in the neck to edit. This would create a zip file that contains an opf file, a simple (x)html file containing the whole fanfic, and optionally, stylesheets and images (cover, etc). Hopefully, that would make manual edits a bit easier and it should be fairly simple to convert from the oebps zip to a valid epub. Hmm, Calibre might actually have a command-line for this available. I'll have to check. If it does, then that's one less headache for me.
• semantically correct mark-up - I have a heavily edited html.codec.php because the default conversion settings for Calibre wasn't playing nice with some of the fanfics and wreaking havoc with automatic chapter detection. Unfortunately, that resulted in a not so semantically correct html file that I am now wanting to fix.
• switch to Calibre 0.6 - Editing and testing affected codec.php files for compatibility with the new conversion structure in Calibre 0.6.
• And finally, aready-made portable XAMPP environment for Windows users so they can run FLAG locally without needing to know how to set-up Apache and PHP on their computers.

Lol, it looks like I'm trying to accomplish too many things at once, but I'd rather have it done in one go so erayd can add all pertinent changes to the subversion.

[EowynCarter]

switch to utf-8 encoding - The current html.codec.php uses Windows-1252 and apparently, Stanza doesn't like it all that much.

On the same kind of things, some of our letters get messed. like œ. What bugs me is that éèâ and some other works.

[ilovejedd]

Originally Posted by EowynCarter

On the same kind of things, some of our letters get messed. like œ. What bugs me is that éèâ and some other works.

Exactly what I'm seeing, hence, the switch to utf-8 which is actually what epub prefers. Some characters are retained because they exist in cp1252 and because the html entity is used instead of the actual character (e.g. é, ê ).

[ilovejedd]

Originally Posted by ilovejedd

Here's a laundry-list of things I'm trying to do related to FanFiction.Net:

• dynamic catalog creation - For those people with iPhones and iPod Touches, no need to manually check story ids and convert to epub. You can browse new and updated fanfics from favorite categories within Stanza directly. Requires Apache, PHP and Calibre (for command-line conversion functions only).
• And finally, a ready-made portable XAMPP environment for Windows users so they can run FLAG locally without needing to know how to set-up Apache and PHP on their computers.

Someone requested a copy of the Stanza catalog scripts and I was able to clean it up enough for a public release. A ready-made XAMPPLite environment is also included. Hopefully, I'll be able to get the cleaned-up FLAG scripts to erayd by next weekend.

Links are here.

[hime]

I am having trouble with the online FLAG Webservice (http://flag.erayd.net/)

When I try to convert a file to LRF I get the following error:

Fanfiction Lightweight Automated Grabber

Please wait while your story is processed...

/usr/bin/php: error while loading shared libraries: libltdl.so.3: cannot open shared object file: No such file or directory


Redirecting...

Then it redirects me to a page which says:

No input file specified.

This used to work for me until recently

Any idea what is wrong?

[Mnementh]

Originally Posted by hime

I am having trouble with the online FLAG Webservice (http://flag.erayd.net/)

When I try to convert a file to LRF I get the following error:

Fanfiction Lightweight Automated Grabber

Please wait while your story is processed...

/usr/bin/php: error while loading shared libraries: libltdl.so.3: cannot open shared object file: No such file or directory


Redirecting...

Then it redirects me to a page which says:

No input file specified.

This used to work for me until recently

Any idea what is wrong?

Have you tried creating a different format such as epub?

If that doesn't work it's possible erayd hasn't updated his source scripts since FF changed certain things (I haven't used the online version for a while).

[erayd]

Originally Posted by hime

Any idea what is wrong?

The autoupdate script on my server enthusiastically updated that library file, but didn't recompile PHP to adjust for the change, and as a result PHP had no idea where to find it. I'm recompiling it now, so the webservice should be back shortly.

Originally Posted by Mnementh

If that doesn't work it's possible erayd hasn't updated his source scripts since FF changed certain things (I haven't used the online version for a while).

I haven't - I'm still waiting on ilovejedd for his modified version - the one on my site is a wee bit out of date. I've had conflicting reports on whether or not that version still works - some say yes, some say no.

ilovejedd: Any progress?

[erayd]

The webservice at http://flag.erayd.net/ is now back up and running, and I've fixed the script there to grab the metadata properly - it didn't seem to be doing this before. As far as I'm aware everything should be working, so please give me a shout if it's not.

[ilovejedd]

Originally Posted by erayd

The autoupdate script on my server enthusiastically updated that library file, but didn't recompile PHP to adjust for the change, and as a result PHP had no idea where to find it. I'm recompiling it now, so the webservice should be back shortly.

I haven't - I'm still waiting on ilovejedd for his modified version - the one on my site is a wee bit out of date. I've had conflicting reports on whether or not that version still works - some say yes, some say no.

ilovejedd: Any progress?

Sorry. Was working on it slowly but now, the hard drive on my primary laptop crashed so work has gone doubly slow. Actually, I think it might be the whole laptop that's acting up...

I have copies of the scripts, but it's pretty hard looking at code on an 8.9" screen for long periods of time.

Hopefully, I might be able to finalize it this weekend.

[erayd]

Originally Posted by ilovejedd

Sorry. Was working on it slowly but now, the hard drive on my primary laptop crashed so work has gone doubly slow. Actually, I think it might be the whole laptop that's acting up...

I have copies of the scripts, but it's pretty hard looking at code on an 8.9" screen for long periods of time.

Hopefully, I might be able to finalize it this weekend.

All good, I'm in no rush - just let me know when you have something .