FLAG (Fanfiction.net Lightweight Automated Grabber)

[Mnementh]

Hi erayd,

First off thanks for writing this great tool in the first place.

You r29 linux version still works (or at least it does with everything I've tried it with so far, however that does not include any crossover fics) it just lacks the nice formatting and naming of the changes ilovejedd has made. I agree it would be great if we could get those changes incorporated into the original as when you try and use ilovejedds versions with your fflag under linux everything breaks.

[erayd]

Hmmm.... maybe I should refactor this thing to eat XML recipes - might make life a tad easier :-).

[ilovejedd]

Originally Posted by Mnementh

That and the bad taste in your mouth from all the girly drinks lol.

Please at least tell me it was coctails and not alcopops...

Basically, it was just 2/3 Hawaiian Punch + 1/3 Malibu Rum. Not sure if that counts as a cocktail. I am a girl so I do happen to like girly drinks. I just don't drink a lot as I don't really socialize much.

Originally Posted by erayd

OK.... things are starting to seem just a tad crazy with all these versions, and every time I go to consolidate them more changes have sprung up!

You can blame FanFiction.Net for that. Most of the changes I post are due to FanFiction.Net changing stuff. I'm guilty of making epub.codec.php a tad too complicated, though.

Originally Posted by erayd

ilovejedd: Can you post a massive diff between whatever (working) version you have at the moment and the files I've attached at the top of this thread? I'll merge them and stick them up the top, that way at least everyone will be working from the same base rather than all these crazy chaotic different bits and pieces, and the files at the top will actually work with ffnet - as you said earlier, the current release is just a tad broken.

Sure no problem. Just one question... how do I create the diff? Also, my source is based off of rgibbo's code. Not sure if it will work on Linux...

Originally Posted by erayd

Hmmm.... maybe I should refactor this thing to eat XML recipes - might make life a tad easier :-).

Not really. They make changes to those, too, and while they announce changes to site layout, updates to the RSS feed tend to go under the radar. Believe me, I've tried.

[erayd]

Originally Posted by ilovejedd

Sure no problem. Just one question... how do I create the diff? Also, my source is based off of rgibbo's code. Not sure if it will work on Linux...

Thanks :-). To create the diff, just have the folder for each version side-by-side and run any kind of program over them than can generate standard diff/patch output on them. On Linux I'd usually use the command 'diff', or use subversion's diff command. Not sure about Windows - I know there are several, but I wouldn't know what they're called sorry.

As far as the working on Linux bit goes, don't worry about that - half the reason I want a diff is so I can debug things as I merge it.

Edit: Doesn't matter if your version is based off rgibbo's - you can still do the diff against r29, and it'll merge the changes rgibbo made too.

[ilovejedd]

Originally Posted by erayd

Thanks :-). To create the diff, just have the folder for each version side-by-side and run any kind of program over them than can generate standard diff/patch output on them. On Linux I'd usually use the command 'diff', or use subversion's diff command. Not sure about Windows - I know there are several, but I wouldn't know what they're called sorry.

As far as the working on Linux bit goes, don't worry about that - half the reason I want a diff is so I can debug things as I merge it.

Hmm, just this once, how about I send you a zip of the full working directory (from clean source)? Should only take me a couple of days to do. Notepad++ doesn't seem to do diffs at the moment and I don't want to waste time installing and learning new software if it's something that's quick and easy for you to do now. After that, I'll experiment on subversion software on the virtual machine so I can just send you diffs in the future.

Ugh. I know I promised I'd clean up the source and stuff but it's just too frakkin hot to think (98F here in SoCal). In the meantime, here's a quick band-aid for ffnet.source.php. Again, this one's messy as hell and contains redundant code here and there but it should work...

PHP Code:

<?php

function ffnet_source_info() {
    return "\tFetches stories from fanfiction.net";
}

function ffnet_get_story($storyid, $meta = true) {
    $ffurl = 'www.fanfiction.net';
    $story['source'] = $ffurl;
    $ffurl = 'http://' . $ffurl;

    //get initial info
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_USERAGENT, 'FLAG/1.0 (Fanfiction.net Lightweight Automated Grabber)');
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
    curl_setopt($ch, CURLOPT_URL, "$ffurl/s/$storyid/1");
    $story['pages'][1] = curl_exec($ch);

    //extract metadata
    $story['meta'] = ffnet_get_meta($story['pages'][1], $meta);

    //fetch remaining pages
    if(!$meta) {
        if(isset($story['meta']['chapters'])) {
            foreach($story['meta']['chapters'] as $key => $chaptitle) {
                if(!isset($story['pages'][$key])) {
                    curl_setopt($ch, CURLOPT_URL, "$ffurl/s/$storyid/$key");
                    $story['pages'][$key] = curl_exec($ch);
                }
            }
        }
        else $story['meta']['chapters'][1] = $story['meta']['title'];

        //clean pages
        foreach($story['pages'] as &$currpage) {
            preg_match("/<!-- start story -->(.+)<!-- end story -->/Usi", $currpage, $matches);
            $currpage = $matches[1];
        }
    }

    return $story;
}

function ffnet_get_meta($page, $nochapters) {
    $matches = array();

    //category and title
    if (preg_match('/.*<a href.+>.+Crossover<\/a> »/Ui', $page, $matches)) {
        preg_match('/.*<a href.+>(.+) Crossover<\/a> » <b>(.+)<\/b>/Ui', $page, $matches);
        $matches[1] = str_ireplace(',', '', $matches[1]);
        $tag = str_ireplace(' and ', ',', $matches[1]);
    }
    else preg_match('/.+ » <a href=.+>(.+)<\/a> » <b>(.+)<\/b>/Ui', $page, $matches);

    //author
    preg_match("/<a href='\/u\/[0-9]+\/.+'>(.+)<\/a>/Ui", $page, $author);
    $meta['title'] = $matches[2];
    $meta['author'] = $author[1];
    $meta['category'] = $matches[1];
    $meta['crosscat'] = $tag;

    //rating, lang, category
    preg_match("/Rated: <a href='http:\/\/www.fictionratings.com\/guide.php' target='rating'> (.+)<\/a> - (.+) - (.+)( - Reviews: <a href='.+'>[0-9]+<\/a>)?( - Updated: ([0-9-]+))? - Published: ([0-9-]+) (- (Complete)+ )?- id:([0-9]+) /Ui", $page, $matches);
    $meta['rating'] = $matches[1];
    $meta['language'] = $matches[2];
    $meta['genre'] = $matches[3];
    $meta['date_pub'] = $matches[7];
    $meta['date_update'] = $matches[6];
    $meta['complete'] = $matches[9] == 'Complete' ? true : false;
    $meta['ficstatus'] = $matches[9] == 'Complete' ? 'Complete' : 'In-Progress';
    $meta['id'] = $matches[10];

    //description
    preg_match('/<meta name="description" content="(.+), (.+)(, pairing: (.+))?,  (.+)">/Ui', $page, $matches);
    $meta['category2'] = $matches[1];
    $meta['genre2'] = $matches[2];
    $meta['char'] = $matches[4];
    $meta['summary'] = $matches[5];
    $meta['summary'] = iconv('UTF-8', 'ISO-8859-15//TRANSLIT//IGNORE', $meta['summary']);

    //chapters
    if($nochapters){
        if(preg_match("/<SELECT title='chapter navigation' Name=chapter onChange=\".+\">(<option.+)<\/select>/Ui", $page, $matches)) {
            preg_match_all("/<option  value=[0-9]+ (selected)?>([0-9]+). (.+)(?=<option|$)/Ui", $matches[1], $matches);
            foreach($matches[2] as $key => $chapnum) $chaparr = $chapnum;
            $meta['chapters'] = $chaparr;
        }
    }
    else {
        if(preg_match("/<SELECT title='chapter navigation' Name=chapter onChange=\".+\">(<option.+)<\/select>/Ui", $page, $matches)) {
            preg_match_all("/<option  value=[0-9]+ (selected)?>([0-9]+). (.+)(?=<option|$)/Ui", $matches[1], $matches);
            foreach($matches[2] as $key => $chapnum) $chaparr[$chapnum] = $matches[3][$key];
            $meta['chapters'] = $chaparr;
        }
    }

    return $meta;
}

?>

[erayd]

Originally Posted by ilovejedd

Hmm, just this once, how about I send you a zip of the full working directory (from clean source)? Should only take me a couple of days to do. Notepad++ doesn't seem to do diffs at the moment and I don't want to waste time installing and learning new software if it's something that's quick and easy for you to do now.

Sure - it takes me all of a second to generate it from a clean working dir, so why not :-).

[Mnementh]

Gah I'm about to start banging my head against a brick wall...

ilovejedd, when using your ffnet.source.php code printed a few messages above to try and create an epub using the cli I get the following error message:

Warning: Invalid argument supplied for foreach() in C:\...\html.codec.php on line 73

The line in question is: foreach($story['meta']['chapters'] as $key => $title) {

I've tried it with every copy of html.codec.php I have and although the line numbers are different it falls over at the same bit of code everytime.

I tried changing $title to $chaptitle as it is in ffnet.source.php just in case it was something as simple as that with no joy.

Sorry to keep bothering you but I'm about to run out of already grabbed fics and I really don't want to have to go back to doing it manually plus your version of the script makes it look so much better.

TIA for any help.

[ilovejedd]

Originally Posted by Mnementh

ilovejedd, when using your ffnet.source.php code printed a few messages above to try and create an epub using the cli I get the following error message:

Warning: Invalid argument supplied for foreach() in C:\...\html.codec.php on line 73

The line in question is: foreach($story['meta']['chapters'] as $key => $title) {

I've tried it with every copy of html.codec.php I have and although the line numbers are different it falls over at the same bit of code everytime.

I tried changing $title to $chaptitle as it is in ffnet.source.php just in case it was something as simple as that with no joy.

Oh, whoops.

My mistake, change:

PHP Code:

function ffnet_get_story($storyid, $meta = true) { 


to

PHP Code:

function ffnet_get_story($storyid, $meta = false) { 


I kinda also use this for creating cover images and $meta=true means it only retrieves metadata but not the actual pages/chapters. I really should change the default behavior in the script to retrieve all chapters unless specified to get metadata only. By the way, undo any changes you made to html.codec.php. That might actually break some things.

*sigh* Really should start working on that clean source.

[Mnementh]

LOL I know how bad it can be to get started on something you just really can't be bothered with I've been needing t put timesheets in for work for the last 6 weeks I just can't be bothered to get off my ar*e and write them

After making that change the error message has disappeared but I still get no epub file created (I changed html.codec.php back) , there must be something wrong on my end I just can't figure out what I've changed to make things suddenly stop working. Even if ffnet.source.php wasn't pulling down the pages properly surely it should still make the epub it would just be empty?

[ilovejedd]

Oi.

It's empty because it's not finding the title. fflag checks validity of a storyid by checking if it has a title. This one's not my fault. MobileRead changed all "& # 1 8 7 ;" (no spaces) to "»". You need to do a manual search and replace for that. As far as I know, those should only be in the regular expression for category and title.

[Mnementh]

You m'lady are a genius.

I should have realised that actually you did post about it before I just forgot about it.

I guess I'm gonna have to spend a while pulling down a load of fics now in case they change things and break it again... I was trying not to do that because of FFNets TOS oh well...

Again many thanks

[emmadiva]

I am so excited about this program! I bought my kindle specifically to read fanfic. LOL -I'm shamelessly addicted to it. Please forgive my lack of knowledge regarding creating code and recipes. (Is that even what it's called?) Is there some sort of help/tutorial available? I've tried to figure it out based on the posts in this forum but it's a bit confusing.

I used the great url link to download off of ffnet and imported the html to my desktop and dragged it into calibre. It worked perfectly!! However I'd like to see if I can find a way to pull fic from another site. I mostly get my fic from Twilighted.net. They have a number of fics that never make it off the site. It's a login protected site. I searched the forums hoping that someone else may have developed something to work with the site but no luck. I'm hoping to find a solution with this grabber.

Last question -- I promise. How do the category/mass downloads work? I think I read that correctly - That it will download all the fics from a specific author or category and then update when there is a new title? LOL - This would be a dream come true.

Even if you can't help me -- THANK YOU for building such an invaluable program!!!!!!

[ilovejedd]

Originally Posted by emmadiva

I am so excited about this program! I bought my kindle specifically to read fanfic. LOL -I'm shamelessly addicted to it. Please forgive my lack of knowledge regarding creating code and recipes. (Is that even what it's called?) Is there some sort of help/tutorial available? I've tried to figure it out based on the posts in this forum but it's a bit confusing.

I will be working on a guide/tutorial for Windows XP. Alas, it's not exactly very simple since it involves running a "web server" on your computer. One big problem is the lack of a graphical user interface. Recipe, I think, is somewhat specific to Python or Calibre. I don't think I've ever heard it in relationship to PHP or C/C++.

Originally Posted by emmadiva

I used the great url link to download off of ffnet and imported the html to my desktop and dragged it into calibre. It worked perfectly!! However I'd like to see if I can find a way to pull fic from another site. I mostly get my fic from Twilighted.net. They have a number of fics that never make it off the site. It's a login protected site. I searched the forums hoping that someone else may have developed something to work with the site but no luck. I'm hoping to find a solution with this grabber.

This one might be difficult. I don't really want to open an account on a site I will never visit again. Would you mind PM-ing me your log-in or maybe creating one for me? The email account I use for forums is already overflowing and I'm somewhat averse to adding more. Also, do you know if they ban users that "hammer" their site? I won't really be hammering but while trying to test data retrieval, etc, I'm going to have to re-download a couple of fanfics over and over again, until I can get the code working. Before I do this, though, why don't you suggest to the owner/moderators to provide a link for a "printable version"?

Originally Posted by emmadiva

Last question -- I promise. How do the category/mass downloads work? I think I read that correctly - That it will download all the fics from a specific author or category and then update when there is a new title? LOL - This would be a dream come true.

This is already sort of doable within Calibre (instructions and recipe here), at least for FanFiction.Net. Some caveats with this method, though:

1. saves all fanfics into one giant file
2. date-based so it might re-download fanfics if there's a time overlap between scheduled downloads
3. no pagebreaks between chapters

There's probably a few others I haven't thought of. The Calibre recipe is somewhat broken right now. FanFiction.Net changed some things Sunday night and I still haven't gotten around to updating my website with the fixed ffnet source script. I should be able to fix this when I go home.

I am planning on creating a FLAG script for mass downloads, however, I'm not sure when I'll be able to work on that. I still need to get a clean source over to erayd and I've yet to start working on it. Surprisingly, I suddenly got a social life. Normally, I never go out.

[emmadiva]

Originally Posted by ilovejedd

I will be working on a guide/tutorial for Windows XP. Alas, it's not exactly very simple since it involves running a "web server" on your computer. One big problem is the lack of a graphical user interface. Recipe, I think, is somewhat specific to Python or Calibre. I don't think I've ever heard it in relationship to PHP or C/C++.

EEK -- I don't even know what a server is. LOL - Perhaps it's not a great solution for me. However, maybe by the time you get a tutorial done, I'll be a little more computer/internet savvy.

Originally Posted by ilovejedd

This one might be difficult. I don't really want to open an account on a site I will never visit again. Would you mind PM-ing me your log-in or maybe creating one for me? The email account I use for forums is already overflowing and I'm somewhat averse to adding more. Also, do you know if they ban users that "hammer" their site? I won't really be hammering but while trying to test data retrieval, etc, I'm going to have to re-download a couple of fanfics over and over again, until I can get the code working. Before I do this, though, why don't you suggest to the owner/moderators to provide a link for a "printable version"?

*blushes* Oops. I feel a bit silly now. I took your advice regarding a printable version and it turns out there is one. All this time I've been copying, pasting and reformatting. In my defense though, it's the teeniest-tiniest little icon and gets lost in the site graphics.

Thanks for the super simple suggestion!!!

Originally Posted by ilovejedd

This is already sort of doable within Calibre (instructions and recipe here), at least for FanFiction.Net. Some caveats with this method, though:

1. saves all fanfics into one giant file
2. date-based so it might re-download fanfics if there's a time overlap between scheduled downloads
3. no pagebreaks between chapters

There's probably a few others I haven't thought of. The Calibre recipe is somewhat broken right now. FanFiction.Net changed some things Sunday night and I still haven't gotten around to updating my website with the fixed ffnet source script. I should be able to fix this when I go home.

WOW -- That's a super fast fix. No worries if you don't get to it today. (Of course, I'm speaking for myself.) It's just greatly appreciated no matter how long it takes you to get to it.

Originally Posted by ilovejedd

I am planning on creating a FLAG script for mass downloads, however, I'm not sure when I'll be able to work on that. I still need to get a clean source over to erayd and I've yet to start working on it. Surprisingly, I suddenly got a social life. Normally, I never go out.

A social life?! Shame on you. I just found this nifty grabber and you're already abandoning it. LOL -- Just kidding. It seems like you put an incredible amount of time and energy into this. You MORE than deserve to have a social life. Go have fun!

Thanks for the super quick reply. I appreciate the suggestions and help.

[ilovejedd]

Originally Posted by emmadiva

EEK -- I don't even know what a server is. LOL - Perhaps it's not a great solution for me. However, maybe by the time you get a tutorial done, I'll be a little more computer/internet savvy.

Basically, you'll be running software for running websites on your home PC. Don't worry, it's not really all that difficult, you just need to follow instructions.

Originally Posted by emmadiva

WOW -- That's a super fast fix. No worries if you don't get to it today. (Of course, I'm speaking for myself.) It's just greatly appreciated no matter how long it takes you to get to it.

Actually, I already had it fixed Sunday night. I was planning on reading some fanfics before I went to sleep and bam, it wasn't working. I had to fix it. I just haven't had the time to upload the fixed script to my webhost.

Originally Posted by emmadiva

A social life?! Shame on you. I just found this nifty grabber and you're already abandoning it. LOL -- Just kidding. It seems like you put an incredible amount of time and energy into this. You MORE than deserve to have a social life. Go have fun!

This isn't actually my project. It's erayd's. I'm just helping out a bit. Besides, the social life will be gone soon. My best friend from Colorado is visiting since she'll be gone to South Korea for a year or so. Her flight leaves next week so we're making the most of the time we have.