FLAG (Fanfiction.net Lightweight Automated Grabber)

[ilovejedd]

Did a Google search for the site and the results that came were not exactly for public consumption. I can't help you right now unless I want to get fired. I can probably check out the site when I get home and see if I can create a source for it. Just PM me for the url if you want.

[Mnementh]

If you were replying to me, you already have a source for adultfanfiction.net, that's the site I was on about.
Some of the stories look good I'm just not so sure I want to read about rimming lol.
I'm just hoping some of the stories might have a few more adult situations and language in them as unfortunately most of the fanfiction sites appear to have the idea that only kids want to read the stuff posted there so they have very strict rules on what can be posted about.
If all the stories are just thinly veiled (or not as the case maybe) attempts at porn then I'm not bothered with the site after all the old saying is true, Participation is always better than observation

[ilovejedd]

Oh, okay. I had no idea what that was so I thought it was a website. Silly me. Pretty sure there are some decent mature fanfics on AFF. For one thing, a lot of the good fanfics from FF.Net made their way to AFF when FF.Net started weeding out NC-17 fics. Unfortunately, finding the gems among all that rubbish might prove difficult.

[EowynCarter]

Originally Posted by Mnementh

Yeah I've got those thanks. Not interested in TheSugarQuill but I've just had a look at adultfanfiction, I can now honestly say I've seen my first warning for Rimming <shudder> , the format of the site is pretty poor but looks like there might be some good stuff on there.

I'll be intersted in the sugar quill code

[ilovejedd]

TheSugarQuill source already exists.
https://www.mobileread.com/forums/sho...752#post358752

[EowynCarter]

Tanks for the link. i hadn't seen it

[ilovejedd]

Grr... FanFiction.Net enjoys making my life miserable. They changed the page layout again. Hopefully, it'll stay the same for some time.

Here's the updated source. Caveat, didn't have the inclination to look for one of the cleaned up ones. This one is probably full of stray code that I use for testing. Oh well, works for me, though. For anyone using this with Calibre recipes, I've also updated the "print version" generator on utterlyinsane.org

PHP Code:

<?php

function ffnet_source_info() {
    return "\tFetches stories from fanfiction.net";
}

function ffnet_get_story($storyid, $meta = false) {
    $ffurl = 'www.fanfiction.net';
    $story['source'] = $ffurl;
    $ffurl = 'http://' . $ffurl;

    //get initial info
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_USERAGENT, 'SIRG/0.1 (Stanza iPhone RSS Generator)');
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
    curl_setopt($ch, CURLOPT_URL, "$ffurl/s/$storyid/1");
    $story['pages'][1] = curl_exec($ch);

    //extract metadata
    $story['meta'] = ffnet_get_meta($story['pages'][1], $meta);

    //fetch remaining pages
    if(!$meta) {
        if(isset($story['meta']['chapters'])) {
            foreach($story['meta']['chapters'] as $key => $chaptitle) {
                if(!isset($story['pages'][$key])) {
                    curl_setopt($ch, CURLOPT_URL, "$ffurl/s/$storyid/$key");
                    $story['pages'][$key] = curl_exec($ch);
                }
            }
        }
        else $story['meta']['chapters'][1] = $story['meta']['title'];

        //clean pages
        foreach($story['pages'] as &$currpage) {
            preg_match("/<!-- start story -->(.+)<!-- end story -->/Usi", $currpage, $matches);
            $currpage = $matches[1];
        }
    }

    return $story;
}

function ffnet_get_meta($page, $nochapters) {
    $matches = array();

    //category and title
    if (preg_match('/.*<a href.+>.+Crossover<\/a> »/Ui', $page, $matches)) {
        preg_match('/.*<a href.+>(.+) Crossover<\/a> » <b>(.+)<\/b>/Ui', $page, $matches);
        $matches[1] = str_ireplace(',', '', $matches[1]);
        $tag = str_ireplace(' and ', ',', $matches[1]);
        //$tag = preg_replace('/<.+>/Ui', '', $tag);
        //$matches[1] = preg_replace('/<.+>/Ui', '', $matches[1]);
    }
    else preg_match('/.+ » <a href=.+>(.+)<\/a> » <b>(.+)<\/b>/Ui', $page, $matches);

    //author
    preg_match("/<a href='\/u\/[0-9]+\/.+'>(.+)<\/a>/Ui", $page, $author);
    $meta['title'] = $matches[2];
    $meta['author'] = $author[1];
    $meta['category'] = $matches[1];
    $meta['crosscat'] = $tag;

    //rating, lang, category
    preg_match("/Rated: <a href='http:\/\/www.fictionratings.com\/guide.php' target='rating'> (.+)<\/a> - (.+) - (.+) - Reviews: <a href='.+'>[0-9]+<\/a>( - Updated: ([0-9-]+))? - Published: ([0-9-]+) (- (Complete)+ )?- id:([0-9]+)/Ui", $page, $matches);
    $meta['rating'] = $matches[1];
    $meta['language'] = $matches[2];
    $meta['genre'] = $matches[3];
    $meta['date_pub'] = $matches[6];
    $meta['date_update'] = $matches[5];
    $meta['complete'] = $matches[8] == 'Complete' ? true : false;
    $meta['ficstatus'] = $matches[8] == 'Complete' ? 'Complete' : 'In-Progress';
    $meta['id'] = $matches[9];

    //description
    preg_match('/<meta name="description" content="(.+), (.+)(, pairing: (.+))?,  (.+)">/Ui', $page, $matches);
    $meta['category2'] = $matches[1];
    $meta['genre2'] = $matches[2];
    $meta['char'] = $matches[4];
    $meta['summary'] = $matches[5];
    $meta['summary'] = iconv('UTF-8', 'ISO-8859-15//TRANSLIT//IGNORE', $meta['summary']);

    //chapters
    if($nochapters){
        if(preg_match("/<SELECT title='chapter navigation' Name=chapter onChange=\".+\">(<option.+)<\/select>/Ui", $page, $matches)) {
            preg_match_all("/<option  value=[0-9]+ (selected)?>([0-9]+). (.+)(?=<option|$)/Ui", $matches[1], $matches);
            foreach($matches[2] as $key => $chapnum) $chaparr = $chapnum;
            $meta['chapters'] = $chaparr;
        }
    }
    else {
        if(preg_match("/<SELECT title='chapter navigation' Name=chapter onChange=\".+\">(<option.+)<\/select>/Ui", $page, $matches)) {
            preg_match_all("/<option  value=[0-9]+ (selected)?>([0-9]+). (.+)(?=<option|$)/Ui", $matches[1], $matches);
            foreach($matches[2] as $key => $chapnum) $chaparr[$chapnum] = $matches[3][$key];
            $meta['chapters'] = $chaparr;
        }
    }

    return $meta;
}

?>

[pilotbob]

You should ask them to create an RSS feed or other type of REST API.

BOb

[ilovejedd]

Originally Posted by pilotbob

You should ask them to create an RSS feed or other type of REST API.

BOb

They already have an RSS feed which contains links to individual stories. Don't see how that has any bearing on fetching stories, unless they start generating RSS feeds for each story. Not practical, imho. A REST API would be useful, though, but something tells me they might not go for it.

Originally Posted by FanFiction.Net Terms of Service

E. You agree not to use or launch any automated system, including without limitation, "robots," "spiders," or "offline readers," that accesses the Website in a manner that sends more request messages to the FanFiction.Net servers in a given period of time than a human can reasonably produce in the same period by using a conventional on-line web browser. Notwithstanding the foregoing, FanFiction.Net grants the operators of public search engines permission to use spiders to copy materials from the site for the sole purpose of and solely to the extent necessary for creating publicly available searchable indices of the materials, but not caches or archives of such materials. FanFiction.Net reserves the right to revoke these exceptions either generally or in specific cases.

The "given period of time" definition is kinda loose. I only download a few fanfics in a 24-hour period. Actually, I download maybe 2~4 in an hour and I can easily reproduce that (and more) on Firefox. Granted, the Calibre recipe is somewhat pushing it, but I only have it scheduled to run once a day to retrieve fanfics updated within a 24-hour period. Basically, the same fanfics I would have read in a 24-hour period, were I to actually browse their website via web browser.

A link for a "printable" version would sure be useful, though, and it seems like something they would consider.

[Mnementh]

Hmmm don't know what I've broken but I now can't produce either an epub file using php cli or even an html version using print.php.

I've tried both versions of epub.codec.php (the one ilovejedd posted as an attachment and the code listing in his post above), in fact I've actually tried every version i can find posted and also amalgamations of them all (admittedly my understanding of php is extremely limited so that probably wasn't a good idea...), I've also tried every version of ffnet.source.php and various amalgamations I can find again to no avail.

I have reinstalled xampp (after uninstalling first in case of any left over problems) and gone through all the above tests again, I've even installed linux on one of my laptops to test different versions on there though thats having other issues before even getting as far as producing the file.

When using the cli fflag appears to download the chapters (though I'm beginning to think it's actually not doing so) but doesn't create the file at the end, it also appears to be missing some of the metadata.

I have also tried to use the cli to create an html version thinking it might be a problem with calibres html2epub but this file is also not created (this is what makes me think maybe the chapters aren't actually being grabbed though i could be wrong about this).

I have also unistalled and reinstalled calibre just to eliminate that as a possible problem.

I am just going around in circles now trying to fix this so if anybody possibly has some ideas of other things I could try I would be extremely gratefull.

TIA

Mnementh

[ilovejedd]

Originally Posted by Mnementh

Hmmm don't know what I've broken but I now can't produce either an epub file using php cli or even an html version using print.php.

I've tried both versions of epub.codec.php (the one ilovejedd posted as an attachment and the code listing in her post above), in fact I've actually tried every version i can find posted and also amalgamations of them all (admittedly my understanding of php is extremely limited so that probably wasn't a good idea...), I've also tried every version of ffnet.source.php and various amalgamations I can find again to no avail.

It wasn't something you broke. FanFiction.Net just changed things around Friday and then again, last night. I haven't had time to upload the fixed version (was drinking at my friend's house from Sat 8:30pm until 5am yesterday, woke up 8:30am to go visit another friend, finally went home 10:30am, slept most of the day, had dinner, tried to read some fanfics on iphone which obviously didn't work, fixed my local copy of ffnet.source.php and finally went to sleep).

By the way, the code posted above is for ffnet.source.php (post #97). I guess I should've specified that. I haven't made any changes to epub.codec.php since FanFiction.Net did the crossover changes (basically changed to reflect both categories in the subject/tag).

In future, though, I suggest you keep whatever source code you have (epub.codec.php, etc) in case something's not functioning. Chances are, the fault lies within ffnet.source.php because FanFiction.Net changed its site layout or something and not because of any of the other modules.

If you're impatient like me, you might want to try your hand at fixing ffnet.source.php, specifically the ffnet_get_meta($page) function since that's the one that usually gets broken when there are changes to the site. I think the commenting should be sufficient to navigate the source code. The algorithm to determine the various metadata is just based on regular expressions so I suggest you read up on that. I've found the following to be a valuable resource:
http://php.net/manual/en/book.pcre.php

[Mnementh]

Thanks for the reply.

I realised the code was for ffnet.source.php the only reason i was on about epub.codec.php is cause I thought that might be the problem because the only thing that appears to be wrong is the files themselves aren't being created.

Unfortunately (or fortunately as the case may be) I've recently gone back to XP from Vista x64 and I honestly can't remember if I've had it working since then.

Guess I'll have to wait till you upload a complete version of everything <hint hint> as I'm now so lost and confused with different versions of everything from rgibbo's version and your own that probably no matter what I do it won't work...

I will wait patiently as all your efforts are appreciated and I know the feeling of a hangover

[ilovejedd]

Originally Posted by Mnementh

I will wait patiently as all your efforts are appreciated and I know the feeling of a hangover

No hangover (all I had were just girly drinks). I was just too tired from the lack of sleep.

[Mnementh]

Originally Posted by ilovejedd

No hangover (all I had were just girly drinks). I was just too tired from the lack of sleep.

That and the bad taste in your mouth from all the girly drinks lol.

Please at least tell me it was coctails and not alcopops...

[erayd]

OK.... things are starting to seem just a tad crazy with all these versions, and every time I go to consolidate them more changes have sprung up!

ilovejedd: Can you post a massive diff between whatever (working) version you have at the moment and the files I've attached at the top of this thread? I'll merge them and stick them up the top, that way at least everyone will be working from the same base rather than all these crazy chaotic different bits and pieces, and the files at the top will actually work with ffnet - as you said earlier, the current release is just a tad broken.