FLAG (Fanfiction.net Lightweight Automated Grabber)

[hime]

with the web FLAG service I get the following error:

* Extracting content...
* Compiling epub data...
/usr/bin/html2epub: line 7: 1458 Segmentation fault $loader "$@"

is it possible to fix this error?

Thanks!

[erayd]

Originally Posted by hime

with the web FLAG service I get the following error:

* Extracting content...
* Compiling epub data...
/usr/bin/html2epub: line 7: 1458 Segmentation fault $loader "$@"

is it possible to fix this error?

Thanks!

Which story ID was causing problems? Did you actually get a book out of it? If I remember correctly, that particular tool has always segfaulted on completion, but does in fact output the correct file first.

[hime]

Originally Posted by erayd

Which story ID was causing problems? Did you actually get a book out of it? If I remember correctly, that particular tool has always segfaulted on completion, but does in fact output the correct file first.

yesterday calibre wouldn't take the lrf and epub files produced from the website, but today i tried adding the stories again and it worked! So something must have been wrong with my calibre, and not the website. Sorry for the trouble and thanks for your very quick respose!

[ilovejedd]

Tiny update, I'll be working on the scripts this afternoon up to tomorrow. This time, I'm going to stop doing simultaneous additions and just work on cleaning up code and incorporating changes I've made to the scripts. I've been trying to create an epub.codec.php that does everything in PHP as calling Calibre takes a really, really long time for some of the long fanfics I read (worst was 700,000 words). However, it seems like that project will take me forever to finish so I'll just incorporate whatever working changes I've made and submit that.

A bit OT:
Need a little help. Does anyone here know javascript? I'm trying to create a Greasemonkey script that will insert links for printable versions of stories on FanFiction.Net.

Output would be similar to the following:
http://www.utterlyinsane.org/ff.net/

Thanks!

[Amalthia]

Originally Posted by ilovejedd

Tiny update, I'll be working on the scripts this afternoon up to tomorrow. This time, I'm going to stop doing simultaneous additions and just work on cleaning up code and incorporating changes I've made to the scripts. I've been trying to create an epub.codec.php that does everything in PHP as calling Calibre takes a really, really long time for some of the long fanfics I read (worst was 700,000 words). However, it seems like that project will take me forever to finish so I'll just incorporate whatever working changes I've made and submit that.

A bit OT:
Need a little help. Does anyone here know javascript? I'm trying to create a Greasemonkey script that will insert links for printable versions of stories on FanFiction.Net.

Output would be similar to the following:
http://www.utterlyinsane.org/ff.net/

Thanks!

I was thinking of doing something similar with inserting a script into LJ that'll create a print view version that's just the story without the LJ header. Sadly, I don't know much about javascript and I'm starting at 0 learning.

[sigizmund]

Chaps, take a look at http://fanfictionloader.appspot.com — it supports all the same as FLAG + FanFicAuthors + FicWad + FictionAlley (almost there), outputs to ePub and HTML and needs no installation. It is powered by open source library http://code.google.com/p/fanficdownloader/ which of all dependencies needs only Python and nothing else - feel free to use it too.

[erayd]

Looks useful sigizmund - thanks for the link .

[araevin]

Does FF.Net have an api I'm stumped as to how to pull info from the site. I'm qurious as to how you guys are doing it are you guys using REST or something simular? I'd like to know because Im trying to do so in either c++ or obj-c.

so guys any help on how to pull info from the site.

[erayd]

Originally Posted by araevin

Does FF.Net have an api?

No (or at least, not that I'm aware of).

I'm stumped as to how to pull info from the site.

Just use a standard HTTP GET request - that will get you everything you need.

I'm curious as to how you guys are doing it are you guys using REST or something similar?

Nope, we're just grabbing the same stuff that you'd see if you browse to the site yourself.

I'd like to know because Im trying to do so in either c++ or obj-c.

Language shouldn't matter - any decent programming language should be able to achieve it. That includes both C++ and Objective C.

...so guys any help on how to pull info from the site?

Just grab it the same way as everyone else is doing, and then use regex to extract the information. You could also combine that with directly parsing the xml tree if you want to improve performance a little.

[araevin]

Thanx erayd ur the best.

[lightn]

I'm interested in this software. I would like to try an improve it a bit. What is the current version? Is it r29 linked from the first comment? Based on the regexps in that code it looks to be outdated. I tried downloading it from subversion, but it is requesting a user name and password.

I would also like to make a grease monkey script to add links to FLAG on fanfiction.net, but it seems that getstory.php only supports POST requests. Could you add GET support to it so that it can be directly linked to?

Thanks

[erayd]

Originally Posted by lightn

I'm interested in this software. I would like to try an improve it a bit.

By all means feel free, I'm sure there are people here who would appreciate it.

What is the current version? Is it r29 linked from the first comment? Based on the regexps in that code it looks to be outdated. I tried downloading it from subversion, but it is requesting a user name and password.

The current version is r29 in the first comment, plus whatever mods ilovejedd has added to it since - it's in a bit of a mess at the moment. He's been promising to clean it up and give me a tidy version I can put into svn, but it's been taking a while. As my own time is rather limited, it's unlikely that I will do this myself.

I would also like to make a grease monkey script to add links to FLAG on fanfiction.net, but it seems that getstory.php only supports POST requests. Could you add GET support to it so that it can be directly linked to?

Can I - yes. Will I - no. The reason for this is that adding GET links could result in search engines crawling it, which in turn could put very heavy load on my server - generating ebook files is a rather CPU-intensive activity.

[Iteria]

The online service is hanging for me for this one story. 2435460 is the id. I think it's just too long (it has 500 chapters). It hangs around chapter 430. It's probably not a normal problem, but I figured you might want to know.

[erayd]

Originally Posted by Iteria

The online service is hanging for me for this one story. 2435460 is the id. I think it's just too long (it has 500 chapters). It hangs around chapter 430. It's probably not a normal problem, but I figured you might want to know.

Are you sure the problem is with the webservice? I just tested it with that story ID, and it worked perfectly. Admittedly it did take rather a long time, but that's to be expected with something 500 chapters long - you're forcing it to hit the ff.net server 500 times to grab all the content, and then consolidate all that into a single file.

Can you describe your issue in a bit more detail?

[Iteria]

Originally Posted by erayd

Are you sure the problem is with the webservice? I just tested it with that story ID, and it worked perfectly. Admittedly it did take rather a long time, but that's to be expected with something 500 chapters long - you're forcing it to hit the ff.net server 500 times to grab all the content, and then consolidate all that into a single file.

Can you describe your issue in a bit more detail?

I tried again and it's a no-go. I chose the format to be epub, I don't know if that makes a difference. Basically what happens is that the screen will be completely blank for a while, then after a few minutes the story chapters will pop up like normal until 430. I have let it sit for 10 after the pause and still it doesn't finish.

I don't know if this will help, but I'm doing this in Firefox ver 3.6 on windows.