FLAG (Fanfiction.net Lightweight Automated Grabber)

[erayd]

Originally Posted by rgibbo

<snip>
After scanning through the code for about 15 minutes, I managed to found the problem, it lies with how the tool gets the page data from the site as it sends out a user agent string to help identify it, it seems fanfiction has blocked the tool's user agent string so the tool can't download anything. I decided to change the user agent string which was FLAG/1.0 or similar to the something differentand it seems to be working again now.

Well that's the solution if anyone else wants it.

Sorry to disillusion you... but the UA string wasn't actually the issue at all. The problem was that the DNS server I was using had crashed (darn thing ran out of memory), which meant that the web service was unable to resolve the hostname for fanfiction.net (and therefore couldn't download the stories). I've restarted the DNS server, and everything now works perfectly again (Thanks heaps for bringing this to my attention ). This means that whatever issue you were having was unrelated, and specific to your particular setup.

What else did you change? As the UA for FLAG isn't blocked, changing it wouldn't have fixed anything.

Edit: Out of interest, how difficult was it to get this runing on Windows? I've used a fair few things that are specific to Linux/Unix systems (various posix functions, assuming that /tmp exists etc). Are you able to send me a diff?

[rgibbo]

oh well, must have been something else on my end then, oh well it works now

as for running it on windows, i had to remove the posix stuff since windows machines dont have posix installed on them (same for php as well), ill send you a diff of it soon after ive made some more changes so its not too different from the original as i almost completly rewrote the first page and the codec page save points to account for no tmp folder

[erayd]

Originally Posted by rgibbo

...ill send you a diff of it soon after ive made some more changes so its not too different from the original as i almost completly rewrote the first page and the codec page save points to account for no tmp folder

Awesome - thanks .

[rgibbo]

Here's my version of flag for windows, it is now similar to the original so it *might* run on linux (i dont have linux so i don't know if it works on there or not)

Some other modifications I have made are: -

• Can detect which platform it is running on, and sorts out folder paths automatically including the temp folder
• Writes the data to a file by default instead of printing to the console, this is easier for me
• Doesn't create the output file if the story id is invalid

Any comments are appreciated

Note: The epub and rtf formats don't work on windows as I don't use these, I will do it if anyone needs them

[Lane]

rgibbo, thank you so much for that Windows version! I would really appreciate it if you could enable the rtf-saving feature, as that's the one I constantly use.

[rgibbo]

Thanks for Lane for pointing this out.

There is no application for the windows version since I have PHP installed on my development machine, and I run a version of the app that erayd first wrote.

I will be rewriting the application so that it is a proper windows application, without any of the PHP libraries being required. This version will hopefully be posted here at the end of the week if i don't get any work from college to do over christmas.

[erayd]

Thanks rgibbo for the work you've put into this - do you want svn commit access to that repo? Alternatively I can merge changes, if you're happy to wait a few days - I'm pretty busy right now.

Regarding saving to rtf, I'm pretty sure there is a windows version of pandoc - so enabling this should be pretty simple.

[Amalthia]

I can't wait to test this out tonight after work. I also really hope the OTW archive can implement something very similar to what you have here. It would really make reading fan fiction on my reader so much easier. (assuming they have some way to customize the output) because I'm kind of particular about the size of font and margins.

[moggie]

I have just posted a message about a sugarquill recipe for calibre that I thought people here might be interested in.

https://www.mobileread.com/forums/sho...2&postcount=78

[Amalthia]

I'm going to have to try this out when I'm home. Do I have to do anything special to add the reciepe? or are the instructions included in the zip file?

Also, is there a way to change the LRF formatting? I'm kind of fond of limiting margin space and increasing font size. I'm not sure if I actually need to try and change the format since I haven't tried it yet but I'd like to know if it's an option ahead of time.

Also, since this seems to pull up the author's stories, if the author has multiple stories is there a way to combine all of them into a single file using the Table of Contents to index the individual stories instead of the chapters?

I've been merging author's stories into single files out of necessity. Most fan fiction stories are short and it's really easy to get over 1500 stories on your reader and well it really slows down at that point.

[moggie]

Originally Posted by Amalthia

I'm going to have to try this out when I'm home. Do I have to do anything special to add the recipe? or are the instructions included in the zip file?

Also, is there a way to change the LRF formatting? I'm kind of fond of limiting margin space and increasing font size. I'm not sure if I actually need to try and change the format since I haven't tried it yet but I'd like to know if it's an option ahead of time.

Basically look up the author id and pass it as the username

You need to have calibre, its a recipe for calibre There are standard options in the calibre GUI or you can use the command line.

    --left-margin=LEFT_MARGIN
                        Left margin of page. Default is 20 px.

    --right-margin=RIGHT_MARGIN
                        Right margin of page. Default is 20 px.

    --top-margin=TOP_MARGIN
                        Top margin of page. Default is 10 px.

    --bottom-margin=BOTTOM_MARGIN
                        Bottom margin of page. Default is 0 px.

Originally Posted by Amalthia

Also, since this seems to pull up the author's stories, if the author has multiple stories is there a way to combine all of them into a single file using the Table of Contents to index the individual stories instead of the chapters?

I've been merging author's stories into single files out of necessity. Most fan fiction stories are short and it's really easy to get over 1500 stories on your reader and well it really slows down at that point.

It already does this basically you get a 2000 odd page book ( well for some authors ) with each story fully downloaded with an index page with an entry for each story.

[Amalthia]

Originally Posted by moggie

Basically look up the author id and pass it as the username

You need to have calibre, its a recipe for calibre There are standard options in the calibre GUI or you can use the command line.

I'm such a dimbat sometimes.

It already does this basically you get a 2000 odd page book ( well for some authors ) with each story fully downloaded with an index page with an entry for each story.

Okay this is starting to sound too good to be true.

Amalthia

[Elfwreck]

Originally Posted by Amalthia

I can't wait to test this out tonight after work. I also really hope the OTW archive can implement something very similar

OTW's stories all have a 1-page-for-printing version;they can be saved out as HTML with no extra processing. (One page vs chapters was one of the debates that went around during coding discussions; feedback was very clear: fans want both. Some want one all the time, some want the other all the time, some want the option depending on their current reading circumstances.) From what I can tell, the default at OTW is the single-page layout; you have to click a button to get separate chapters.

[erayd]

Originally Posted by Amalthia

I'm going to have to try this out when I'm home. Do I have to do anything special to add the reciepe? or are the instructions included in the zip file?

It will need to be rewritten in PHP, and in the format that FLAG uses for a source. Not particularly hard, but you may want to have the ffnet one open as a reference.

Also, is there a way to change the LRF formatting? I'm kind of fond of limiting margin space and increasing font size. I'm not sure if I actually need to try and change the format since I haven't tried it yet but I'd like to know if it's an option ahead of time.

This is currently hardcoded into the LRF output, but as it uses calibre's html2lrf for this you should find it reasonably easy to change.

Also, since this seems to pull up the author's stories, if the author has multiple stories is there a way to combine all of them into a single file using the Table of Contents to index the individual stories instead of the chapters?

This doesn't pull up the author's stories, it pulls up all the chapters of the particular story you ask it to grab. Pulling up all stories by an author would require substantial rewriting of the ffnet source - doable, but honestly not something that is a priority for me right now. I would be happy to merge changes though, if someone else adds this.

[Amalthia]

Originally Posted by erayd

It will need to be rewritten in PHP, and in the format that FLAG uses for a source. Not particularly hard, but you may want to have the ffnet one open as a reference.

This is currently hardcoded into the LRF output, but as it uses calibre's html2lrf for this you should find it reasonably easy to change.

This doesn't pull up the author's stories, it pulls up all the chapters of the particular story you ask it to grab. Pulling up all stories by an author would require substantial rewriting of the ffnet source - doable, but honestly not something that is a priority for me right now. I would be happy to merge changes though, if someone else adds this.

Thanks for getting back to me so quickly.