FLAG (Fanfiction.net Lightweight Automated Grabber)

[hime]

Originally Posted by erayd

The webservice at http://flag.erayd.net/ is now back up and running, and I've fixed the script there to grab the metadata properly - it didn't seem to be doing this before. As far as I'm aware everything should be working, so please give me a shout if it's not.

Thanks for getting it up and running It did work for a little while but now when I try to use it to get fanfiction in LRF format, I get the following error:

Please wait while your story is processed...

which: no html2lrf in ((null))
Dependency html2lrf (part of Calibre) is missing - is Calibre installed?


Redirecting...

[erayd]

Originally Posted by hime

Thanks for getting it up and running It did work for a little while but now when I try to use it to get fanfiction in LRF format, I get the following error:

Please wait while your story is processed...

which: no html2lrf in ((null))
Dependency html2lrf (part of Calibre) is missing - is Calibre installed?


Redirecting...

Thank you - fixed.

[hime]

Originally Posted by erayd

Thank you - fixed.

Thank you!

[ilovejedd]

Originally Posted by erayd

All good, I'm in no rush - just let me know when you have something .

Gah. Gave up trying to code on the netbook. The small screen and tiny keyboard just isn't conducive to sanity. Not to mention I recently started watching Bones and have just finished a marathon session of season 1 to 4.

I'm scavenging for my old laptop and will finish the work there.

[rfresa]

Thank you so much for this great little tool. Now I can simply connect my iPod touch to WiFi, go to the online catalog in Stanza, enter a story ID, and download the whole thing. I can read at work where there is no WiFi! I appreciate how streamlined and smooth the process is. Are there any similar tools I could use to download from other story sites? Previous to finding this, I had made a little apple script to try to accomplish the same thing, so I know a bit about what's needed. Perhaps a little form where you could enter the URL, and then select the part of it that indicates the chapter number? Then input the number of chapters and the program could figure out how to do it. It would be messy, of course, because there wouldn't be any way to get rid of the header/footer and other extra stuff without knowing how that specific site works, but it would be better than nothing. Anyway, those are just my thoughts on the subject.

Thanks so much for making FLAG!

[hime]

I tried to use the FLAG webservice and I get the following error:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 3053948936 bytes) in /var/www/net/erayd/flag/pub/flag/codec/html.codec.php on line 84

[rfresa]

I am getting the same thing. Tried it on several stories, but it won't work. Does the downloaded program work on mac? I haven't tried it yet.

[erayd]

Originally Posted by hime

I tried to use the FLAG webservice and I get the following error:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 3053948936 bytes) in /var/www/net/erayd/flag/pub/flag/codec/html.codec.php on line 84

Fixed - it seems the latest version of htmltidy has a rather bad memory leak. I've reverted to the previous version; that seems to have resolved the issue.

[ilovejedd]

Heya! Sorry I've been MIA for a long time. Anyhow, I just got my new laptop (ASUS UL30A-X5) so I can start on coding again. I still can't type as fast as I did on my old laptop, but both the keyboard and screen are better than the netbook, at least.

[Mnementh]

Welcome back, looking forward to your efforts

Just looked up the specs on your laptop, looks pretty sweet, something tells me you won't really be getting 12 hours of battery life though I'd be interested to hear if that is actually semi accurate, might get one myself if it is, my small laptop only gets about two hours which makes it pretty useless for work.

[ilovejedd]

Originally Posted by Mnementh

Welcome back, looking forward to your efforts

Just looked up the specs on your laptop, looks pretty sweet, something tells me you won't really be getting 12 hours of battery life though I'd be interested to hear if that is actually semi accurate, might get one myself if it is, my small laptop only gets about two hours which makes it pretty useless for work.

On a custom power saving profile at 25% brightness, I got 9 hours use out of it yesterday. Pretty darn good, imho. With wifi turned off, I have no doubt it can go up to more than 10 hours. There's a more expensive model (UL30A-A2) which has better battery life (84WHr vs 63WHr). Unfortunately, aesthetics made the choice for me. The other one was only available in silver.

[Dr. Wily]

If this works, you have made my life much easier. I have converted several fanfics from fanfiction.net, including one I'd rather like to upload here unless they don't fall under Free Use. I first tried converting one using Copy/paste, Notepad, HTML, Calibre, and lots of diligence, and now I use Copy/paste, Microsoft Word, and Calibre. If this program can automatically get me an HTML version of the fic, that'd be just awesome. I shall have to try it out when I get home...

[Elfwreck]

Originally Posted by Dr. Wily

If this works, you have made my life much easier. I have converted several fanfics from fanfiction.net, including one I'd rather like to upload here unless they don't fall under Free Use.

Fanfics can't be reposted w/o permission of the author. They're not in the public domain, and very few of them have been released under Creative Commons. (That said, I believe the majority of fanfic authors don't mind having their stuff reposted elsewhere--IF they've been asked, and don't mind the hosting site. For MR's rules, you definitely need to get permission first.)

To create fanfic ebooks, you can also use the Fanfiction Downloader, which has to be installed on your home machine, but works on several different fanfiction sites. (The website doesn't say so, but I believe it also works on Skyhawke.)

[araevin]

I've been here for a bit and I've been going over the work you guys have been doing and I wonder does FF.net have an API? I've been mulling over some of whats been said on this forum for some time and I've really had no luck with stanza I've all but quit.

I've actually got an idea for an iPhone app but was wondering how to interface with the site. What I had wanted to do is quite similar exempt this will be a native app not a php or cgi script.

What I wanted was to be able to basically pull the site down and display it optimized for iPhone screen sizes. cache stories for offline reading or just for speed using CoreData. Basically since FF.net will not optimize their site with css for small screens I want the user to have the option of pulling the whole site into something like a SSB (site specific browser).

Any thoughts or ideas?

[ilovejedd]

Originally Posted by araevin

I've been here for a bit and I've been going over the work you guys have been doing and I wonder does FF.net have an API? I've been mulling over some of whats been said on this forum for some time and I've really had no luck with stanza I've all but quit.

I've actually got an idea for an iPhone app but was wondering how to interface with the site. What I had wanted to do is quite similar exempt this will be a native app not a php or cgi script.

What I wanted was to be able to basically pull the site down and display it optimized for iPhone screen sizes. cache stories for offline reading or just for speed using CoreData. Basically since FF.net will not optimize their site with css for small screens I want the user to have the option of pulling the whole site into something like a SSB (site specific browser).

Any thoughts or ideas?

Alas, no API. Just plain scraping.

If you're willing to give Stanza another try, though, I've coded some PHP scripts that automatically generates Stanza compatible RSS feeds and calls FLAG to convert stories to EPUB format.

Imho, it's much easier to accomplish things via server side scripting instead of coding an actual Apple-sanctioned iPhone app. For one thing, I believe you need to be running OS X for the iPhone SDK, then there's the whole mystical Apple review process. Even what you want to accomplish can be done via server side scripting. I've actually done something similar, but only for a full browser (basically, the script injects links for epub, lit, mobi and plain/printable html formats). Albeit, I'm planning on modifying the PHP scripts to create an iPhone-friendly output, too. Server-side scripting also makes it faster and easier to release updates for when FanFiction.Net changes something on their website. Much to my dismay, they tend to do that quite a bit.