FLAG (Fanfiction.net Lightweight Automated Grabber) - Page 13

erayd · 03-05-2010, 04:24 PM

Quote:

Originally Posted by Iteria

I tried again and it's a no-go. I chose the format to be epub, I don't know if that makes a difference. Basically what happens is that the screen will be completely blank for a while, then after a few minutes the story chapters will pop up like normal until 430. I have let it sit for 10 after the pause and still it doesn't finish.

I don't know if this will help, but I'm doing this in Firefox ver 3.6 on windows.

I don't think it's your browser. I've done some more testing, and it looks like LRF and HTML work fine (my first test was with LRF), but ePub fails at chapter 430 each time.

Noting that you have a PRS-505, might I suggest grabbing the LRF version in the meantime until this is fixed?

skb · 03-20-2010, 02:43 AM

I have just stumbled upon this thread and these wonderful gadgets to download fanfic.

Thank you so much to all who have provided them - they're marvellous!

JaneD · 05-18-2010, 05:43 PM

Quote:

Originally Posted by sigizmund

Chaps, take a look at http://fanfictionloader.appspot.com — it supports all the same as FLAG + FanFicAuthors + FicWad + FictionAlley (almost there), outputs to ePub and HTML and needs no installation. It is powered by open source library http://code.google.com/p/fanficdownloader/ which of all dependencies needs only Python and nothing else - feel free to use it too.

Oh wow, I just used this and it's life-changing. I just put a 25-chapter fanfic into my Kindle using this - a fic I've been meaning to read forever, but just didn't want to spend the time copying the whole damn thing! And it took basically two clicks. Love it!

AgentBEATS · 05-18-2010, 08:12 PM

Quote:

Originally Posted by sigizmund

Chaps, take a look at http://fanfictionloader.appspot.com — it supports all the same as FLAG + FanFicAuthors + FicWad + FictionAlley (almost there), outputs to ePub and HTML and needs no installation. It is powered by open source library http://code.google.com/p/fanficdownloader/ which of all dependencies needs only Python and nothing else - feel free to use it too.

This is great, just wish it supported the auto-creation of the table of contents like FLAG (Or at least the FLAG online service, as I have no clue how to run the actual program) does. But being able to download from those other sources I guess makes up for it.

Davimee · 05-21-2010, 03:50 PM

Oh, wow! This is incredible! Thank you!

Just like everyone else here, I've found I enjoy reading fanfic, but it takes so much time to get all those chapters into a format that I can read on my Pocket Pro. Thank you, thank you, thank you!

A bit off topic, but does anyone know if there's a quick and easy way to grab transcripts off of web pages into an ebook? For example, gateworld.com (I think that's the site) has Stargate transcripts, and for SGA I've just copied each one into Word and saved it as .rtf. But is there an easier way, like maybe something that can grab an entire season and make each episode a chapter, or something like that?

Thanks again for this wonderful program!

emmadiva · 06-05-2010, 12:34 AM

Eek! I'm in desperate need of some help. Ilovejedd had set up my website with a slightly tweaked version of flag that she had been using on utterlyinsane.org. It had no black bar at the top and included a simple table of contents, the status of the story (complete or in progress) as well as the category. I gave her an amazon gift certificate for her time and she uploaded all the flag codes on to my server for me because I'm not familiar with any sort of php/code or how any of that works. It's actually the only reason I got my own website. LOL - Yes, I like fanfic a bit too much.

Well, it's always worked perfectly until today. I think ff.net must have changed something because neither my web version of flag nor hers will pull a story. Instead it tries to fetch the story but comes back absolutely blank.

I don't have a clue how to get this working again.

I've tried other ff downloaders but none compare to flag. If anyone has any suggestions, I'd truly appreciate it. Or if Ilovejedd is still on the forum, I'd love to get your help again.

Thanks in advance to everyone for any advice/help.

emmadiva · 06-12-2010, 11:11 AM

Hi all! Looks like ff.net changed their page structure again. FLAG pulls up the requested story but only the chapter titles appear, no story content.

kaotic · 06-13-2010, 10:00 PM

Quote:

Originally Posted by emmadiva

Hi all! Looks like ff.net changed their page structure again. FLAG pulls up the requested story but only the chapter titles appear, no story content.

Yeah it seems they do that quite often -- well often enough that it gets annoying.

sourcejedi · 06-14-2010, 04:13 PM

Pity, I wasn't expecting this to break so soon after I started using it.

I found a dumb-as-rocks incantation for Unix, if anyone's interested. The catch is you have to know Unix first :-). I can't tell you how to do this under Windows.

Code:

curl -o "#1.html" http://www.fanfiction.net/s/5782108/[1-2]

That's the exact command to download chapters 1-2 of story ID 5782108 from FanFiction.net. You'll get several files, e.g. 1.html and 2.html. It'll have all the FanFiction.net buttons / links etc - most of them won't work.

Amazingly the font, size, and "night mode" buttons still work if you open it in a web browser. Oh, and the adverts ;-).
For reasons unknown to man, FBReader based viewers will cleverly hide everything except the story text itself, and the "Review this Chapter" link at the end.
If you're unfortunate enough not to be using FBReader, and you don't like the margins or other formatting, try using the Mobile version of FanFiction.Net. Just replace "www." with "m.".
lynx complains about the HTML, as does the w3c validator. If you convert this to ePUB using a braindead tool that just zips up what it's given, you will get a very non-compliant ePUB. There is no guarantee it to work on your device.

Sorry if your system can't handle directories of html files. (IMO that makes it a proprietary document viewer, not an e-reader :-p). Something like Calibre may be able to convert for you (or not; see above).

You can modify the command to download chapters from any story. Replace the URL pattern with the web address of the first chapter you want. I recommend you copy+paste the address from the location bar of your web browser.

The trick to get multiple chapters at once is to manually add square brackets round the chapter number at the end, and replace the number with a range, such as [1-2] or [23-25].

FanFiction.Net don't really want you to do this, but the same applies to FLAG.

ilovejedd · 06-15-2010, 09:21 PM

By the way, I already fixed the version on my server in case anyone's interested:

Code:

http://utterlyinsane.org/fflag/print.php?storyid=

Needless to say, you need to add the story id at the end of the url.

Alas, FanFiction Downloader on appspot seems to be working quite well so I kinda lost the initiative to clean up my messed up scripts to submit to erayd.

skb · 06-18-2010, 05:32 AM

Quote:

Originally Posted by ilovejedd

By the way, I already fixed the version on my server in case anyone's interested:

Code:

http://utterlyinsane.org/fflag/print.php?storyid=

I am! I use your version quite a bit. Thank you for taking the time to update.

kaotic · 06-19-2010, 05:07 AM

Quote:

Originally Posted by ilovejedd

By the way, I already fixed the version on my server in case anyone's interested:

Code:

http://utterlyinsane.org/fflag/print.php?storyid=

Needless to say, you need to add the story id at the end of the url.

Alas, FanFiction Downloader on appspot seems to be working quite well so I kinda lost the initiative to clean up my messed up scripts to submit to erayd.

Is there a way to make an epub using the link above? I'd rather not use html.

sigizmund · 06-20-2010, 10:18 AM

Quote:

Originally Posted by kaotic

Is there a way to make an epub using the link above? I'd rather not use html.

Use Fanfiction Downloader

It supports changed ffnet and also features Twilighted support which practically no one else does.

erayd · 07-02-2010, 10:06 AM

Someone has just brought this issue to my attention off-forum via email (thanks

). I have fixed the parser to take ff.net's recent changes into account, and it should now extract all metadata and story content correctly.

Edit: I've also added support for PDF output.

Kaotic: The online version of FLAG will build ePub files for you.

Please don't hesitate to let me know if you have any further issues with it.

octarineblues · 07-08-2010, 11:12 AM

I've been having some trouble with story ID 2609602 when using the online version of FLAG. For some reason, chapters 9 and 10 (the two epilogues) are generated with only the title and two separators. I tried HTML, ePub, and PDF and got the same result on all three. I tried the version that ilovejedd linked to a few posts up, and that generates the chapters fine, but I'd really like to be able to make an ePub of this story. I haven't used FLAG enough to notice if this happens with any other stories, so I don't really know what's causing it.

06-14-2010, 04:13 PM	#189
sourcejedi Groupie Posts: 155 Karma: 200000 Join Date: Dec 2009 Location: Britania Device: Android	Pity, I wasn't expecting this to break so soon after I started using it. I found a dumb-as-rocks incantation for Unix, if anyone's interested. The catch is you have to know Unix first :-). I can't tell you how to do this under Windows. Code: curl -o "#1.html" http://www.fanfiction.net/s/5782108/[1-2] That's the exact command to download chapters 1-2 of story ID 5782108 from FanFiction.net. You'll get several files, e.g. 1.html and 2.html. It'll have all the FanFiction.net buttons / links etc - most of them won't work. Amazingly the font, size, and "night mode" buttons still work if you open it in a web browser. Oh, and the adverts ;-). For reasons unknown to man, FBReader based viewers will cleverly hide everything except the story text itself, and the "Review this Chapter" link at the end. If you're unfortunate enough not to be using FBReader, and you don't like the margins or other formatting, try using the Mobile version of FanFiction.Net. Just replace "www." with "m.". lynx complains about the HTML, as does the w3c validator. If you convert this to ePUB using a braindead tool that just zips up what it's given, you will get a very non-compliant ePUB. There is no guarantee it to work on your device. Sorry if your system can't handle directories of html files. (IMO that makes it a proprietary document viewer, not an e-reader :-p). Something like Calibre may be able to convert for you (or not; see above). You can modify the command to download chapters from any story. Replace the URL pattern with the web address of the first chapter you want. I recommend you copy+paste the address from the location bar of your web browser. The trick to get multiple chapters at once is to manually add square brackets round the chapter number at the end, and replace the number with a range, such as [1-2] or [23-25]. FanFiction.Net don't really want you to do this, but the same applies to FLAG.

06-15-2010, 09:21 PM	#190
ilovejedd hopeless n00b Posts: 5,126 Karma: 19597086 Join Date: Jan 2009 Location: in the middle of nowhere Device: PW4, PW3, Libra H2O, iPad 10.5, iPad 11, iPad 12.9	By the way, I already fixed the version on my server in case anyone's interested: Code: http://utterlyinsane.org/fflag/print.php?storyid= Needless to say, you need to add the story id at the end of the url. Alas, FanFiction Downloader on appspot seems to be working quite well so I kinda lost the initiative to clean up my messed up scripts to submit to erayd.

07-02-2010, 10:06 AM	#194
erayd Zealot Posts: 134 Karma: 146 Join Date: Apr 2008 Device: Onyx Boox Poke 2	Someone has just brought this issue to my attention off-forum via email (thanks ). I have fixed the parser to take ff.net's recent changes into account, and it should now extract all metadata and story content correctly. Edit: I've also added support for PDF output. Kaotic: The online version of FLAG will build ePub files for you. Please don't hesitate to let me know if you have any further issues with it. Last edited by erayd; 07-02-2010 at 11:55 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Fanfiction.net on Kindle	forkyfork	Amazon Kindle	26	08-07-2011 08:42 AM
bookmarks/notes grabber	Reader2	Android Developer's Corner	0	10-02-2010 09:24 AM
EASY fanfiction grabber?	sherryg	Workshop	19	01-08-2010 03:13 AM
FLAG (Fanfiction.net Lightweight Automated Grabber) and Calibre?	malkie13	Calibre	1	02-10-2009 05:43 PM

03-20-2010, 02:43 AM	#182
skb Evangelist Posts: 401 Karma: 1597305 Join Date: Mar 2010 Device: Ipod G4, MacOS 10.12, Calibre, Pocketbook Touch HD 3	I have just stumbled upon this thread and these wonderful gadgets to download fanfic. Thank you so much to all who have provided them - they're marvellous!

05-21-2010, 03:50 PM	#185
Davimee Happy Wife & Momma Posts: 532 Karma: 1634 Join Date: Jan 2010 Location: USA Device: Purple Astak Pocket Pro! :)	Oh, wow! This is incredible! Thank you! Just like everyone else here, I've found I enjoy reading fanfic, but it takes so much time to get all those chapters into a format that I can read on my Pocket Pro. Thank you, thank you, thank you! A bit off topic, but does anyone know if there's a quick and easy way to grab transcripts off of web pages into an ebook? For example, gateworld.com (I think that's the site) has Stargate transcripts, and for SGA I've just copied each one into Word and saved it as .rtf. But is there an easier way, like maybe something that can grab an entire season and make each episode a chapter, or something like that? Thanks again for this wonderful program!

06-05-2010, 12:34 AM	#186
emmadiva Junior Member Posts: 7 Karma: 10 Join Date: Apr 2009 Device: kindle 2	Eek! I'm in desperate need of some help. Ilovejedd had set up my website with a slightly tweaked version of flag that she had been using on utterlyinsane.org. It had no black bar at the top and included a simple table of contents, the status of the story (complete or in progress) as well as the category. I gave her an amazon gift certificate for her time and she uploaded all the flag codes on to my server for me because I'm not familiar with any sort of php/code or how any of that works. It's actually the only reason I got my own website. LOL - Yes, I like fanfic a bit too much. Well, it's always worked perfectly until today. I think ff.net must have changed something because neither my web version of flag nor hers will pull a story. Instead it tries to fetch the story but comes back absolutely blank. I don't have a clue how to get this working again. I've tried other ff downloaders but none compare to flag. If anyone has any suggestions, I'd truly appreciate it. Or if Ilovejedd is still on the forum, I'd love to get your help again. Thanks in advance to everyone for any advice/help.

06-12-2010, 11:11 AM	#187
emmadiva Junior Member Posts: 7 Karma: 10 Join Date: Apr 2009 Device: kindle 2	Hi all! Looks like ff.net changed their page structure again. FLAG pulls up the requested story but only the chapter titles appear, no story content.

07-08-2010, 11:12 AM	#195
octarineblues Member Posts: 12 Karma: 10 Join Date: Jul 2010 Device: none	I've been having some trouble with story ID 2609602 when using the online version of FLAG. For some reason, chapters 9 and 10 (the two epilogues) are generated with only the title and two separators. I tried HTML, ePub, and PDF and got the same result on all three. I tried the version that ilovejedd linked to a few posts up, and that generates the chapters fine, but I'd really like to be able to make an ePub of this story. I haven't used FLAG enough to notice if this happens with any other stories, so I don't really know what's causing it.