Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Formats > Workshop

Notices

Reply
 
Thread Tools Search this Thread
Old 12-05-2008, 04:21 PM   #31
erayd
Zealot
erayd doesn't littererayd doesn't litter
 
Posts: 128
Karma: 146
Join Date: Apr 2008
Device: Sony PRS-650
Quote:
Originally Posted by rgibbo View Post
<snip>
After scanning through the code for about 15 minutes, I managed to found the problem, it lies with how the tool gets the page data from the site as it sends out a user agent string to help identify it, it seems fanfiction has blocked the tool's user agent string so the tool can't download anything. I decided to change the user agent string which was FLAG/1.0 or similar to the something differentand it seems to be working again now.

Well that's the solution if anyone else wants it.
Sorry to disillusion you... but the UA string wasn't actually the issue at all. The problem was that the DNS server I was using had crashed (darn thing ran out of memory), which meant that the web service was unable to resolve the hostname for fanfiction.net (and therefore couldn't download the stories). I've restarted the DNS server, and everything now works perfectly again (Thanks heaps for bringing this to my attention ). This means that whatever issue you were having was unrelated, and specific to your particular setup.

What else did you change? As the UA for FLAG isn't blocked, changing it wouldn't have fixed anything.

Edit: Out of interest, how difficult was it to get this runing on Windows? I've used a fair few things that are specific to Linux/Unix systems (various posix functions, assuming that /tmp exists etc). Are you able to send me a diff?

Last edited by erayd; 12-05-2008 at 04:26 PM.
erayd is offline   Reply With Quote
Old 12-05-2008, 08:23 PM   #32
rgibbo
Junior Member
rgibbo began at the beginning.
 
Posts: 5
Karma: 14
Join Date: Dec 2008
Device: Sony PRS-505
oh well, must have been something else on my end then, oh well it works now

as for running it on windows, i had to remove the posix stuff since windows machines dont have posix installed on them (same for php as well), ill send you a diff of it soon after ive made some more changes so its not too different from the original as i almost completly rewrote the first page and the codec page save points to account for no tmp folder
rgibbo is offline   Reply With Quote
Old 12-05-2008, 09:29 PM   #33
erayd
Zealot
erayd doesn't littererayd doesn't litter
 
Posts: 128
Karma: 146
Join Date: Apr 2008
Device: Sony PRS-650
Quote:
Originally Posted by rgibbo View Post
...ill send you a diff of it soon after ive made some more changes so its not too different from the original as i almost completly rewrote the first page and the codec page save points to account for no tmp folder
Awesome - thanks .
erayd is offline   Reply With Quote
Old 12-06-2008, 02:58 PM   #34
rgibbo
Junior Member
rgibbo began at the beginning.
 
Posts: 5
Karma: 14
Join Date: Dec 2008
Device: Sony PRS-505
Here's my version of flag for windows, it is now similar to the original so it *might* run on linux (i dont have linux so i don't know if it works on there or not)

Some other modifications I have made are: -
  • Can detect which platform it is running on, and sorts out folder paths automatically including the temp folder
  • Writes the data to a file by default instead of printing to the console, this is easier for me
  • Doesn't create the output file if the story id is invalid

Any comments are appreciated

Note: The epub and rtf formats don't work on windows as I don't use these, I will do it if anyone needs them
Attached Files
File Type: zip flag-win.zip (43.8 KB, 360 views)

Last edited by rgibbo; 12-06-2008 at 03:01 PM.
rgibbo is offline   Reply With Quote
Old 12-12-2008, 08:27 AM   #35
Lane
Junior Member
Lane began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Nov 2008
Device: HP iPaq 210
rgibbo, thank you so much for that Windows version! I would really appreciate it if you could enable the rtf-saving feature, as that's the one I constantly use.
Lane is offline   Reply With Quote
Old 12-16-2008, 08:28 AM   #36
rgibbo
Junior Member
rgibbo began at the beginning.
 
Posts: 5
Karma: 14
Join Date: Dec 2008
Device: Sony PRS-505
Thanks for Lane for pointing this out.

There is no application for the windows version since I have PHP installed on my development machine, and I run a version of the app that erayd first wrote.

I will be rewriting the application so that it is a proper windows application, without any of the PHP libraries being required. This version will hopefully be posted here at the end of the week if i don't get any work from college to do over christmas.
rgibbo is offline   Reply With Quote
Old 12-16-2008, 02:51 PM   #37
erayd
Zealot
erayd doesn't littererayd doesn't litter
 
Posts: 128
Karma: 146
Join Date: Apr 2008
Device: Sony PRS-650
Thanks rgibbo for the work you've put into this - do you want svn commit access to that repo? Alternatively I can merge changes, if you're happy to wait a few days - I'm pretty busy right now.

Regarding saving to rtf, I'm pretty sure there is a windows version of pandoc - so enabling this should be pretty simple.
erayd is offline   Reply With Quote
Old 12-16-2008, 04:02 PM   #38
Amalthia
Guru
Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.
 
Amalthia's Avatar
 
Posts: 937
Karma: 1960
Join Date: Jan 2007
Location: Anchorage, AK
Device: Sony Reader PRS-505, PRS-650
I can't wait to test this out tonight after work. I also really hope the OTW archive can implement something very similar to what you have here. It would really make reading fan fiction on my reader so much easier. (assuming they have some way to customize the output) because I'm kind of particular about the size of font and margins.
Amalthia is offline   Reply With Quote
Old 12-23-2008, 08:14 AM   #39
moggie
Enthusiast
moggie is on a distinguished road
 
moggie's Avatar
 
Posts: 34
Karma: 74
Join Date: Nov 2008
Location: Cambridge,UK
Device: sony 505
Sugarquill

I have just posted a message about a sugarquill recipe for calibre that I thought people here might be interested in.

http://www.mobileread.com/forums/sho...2&postcount=78
moggie is offline   Reply With Quote
Old 12-23-2008, 12:58 PM   #40
Amalthia
Guru
Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.
 
Amalthia's Avatar
 
Posts: 937
Karma: 1960
Join Date: Jan 2007
Location: Anchorage, AK
Device: Sony Reader PRS-505, PRS-650
I'm going to have to try this out when I'm home. Do I have to do anything special to add the reciepe? or are the instructions included in the zip file?

Also, is there a way to change the LRF formatting? I'm kind of fond of limiting margin space and increasing font size. I'm not sure if I actually need to try and change the format since I haven't tried it yet but I'd like to know if it's an option ahead of time.

Also, since this seems to pull up the author's stories, if the author has multiple stories is there a way to combine all of them into a single file using the Table of Contents to index the individual stories instead of the chapters?

I've been merging author's stories into single files out of necessity. Most fan fiction stories are short and it's really easy to get over 1500 stories on your reader and well it really slows down at that point.
Amalthia is offline   Reply With Quote
Old 12-23-2008, 01:13 PM   #41
moggie
Enthusiast
moggie is on a distinguished road
 
moggie's Avatar
 
Posts: 34
Karma: 74
Join Date: Nov 2008
Location: Cambridge,UK
Device: sony 505
Quote:
Originally Posted by Amalthia View Post
I'm going to have to try this out when I'm home. Do I have to do anything special to add the recipe? or are the instructions included in the zip file?

Also, is there a way to change the LRF formatting? I'm kind of fond of limiting margin space and increasing font size. I'm not sure if I actually need to try and change the format since I haven't tried it yet but I'd like to know if it's an option ahead of time.
Basically look up the author id and pass it as the username

You need to have calibre, its a recipe for calibre There are standard options in the calibre GUI or you can use the command line.

Code:
    --left-margin=LEFT_MARGIN
                        Left margin of page. Default is 20 px.

    --right-margin=RIGHT_MARGIN
                        Right margin of page. Default is 20 px.

    --top-margin=TOP_MARGIN
                        Top margin of page. Default is 10 px.

    --bottom-margin=BOTTOM_MARGIN
                        Bottom margin of page. Default is 0 px.
Quote:
Originally Posted by Amalthia View Post
Also, since this seems to pull up the author's stories, if the author has multiple stories is there a way to combine all of them into a single file using the Table of Contents to index the individual stories instead of the chapters?

I've been merging author's stories into single files out of necessity. Most fan fiction stories are short and it's really easy to get over 1500 stories on your reader and well it really slows down at that point.
It already does this basically you get a 2000 odd page book ( well for some authors ) with each story fully downloaded with an index page with an entry for each story.
moggie is offline   Reply With Quote
Old 12-23-2008, 01:55 PM   #42
Amalthia
Guru
Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.
 
Amalthia's Avatar
 
Posts: 937
Karma: 1960
Join Date: Jan 2007
Location: Anchorage, AK
Device: Sony Reader PRS-505, PRS-650
Quote:
Originally Posted by moggie View Post
Basically look up the author id and pass it as the username

You need to have calibre, its a recipe for calibre There are standard options in the calibre GUI or you can use the command line.
I'm such a dimbat sometimes.

Quote:




It already does this basically you get a 2000 odd page book ( well for some authors ) with each story fully downloaded with an index page with an entry for each story.
Okay this is starting to sound too good to be true.

Amalthia
Amalthia is offline   Reply With Quote
Old 12-23-2008, 06:14 PM   #43
Elfwreck
Grand Sorcerer
Elfwreck ought to be getting tired of karma fortunes by now.Elfwreck ought to be getting tired of karma fortunes by now.Elfwreck ought to be getting tired of karma fortunes by now.Elfwreck ought to be getting tired of karma fortunes by now.Elfwreck ought to be getting tired of karma fortunes by now.Elfwreck ought to be getting tired of karma fortunes by now.Elfwreck ought to be getting tired of karma fortunes by now.Elfwreck ought to be getting tired of karma fortunes by now.Elfwreck ought to be getting tired of karma fortunes by now.Elfwreck ought to be getting tired of karma fortunes by now.Elfwreck ought to be getting tired of karma fortunes by now.
 
Elfwreck's Avatar
 
Posts: 5,142
Karma: 24387852
Join Date: Nov 2008
Location: SF Bay Area, California, USA
Device: Clié; PRS-505; EZR Pocket Pro, PRS-600, Kobo Mini
Quote:
Originally Posted by Amalthia View Post
I can't wait to test this out tonight after work. I also really hope the OTW archive can implement something very similar
OTW's stories all have a 1-page-for-printing version;they can be saved out as HTML with no extra processing. (One page vs chapters was one of the debates that went around during coding discussions; feedback was very clear: fans want both. Some want one all the time, some want the other all the time, some want the option depending on their current reading circumstances.) From what I can tell, the default at OTW is the single-page layout; you have to click a button to get separate chapters.
Elfwreck is offline   Reply With Quote
Old 12-23-2008, 06:16 PM   #44
erayd
Zealot
erayd doesn't littererayd doesn't litter
 
Posts: 128
Karma: 146
Join Date: Apr 2008
Device: Sony PRS-650
Quote:
Originally Posted by Amalthia View Post
I'm going to have to try this out when I'm home. Do I have to do anything special to add the reciepe? or are the instructions included in the zip file?
It will need to be rewritten in PHP, and in the format that FLAG uses for a source. Not particularly hard, but you may want to have the ffnet one open as a reference.

Quote:
Also, is there a way to change the LRF formatting? I'm kind of fond of limiting margin space and increasing font size. I'm not sure if I actually need to try and change the format since I haven't tried it yet but I'd like to know if it's an option ahead of time.
This is currently hardcoded into the LRF output, but as it uses calibre's html2lrf for this you should find it reasonably easy to change.

Quote:
Also, since this seems to pull up the author's stories, if the author has multiple stories is there a way to combine all of them into a single file using the Table of Contents to index the individual stories instead of the chapters?
This doesn't pull up the author's stories, it pulls up all the chapters of the particular story you ask it to grab. Pulling up all stories by an author would require substantial rewriting of the ffnet source - doable, but honestly not something that is a priority for me right now. I would be happy to merge changes though, if someone else adds this.

Last edited by erayd; 12-23-2008 at 06:25 PM.
erayd is offline   Reply With Quote
Old 12-23-2008, 06:58 PM   #45
Amalthia
Guru
Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.Amalthia once ate a cherry pie in a record 7 seconds.
 
Amalthia's Avatar
 
Posts: 937
Karma: 1960
Join Date: Jan 2007
Location: Anchorage, AK
Device: Sony Reader PRS-505, PRS-650
Quote:
Originally Posted by erayd View Post
It will need to be rewritten in PHP, and in the format that FLAG uses for a source. Not particularly hard, but you may want to have the ffnet one open as a reference.

This is currently hardcoded into the LRF output, but as it uses calibre's html2lrf for this you should find it reasonably easy to change.

This doesn't pull up the author's stories, it pulls up all the chapters of the particular story you ask it to grab. Pulling up all stories by an author would require substantial rewriting of the ffnet source - doable, but honestly not something that is a priority for me right now. I would be happy to merge changes though, if someone else adds this.
Thanks for getting back to me so quickly.
Amalthia is offline   Reply With Quote
Reply

Tags
converter, fanfiction, fanfiction.net, grabber, lrf

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Fanfiction.net on Kindle forkyfork Amazon Kindle 26 08-07-2011 08:42 AM
bookmarks/notes grabber Reader2 Android Developer's Corner 0 10-02-2010 09:24 AM
EASY fanfiction grabber? sherryg Workshop 19 01-08-2010 03:13 AM
FLAG (Fanfiction.net Lightweight Automated Grabber) and Calibre? malkie13 Calibre 1 02-10-2009 05:43 PM


All times are GMT -4. The time now is 05:28 AM.


MobileRead.com is a privately owned, operated and funded community.