[GUI Plugin] FanFictionDownLoader

[leistnerm]

Jim -- Thanks for the link. That definetly helps point me in the correct direction. I've been looking over the code to get an idea of how it's structured and how python works. So far it seems pretty straight forward.

I did notice one thing that I'll probably work on, but just in case I can't figure it out, I'll let you know too It doesn't appear you are getting urls using gzip or deflate from the server. I have a similar progam I wrote late last year that DL's any story on FF.net over 25k words and matching a few other criteria automatically. When I started using gzip over just normal requests it started running at least an order of magnitude faster.

I did manage to find a few pages on how to do this in python, but not with the method you are using to retireve urls. So I'll probably play around with that for a starter... something easy before I try to do a whole file myself.

http://stackoverflow.com/questions/3...-fetch-webpage
http://techknack.net/python-urllib2-handlers/

Anyways, thought I'd at least metion my previous experience with a progam doing something similar to this and the speed increase I got.

Looking forward to hopefully being about to eventually contribute to this already great plugin.

Mark

[JimmXinu]

Originally Posted by leistnerm

It doesn't appear you are getting urls using gzip or deflate from the server. I have a similar progam I wrote late last year that DL's any story on FF.net over 25k words and matching a few other criteria automatically. When I started using gzip over just normal requests it started running at least an order of magnitude faster.

That's a very good point.

For a very long time, our focus for this code was primarily on the web version that runs on Google AppEngine. I'm pretty sure I remember confirming that AppEngine did "Content-Encoding: gzip" automatically on the ebook files when they were downloaded from the web app.

But I don't remember ever thinking about it on our client side. It's certainly worth a look, thanks for the suggestion.

BTW, ff.net in particular is slower than it could be--there's a hard coded 0.5sec sleep between hits (plus the slow_down_sleep_time, if set). We have first hand evidence from one of the other developers that ff.net will restrict your IP if you hit them "too fast". See http://www.fanfiction.net/tos/ section 4E.

Jim

[leistnerm]

Originally Posted by JimmXinu

That's a very good point.

For a very long time, our focus for this code was primarily on the web version that runs on Google AppEngine. I'm pretty sure I remember confirming that AppEngine did "Content-Encoding: gzip" automatically on the ebook files when they were downloaded from the web app.

Jim

Hmm. I've been pegging them pretty hard for a while and not been blocked. I guess I should be careful on that. And after further digging, it looks like it has code to support gzip, in the __decode method/function/whatever python calls them. Perhaps the slowdown is just due to that delay.

Anyways thanks for your time

[JimmXinu]

Using the GZipProcessor from http://techknack.net/python-urllib2-handlers/, it's actually really easy to implement.

Some fairly cursory testing seems to indicate that there's no real improvement for ff.net. And several of our supported sites won't send us gzip (looks like it's due to User-Agent). I did see ~30% improvement with TtH, though.

That's as much effort as I'm going to put into it tonight.

[ilovejedd]

Just tried it last night. Excellent plugin. Wish it was available 3 years ago when I started converting fanfics to ebooks.

[iatheia]

Originally Posted by leistnerm

Any chance of getting any of the following sites supported?

http://www.fanficauthors.net/
http://www.hpfanficarchive.com/
http://thequidditchpitch.org/
http://www.phoenixsong.net/
http://restrictedsection.org/
http://www.mugglenet.com/
http://checkmated.com/
http://mvsfiction.com/
http://sugarquill.net/

... or any documentation on how you go about creating a new site adapter? I can scrape web pages easily enough in C# or similar languages, but don't have much experience with Python.

I can take a look at a few more populated ones in a bit.

Ida.

[Ayiana]

First off, thank you so much for crafting this plugin. In the past I've spent untold hours downloading fics manually, adding them to Calibre, and then filling in all the fields. This plugin, if I can get it to work right, is the answer to a prayer.

Here's what's going on, though. I installed the plugin, customized my preferences so that the downloader icon should appear in my menubar whether or not my Kindle is connected, and restarted Calibre. When Calibre first starts, everything looks fine. Then the ffdownloader icon vanishes, so I'm assuming it's crashing? If I click on it before it crashes, I get this error message:

calibre, version 0.8.38
ERROR: Unhandled exception: <b>IndexError</b>:list index out of range

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 241, in do_loop
IndexError: list index out of range

I've tried uninstalling and reinstalling the plugin several times, even with reboots/restarts in between, to no avail. Any ideas?

[JimmXinu]

Originally Posted by Ayiana

Here's what's going on, though. I installed the plugin, customized my preferences so that the downloader icon should appear in my menubar whether or not my Kindle is connected, and restarted Calibre. When Calibre first starts, everything looks fine. Then the ffdownloader icon vanishes, so I'm assuming it's crashing?

I don't think I've ever seen the button disappear after it was on screen--and believe me, I crashed it a lot during development.

Originally Posted by Ayiana

If I click on it before it crashes, I get this error message:

calibre, version 0.8.38
ERROR: Unhandled exception: <b>IndexError</b>:list index out of range

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 241, in do_loop
IndexError: list index out of range

The only way I know to get that error is to click the plug in with no books selected (or choose 'Add New from URL(s)'), then *not* enter any URLs and click OK.

If you could start calibre in debug mode (calibre-debug -g) and post the debug output, that would be helpful.

Jim

[Ayiana]

Well, I'm utterly baffled now.

I uninstalled the plugin and shut down my computer last night right after I posted here. Reinstalled this morning, set my preferences, and ... everything's fine. No crash. No error message. Color me confused but relieved.

And thank you again. This plugin is a fantastic addition to Calibre's functionality.

[JimmXinu]

@Ayiana - Glad you like it. Hopefully it stays fixed.

[Laia]

Hi Jim,
I downloaded the link "FanFictionDownLoaderPlugin-1.3.4.zip" but it's a .zip, where I had to click to open the program?

Thank you

[davidfor]

Originally Posted by Laia

I downloaded the link "FanFictionDownLoaderPlugin-1.3.4.zip" but it's a .zip, where I had to click to open the program?

Save the file to disk. Then open Calibre and it's preferences. Click on "Plugins" (near the bottom-left of the window) and then click on "Load plugin from file". In the file picker window, find the pligin zip file, select it and press "Open". Then follow the rest of the prompts to install and configure.

[JimmXinu]

@Laia - What davidfor said is correct.

Or, you don't need to download it yourself--calibre can do it for you.

For full instructions on installing plugins, see:
https://www.mobileread.com/forums/sho...d.php?t=118680

[jjm6383]

Hi,

I really like this plugin. It's making my life easier.

I haven't been a big user of FanFiction Downloader. I use Deffnet. With Deffnet, you could set a simple CSS file, so stories came out lightly formated. Can you do this with the plugin? Or could this be added to the plugin at a later time?

Thanks,
Joyslin

[JimmXinu]

Originally Posted by jjm6383

I haven't been a big user of FanFiction Downloader. I use Deffnet. With Deffnet, you could set a simple CSS file, so stories came out lightly formated. Can you do this with the plugin? Or could this be added to the plugin at a later time?

Glad you like it.

Do you use the HTML output? epub, mobi and obviously txt are more limited in what they can use for CSS. And the story text HTML is taken from the source sites as untouched as possible. Deffnetizer only deals with ffnet, which for all its sins is extremely consistent in it's HTML. Trying to wrap custom html and CSS around all the different sites would be... interesting, I fear.

I'll add user CSS to the wish list, but my free time has largely dried up again for a while, so don't hold your breath.