02-02-2012, 10:20 PM | #76 |
Junior Member
Posts: 3
Karma: 10
Join Date: Feb 2012
Device: Kindle, iPad, Blackberry
|
Jim -- Thanks for the link. That definetly helps point me in the correct direction. I've been looking over the code to get an idea of how it's structured and how python works. So far it seems pretty straight forward.
I did notice one thing that I'll probably work on, but just in case I can't figure it out, I'll let you know too It doesn't appear you are getting urls using gzip or deflate from the server. I have a similar progam I wrote late last year that DL's any story on FF.net over 25k words and matching a few other criteria automatically. When I started using gzip over just normal requests it started running at least an order of magnitude faster. I did manage to find a few pages on how to do this in python, but not with the method you are using to retireve urls. So I'll probably play around with that for a starter... something easy before I try to do a whole file myself. http://stackoverflow.com/questions/3...-fetch-webpage http://techknack.net/python-urllib2-handlers/ Anyways, thought I'd at least metion my previous experience with a progam doing something similar to this and the speed increase I got. Looking forward to hopefully being about to eventually contribute to this already great plugin. Mark |
02-02-2012, 10:57 PM | #77 | |
Plugin Developer
Posts: 6,534
Karma: 4277329
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
Quote:
For a very long time, our focus for this code was primarily on the web version that runs on Google AppEngine. I'm pretty sure I remember confirming that AppEngine did "Content-Encoding: gzip" automatically on the ebook files when they were downloaded from the web app. But I don't remember ever thinking about it on our client side. It's certainly worth a look, thanks for the suggestion. BTW, ff.net in particular is slower than it could be--there's a hard coded 0.5sec sleep between hits (plus the slow_down_sleep_time, if set). We have first hand evidence from one of the other developers that ff.net will restrict your IP if you hit them "too fast". See http://www.fanfiction.net/tos/ section 4E. Jim |
|
Advert | |
|
02-02-2012, 11:17 PM | #78 | |
Junior Member
Posts: 3
Karma: 10
Join Date: Feb 2012
Device: Kindle, iPad, Blackberry
|
Quote:
Anyways thanks for your time |
|
02-03-2012, 12:35 AM | #79 |
Plugin Developer
Posts: 6,534
Karma: 4277329
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
Using the GZipProcessor from http://techknack.net/python-urllib2-handlers/, it's actually really easy to implement.
Some fairly cursory testing seems to indicate that there's no real improvement for ff.net. And several of our supported sites won't send us gzip (looks like it's due to User-Agent). I did see ~30% improvement with TtH, though. That's as much effort as I'm going to put into it tonight. |
02-03-2012, 10:53 AM | #80 |
hopeless n00b
Posts: 5,110
Karma: 19597086
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, Libra H2O, iPad 10.5, iPad 11, iPad 12.9
|
Just tried it last night. Excellent plugin. Wish it was available 3 years ago when I started converting fanfics to ebooks.
|
Advert | |
|
02-05-2012, 10:38 PM | #81 | |
Zealot
Posts: 139
Karma: 21754
Join Date: Mar 2011
Device: Kindle 3, Icarus Illumina XL, Boyue T62
|
Quote:
Ida. |
|
02-05-2012, 10:59 PM | #82 |
Junior Member
Posts: 2
Karma: 10
Join Date: Feb 2012
Device: Kindle 2
|
First off, thank you so much for crafting this plugin. In the past I've spent untold hours downloading fics manually, adding them to Calibre, and then filling in all the fields. This plugin, if I can get it to work right, is the answer to a prayer.
Here's what's going on, though. I installed the plugin, customized my preferences so that the downloader icon should appear in my menubar whether or not my Kindle is connected, and restarted Calibre. When Calibre first starts, everything looks fine. Then the ffdownloader icon vanishes, so I'm assuming it's crashing? If I click on it before it crashes, I get this error message: calibre, version 0.8.38 ERROR: Unhandled exception: <b>IndexError</b>:list index out of range Traceback (most recent call last): File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 241, in do_loop IndexError: list index out of range I've tried uninstalling and reinstalling the plugin several times, even with reboots/restarts in between, to no avail. Any ideas? |
02-06-2012, 10:17 AM | #83 | ||
Plugin Developer
Posts: 6,534
Karma: 4277329
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
Quote:
Quote:
If you could start calibre in debug mode (calibre-debug -g) and post the debug output, that would be helpful. Jim |
||
02-06-2012, 10:53 AM | #84 |
Junior Member
Posts: 2
Karma: 10
Join Date: Feb 2012
Device: Kindle 2
|
Well, I'm utterly baffled now.
I uninstalled the plugin and shut down my computer last night right after I posted here. Reinstalled this morning, set my preferences, and ... everything's fine. No crash. No error message. Color me confused but relieved. And thank you again. This plugin is a fantastic addition to Calibre's functionality. |
02-06-2012, 11:15 PM | #85 |
Plugin Developer
Posts: 6,534
Karma: 4277329
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
@Ayiana - Glad you like it. Hopefully it stays fixed.
|
02-07-2012, 01:13 PM | #86 |
Junior Member
Posts: 1
Karma: 10
Join Date: Feb 2012
Device: none
|
Hi Jim,
I downloaded the link "FanFictionDownLoaderPlugin-1.3.4.zip" but it's a .zip, where I had to click to open the program? Thank you |
02-07-2012, 08:45 PM | #87 |
Grand Sorcerer
Posts: 24,905
Karma: 47303822
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Save the file to disk. Then open Calibre and it's preferences. Click on "Plugins" (near the bottom-left of the window) and then click on "Load plugin from file". In the file picker window, find the pligin zip file, select it and press "Open". Then follow the rest of the prompts to install and configure.
|
02-07-2012, 08:50 PM | #88 |
Plugin Developer
Posts: 6,534
Karma: 4277329
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
@Laia - What davidfor said is correct.
Or, you don't need to download it yourself--calibre can do it for you. For full instructions on installing plugins, see: https://www.mobileread.com/forums/sho...d.php?t=118680 |
02-07-2012, 09:23 PM | #89 |
Enthusiast
Posts: 33
Karma: 50
Join Date: Nov 2007
Location: Texas
Device: Pandigital Nova, Aluratek Libre Pro, Dell Axim x51v
|
Hi,
I really like this plugin. It's making my life easier. I haven't been a big user of FanFiction Downloader. I use Deffnet. With Deffnet, you could set a simple CSS file, so stories came out lightly formated. Can you do this with the plugin? Or could this be added to the plugin at a later time? Thanks, Joyslin |
02-08-2012, 06:13 PM | #90 | |
Plugin Developer
Posts: 6,534
Karma: 4277329
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
Quote:
Do you use the HTML output? epub, mobi and obviously txt are more limited in what they can use for CSS. And the story text HTML is taken from the source sites as untouched as possible. Deffnetizer only deals with ffnet, which for all its sins is extremely consistent in it's HTML. Trying to wrap custom html and CSS around all the different sites would be... interesting, I fear. I'll add user CSS to the wish list, but my free time has largely dried up again for a while, so don't hold your breath. |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
[GUI Plugin] Find Duplicates | kiwidude | Plugins | 1110 | 07-30-2024 06:59 PM |
[GUI Plugin] Count Pages | kiwidude | Plugins | 1750 | 04-07-2024 01:20 AM |
[GUI Plugin] Open With | kiwidude | Plugins | 403 | 04-01-2024 08:39 AM |
[GUI Plugin] Resize Cover | kiwidude | Plugins | 95 | 03-16-2024 11:55 PM |
[GUI Plugin] Plugin Updater **Deprecated** | kiwidude | Plugins | 159 | 06-19-2011 12:27 PM |