Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 02-02-2012, 11:20 PM   #76
leistnerm
Junior Member
leistnerm began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Feb 2012
Device: Kindle, iPad, Blackberry
Jim -- Thanks for the link. That definetly helps point me in the correct direction. I've been looking over the code to get an idea of how it's structured and how python works. So far it seems pretty straight forward.

I did notice one thing that I'll probably work on, but just in case I can't figure it out, I'll let you know too It doesn't appear you are getting urls using gzip or deflate from the server. I have a similar progam I wrote late last year that DL's any story on FF.net over 25k words and matching a few other criteria automatically. When I started using gzip over just normal requests it started running at least an order of magnitude faster.

I did manage to find a few pages on how to do this in python, but not with the method you are using to retireve urls. So I'll probably play around with that for a starter... something easy before I try to do a whole file myself.

http://stackoverflow.com/questions/3...-fetch-webpage
http://techknack.net/python-urllib2-handlers/

Anyways, thought I'd at least metion my previous experience with a progam doing something similar to this and the speed increase I got.

Looking forward to hopefully being about to eventually contribute to this already great plugin.

Mark
leistnerm is offline   Reply With Quote
Old 02-02-2012, 11:57 PM   #77
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,820
Karma: 510215
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by leistnerm View Post
It doesn't appear you are getting urls using gzip or deflate from the server. I have a similar progam I wrote late last year that DL's any story on FF.net over 25k words and matching a few other criteria automatically. When I started using gzip over just normal requests it started running at least an order of magnitude faster.
That's a very good point.

For a very long time, our focus for this code was primarily on the web version that runs on Google AppEngine. I'm pretty sure I remember confirming that AppEngine did "Content-Encoding: gzip" automatically on the ebook files when they were downloaded from the web app.

But I don't remember ever thinking about it on our client side. It's certainly worth a look, thanks for the suggestion.

BTW, ff.net in particular is slower than it could be--there's a hard coded 0.5sec sleep between hits (plus the slow_down_sleep_time, if set). We have first hand evidence from one of the other developers that ff.net will restrict your IP if you hit them "too fast". See http://www.fanfiction.net/tos/ section 4E.

Jim
JimmXinu is offline   Reply With Quote
 
Advertisement
Old 02-03-2012, 12:17 AM   #78
leistnerm
Junior Member
leistnerm began at the beginning.
 
Posts: 3
Karma: 10
Join Date: Feb 2012
Device: Kindle, iPad, Blackberry
Quote:
Originally Posted by JimmXinu View Post
That's a very good point.

For a very long time, our focus for this code was primarily on the web version that runs on Google AppEngine. I'm pretty sure I remember confirming that AppEngine did "Content-Encoding: gzip" automatically on the ebook files when they were downloaded from the web app.

Jim
Hmm. I've been pegging them pretty hard for a while and not been blocked. I guess I should be careful on that. And after further digging, it looks like it has code to support gzip, in the __decode method/function/whatever python calls them. Perhaps the slowdown is just due to that delay.

Anyways thanks for your time
leistnerm is offline   Reply With Quote
Old 02-03-2012, 01:35 AM   #79
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,820
Karma: 510215
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Using the GZipProcessor from http://techknack.net/python-urllib2-handlers/, it's actually really easy to implement.

Some fairly cursory testing seems to indicate that there's no real improvement for ff.net. And several of our supported sites won't send us gzip (looks like it's due to User-Agent). I did see ~30% improvement with TtH, though.

That's as much effort as I'm going to put into it tonight.
JimmXinu is offline   Reply With Quote
Old 02-03-2012, 11:53 AM   #80
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 2,312
Karma: 5761596
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PRS-350, Nexus S, Galaxy S, Nook Color, iPhone4, iPT4, iPad 2012
Just tried it last night. Excellent plugin. Wish it was available 3 years ago when I started converting fanfics to ebooks.
ilovejedd is offline   Reply With Quote
Old 02-05-2012, 11:38 PM   #81
iatheia
Connoisseur
iatheia is as sexy as a twisted cruller doughtnut.iatheia is as sexy as a twisted cruller doughtnut.iatheia is as sexy as a twisted cruller doughtnut.iatheia is as sexy as a twisted cruller doughtnut.iatheia is as sexy as a twisted cruller doughtnut.iatheia is as sexy as a twisted cruller doughtnut.iatheia is as sexy as a twisted cruller doughtnut.iatheia is as sexy as a twisted cruller doughtnut.iatheia is as sexy as a twisted cruller doughtnut.iatheia is as sexy as a twisted cruller doughtnut.iatheia is as sexy as a twisted cruller doughtnut.
 
Posts: 90
Karma: 15316
Join Date: Mar 2011
Device: Kindle 3
Quote:
Originally Posted by leistnerm View Post
Any chance of getting any of the following sites supported?

http://www.fanficauthors.net/
http://www.hpfanficarchive.com/
http://thequidditchpitch.org/
http://www.phoenixsong.net/
http://restrictedsection.org/
http://www.mugglenet.com/
http://checkmated.com/
http://mvsfiction.com/
http://sugarquill.net/

... or any documentation on how you go about creating a new site adapter? I can scrape web pages easily enough in C# or similar languages, but don't have much experience with Python.
I can take a look at a few more populated ones in a bit.

Ida.
iatheia is offline   Reply With Quote
Old 02-05-2012, 11:59 PM   #82
Ayiana
Junior Member
Ayiana began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Feb 2012
Device: Kindle 2
First off, thank you so much for crafting this plugin. In the past I've spent untold hours downloading fics manually, adding them to Calibre, and then filling in all the fields. This plugin, if I can get it to work right, is the answer to a prayer.

Here's what's going on, though. I installed the plugin, customized my preferences so that the downloader icon should appear in my menubar whether or not my Kindle is connected, and restarted Calibre. When Calibre first starts, everything looks fine. Then the ffdownloader icon vanishes, so I'm assuming it's crashing? If I click on it before it crashes, I get this error message:

calibre, version 0.8.38
ERROR: Unhandled exception: <b>IndexError</b>:list index out of range

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 241, in do_loop
IndexError: list index out of range

I've tried uninstalling and reinstalling the plugin several times, even with reboots/restarts in between, to no avail. Any ideas?
Ayiana is offline   Reply With Quote
Old 02-06-2012, 11:17 AM   #83
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,820
Karma: 510215
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by Ayiana View Post
Here's what's going on, though. I installed the plugin, customized my preferences so that the downloader icon should appear in my menubar whether or not my Kindle is connected, and restarted Calibre. When Calibre first starts, everything looks fine. Then the ffdownloader icon vanishes, so I'm assuming it's crashing?
I don't think I've ever seen the button disappear after it was on screen--and believe me, I crashed it a lot during development.
Quote:
Originally Posted by Ayiana View Post
If I click on it before it crashes, I get this error message:

calibre, version 0.8.38
ERROR: Unhandled exception: <b>IndexError</b>:list index out of range

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 241, in do_loop
IndexError: list index out of range
The only way I know to get that error is to click the plug in with no books selected (or choose 'Add New from URL(s)'), then *not* enter any URLs and click OK.

If you could start calibre in debug mode (calibre-debug -g) and post the debug output, that would be helpful.

Jim
JimmXinu is offline   Reply With Quote
Old 02-06-2012, 11:53 AM   #84
Ayiana
Junior Member
Ayiana began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Feb 2012
Device: Kindle 2
Well, I'm utterly baffled now.

I uninstalled the plugin and shut down my computer last night right after I posted here. Reinstalled this morning, set my preferences, and ... everything's fine. No crash. No error message. Color me confused but relieved.

And thank you again. This plugin is a fantastic addition to Calibre's functionality.
Ayiana is offline   Reply With Quote
Old 02-07-2012, 12:15 AM   #85
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,820
Karma: 510215
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
@Ayiana - Glad you like it. Hopefully it stays fixed.
JimmXinu is offline   Reply With Quote
Old 02-07-2012, 02:13 PM   #86
Laia
Junior Member
Laia began at the beginning.
 
Posts: 1
Karma: 10
Join Date: Feb 2012
Device: none
Hi Jim,
I downloaded the link "FanFictionDownLoaderPlugin-1.3.4.zip" but it's a .zip, where I had to click to open the program?

Thank you
Laia is offline   Reply With Quote
Old 02-07-2012, 09:45 PM   #87
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 6,087
Karma: 6238033
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo Touch, Kobo Glo
Quote:
Originally Posted by Laia View Post
I downloaded the link "FanFictionDownLoaderPlugin-1.3.4.zip" but it's a .zip, where I had to click to open the program?
Save the file to disk. Then open Calibre and it's preferences. Click on "Plugins" (near the bottom-left of the window) and then click on "Load plugin from file". In the file picker window, find the pligin zip file, select it and press "Open". Then follow the rest of the prompts to install and configure.
davidfor is online now   Reply With Quote
Old 02-07-2012, 09:50 PM   #88
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,820
Karma: 510215
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
@Laia - What davidfor said is correct.

Or, you don't need to download it yourself--calibre can do it for you.

For full instructions on installing plugins, see:
http://www.mobileread.com/forums/sho...d.php?t=118680
JimmXinu is offline   Reply With Quote
Old 02-07-2012, 10:23 PM   #89
jjm6383
Enthusiast
jjm6383 is on a distinguished road
 
Posts: 31
Karma: 50
Join Date: Nov 2007
Location: Texas
Device: Pandigital Nova, Aluratek Libre Pro, Dell Axim x51v
Hi,

I really like this plugin. It's making my life easier.

I haven't been a big user of FanFiction Downloader. I use Deffnet. With Deffnet, you could set a simple CSS file, so stories came out lightly formated. Can you do this with the plugin? Or could this be added to the plugin at a later time?

Thanks,
Joyslin
jjm6383 is offline   Reply With Quote
Old 02-08-2012, 07:13 PM   #90
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,820
Karma: 510215
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by jjm6383 View Post
I haven't been a big user of FanFiction Downloader. I use Deffnet. With Deffnet, you could set a simple CSS file, so stories came out lightly formated. Can you do this with the plugin? Or could this be added to the plugin at a later time?
Glad you like it.

Do you use the HTML output? epub, mobi and obviously txt are more limited in what they can use for CSS. And the story text HTML is taken from the source sites as untouched as possible. Deffnetizer only deals with ffnet, which for all its sins is extremely consistent in it's HTML. Trying to wrap custom html and CSS around all the different sites would be... interesting, I fear.

I'll add user CSS to the wish list, but my free time has largely dried up again for a while, so don't hold your breath.
JimmXinu is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Open With kiwidude Plugins 251 12-15-2014 01:07 PM
[GUI Plugin] Count Pages kiwidude Plugins 701 11-26-2014 06:34 AM
[GUI Plugin] Find Duplicates kiwidude Plugins 510 11-16-2014 12:37 AM
[GUI Plugin] Resize Cover kiwidude Plugins 34 07-13-2014 08:15 PM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 01:27 PM


All times are GMT -4. The time now is 07:32 AM.


MobileRead.com is a privately owned, operated and funded community.