[GUI Plugin] FanFictionDownLoader

[FicReader81]

Originally Posted by jesterod

i just updated and i get an error when trying to update

Hi. I am getting a similar error message while trying to update stories (The number doesn't matter nor which stories I try to update. Could be 5, could be 50.) after I updated the plugin on calibre today. Can someone please suggest a solution.

[JimmXinu]

Originally Posted by FicReader81

Hi. I am getting a similar error message while trying to update stories (The number doesn't matter nor which stories I try to update. Could be 5, could be 50.) after I updated the plugin on calibre today. Can someone please suggest a solution.

Don't try to update books that don't have story URLs.

When you are looking at the update list, remove the ones that say 'No story url found' before hitting OK.

[fatalerror328]

I was interested if there are any plans on improving mobi generating? Like TOC and fixing unreadable parts? If not, then maybe I can try to help? Is there any prefered method to help you with development?

[FicReader81]

Originally Posted by JimmXinu

Don't try to update books that don't have story URLs.

When you are looking at the update list, remove the ones that say 'No story url found' before hitting OK.

Thank you for the reply and the solution. It worked. I had quite a few stories that had no URLs or those that had been deleted on FF.net and once I got rid of those I stopped receiving the error.

However now their is a different issue. Whenever I try to update the stories (Again the no doesn't matter, I have tried it with groups numbering 5 to 30 of different stories from my collection of 400 odd), the plugin starts fetching metadata for the stories but it doesn't finish. Calibre ends up freezing somewhere during this step of the updating and when I try to retrieve it "Windows Explorer" gives me the error message that Calibre will now be closing. Calibre hasn't been able to successfully finish updating the stories I select (As I mentioned I have tried different size bunches from the collection) ever since I was able to get rid of the earlier error.

I have been using the plugin on Calibre for 4-5 months now and this is the first time I have experienced any issues. It seems these issues have surfaced right after I updated the plugin today.

I would appreciate if you could look into the problems and suggest a solution again. Would truly appreciate it.

[JimmXinu]

Originally Posted by fatalerror328

I was interested if there are any plans on improving mobi generating? Like TOC and fixing unreadable parts? If not, then maybe I can try to help? Is there any prefered method to help you with development?

I don't have any plans to, no.

The Google code project we keep it in has a facility for people to clone the repository (not the whole project) for their own use.

I'd suggest starting with a clone and then once you've made some progress let me know and I'll review it for inclusion.

[JimmXinu]

Attached is a test version with a potential fix for the KeyError: u'site' problem.

I'd appreciate it if a couple of the people who saw that problem would give it a try.

Thanks.

UPDATE Feb 28, 2014 -- Remove obsolete beta versions

[JimmXinu]

Originally Posted by FicReader81

However now their is a different issue. Whenever I try to update the stories (Again the no doesn't matter, I have tried it with groups numbering 5 to 30 of different stories from my collection of 400 odd), the plugin starts fetching metadata for the stories but it doesn't finish. Calibre ends up freezing somewhere during this step of the updating and when I try to retrieve it "Windows Explorer" gives me the error message that Calibre will now be closing. Calibre hasn't been able to successfully finish updating the stories I select (As I mentioned I have tried different size bunches from the collection) ever since I was able to get rid of the earlier error.

If you just leave it alone for several minutes does it eventually finish?

To prevent throttling on ffnet, the sleep time between requests was significantly increased version before the last. Long enough that calibre was entering the 'not responding' state according to windows.

The last version included code to reduce those sleep times for small update sets.

Quoting myself from an earlier post:

Originally Posted by JimmXinu

Checking for updates on hundreds* of stories at once causes excessive traffic on the source site. fanfiction.net in particular has been known to block people's IP address if they generate too much traffic.

Instead, I highly recommend that you 'follow' stories and use the site's email notifications to inform you about updates and then update those specific stories. I also recommend using Update EPUB, not Overwrite, unless you have reason to believe that a particular story has changes in previous chapters.

* The original post said thousands.

Throttling by ffnet is no longer a theory, but an established fact.

[FicReader81]

Originally Posted by JimmXinu

Attached is a test version with a potential fix for the KeyError: u'site' problem.

I'd appreciate it if a couple of the people who saw that problem would give it a try.

Thanks.

Hi. I would like to help but I don't know what to do with the contents of the zip file. If you could explain, perhaps I may be able to help.

[FicReader81]

Originally Posted by JimmXinu

If you just leave it alone for several minutes does it eventually finish?

To prevent throttling on ffnet, the sleep time between requests was significantly increased version before the last. Long enough that calibre was entering the 'not responding' state according to windows.

The last version included code to reduce those sleep times for small update sets.

Quoting myself from an earlier post:


* The original post said thousands.

Throttling by ffnet is no longer a theory, but an established fact.

It doesn't matter how long I allow the process to continue, it just doesn't finish. As an example I take a sample of 5 fics from FF.net in my library and try to update them. The process would start and the plugin would try to fetch the metadata for the stories. Sometimes it would get completed, but most of the times it would just get stuck at lets say 3rd out of the 5 stories (I would say out of 10 tries, 2 would succeed and 8 would freeze) and when I try to retrieve the calibre program, "Windows Explorer" would come up with the message that calibre is not responding and it would then close the program. Prior to this update I could update my entire library in one go, but after failing a couple of times yesterday, I tried to update in batches of 10-20 fics. But as I said, 8 out of 10 times, calibre would just freeze (No matter how long I allowed it) and then I would have to restart the program and start again.

I usually update my fics once in a week or 10 days and haven't encountered any problems prior to this update (Although the update of the entire library did take a long time), but trying to select 10 and going from there is a bit tedious, esp if you don't know whether the update would succeed or not. I have made the changes you suggested wrt choosing update instead of overwrite, but the issue is the process doesn't go beyond the stage of fetching metadata more often than not.

Hope this would be of some use to you. Thank you once again for the reply and hope you can provide a solution.

[JimmXinu]

Originally Posted by FicReader81

Hi. I would like to help but I don't know what to do with the contents of the zip file. If you could explain, perhaps I may be able to help.

See this "Introduction to plugins" posting and scroll down to the section titled "How do I install a plugin manually?"

[JimmXinu]

Originally Posted by FicReader81

It doesn't matter how long I allow the process to continue, it just doesn't finish. As an example I take a sample of 5 fics from FF.net in my library and try to update them.
...

You're probably on ffnet's throttle list. They've been blocking (errors) more than hanging lately, but they've done both in past.

Evidence from other users is that it generally takes a couple days for the throttle to expire. After that, you're generally okay again as long as you don't do too many updates at once. Personally, I define 'too many' as about 10.

[AmberDragon]

Originally Posted by JimmXinu

Evidence from other users is that it generally takes a couple days for the throttle to expire. After that, you're generally okay again as long as you don't do too many updates at once. Personally, I define 'too many' as about 10.

Interesting. I got caught in the throttle problem a while back and was confused because I try and keep my updating number to three or less at time. I chalked it up to FFNet's general flakiness, but I've just checked through my FFNet anthologies and found one that has eight stories in it. Could an update of a large anthology trigger the throttle?

[JimmXinu]

Originally Posted by AmberDragon

Interesting. I got caught in the throttle problem a while back and was confused because I try and keep my updating number to three or less at time. I chalked it up to FFNet's general flakiness, but I've just checked through my FFNet anthologies and found one that has eight stories in it. Could an update of a large anthology trigger the throttle?

Under the covers, updating an anthology is really an update of each individual story in the anthology. So yes, updating anthologies could if the sleep times aren't long enough.

[Robin42069]

Status
Title
Author
Comment
URL
Bad
Unknown
Unknown
Story does not exist: (http://www.fanfiction.net/s/5904185/1/)
https://www.fanfiction.net/s/5904185/1/


I've been seeing this since i tried updating or even adding new ones to my library. could this mean i'm on they're throttle list or something cuz its not the only story that says that and i've then checked on them and the story is there?

[FicReader81]

Originally Posted by JimmXinu

See this "Introduction to plugins" posting and scroll down to the section titled "How do I install a plugin manually?"

Hi I installed the plugin as per the instructions 2 days ago. I have been updating and adding new fics as usual. Their doesn't seem to be any problems other than the program hanging as I described in my earlier posts. (Which you suggested could be from the FF.net servers throttling. Although for me this has happened for the first time after the recent update. I have always updated fics once a week or 10 days and didn't face this issue before.)

Is their anything specific you want me to check for or a test to run? If so, please let me know.