[GUI Plugin] FanFictionDownLoader - Page 122

riothamus · 09-08-2013, 06:59 PM

Not sure if its a fluke or not, but I'm getting the 403 error with ff.net again. A few hours ago everything was working fine and I got several stories downloaded. Just tried to do an update and no go.

cryzed · 09-08-2013, 07:44 PM

That's probably because FanFiction.Net doesn't want something that looks like an entirely automated bot network (spanning hundreds of IPs) crawling their website -- It currently looks like that, because the same exact non-default user-agent header is used by a wide range of users, thus they blocked it, just like they did with the Python urllib default user-agent.

An easy fix for this would be to simply allow users to customize the user-agent in their personal.ini configuration file, and set it to something commonly used.

RenoA · 09-08-2013, 08:00 PM

Actually, my FFDL was working before I updated it to 1.7.41. I think I had 1.7.39 before that, and it worked fine, then I updated and it stopped. Incidentally, I also updated Calibre from 1.1 to 1.2 at the same time. I tried the fix on the previous page, and it didn't work :/

JimmXinu · 09-08-2013, 08:05 PM

Quote:

Originally Posted by riothamus

Not sure if its a fluke or not, but I'm getting the 403 error with ff.net again. A few hours ago everything was working fine and I got several stories downloaded. Just tried to do an update and no go.

Hrm. This is what I've been afraid of. Looks like they are now specifically blocking our new FFDL User-agent, too.

I need to think about how I want to handle this for a bit. Morally, not technologically. I have a whole bag full of tricks I could apply (including cryzed's suggestion).

At a minimum, however, downloading ffnet stories will be slowed down significantly-- probably 5x slower.

cryzed · 09-08-2013, 08:05 PM

It's definitely the blocked user-agent, I know because when changing it to something more common in the "base_adapter.py" file it works perfectly again.

JimmXinu · 09-08-2013, 08:11 PM

Quote:

Originally Posted by cryzed

It's definitely the blocked user-agent, I know because when changing it to something more common in the "base_adapter.py" file it works perfectly again.

Yep. It's the 'FFDL' part now, not even the specific string. (The Modify Headers add-on in Firefox is extremely useful for web developers.)

JimmXinu · 09-08-2013, 08:56 PM

Attached is another version to try. It significantly increases the wait between chapter fetches on ffnet.

User agent remains configured as:

Code:

## User-agent
#user_agent:FFDL/1.7

UPDATE Sep 14 2013 - Remove obsolete beta versions

skb · 09-09-2013, 03:05 AM

I gave it a go and it works, thank you. It wasn't that much slower for me (but then I'm on wireless and my connection speed is ordinary).

And I learned how to manually install a plugin! Go me!

Patricia1066 · 09-09-2013, 07:12 AM

I installed fanfictiondownloader from the downloaded zip. It still shows 1.7.41 - am I doing something wrong? I deleted 1.7.41 then installed plugin from zip file.

I tried to install a new story from ff.net and got the and got the following log. Thats when I realised that the plugin had the same version number as the recent update.

Status Title Author Comment URL
Bad Unknown Unknown HTTP Error 403: Forbidden
http://www.fanfiction.net/s/9250468/1/

Ignore this comment, I hadn't restarted calibre. Its downloading now.

Patricia1066 · 09-09-2013, 07:41 AM

The plugin has not changed its version number after restart. The FFDL identified a valid URL, and started download. But no file was created.

The log says its been created. But I cannot find it.

Pastebin Paste ID: j8zRG0k3

BaconButty · 09-09-2013, 08:08 AM

The recurrent 403 error from Fanfiction.net is resolved after a manual installation of JimmXinu's latest zip file (see comment #1822).

Thank you.

No noticeable speed reduction here.

seabream · 09-09-2013, 08:20 AM

@BaconBuddy - I think that was the one Patricia1066 installed. Otherwise the version number question wouldn't make sense right? (ignore this comment if your comment at #1826 wasn't a reply to Patricia1066)

@Patricia1066 - JimmXinu generally doesn't give plugin versions an official higher number unless they are official releases. Posts announcing those look like #1815 or #1808. And unlike #1822, don't contain an attached version. So the lack of change in version number doesn't mean that you're doing anything wrong. It's supposed to look that way. Can you post the log file from the one where it says it created a file that doesn't show up?

Patricia1066 · 09-09-2013, 08:27 AM

Quote:

Originally Posted by seabream

@BaconBuddy - I think that was the one Patricia1066 installed. Otherwise the version number question wouldn't make sense right?

Yes it is the downloaded zip that I have installed.

@Patricia1066 - JimmXinu generally doesn't give plugin versions an official higher number unless they are official releases. Posts announcing those look like #1815 or #1808. And unlike #1822, don't contain an attached version. So the lack of change in version number doesn't mean that you're doing anything wrong. It's supposed to look that way. Can you post the log file from the failed download please?

Can you access it from the pastebin post #1825, or shall I paste it here?

Thanks for your help.

Patricia1066 · 09-09-2013, 08:34 AM

Stranger and stranger.

One successful download, which has not shown up.
One successful download which has shown up in the calibre library.

Same author, same site but different stories (ff.net)

seabream · 09-09-2013, 08:36 AM

Quote:

Originally Posted by Patricia1066

Can you access it from the pastebin post #1825, or shall I paste it here?

Ah, so that's what that is. I'm not familiar with pastebin, so thought it might have been a signature I didn't understand.

Hmm. Reading the post, I don't have a better idea of what didn't work. Sorry. Maybe someone else will?

09-08-2013, 07:44 PM	#1817
cryzed Evangelist Posts: 408 Karma: 1050547 Join Date: Mar 2011 Device: Kindle Oasis 2	That's probably because FanFiction.Net doesn't want something that looks like an entirely automated bot network (spanning hundreds of IPs) crawling their website -- It currently looks like that, because the same exact non-default user-agent header is used by a wide range of users, thus they blocked it, just like they did with the Python urllib default user-agent. An easy fix for this would be to simply allow users to customize the user-agent in their personal.ini configuration file, and set it to something commonly used. Last edited by cryzed; 09-08-2013 at 07:49 PM.

09-08-2013, 08:56 PM	#1822
JimmXinu Plugin Developer Posts: 6,306 Karma: 3966249 Join Date: Dec 2011 Location: Midwest USA Device: Kindle Paperwhite(10th)	Attached is another version to try. It significantly increases the wait between chapter fetches on ffnet. User agent remains configured as: Code: ## User-agent #user_agent:FFDL/1.7 UPDATE Sep 14 2013 - Remove obsolete beta versions Last edited by JimmXinu; 09-14-2013 at 06:11 PM. Reason: Remove obsolete beta versions

09-09-2013, 07:12 AM	#1824
Patricia1066 Zealot Posts: 131 Karma: 19894 Join Date: Mar 2010 Location: Bromley, UK Device: Kindle PW 10th gen, Kindle Fire 7 9th gen, Kindle KB 3rd gen	I installed fanfictiondownloader from the downloaded zip. It still shows 1.7.41 - am I doing something wrong? I deleted 1.7.41 then installed plugin from zip file. I tried to install a new story from ff.net and got the and got the following log. Thats when I realised that the plugin had the same version number as the recent update. Status Title Author Comment URL Bad Unknown Unknown HTTP Error 403: Forbidden http://www.fanfiction.net/s/9250468/1/ Ignore this comment, I hadn't restarted calibre. Its downloading now. Last edited by Patricia1066; 09-09-2013 at 07:16 AM.

09-09-2013, 07:41 AM	#1825
Patricia1066 Zealot Posts: 131 Karma: 19894 Join Date: Mar 2010 Location: Bromley, UK Device: Kindle PW 10th gen, Kindle Fire 7 9th gen, Kindle KB 3rd gen	The plugin has not changed its version number after restart. The FFDL identified a valid URL, and started download. But no file was created. The log says its been created. But I cannot find it. Pastebin Paste ID: j8zRG0k3 Last edited by Patricia1066; 09-09-2013 at 07:43 AM.

09-09-2013, 08:20 AM	#1827
seabream Enthusiast Posts: 48 Karma: 1056 Join Date: Jul 2012 Location: Central Canada Device: Sony PRS-T1, RIM Playbook & Z10, Samsung Note, MacOSX	@BaconBuddy - I think that was the one Patricia1066 installed. Otherwise the version number question wouldn't make sense right? (ignore this comment if your comment at #1826 wasn't a reply to Patricia1066) @Patricia1066 - JimmXinu generally doesn't give plugin versions an official higher number unless they are official releases. Posts announcing those look like #1815 or #1808. And unlike #1822, don't contain an attached version. So the lack of change in version number doesn't mean that you're doing anything wrong. It's supposed to look that way. Can you post the log file from the one where it says it created a file that doesn't show up? Last edited by seabream; 09-09-2013 at 08:27 AM. Reason: typo correction, clarification

09-08-2013, 06:59 PM	#1816
riothamus Connoisseur Posts: 83 Karma: 100000 Join Date: Nov 2010 Device: None	Not sure if its a fluke or not, but I'm getting the 403 error with ff.net again. A few hours ago everything was working fine and I got several stories downloaded. Just tried to do an update and no go.

09-08-2013, 08:00 PM	#1818
RenoA Junior Member Posts: 5 Karma: 10 Join Date: Sep 2013 Device: Calibre	Actually, my FFDL was working before I updated it to 1.7.41. I think I had 1.7.39 before that, and it worked fine, then I updated and it stopped. Incidentally, I also updated Calibre from 1.1 to 1.2 at the same time. I tried the fix on the previous page, and it didn't work :/

09-08-2013, 08:05 PM	#1820
cryzed Evangelist Posts: 408 Karma: 1050547 Join Date: Mar 2011 Device: Kindle Oasis 2	It's definitely the blocked user-agent, I know because when changing it to something more common in the "base_adapter.py" file it works perfectly again.

09-09-2013, 03:05 AM	#1823
skb Evangelist Posts: 401 Karma: 1597305 Join Date: Mar 2010 Device: Ipod G4, MacOS 10.12, Calibre, Pocketbook Touch HD 3	I gave it a go and it works, thank you. It wasn't that much slower for me (but then I'm on wireless and my connection speed is ordinary). And I learned how to manually install a plugin! Go me!

09-09-2013, 08:08 AM	#1826
BaconButty Connoisseur Posts: 69 Karma: 520 Join Date: Jun 2012 Location: France Device: Kindle Paperwhite 5 & iPad Mini & Galaxy Tab10	The recurrent 403 error from Fanfiction.net is resolved after a manual installation of JimmXinu's latest zip file (see comment #1822). Thank you. No noticeable speed reduction here.

09-09-2013, 08:34 AM	#1829
Patricia1066 Zealot Posts: 131 Karma: 19894 Join Date: Mar 2010 Location: Bromley, UK Device: Kindle PW 10th gen, Kindle Fire 7 9th gen, Kindle KB 3rd gen	Stranger and stranger. One successful download, which has not shown up. One successful download which has shown up in the calibre library. Same author, same site but different stories (ff.net)

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] Count Pages	kiwidude	Plugins	1750	04-07-2024 01:20 AM
[GUI Plugin] Open With	kiwidude	Plugins	403	04-01-2024 08:39 AM
[GUI Plugin] Resize Cover	kiwidude	Plugins	95	03-16-2024 11:55 PM
[GUI Plugin] Find Duplicates	kiwidude	Plugins	1096	03-16-2024 11:28 PM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 12:27 PM

Advert

Advert