[GUI Plugin] FanFictionDownLoader

[Emmi]

Hi,
I just came by to let you know, that fanfiction.net appears to have broken down again. I get a

'NoneType' object has no attribute '__getitem__'

error message when trying to update existing or adding new stories. Other sites seem to be uneffected.

Thank you for your continued work,
Emmi

[JimmXinu]

Originally Posted by Emmi

Hi,
I just came by to let you know, that fanfiction.net appears to have broken down again.

It's working fine for me.

Do you have specific story URLs it's not working for?

Debug output (run from command line as calibre-debug -g) would also be helpful.

[Emmi]

It happens with all stories I've tried so far.
As for the debuglog:

calibre Debug log
calibre 1.21 [64bit]  isfrozen: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.5
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Starting up...
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(155):Plugin FanFictionDownLoader macmenuhack file_path:C:\Users\Emmi\AppData\Roaming\calibre\plugins\fanfictiondownloader_macmenuhack.txt
Started up in 22.34 seconds with 764 books
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(722):FanFictionDownLoader v1.8.9
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(757):url:https://www.fanfiction.net/s/8968909/1/
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.adapter_fanfictionnet(88):URL: http://www.fanfiction.net/s/8968909/1/
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.adapter_fanfictionnet(121):=Trying newer chapter: https://www.fanfiction.net/s/8968909/2/
FFDL:WARNING:calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.base_adapter(182):Caught an exception reading URL: https://www.fanfiction.net/s/8968909/2/  Exception <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>.
FFDL:WARNING:calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.base_adapter(182):Caught an exception reading URL: https://www.fanfiction.net/s/8968909/2/  Exception <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>.
FFDL:WARNING:calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.base_adapter(182):Caught an exception reading URL: https://www.fanfiction.net/s/8968909/2/  Exception <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>.
FFDL:WARNING:calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.base_adapter(182):Caught an exception reading URL: https://www.fanfiction.net/s/8968909/2/  Exception <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>.
FFDL:ERROR:calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.base_adapter(184):Giving up on https://www.fanfiction.net/s/8968909/2/
FFDL:ERROR:__init__.py(1182):<urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>
Traceback (most recent call last):
  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.base_adapter", line 179, in _fetchUrl
  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.base_adapter", line 168, in _fetchUrlRaw
  File "urllib2.py", line 404, in open
  File "urllib2.py", line 422, in _open
  File "urllib2.py", line 382, in _call_chain
  File "urllib2.py", line 1222, in https_open
  File "urllib2.py", line 1184, in do_open
URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>
FFDL:ERROR:calibre_plugins.fanfictiondownloader_plugin.dialogs(600):Exception: {u'comment': u"'NoneType' object has no attribute '__getitem__'", u'publisher': u'www.fanfiction.net', u'begin': None, u'good': False, u'end': None, u'author': [u'rightforlife'], u'url': u'https://www.fanfiction.net/s/8968909/1/', u'listorder': 0, u'author_sort': u'rightforlife', u'comments': u'', u'calibre_id': 872, u'added': False, u'title': u'A Wrinkle in Time'}:'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
  File "calibre_plugins.fanfictiondownloader_plugin.dialogs", line 590, in do_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 823, in prep_download_loop
  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.base_adapter", line 243, in getStoryMetadataOnly
  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.adapter_fanfictionnet", line 132, in extractChapterUrlsAndMetadata
TypeError: 'NoneType' object has no attribute '__getitem__'

[JimmXinu]

Originally Posted by Emmi

It happens with all stories I've tried so far.
As for the debuglog:

...
Exception <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>
...

That's an SSL error that indicates the response you're getting isn't even valid SSL, let alone HTTP or HTML.

Were you downloading/updating a large (more than 12) number of stories? Because we have seen ffnet throttling requests from users who make too many hits to their servers. (The value of 'too many' is not known.) But I thought they were doing it more gracefully.

Any other causes I try to speculate would only be guesses. In your place, I'd try making sure I could read the stories from a browser, rebooting, and trying from a different network.

[Robin42069]

ok i've tried to download this story http://storiesonline.net/s/69347 and i get this message
Status
Title
Author
Comment
URL
Error
The Earl's Man
FantasyLover
list index out of range
http://storiesonline.net/s/69347


i've had no problem downloading anything else really but this one story. can someone explain why it wont work.

heres the debug log as well to help you to tell me if its my problem or just a bug of some sort

[davidfor]

Originally Posted by Robin42069

ok i've tried to download this story http://storiesonline.net/s/69347 and i get this message
Status
Title
Author
Comment
URL
Error
The Earl's Man
FantasyLover
list index out of range
http://storiesonline.net/s/69347


i've had no problem downloading anything else really but this one story. can someone explain why it wont work.

heres the debug log as well to help you to tell me if its my problem or just a bug of some sort

It is an error in Chapter 12. There is an extra "</div>" just before the story text. If you open the page, you can see what is happening. The background colour for the story text and the margins are different to the other pages.

I'm not sure how to fix this. I'll have a think and see if I can come up with a solution. Of course, reporting it to the author or the site admin might get the page fixed so that the problem goes away

And for the problem reported earlier in the week, I have fix. But, I haven't had a chance to test it enough. Hopefully tonight.

[Robin42069]

well damn i didnt think i would get a reply back so soon. and you figured it out quick as well. thanks for the explanation as well. i see what you mean on chapter 12 with the margins and such. i'll drop a email to the site manager to see if they will fix it not sure if they would see the need or not but doesn't hurt to try though. but thank you for the help.

[davidfor]

Originally Posted by JimmXinu

davidfor wrote the adapter for that site, so I'd like to hear from him before doing a permanent solution.

I have attached a version that includes that content, but it might include other stuff that isn't wanted, too.

I have finally had a chance to finish this and test it. The page retrieval code has been rewritten. The problems it fixes include:

- Headings within the story where being stripped
- For multi-page stories, the second page and further were in a div inside the main story div. This worked but it looked wrong.
- Spans used for styles in the story were stripped along with the text they containted.
- Any br tags inside the story were also stripped.

This mainly meant some structure was being lost. The spans were probably the worst bit as if they were used, they tended to be used a lot.

I haven't made any attempt to fix the problem reported on the weekend where there was a "</div>" in the wrong place.

The finestories.com plugin probably should have the same changes, but it will be a few days before I can look at it.

[JimmXinu]

Here's a test version with davidfor's changes included.

UPDATE Feb 7, 2014 - Remove obsolete beta versions

[isubs111]

Hi,

Would it be possible to have a button made for your browser that would grab the current page address and start the download of fanfiction in Calibre without you having to go and copy and paste the address and add the download separately? The button could be saved onto your Favourites bar in Firefox for example. You could maybe also have all your options set and pre-saved so you won't have to go into Calibre to say 'yes' to add to library, etc.

Thanks & kind regards,

Nic

[TrulyInnovative]

Originally Posted by JimmXinu

Here's a test version with davidfor's changes included.

Thanks again to both you and davidfor. Based on a very brief test, this seems to solve the problems I had. The only minor nitpick I have is that there are no colors on the epub, but I don't even know if it's possible to include them. Besides, I mostly read on a black & white reader so that's really not a big issue.

[Robin42069]

well damn man. you fixed that one problem with the story margins and now it downloaded the story i wanted so thank you very much. great work!

[JimmXinu]

Originally Posted by isubs111

Would it be possible to have a button made for your browser that would grab the current page address and start the download of fanfiction in Calibre without you having to go and copy and paste the address and add the download separately? The button could be saved onto your Favourites bar in Firefox for example. You could maybe also have all your options set and pre-saved so you won't have to go into Calibre to say 'yes' to add to library, etc.

Yeah, that would be nice, wouldn't it. But it's not that easy.

Calibre uses the GUI thread as the lock on the database. So there's no way (that I know of) to properly invoke a plugin in the GUI, from outside the GUI.

You could make a bookmarklet that used the FFDL web service version and save it to calibre's auto-add directory. Or use a browser add-on to call the FFDL CLI (or plugin CLI) to download and save it to calibre's auto-add directory. But that wouldn't help with updates and couldn't call the other plugins that FFDL integrates with.

[davidfor]

Originally Posted by TrulyInnovative

Thanks again to both you and davidfor. Based on a very brief test, this seems to solve the problems I had. The only minor nitpick I have is that there are no colors on the epub, but I don't even know if it's possible to include them. Besides, I mostly read on a black & white reader so that's really not a big issue.

The colours are in the stylesheet. If you use the storiesonline stylesheet, it will show them. You can copy that into your personal.ini, but it then gets used for all sites.

[davidfor]

Originally Posted by Robin42069

well damn man. you fixed that one problem with the story margins and now it downloaded the story i wanted so thank you very much. great work!

Sorry, but it didn't. It just didn't show throw any errors. All the chapters that are in "Book III" are wrong. The first only has the second page. The others are blank. They all have the same error in the page source.

I haven't looked a how to handle this error yet.