[GUI Plugin] FanFictionDownLoader

[Linwe]

Ok then! This is why I'm not in programming... everything works good and fine now. I didn't remove the # which was apparently what I was doing wrong the first (several) times around.

Thank you for all the help!

[jessifanfic]

Originally Posted by iatheia

I attempted to do an adapter for National Library prior, though, I decided to leave it till later. But since you are here, would it be possible for you to include an actual link to the author's page and/or numerical things such as word count and the date in the header of a story? Because that was the biggest thing stopping me so far.

Both sites' (National-library.net and ncisfic.com) story pages have been updated to include the author page, chapter count and word count. Sorry it took so long, life has been crazy.

Let me know how that works for you!

[iatheia]

Originally Posted by jessifanfic

Both sites' (National-library.net and ncisfic.com) story pages have been updated to include the author page, chapter count and word count. Sorry it took so long, life has been crazy.

Let me know how that works for you!

Awesome! I'll get these done probably around Saturday-ish, then =)

[JimmXinu]

Version 1.5.39 - 28 Jun 2012
Another fix for fanfiction.net's latest changes--prevent 'Status: Complete' from appearing as a character.

As usual, there may be a delay before calibre notices the new version. You can get the latest by going in and telling calibre to install it again.

[ADevilSee]

Amazing work! Great plugin, Truly... Thanks!

Got the update from yesterday changes after that and another one stopped working.. Thanks for the quick work!

[Firedancer885]

Originally Posted by JimmXinu

Version 1.5.38 - 28 Jun 2012
New Site: www.fanfiktion.de

Tiny problem here. I have 16 stories from this site and when I tried to update them only 2 downloaded properly. The other 14 give a 'NoneType' object has no attribute 'string' error. Could you look into it? I'm using both the latest calibre and the latest plugin.
I put my username and password into personal.ini.

And then I went to the site and discovered, that my trying to batchdownload 16 stories caused the site to lock me out with a floodprotect message.

[JimmXinu]

Originally Posted by Firedancer885

Tiny problem here. I have 16 stories from this site and when I tried to update them only 2 downloaded properly. The other 14 give a 'NoneType' object has no attribute 'string' error. Could you look into it? I'm using both the latest calibre and the latest plugin.
I put my username and password into personal.ini.

Please, always, always, always include story URLs when reporting problems.

Originally Posted by Firedancer885

And then I went to the site and discovered, that my trying to batchdownload 16 stories caused the site to lock me out with a floodprotect message.

The site's FAQ (after auto-translation) says:

I get the message "Protect Flood" and I'm no longer on the page. What's up?

The flood protection is an automatic safety system that caused a temporary closure, when a large number of heavily polluting server requests sent by a user, for example, if a lot of search commands are sent consecutively. The closure is temporary and will be automatically canceled after at least two hours.

Perhaps a default slow_down_sleep_time for www.fanfiktion.de is in order. And a more specific error message for "Protect Flood"?

[calvertvl]

There still seems to be a small issue in the way fanfiction.net metadata is parsed for single-chapter stories; both in-progress and completed stories with one chapter (such as the one below) fail with an error:
http://www.fanfiction.net/s/5038877/1/
'NoneType' object has no attribute 'date'

If it is completed and not in the library, it will retrieve the story, but does not populate the Updated date.
ETA: I did not test in progress and not in the library.

[JimmXinu]

@calvertvl - Are you sure you have the latest version of the plugin? Does the 'About Plugin' box show v1.5.39? Because ffnet updated their format recently and it required FFDL to change.

The URL given and a completed one shot I found both work fine for me.

One-shot/single chapter stories on ffnet, do not generally have an updated date, only a published date. So there's nothing to populate.

If you still have problems, run calibre in debug mode by running from a command prompt window with 'calibre-debug -g' and then post the debug output from the command prompt window.

[iatheia]

Originally Posted by Firedancer885

Tiny problem here. I have 16 stories from this site and when I tried to update them only 2 downloaded properly. The other 14 give a 'NoneType' object has no attribute 'string' error. Could you look into it? I'm using both the latest calibre and the latest plugin.
I put my username and password into personal.ini.

And then I went to the site and discovered, that my trying to batchdownload 16 stories caused the site to lock me out with a floodprotect message.

Are there stories that are password protected? If yes, link would be appreciated.

[calvertvl]

Hi Jimm,

I thought I had the latest version and confirmed it (with a freshly-started instance of calibre); it still doesn't work for books in the library, except that updating the calibre metadata only works (logs for both of these below).

As regards the update date, in the past the plugin populated it for me; my primary concern was that the behavior changed.

As a test I did also try an 'overwrite always' for this story (which was successful), but I generally prefer to only overwrite updates in case I accidentally select a large fanfic that hasn't been updated.

Output from 'overwrite if newer' (failure):
URL retrieved 1 of 1
FanFictionDownLoader v1.5.39
url:http://www.fanfiction.net/s/5038877/1/
=Trying newer chapter: http://www.fanfiction.net/s/5038877/2/
update existing id:3607
Exception: {u'comment': u"'NoneType' object has no attribute 'date'", u'publisher': u'www.fanfiction.net', u'begin': None, u'good': False, u'end': None, u'pubdate': datetime.datetime(2009, 5, 4, 0, 0, tzinfo=SafeLocalTimeZone()), u'author': u'JE1991', u'url': u'http://www.fanfiction.net/s/5038877/1/', u'series': None, u'username': 'NoneGiven', u'author_sort': u'JE1991', u'all_metadata': {'rating': u'M', 'langcode': u'en', 'site': u'www.fanfiction.net', 'datePublished': u'2009-05-04', 'storyUrl': u'http://www.fanfiction.net/s/5038877/1/', 'storyId': u'5038877', 'category': u'Harry Potter', 'formatext': u'.epub', 'author': u'JE1991', 'formatname': u'epub', 'version': u'4.4', 'numChapters': '1', 'status': u'In-Progress', 'numWords': u'2,978', 'description': u'How will a hidden prophecy change the life of hermione granger and the one she loves?', 'dateCreated': u'2012-06-29 19:48:43', 'authorId': u'1837665', 'authorUrl': u'http://www.fanfiction.net/u/1837665/JE1991', 'characters': u'Harry P., Hermione G.', 'genre': u'Horror, Romance', 'language': u'English', 'title': u'Fallen Hero', 'siteabbrev': u'ffnet', 'extratags': u'FanFiction'}, u'comments': u'How will a hidden prophecy change the life of hermione granger and the one she loves?', u'is_adult': False, u'calibre_id': 3607, u'added': False, u'timestamp': None, u'title': u'Fallen Hero', u'password': '', u'tags': [u'FanFiction'], u'icon': u'plus.png'}:'NoneType' object has no attribute 'date'
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 308, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 582, in get_metadata_for_book
AttributeError: 'NoneType' object has no attribute 'date'
Fetched metadata for 1 of 1

Same URL, update metadata only (success):
URL retrieved 1 of 1
FanFictionDownLoader v1.5.39
url:http://www.fanfiction.net/s/5038877/1/
=Trying newer chapter: http://www.fanfiction.net/s/5038877/2/
update existing id:3607
Fetched metadata for 1 of 1
add/update Fallen Hero http://www.fanfiction.net/s/5038877/1/
No value for dateUpdated, skipping custom column(Content Updated) update.
Updated 1 of 1
all done, remove temp dir.

[JimmXinu]

@calvertvl - Okay, I understand what's happening here now.

First, you're doing an 'overwrite if newer', not update. That's why it's failing outright for you, but working for me--I generally assume 'update epub on new chapters' for existing stories unless told otherwise.

Second, you're right that it used to fill in updated all the time. The hidden field date_u that FFDL used to use for ffnet was always there, even if Updated didn't appear on the screen. FFDL didn't used to have to do anything special for it, and I thought I found an old example without Updated date in my library, so I assumed that it always worked that way. Clearly, I was wrong.

Please try the attached version and confirm that it corrects the problem for you.

[trianglekitty]

Hello! I'm very new to using FanFictionDownLoader, but I'm already seeing the huge value of it. In just a few days I've saved almost 2000 works.

I am, however, running into a small problem saving works from AO3. Attempting to save restricted stories returns the "nontype object has no attribute string' error. I actually did not realize this was happening for some time, which means I probably thought I had saved works that returned this error and were not actually saved.

Here's an example of a story that returns this error:

http://archiveofourown.org/works/342728/chapters/555799

I attempted to save my username/password in the personal.ini, thinking that if I were 'signed in' it would bypass the restriction, but this did not work. I'm probably missing something very simple.

[Firedancer885]

Originally Posted by JimmXinu

Please, always, always, always include story URLs when reporting problems.

Sorry, as I later discovered my first problem was a direct result of the Floodprotection lockout. After I restarted my router and was allowed back to the site, I managed to download the majority of the other stories without problems. The ones I couldn't weren't a plugin problem. (They had been deleted from the site.)

[JimmXinu]

Originally Posted by trianglekitty

I am, however, running into a small problem saving works from AO3. Attempting to save restricted stories returns the "nontype object has no attribute string' error. I actually did not realize this was happening for some time, which means I probably thought I had saved works that returned this error and were not actually saved.

Here's an example of a story that returns this error:

http://archiveofourown.org/works/342728/chapters/555799

I attempted to save my username/password in the personal.ini, thinking that if I were 'signed in' it would bypass the restriction, but this did not work. I'm probably missing something very simple.

The adapter for AO3 doesn't have support for user/pass yet. (I don't think AO3 had user/pass like that when it was implemented.)

I don't know that we'll be able add that right away because I don't know that we (the FFDL devs) can get AO3 accounts. The AO3 site just told me "You are currently number 25745 on our waiting list! At our current rate, you should receive an invitation on or around: December 19, 2012"