[GUI Plugin] FanFictionDownLoader

[PandathePanda]

Originally Posted by Robin42069

i just downloaded and installed the new plugin and still same problem.its most likley a problem on my end as i said i got a crappy older computer and running windows xp and that messes up on me often sometimes so im not gonna cry over it to much. it still downloads most things. so not a huge problem

if it's just fanfiction, you could use ficsave.com as a way to bypass your problem

[Robin42069]

oddly enough its not fanfiction.net all together cuz im able to update and download stuff fine for the most part it just oddly fails sometimes on some stories. its why im not to upset by the failure. but thank you for the ficsave link.

[JimmXinu]

Originally Posted by thecookiemommma

I'm getting a fun error today on both FF.net and AO3. Here's what I copied from my download log.

That's a bug. There's a debug statement that refers to the wrong variable. In my usual set up, the variable is set by some unrelated above it, so it doesn't error.

Attached is a test version that should fix the problem.

[JimmXinu]

Originally Posted by Robin42069

maybe you can tell me if this is my computers fault due to it being crappy and old as dirt or if something else is going on. It updates some stories but then i get this message on others

I tried it with calibre 1.48 (portable) on Win8 and didn't get any error on initial download of that story.

I tried updating to see what would happen and I did get an error, but it was a different one dealing with unicode vs non-unicode and BeautifulSoup4.

The attached version works better--it explicitly tells BS4 to use html5lib while re-interpreting existing chapters during update. It should work fine, but unicode can be a funny thing, so I'm not suggesting everybody try this version yet--just the adventurous types who want to help test things.

I honestly no idea if this will solve Robin42069's original problem because I can't fathom how that error could have happened. 'Crappy old computer' is a possible answer.

[Ealdwulf]

Originally Posted by JimmXinu

I tried it with calibre 1.48 (portable) on Win8 and didn't get any error on initial download of that story.

I tried updating to see what would happen and I did get an error, but it was a different one dealing with unicode vs non-unicode and BeautifulSoup4.

The attached version works better--it explicitly tells BS4 to use html5lib while re-interpreting existing chapters during update. It should work fine, but unicode can be a funny thing, so I'm not suggesting everybody try this version yet--just the adventurous types who want to help test things.

I honestly no idea if this will solve Robin42069's original problem because I can't fathom how that error could have happened. 'Crappy old computer' is a possible answer.

unfortunately... crappy old computer is not the answer.... my system is only a couple of years old (i7-3820 3.60Ghz, 16gb RAM, Win 8.1 64-bit)

I have the same problem....

calibre, version 2.22.0
ERROR: Error Updating Metadata: <p>An error has occurred while FFDL was updating calibre's metadata for <a href='https://www.fanfiction.net/s/8914586/1/'>Harry Potter: Junior Inquisitor</a>.</p>The ebook has been updated, but the metadata has not.

Traceback (most recent call last):
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1267, in update_books_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1791, in update_metadata
UnboundLocalError: local variable 'line' referenced before assignment

Story Details:calibre_id:
     1544
updatedate:
     2015-03-28 15:16:10-05:00
pubdate:
     2013-01-16 15:53:21-06:00
site:
     www.fanfiction.net
comments:
     <p>Before the start of fifth year Dumbledore changes the plans. Unfortunately he didn't bother to inform Harry. At his trial, Harry realises that it is down to him to save his own skin. To do so his Slytherin side must come out to play, and once it's out it sticks around turning life at Hogwarts on its head. Warnings: EvilDumbles, SheepOrder/Staff, GoodGuysDontGetEverythingTheirWay</p>
status:
     Meta
icon:
     metadata.png
added:
     False
comment:
     Metadata collected.
author_sort:
     [u'sprinter1988']
series:
     
username:
     (was set, removed for security)
good:
     True
publisher:
     www.fanfiction.net
listorder:
     0
is_adult:
     False
all_metadata:
     numWords:
          171,682
     site:
          www.fanfiction.net
     status:
          In-Progress
     authorId:
          2936579
     numChapters:
          30
     seriesUrl:
          
     storyUrl:
          https://www.fanfiction.net/s/8914586/1/
     genre:
          Adventure, Drama
     series:
          
     category:
          Harry Potter
     dateCreated:
          2015-03-29 01:24:23
     siteabbrev:
          ffnet
     warnings:
          
     language:
          English
     authorHTML:
          <a class='authorlink' href='https://www.fanfiction.net/u/2936579/sprinter1988'>sprinter1988</a>
     description:
          Before the start of fifth year Dumbledore changes the plans. Unfortunately he didn't bother to inform Harry. At his trial, Harry realises that it is down to him to save his own skin. To do so his Slytherin side must come out to play, and once it's out it sticks around turning life at Hogwarts on its head. Warnings: EvilDumbles, SheepOrder/Staff, GoodGuysDontGetEverythingTheirWay
     favs:
          9,074
     authorUrl:
          https://www.fanfiction.net/u/2936579/sprinter1988
     extratags:
          FanFiction
     datePublished:
          2013-01-16 15:53:21
     version:
          FanFictionDownLoader v2.1.5
     seriesHTML:
          
     storyId:
          8914586
     lastupdate:
          Last Update Year/Month: 2015/03, Last Update: 2015/03/28
     ships:
          
     characters:
          Amelia B., Hannah A., Harry P., Susan B.
     title:
          Harry Potter: Junior Inquisitor
     output_css:
          
     langcode:
          en
     formatext:
          
     rating:
          T
     dateUpdated:
          2015-03-28 15:16:10
     formatname:
          
     follows:
          12,045
     author:
          sprinter1988
     reviews:
          6,580
password:
     
url:
     https://www.fanfiction.net/s/8914586/1/
begin:
     None
title:
     Harry Potter: Junior Inquisitor
timestamp:
     2015-03-29 01:24:23.221000-05:00
tags:
     ['FanFiction', u'Drama', u'Adventure', u'In-Progress']
author:
     [u'sprinter1988']
end:
     None

and not just with ff.net

calibre, version 2.22.0
ERROR: Error Updating Metadata: <p>An error has occurred while FFDL was updating calibre's metadata for <a href='http://www.siye.co.uk/siye/viewstory.php?sid=129928'>Homecoming</a>.</p>The ebook has been updated, but the metadata has not.

Traceback (most recent call last):
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1267, in update_books_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1791, in update_metadata
UnboundLocalError: local variable 'line' referenced before assignment

Story Details:calibre_id:
     3584
updatedate:
     2015-03-28 00:00:00-05:00
pubdate:
     2015-02-17 00:00:00-06:00
site:
     www.siye.co.uk
comments:
     <p>After four years working overseas for the shadowy Department M, a world-weary and dispirited Harry Potter returns to the land of his birth. He meets some old friends and makes some new ones, as he learns that much has changed since he left home. AU, a ‘Harry never went to Hogwarts’ story.</p>
status:
     Meta
icon:
     metadata.png
added:
     False
comment:
     Metadata collected.
author_sort:
     [u'Brennus']
series:
     
username:
     (was set, removed for security)
good:
     True
publisher:
     www.siye.co.uk
listorder:
     0
is_adult:
     False
all_metadata:
     numWords:
          58,802
     site:
          www.siye.co.uk
     status:
          In-Progress
     authorId:
          16184
     numChapters:
          9
     seriesUrl:
          
     storyUrl:
          http://www.siye.co.uk/siye/viewstory.php?sid=129928
     genre:
          Action/Adventure
     series:
          
     category:
          Alternate Universe, Harry Potter
     dateCreated:
          2015-03-29 01:27:38
     siteabbrev:
          siye
     warnings:
          Death, Extreme Language, Intimate Sexual Situations, Negative Alcohol Use, Violence
     language:
          
     authorHTML:
          <a class='authorlink' href='http://www.siye.co.uk/siye/viewuser.php?uid=16184'>Brennus</a>
     description:
          After four years working overseas for the shadowy Department M, a world-weary and dispirited Harry Potter returns to the land of his birth. He meets some old friends and makes some new ones, as he learns that much has changed since he left home. AU, a ‘Harry never went to Hogwarts’ story.
     authorUrl:
          http://www.siye.co.uk/siye/viewuser.php?uid=16184
     extratags:
          FanFiction
     datePublished:
          2015-02-17
     version:
          FanFictionDownLoader v2.1.5
     seriesHTML:
          
     storyId:
          129928
     lastupdate:
          Last Update Year/Month: 2015/03, Last Update: 2015/03/28
     ships:
          Harry Potter/Ginny Weasley
     characters:
          Ginny, Ginny Weasley, Harry, Harry Potter, Nymphadora Tonks
     title:
          Homecoming
     output_css:
          
     langcode:
          
     formatext:
          
     rating:
          R
     dateUpdated:
          2015-03-28
     formatname:
          
     author:
          Brennus
password:
     
url:
     http://www.siye.co.uk/siye/viewstory.php?sid=129928
begin:
     None
title:
     Homecoming
timestamp:
     2015-03-29 01:27:38.477000-05:00
tags:
     ['FanFiction', u'Action/Adventure', u'In-Progress']
author:
     [u'Brennus']
end:
     None

and

calibre, version 2.22.0
ERROR: Error Updating Metadata: <p>An error has occurred while FFDL was updating calibre's metadata for <a href='http://hpfanficarchive.com/stories/viewstory.php?sid=1399'>Witches of Westfield</a>.</p>The ebook has been updated, but the metadata has not.

Traceback (most recent call last):
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1267, in update_books_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1791, in update_metadata
UnboundLocalError: local variable 'line' referenced before assignment

Story Details:calibre_id:
     3586
updatedate:
     2015-03-24 00:00:00-05:00
pubdate:
     2015-02-24 00:00:00-06:00
site:
     hpfanficarchive.com
comments:
     <p>The war is over, Voldemort was defeated and the summer draws to a close as the new school term begins. Harry must decide what to do with his life. Will he return to school for his 7th year, be an Auror…or something else? Read along as Harry finally has control of his own life…or does he?</p><p></p>
status:
     Meta
icon:
     metadata.png
added:
     False
comment:
     Metadata collected.
author_sort:
     [u'EJ Daniels']
series:
     
username:
     (was set, removed for security)
good:
     True
publisher:
     hpfanficarchive.com
listorder:
     0
is_adult:
     False
all_metadata:
     numWords:
          30,812
     site:
          hpfanficarchive.com
     status:
          In-Progress
     authorId:
          9669
     numChapters:
          5
     seriesUrl:
          
     storyUrl:
          http://hpfanficarchive.com/stories/viewstory.php?sid=1399
     genre:
          Fluff, Friendship, Humor
     series:
          
     category:
          Harem/Multi pairing, Harry Potter
     dateCreated:
          2015-03-29 01:35:32
     siteabbrev:
          hpffa
     warnings:
          Adult Themes
     language:
          
     authorHTML:
          <a class='authorlink' href='http://hpfanficarchive.com/stories/viewuser.php?uid=9669'>EJ Daniels</a>
     description:
          <p>The war is over, Voldemort was defeated and the summer draws to a close as the new school term begins. Harry must decide what to do with his life. Will he return to school for his 7th year, be an Auror…or something else? Read along as Harry finally has control of his own life…or does he?</p> <br/>
     authorUrl:
          http://hpfanficarchive.com/stories/viewuser.php?uid=9669
     extratags:
          FanFiction
     datePublished:
          2015-02-24
     version:
          FanFictionDownLoader v2.1.5
     seriesHTML:
          
     storyId:
          1399
     lastupdate:
          Last Update Year/Month: 2015/03, Last Update: 2015/03/24
     ships:
          Harry/Multi
     characters:
          Daphne Greengrass, Harry James Potter, Hermione Granger, Luna Lovegood, Tracey Davis
     title:
          Witches of Westfield
     output_css:
          
     langcode:
          
     formatext:
          
     rating:
          NC-17 - No One 17 and Under Admitted
     dateUpdated:
          2015-03-24
     formatname:
          
     author:
          EJ Daniels
password:
     
url:
     http://hpfanficarchive.com/stories/viewstory.php?sid=1399
begin:
     None
title:
     Witches of Westfield
timestamp:
     2015-03-29 01:35:32.900000-05:00
tags:
     [u'Fluff', 'FanFiction', u'In-Progress', u'Friendship', u'Humor']
author:
     [u'EJ Daniels']
end:
     None

I have also notice that for some reason that some of my ff.net stories have somehow ended up with the cover being the ff.net cover image instead of what I have it set to use. not 100% certain when that started but the metadata update failure started after I updated to ffdl plugin 2.1.5

although I just copied those three stories to a different library and tried them... and it worked fine.

[Sunlite]

Originally Posted by Ealdwulf

unfortunately... crappy old computer is not the answer.... my system is only a couple of years old (i7-3820 3.60Ghz, 16gb RAM, Win 8.1 64-bit)

There are two issues discussed. Yours looks like the issue of thecookiemommma. For that issue JimmXinu attached a new version of the plug-in with a bugfix. You could test if that fixes your issue as well and give some feedback for JimmXinu.

The other issue from Robin42069 was attribute to the 'crappy old computer'.

[cryzed]

@Robin42069 I'm pretty sure your error occurs when fanfiction.net is unresponsive or a bit slow, it sometimes decides to deliver exactly half a page a few times in a row. Did you try downloading the story you mentioned a few times at different dates and times?

[JimmXinu]

Originally Posted by Ealdwulf

unfortunately... crappy old computer is not the answer...

I have the same problem....

  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1791, in update_metadata
UnboundLocalError: local variable 'line' referenced before assignment

I have also notice that for some reason that some of my ff.net stories have somehow ended up with the cover being the ff.net cover image instead of what I have it set to use. not 100% certain when that started but the metadata update failure started after I updated to ffdl plugin 2.1.5

although I just copied those three stories to a different library and tried them... and it worked fine.

As Sunlite pointed out, that problem has been reported and is already fixed in the test version posted here.

The cover being set or not is part of the same problem and it working in one library and not another makes sense because the error doesn't happen if you have any custom_columns_settings applied.

[thecookiemommma]

Originally Posted by JimmXinu

That's a bug. There's a debug statement that refers to the wrong variable. In my usual set up, the variable is set by some unrelated above it, so it doesn't error.

Attached is a test version that should fix the problem.

It does. Thank you so much. You put so much effort into this, and it's such a joy to use. Makes things so much easier.

[JimmXinu]

Originally Posted by thecookiemommma

It does. Thank you so much. You put so much effort into this, and it's such a joy to use. Makes things so much easier.

Reminder for everybody: Name Change Survey

If you haven't yet, remember to vote for one of the new names under consideration.

That is all.

[JimmXinu]

Test version with previous changes plus new date format to fix onedirectionfanfiction.com.

[aleyx]

It occurred to me only after submitting the survey, but it would have been nice to have a choice for a name more fiction-neutral. As in, FFDL can be used for web fiction in general, not just fanfiction.

Okay, so fanfiction makes the overwhelming majority of use cases, but still...

Ah well, too late for this round. Maybe for the next revamp ^_^

N.

[cryzed]

(I just want to note that my original suggestion was still Ficfare which would have been a bit more neutral, I am however also liking Fanficfare.)

[thatblondelady]

I've noticed a new issue in the current release version of FFDL. I can only update stories once per launch of calibre. I've replicated this several times.

Steps to reproduce:

1) Either update my anthology that has a change, OR update my 28 or so in progress stories.

2) Try to update the one I didn't in step 1.

3) I get a screen that looks like this, and NEVER changes. (I let it go for hours yesterday by accident)

https://www.evernote.com/shard/s10/s...mall&width=832

Quitting Calibre and repeating whatever action hung before seems to work to solve the problem, but I thought I would mention it in case it was some sort of problem for everyone. If it isn't, I'm not that bothered. I rarely perform two actions on the same launch of Calibre.

Edit 2: I should mention that the first fic on my in progress list is FFN, and the anthology is AO3, so I don't *think* it's a case of rate limiting. Not sure that's even relevant, but I thought I would toss it out there.

[thatblondelady]

It looks like the most recent test version (which I missed on my first read) solved the problem. Nothing to see here.