[GUI Plugin] FanFictionDownLoader

[Ealdwulf]

I don't know if there is a maximum... but I did something like 400 links this morning and it went fine... other than the long time to process everything...

[JimmXinu]

Originally Posted by texasnightowl

Was wondering if there is a max number of links you should copy into it for downloading? I've been doing batches of 20-25 and it seems fine so far as I can tell?

There is no fixed limit. On my computer, if I starting trying to do more than ~50 at a time, it seems to slow down more than the increase in URLs would justify. But I have (for testing purposes) done batches of 100s.

FYI, to avoid wasting bandwidth and CPU on the fanfic sites, I strongly recommend that users don't use FFDL's 'update' or 'overwrite newer' features to check for updates. I believe that you should use the email notification or RSS feeds from the sites/stories you follow. Then when you know there's an update, use FFDL to get the update.

[texasnightowl]

Originally Posted by JimmXinu

New Version 1.5.31 - 09 Jun 2012
Fix for ashwinder.sycophanthex.com login.

The ashwinder fix for the reported issue worked great! No more issues with the login prompt.

However, I came across another couple stories I had a problem with. The comment on the failure is: 'NoneType' object has no attribute 'nextSibling'. Here is the link to one:
http://ashwinder.sycophanthex.com/vi...y.php?sid=7879

Finally, is www.thepetulantpoetess.com under consideration for support?

[BaconButty]

Error code when updating from fanfiction.net:

Download FanFiction Book
FanFictionDownLoader v1.5.3
Adding jobs for URLs:
http://www.fanfiction.net/s/7879421/1/
http://www.fanfiction.net/s/8092082/1/
http://www.fanfiction.net/s/7584064/1/

UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 58: invalid continuation byte

Edit:
the error seems to appear when using "Add new from URL(s)" and not when updating from "Update Existing List".
Edit:
Error seems intermittent, as another ff.net story updated without problem.
Can you explain what the code refers to, in case it's at my end...?

[JimmXinu]

Originally Posted by BaconButty

Error code when updating from fanfiction.net:

Download FanFiction Book
FanFictionDownLoader v1.5.3

You're 28 versions behind. Upgrade to the latest version and those work fine.

[BaconButty]

Originally Posted by JimmXinu

You're 28 versions behind. Upgrade to the latest version and those work fine.

Weird...
I was prompted to update this morning and I did...

The Calibre log says "lastet version installed" 1.5.31 released 09 June 2012, and I forced an update before posting.

Edit: This is from the job log itself:

Originally Posted by BaconButty

Download FanFiction Book
FanFictionDownLoader v1.5.3

[JimmXinu]

Did you restart calibre?

[BaconButty]

Originally Posted by JimmXinu

Did you restart calibre?

Yes, several times...

[BaconButty]

Download FanFiction Book
FanFictionDownLoader v1.5.31
Adding jobs for URLs:
http://www.fanfiction.net/s/8117294/1/
Logfile for book ID 996 (Beyond the Scope)
urlhttp://www.fanfiction.net/s/8117294/1/) id996)
=Trying newer chapter: http://www.fanfiction.net/s/8117294/24/
=======Found newer chapter: http://www.fanfiction.net/s/8117294/24/
no calibre_series
write to C:\Users\XXXXX\AppData\Local\Temp\calibre_0.8.55_t mp_ztp7b3\fanfictiondownloader_gitq4u\new-996-zdvdus.epub
Exception: {u'comment': u"'utf8' codec can't decode byte 0xe9 in position 58: invalid continuation byte", u'author_sort': u'Catastrophia', u'pubdate': datetime.datetime(2012, 5, 14, 0, 0, tzinfo=SafeLocalTimeZone()), u'series': None, u'is_adult': False, u'added': False, u'end': None, u'author': u'Catastrophia', u'comments': u"The job market is bad, and recent college grad Bella's dreams of being a teacher are crumbling away. Edward is a single father and doctor, struggling to raise his seven year old daughter by himself. Fate intervenes, changing them both forever. Drabble fic", u'calibre_id': 996, u'username': 'NoneGiven', u'begin': None, u'good': False, u'tags': [u'In-Progress', u'Bella', u'Romance', u'Edward', u'FanFiction', u'Twilight'], u'timestamp': datetime.datetime(2012, 6, 10, 17, 37, 52, 223000, tzinfo=SafeLocalTimeZone()), u'outfile': u'C:\\Users\\Cayrol\\AppData\\Local\\Temp\\calibre _0.8.55_tmp_ztp7b3\\fanfictiondownloader_gitq4u\\n ew-996-zdvdus.epub', u'password': '', u'icon': u'dialog_error.png', u'publisher': u'www.fanfiction.net', u'url': u'http://www.fanfiction.net/s/8117294/1/', u'all_metadata': {'rating': u'M', 'numWords': u'15,881', 'site': u'www.fanfiction.net', 'datePublished': u'2012-05-14', 'storyUrl': u'http://www.fanfiction.net/s/8117294/1/', 'storyId': u'8117294', 'category': u'Twilight', 'formatext': u'.epub', 'author': u'Catastrophia', 'formatname': u'epub', 'version': u'4.4', 'dateUpdated': u'2012-06-10', 'numChapters': '24', 'status': u'In-Progress', 'langcode': u'en', 'description': u"The job market is bad, and recent college grad Bella's dreams of being a teacher are crumbling away. Edward is a single father and doctor, struggling to raise his seven year old daughter by himself. Fate intervenes, changing them both forever. Drabble fic", 'dateCreated': u'2012-06-10 17:37:52', 'authorId': u'2262046', 'authorUrl': u'http://www.fanfiction.net/u/2262046/Catastrophia', 'characters': u'Bella, Edward', 'genre': u'Romance', 'language': u'English', 'title': u'Beyond the Scope', 'siteabbrev': u'ffnet', 'extratags': u'FanFiction'}, u'title': u'Beyond the Scope'}:'utf8' codec can't decode byte 0xe9 in position 58: invalid continuation byte
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.jobs" , line 140, in do_download_for_worker
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.writers.base_writer", line 217, in writeStory
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.base_adapter", line 207, in getStory
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.adapter_fanfictionnet", line 253, in getChapterText
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.base_adapter", line 178, in _fetchUrl
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 58: invalid continuation byte



Edit: story now updated without problem

[iatheia]

Originally Posted by texasnightowl

The ashwinder fix for the reported issue worked great! No more issues with the login prompt.

However, I came across another couple stories I had a problem with. The comment on the failure is: 'NoneType' object has no attribute 'nextSibling'. Here is the link to one:
http://ashwinder.sycophanthex.com/vi...y.php?sid=7879

Finally, is www.thepetulantpoetess.com under consideration for support?

Pushed the fix for the issue and added the site to the list of under consideration.

[JimmXinu]

Originally Posted by BaconButty

Download FanFiction Book
FanFictionDownLoader v1.5.31
Adding jobs for URLs:
http://www.fanfiction.net/s/8117294/1/
Logfile for book ID 996 (Beyond the Scope)
urlhttp://www.fanfiction.net/s/8117294/1/) id996)
=Trying newer chapter: http://www.fanfiction.net/s/8117294/24/
=======Found newer chapter: http://www.fanfiction.net/s/8117294/24/

There's been some past problems with entity encoding. If you run into "'utf8' codec can't decode" errors on an update, trying doing an Overwrite. That can fix some of these problems.

[BaconButty]

Originally Posted by JimmXinu

There's been some past problems with entity encoding. If you run into "'utf8' codec can't decode" errors on an update, trying doing an Overwrite. That can fix some of these problems.

OK, thanks for the tip.

[Jade Aislin]

Originally Posted by JimmXinu

You can avoid the copy/paste at least by just opening the Add New dialog and dragging each link from the browser onto it. I find that acceptable quick for my purposes.

Unfortunately, my computer doesn't let me drag and drop. Nor did the one before it. I think it has something to do with the fact that it is Windows 7, but I haven't figured out how to fix it. It'd be really nice now because I'm trying to get stories from archive of our own, which keeps timing out when I browse the pages, but I have to copy the url shortcut and than past it to excel.


On another note, I noticed a couple things when I tried downloading from WWOMB. First when I followed the link provided in the supported sites section, I was taken to a url that had /~peja/ and not /peja/ like in the adapter. The first one seems to be a mirror site. It might be helpful to list the alternative as a possible url to get the stories from.

Also, the word count is wrong. Although it shows what is shown on the site, it seems that the site is only counting the first chapter when it gives the word count. I downloaded a story that has fifteen chapters and the word count said it only had 2606 words, which was how big the first chapter was. I used the count plugin to confirm that the word count was bigger and got 35963. I'm not sure if there is any way to fix this. If there isn't I'll just run the count word plugin for these stories to get a more accurate word count.

[JimmXinu]

Originally Posted by iatheia

Pushed the fix for the issue and added the site to the list of under consideration.

@texasnightowl - Attached is a version with Ida's fix, please try it out.

[texasnightowl]

Originally Posted by JimmXinu

@texasnightowl - Attached is a version with Ida's fix, please try it out.

Done! Appeared to work great. All 4 of the stories I experienced the problem with downloaded fine. Thanks to you both!