[GUI Plugin] FanFictionDownLoader

[JimmXinu]

Version 1.7.43 - 11 Sept 2013

• Fix dates for fimfiction.net
• Additional groups as site specific data for fimfiction.net.
• Add default_value_status (etc) feature.

[Sefiriot]

Hmm. calibre doesn't seem to see it on the updated list. Not out yet?

[JimmXinu]

Originally Posted by Sefiriot

Hmm. calibre doesn't seem to see it on the updated list. Not out yet?

calibre doesn't see plugin updates until the Index thread has been updated. That's a manual process handled by the forum moderators. Sometimes it takes a while, which is why I also announce it.

You can go into calibre's plugin list and manually install the plugin and you'll get the latest version even if it doesn't show a new version on the list.

[kovidgoyal]

Originally Posted by JimmXinu

calibre doesn't see plugin updates until the Index thread has been updated.

That's actually no longer true for calibre 1.x releases, all plugins are downloaded from http://plugins.calibre-ebook.com which mirrors this site and automatically scans for plugin updates every hour. So if your users are using newer calibre releases they should be notified of plugin updates within an hour of you updating the zip file in the first post.

[JimmXinu]

Originally Posted by kovidgoyal

That's actually no longer true for calibre 1.x releases, all plugins are downloaded from http://plugins.calibre-ebook.com which mirrors this site and automatically scans for plugin updates every hour. So if your users are using newer calibre releases they should be notified of plugin updates within an hour of you updating the zip file in the first post.

Good to know. Thanks.

[thecookiemommma]

Originally Posted by Sefiriot

Not all reader apps either (well, not all handle them gracefully)-- and if reading with e-ink, graphics might not be the greatest idea, but since I've never used an e-ink device, others will be better able to judge. Still, to hear that FFN are actually considering compilation-- that's amazing! I should note however that there are authors out there who are against having their fics available in one easy download by default from a site, because of plagiarism fears. I know a person who refused to join AO3 precisely because of the availability of downloads, for the above reasons.

I am personally slightly frustrated by this attitude, because with Copy Paste, and GrabMyBooks and several other means available to download and capture the text, it's not a sure sign that someone's not going to be able to access their work in a downloadable format.

I understand the fears. I've had some of my own fic ... not verbatim plagiarized, but poorly riffed (and I was pretty offended by it), but I still want as many readers as possible to read my stuff. But I, too have seen a couple authors who have switched sites like four times to avoid downloadable content.

[Sefiriot]

Originally Posted by thecookiemommma

I am personally slightly frustrated by this attitude, because with Copy Paste, and GrabMyBooks and several other means available to download and capture the text, it's not a sure sign that someone's not going to be able to access their work in a downloadable format.

I understand the fears. I've had some of my own fic ... not verbatim plagiarized, but poorly riffed (and I was pretty offended by it), but I still want as many readers as possible to read my stuff. But I, too have seen a couple authors who have switched sites like four times to avoid downloadable content.

Agreed. No predicting whether somewhere out there someone didn't save that embarassing fic you thought you'd managed to obliterate off the 'Net, really: I found I have a few of those taken down fics saved back in the early 2000's when I was going through the saved fic folders trying to import to calibre. I made an argument for why fear of easier duplication shouldn't be a reason to spurn more modern services than FFN, but well, no convincing cattle that won't drink the water from your trough!

[El_Pirato]

could you use this for the age?

import time
date = input('Date (mm/dd/yyyy): ')
try:
valid_date = time.strptime(date, '%m/%d/%Y')
except ValueError:
print('Invalid date!')

[JimmXinu]

Originally Posted by El_Pirato

could you use this for the age?

import time
date = input('Date (mm/dd/yyyy): ')
try:
valid_date = time.strptime(date, '%m/%d/%Y')
except ValueError:
print('Invalid date!')

If you're trying to suggest we support adult-fanfiction.org, we don't support that site because of the legal warning and requirements, not because of technical difficulty.

[El_Pirato]

Originally Posted by JimmXinu

If you're trying to suggest we support adult-fanfiction.org, we don't support that site because of the legal warning and requirements, not because of technical difficulty.

the legal warning I'm working on that but you know how to bypass the verification page?

[Firedancer885]

Jimm,

I just tried to add http://www.ncisfiction.com/story.php?stid=05113, but I got an error.

calibre, version 1.2.0
HIBA: Error Updating Metadata: <p>An error has occurred while FFDL was updating calibre's metadata for <a href='http://www.ncisfiction.net/chapters.php?stid=05113'>Paws for Christmas</a>.</p>The ebook has been updated, but the metadata has not.

Traceback (most recent call last):
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1113, in update_books_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1472, in update_metadata
  File "site-packages/calibre/db/legacy.py", line 646, in set_custom
  File "site-packages/calibre/db/cache.py", line 939, in set_field
  File "site-packages/calibre/db/write.py", line 501, in set_books
  File "site-packages/calibre/db/write.py", line 501, in <dictcomp>
  File "site-packages/calibre/db/write.py", line 88, in adapt_number
ValueError: invalid literal for int() with base 10: ''

Story Details:
comment:
     Download epub completed, 1 chapters.
status:
     Add
pubdate:
     2010-12-14 00:00:00+01:00
listorder:
     0
is_adult:
     False
added:
     True
end:
     None
author:
     [u'Kath']
comments:
     <p>Gibbs gets a hero's reward, no matter if he wants it or not.</p>
calibre_id:
     5594
series:
     
username:
     <was set, removed for security>
begin:
     None
good:
     True
tags:
     [u'FanFiction']
timestamp:
     2013-09-12 20:16:38.320922+02:00
outfile:
     /tmp/calibre_1.2.0_tmp_HOfFeZ/fanfictiondownloader_hB5Z1C/Paws for Christmas-Kath-pd7F8t.epub
updatedate:
     2010-12-14 00:00:00+01:00
all_metadata:
     rating:
          NC-17
     series:
          
     numWords:
          
     site:
          www.ncisfiction.net
     datePublished:
          2010-12-14
     authorHTML:
          <a class='authorlink' href='http://www.ncisfiction.net/profile.php?user_id=112761'>Kath</a>
     storyUrl:
          http://www.ncisfiction.net/chapters.php?stid=05113
     storyId:
          05113
     category:
          NCIS, Slash >> Gibbs/DiNozzo
     lastupdate:
          Last Update Year/Month: 2010/12, Last Update: 2010/12/14
     formatext:
          
     author:
          Kath
     ships:
          Gibbs/DiNozzo
     formatname:
          
     seriesHTML:
          
     version:
          FanFictionDownLoader v1.7.43
     dateUpdated:
          2010-12-14
     numChapters:
          1
     status:
          Completed
     langcode:
          
     description:
          <td>Gibbs gets a hero's reward, no matter if he wants it or not.</td>
     warnings:
          
     dateCreated:
          2013-09-12 20:16:38
     output_css:
          
     authorId:
          112761
     authorUrl:
          http://www.ncisfiction.net/profile.php?user_id=112761
     characters:
          Abby Sciuto, Anthony DiNozzo, Donald Mallard, Jenny Shephard, Kate Todd, Leroy Jethro Gibbs, Timothy McGee
     genre:
          Alternate Universe, First Time, Holiday, Romance
     language:
          
     seriesUrl:
          
     siteabbrev:
          ncisfn
     title:
          Paws for Christmas
     extratags:
          FanFiction
password:
     
icon:
     plus.png
publisher:
     www.ncisfiction.net
url:
     http://www.ncisfiction.net/chapters.php?stid=05113
author_sort:
     [u'Kath']
title:
     Paws for Christmas

[JimmXinu]

Originally Posted by Firedancer885

Jimm,

I just tried to add http://www.ncisfiction.com/story.php?stid=05113, but I got an error.

...
  File "site-packages/calibre/db/write.py", line 88, in adapt_number
ValueError: invalid literal for int() with base 10: ''
...
     numWords:
          
     site:
          www.ncisfiction.net
...

It's failing because there's no word count collected and it's failing when trying to set it to an empty string. I'd blame it on calibre 1.x, but 0.9 does the same thing. Apparently it changed on one end or the other somewhere along the way.

Attached is a version of the PI that skips setting int/float columns when the data is empty.

UPDATE Sep 14 2013 - Remove obsolete beta versions

[JimmXinu]

Originally Posted by JimmXinu

It's failing because there's no word count collected and it's failing when trying to set it to an empty string.

FYI, FFDL doesn't collect word count from ncisfiction.net because it's a nuisance to get.

Author list shows only series and stories not in series. And word count per story is only show on the series list page. And FFDL has no way to get the correct series list page from the story URL. I'm not interested in scanning all of an author's series pages to find the right one.

[Linwe]

I've recently been downloading a bunch of stories from fanfiction.portkey.org for friend (she doesn't believe me about the awesomeness that is this plugin) and she's come back to me with the complaint that several stories are missing chapters. For example, downloading this one http://fanfiction.portkey.org/story/18 results in chapters 1, 3, 5, 6 and 8 to be missing. The chapter headers are there but there's no chapter content whatsoever. There are no error messages that pop up during download.

Any ideas anyone?

[JimmXinu]

@Linwe - This is a problem that I've seen before with equally old stories, but on fictionalley.org. The HTML as uploaded by the author originally is flawed and confuses the HTML parser FFDL uses.

The attached version does some brute-force fixes that I don't think will break anything else.

UPDATE Sep 14 2013 - Remove obsolete beta versions