[GUI Plugin] FanFictionDownLoader

[iatheia]

Originally Posted by JimmXinu

The adapter for AO3 doesn't have support for user/pass yet. (I don't think AO3 had user/pass like that when it was implemented.)

I don't know that we'll be able add that right away because I don't know that we (the FFDL devs) can get AO3 accounts. The AO3 site just told me "You are currently number 25745 on our waiting list! At our current rate, you should receive an invitation on or around: December 19, 2012"

Ah, I have an account. I'll fix it later today when working on more adapters.

[JimmXinu]

Originally Posted by iatheia

Ah, I have an account. I'll fix it later today when working on more adapters.

Awesome.

[cryzed]

By the way, Jimm, I found the bug that prevents the fallback labels in base_writer.py (BaseStoryWriter.titleLabes) from being used when there is no title label specified in the configuration files.

Your call getConfigList(entry) in BaseStoryWriter.writeTitlePage always evaluates to true, because self.getConfig(key).split(',') in Configurable.getConfigList returns "['']" instead of "[]" even if empty due to your use of string.split, I added a simple if-statement checking if vlist equals [''], and in case it does set it to [] to circumvent that, you might prefer checking the length of the list and then the 0-index value.

Also, I'm not sure if this sounds useful to you (it certainly did to me though), a really easy way to implement being able to use HTML on the title page is this (line 20-21) in BaseStoryWriter.writeTitlePage; this way it is possible to relatively easy change the appearance of the title page, like so by modifying the "titlepage_entries" in the configuration file (I'm specifically talking about the line break between "Words" and "Summary").

[calvertvl]

Originally Posted by JimmXinu

Please try the attached version and confirm that it corrects the problem for you.

The power went out shortly after my last post, and I'm currently on a different machine a couple of hours away; based on the power company reports it may be Wednesday before I have power back (hopefully internet will be working as well).

I will try as soon as I can, and let you know.

[JimmXinu]

@cryzed--Thanks for pointing out the bug with fallback labels. It's a real corner case--you have to be practically trying to hit the fallback labels and it can only happen to CLI users who edit or delete defaults.ini, so thankfully it hasn't been biting many people.

The problem isn't really with getConfigList returning "['']", but with the fact BaseStoryWriter.writeTitlePage should have been checking for the existence of "<entry>_label" all along. It'll be fixed in the next version.

As for additional HTML included in titlepage_entries, OTOneH, you can accomplish the same thing by adding HTML to the label, like "description_label:<br/>Summary".

OTOtherH, simply outputting non-matching entries allows for some extra flexibility and readability, and will show typo'ed metadata entries that might otherwise escape notice.

So I'll go one better--rather than outputting anything in titlepage_entries that starts with '<', in the next version I'll output anything that isn't a valid metadata entry name. That way it will work with text format, too. Thanks for the suggestion.

[JimmXinu]

Originally Posted by calvertvl

The power went out shortly after my last post, and I'm currently on a different machine a couple of hours away; based on the power company reports it may be Wednesday before I have power back (hopefully internet will be working as well).

Ugh. You have bigger problems to worry about. I'm 98% sure I've fixed it, so it will be in the next release anyway.

[ILB]

If I try to download "http://www.fanfiktion.de/s/4a0b18160000c64e06514438" I get the following error message: 'NoneType' object has no attribute 'string'

debug output:
urlhttp://www.fanfiktion.de/s/4a0b18160000c64e06514438) beginNone) endNone)
FanFictionDownLoader v1.5.39
url:http://www.fanfiktion.de/s/4a0b18160000c64e06514438/1
Exception: {u'comment': u"'NoneType' object has no attribute 'string'", u'begin': None, u'good': False, u'end': None, u'author': u'Unknown', u'url': u'http://www.fanfiktion.de/s/4a0b18160000c64e06514438/1', u'author_sort': u'Unknown', u'calibre_id': None, u'added': False, u'title': u'Unknown'}:'NoneType' object has no attribute 'string'
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 308, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 462, in get_metadata_for_book
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.base_adapter", line 237, in getStoryMetadataOnly
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.adapter_fanfiktionde", line 95, in extractChapterUrlsAndMetadata
AttributeError: 'NoneType' object has no attribute 'string'


Is it possible related to the fact that I have to identify myself as adult on fanfiktion.de to read the story direct on fanfiktion.de?
To identify I need my personal ID card (not just to click on "Yes, I am adult" or similar)

[cryzed]

@JimmXinu, oh that might be, I'm not _that_ familiar with the whole architecture of the code, and good call on outputting all the non-valid entries. And yep, I only noticed the bug because I added parsing for the review count.

Also I feel that I should mention this: in FanFictionNetSiteAdapter.extractChapterUrlsAndMeta data at the very end you retrieve the chapter title, but this always includes the prefix, meaning: "X. XXXXX"; I did this to fix it: "title = title.split(None, 1)[1]". Related to this bug (I assume) I changed something in EpubWriter.writeStoryImpl around line 350 during the story cover generation:

PHP Code:

items.append(("file%04d"%i,
              "OEBPS/file%04d.xhtml"%i,
              "application/xhtml+xml",
              "%d. %s" % (index + 1, title))) 


Namely the last line so it includes the now missing prefix. I assume that this is usually not included so it's your call to make, but don't you think that having index numbers in front of the chapter titles in the table of contents might make navigating easier? Example on how this all looks. Notice the missing number prefix for chapter titles and the added numbers in the table of contents.

[iatheia]

Originally Posted by ILB

If I try to download "http://www.fanfiktion.de/s/4a0b18160000c64e06514438" I get the following error message: 'NoneType' object has no attribute 'string'

debug output:
urlhttp://www.fanfiktion.de/s/4a0b18160000c64e06514438) beginNone) endNone)
FanFictionDownLoader v1.5.39
url:http://www.fanfiktion.de/s/4a0b18160000c64e06514438/1
Exception: {u'comment': u"'NoneType' object has no attribute 'string'", u'begin': None, u'good': False, u'end': None, u'author': u'Unknown', u'url': u'http://www.fanfiktion.de/s/4a0b18160000c64e06514438/1', u'author_sort': u'Unknown', u'calibre_id': None, u'added': False, u'title': u'Unknown'}:'NoneType' object has no attribute 'string'
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 308, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 462, in get_metadata_for_book
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.base_adapter", line 237, in getStoryMetadataOnly
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.adapter_fanfiktionde", line 95, in extractChapterUrlsAndMetadata
AttributeError: 'NoneType' object has no attribute 'string'


Is it possible related to the fact that I have to identify myself as adult on fanfiktion.de to read the story direct on fanfiktion.de?
To identify I need my personal ID card (not just to click on "Yes, I am adult" or similar)

Yes, actually, that's exactly it.
I've added the login portion to the adapter which would be available in the next update. However, since I do not have a German ID, I cannot actually test if it ever gets to the story.

[JimmXinu]

Re: fanfiktion.de - I think we're going to have to have a beta version again this time. I should have it posted here with these changes later today.

[JimmXinu]

@cryzed - Essentially, you're asking for the leading chapter numbers to be removed from the chapter titles at the start of each chapter, but be kept (or even added) in the TOC.

It's not a bad idea, but it's more complex than just changing the ffnet adapter and epub writer.

First, there are 50+ sites, many of which include leading chapter numbers, many of which don't, and a number of which do sometimes--and sometimes switch back and forth in the same story. So removal must be smarter and should be done in the common code somewhere.

Second, putting the numbers back in the epub TOC is fine, but they need to be added to the explicit TOC page, too, since that's what mobi/html/txt use.

Third, not everyone's going to agree that the numbers should be removed or added. So I'm thinking three new configuration options (names subject to change):

• remove_leading_chapter_numbers: true/false
• number_toc_chapters: true/false
• number_chapter_titles: true/false

Opinions, everybody?

[cryzed]

Ah yeah, I feared something like that (then on and off existing chapter numbers). Your suggestion sounds great to me, but the code that detects and possibly removes the leading chapter numbers is probably going to be somewhat complex, I think...

One way would be writing a RegEx that checks if the first few characters are digits followed by a delimiter ('.', ' ', ':' ...) and if that is true for every chapter title in the list, remove this part. The last condition (true for every chapter title) is a precaution so no part of the actual title is accidentally removed... I'm sure there are some corner case where a chapter might be named "5. Night" instead of "5th Night" or something similiar, but if it is true for every chapter the chance that the chapter number is added by the site itself is higher. Or am I making things too complicated?

About adding the numbers to the other writers, yes, of course, I expected as much -- these simply weren't that relevant to me since I use your software mostly for downloading fanfiction.net stories and converting them to EPUB afterwards, I expected that you'd add them to the other writers if you thought it was a good idea.

Also: I'll be honest, I'm not sure if this is already implemented, but what about a custom:

• connection_timeout: integer
• connection_timeout_retries: integer(/string [unlimited]?)

Or something among those lines. I already know that you do something similar to that when you fetch the chapter text, but do you also do it during the update process when the metadata is checked? This way it would also allow the user to configure those options manually.

[JimmXinu]

New Beta version with:

• New Site: ncisfic.com
• New Site: national-library.net
• Added Support for www.fanfiktion.de and archiveofourown.org username and password.
• Set Updated Date to Published Date when not found (fix for recent FFNet changes.)
• Fix for fallback metadata labels, allow other text in titlepage_entries
• New 'Error Column' feature - On update/overwrite fail, save error msg to a custom column.

Note that www.fanfiktion.de doesn't just require you to say 'yes, I'm an adult'. To see adult content there, you have to have an official government German, Austrian or Swiss ID card or passport number. We (the developers) don't have any of those, so some one who does, please test it.

Note: As part of the archiveofourown.org fix, the configuration section for AO3 is now [archiveofourown.org], not [www.archiveofourown.org]. Please change your personal configuration accordingly, if you have customized your settings for archiveofourown.org.

As usual, thanks to Ida for the new sites.

[BaconButty]

Originally Posted by BaconButty

http://www.thewriterscoffeeshop.com/...y.php?sid=6543
...when I try to download (and I have tried all the options, including changing the source path), I get the following message:
Title unknown - Author Unknown - Comment 'NoneType' object has no attribute 'nextSibling"

Quick update on this: I have now updated the same without any problem.

Thanks for your quick response anyway

[iatheia]

Originally Posted by JimmXinu

Note that www.fanfiktion.de doesn't just require you to say 'yes, I'm an adult'. To see adult content there, you have to have an official government German, Austrian or Swiss ID card or passport number. We (the developers) don't have any of those, so some one who does, please test it.

Apparently you can use the website during the night without showing the papers. So it does appear to be working properly.