[GUI Plugin] FanFictionDownLoader

[JimmXinu]

Bad HTML in the story summary can break HTML parsing.

Luckily, I've seen the same problem before on another site and already had a handy fix.

I've updated the test version on the earlier post. Give it a try.

[chaley]

Originally Posted by JimmXinu

If that's the official solution, I can certainly live with it.

But it would have been a lot easier to find if the behavior were consistent, either one way or the other.

A simple custom column and a composite column with a simple template ("{#characters}, {#genre}"): both work without the refresh.

But a composite column with a template function ("{#genre:uppercase()}") doesn't work without the refresh.

I found an easy & efficient way to make composite columns refresh when "normal" columns are changed, satisfying your consistency goal. The change is in trunk source and will be released this week.

You should no longer need to do the refresh_ids before calling get_metadata. It doesn't hurt anything if you continue to call it.

[JimmXinu]

@chaley - Great, thanks!

[Firedancer885]

http://www.wraithbait.com/viewstory.php?sid=14182

Jimm, please check this story. FFDL only downloads the first 4 pages. It get's cut off right after the second "chapter indicator", the bold line "handbook". I think there is something wrong with the text formatting of that section.

There is a blockquote tag right where FFDL cuts off the story.

[Zauberlehrling!]

Originally Posted by JimmXinu

Bad HTML in the story summary can break HTML parsing.

Luckily, I've seen the same problem before on another site and already had a handy fix.

I've updated the test version on the earlier post. Give it a try.

I did and it worked. The error in my second screenshot happened after I included that updated version.

And here I came across something new (sorry if I am going on your nerves ). I tried to update my downloaded stories and one story came up with a "'NoneType'object has no attribute 'string'". See screenshot.

[Firedancer885]

Originally Posted by Zauberlehrling!

I did and it worked. The error in my second screenshot happened after I included that updated version.

And here I came across something new (sorry if I am going on your nerves ). I tried to update my downloaded stories and one story came up with a "'NoneType'object has no attribute 'string'". See screenshot.

That story is already complete. If you make a custom column titled something like complete or fertig or something similar and then use the "custom column" tab of the FFDL settings, you can make Calibre display if a story is completed or not. This way you only need to update the stories tagged "in-progress".

[Zauberlehrling!]

Originally Posted by Firedancer885

That story is already complete. If you make a custom column titled something like complete or fertig or something similar and then use the "custom column" tab of the FFDL settings, you can make Calibre display if a story is completed or not. This way you only need to update the stories tagged "in-progress".

That is exactly what I want, update stories "in-progress".
I will do this from home. Thanks for the idea.

[Firedancer885]

@ Zauberlehrling
You will have to do an "update metadata only" on all of your downloaded stories after you set up the column. Unfortunately, it's not automatic for stories already in your library. And it won't work on fanfiktion.de, because of the german tags.

[calvertvl]

@JimmXinu

That update you released after I had reported the FFnet date issue did resolve the problem. Thank you for the quick resolution on that one.

I did find another issue with FFnet today when running some updates. I'm running 1.5.43 and when trying to update an in-progress story that had been deleted, the error message is the following unfriendly statement:

'NoneType' object has no attribute '__getitem__'

It happens for any story that's missing, on the metadata fetch (for which the FFnet page says "Unable to locate story"). Here's a sample URL:
http://www.fanfiction.net/s/5901296/1/

I don't remember what the message was before, but I think it was something more obvious. This was on a fresh restart of calibre after making sure the latest version was installed.
ETA: Previous message was "Story does not exist:http://www.fanfiction.net/s/5901296/1/" (with the appropriate URL for the story in question; I used the URL for the above example).

Calibre debug is below:
FanFictionDownLoader v1.5.43
url:http://www.fanfiction.net/s/5901296/1/
=Trying newer chapter: http://www.fanfiction.net/s/5901296/2/
=======Found newer chapter: http://www.fanfiction.net/s/5901296/2/
Exception: {u'comment': u"'NoneType' object has no attribute '__getitem__'", u'begin': None, u'good': False, u'end': None, u'author': u'sheltie', u'url': u'http://www.fanfiction.net/s/5901296/1/', u'author_sort': u'sheltie', u'calibre_id': 499, u'added': False, u'title': u'Two Girls with Harry in the Middle'}:'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialo gs", line 308, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 462, in get_metadata_for_book
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.base_adapter", line 237, in getStoryMetadataOnly
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.adapter_fanfictionnet", line 122, in extractChapterUrlsAndMetadata
TypeError: 'NoneType' object has no attribute '__getitem__'
Fetched metadata for 1 of 1

[JimmXinu]

Originally Posted by calvertvl

It happens for any story that's missing, on the metadata fetch (for which the FFnet page says "Unable to locate story"). Here's a sample URL:
http://www.fanfiction.net/s/5901296/1/

I don't remember what the message was before, but I think it was something more obvious.

It was. They changed the text on the Story Not Found page. Fix in next version.

[JimmXinu]

Originally Posted by Firedancer885

And it won't work on fanfiktion.de, because of the german tags.

I didn't know that. Hmm.

Since we don't have German for the different metadata labels for fanfiktion.de, I think the fanfiktion.de adapter should be using 'Completed' and 'In Progress' for status, not 'Fertiggestellt' and 'In Arbeit'. Plus the precedent was already set in www.ficbook.net.

Changed in next version unless somebody talks me out of it.

[JimmXinu]

Originally Posted by Zauberlehrling!

And here I came across something new (sorry if I am going on your nerves ). I tried to update my downloaded stories and one story came up with a "'NoneType'object has no attribute 'string'". See screenshot.

Please include story URLs in problem reports.

[JimmXinu]

Originally Posted by Firedancer885

http://www.wraithbait.com/viewstory.php?sid=14182

Jimm, please check this story. FFDL only downloads the first 4 pages. It get's cut off right after the second "chapter indicator", the bold line "handbook". I think there is something wrong with the text formatting of that section.

There is a blockquote tag right where FFDL cuts off the story.

Almost certainly has something to do with weird (or flat out wrong) HTML from the original author. It's going to be a couple days before I have time to look at it in depth.

[ilovejedd]

Question, is there a debug option available for FFDL to save the original source HTML files in a user-defined directory (e.g. "/debug/SiteAbbrev/StoryID") as well as intermediate files? If not, would it be possible to add this option?

e.g.
/debug/SiteAbbrev/StoryID/input: downloaded source html (1 file per index/chapter)
/debug/SiteAbbrev/StoryID/parsed: parsed metadata in either txt, csv or xml format and clean html version of the story
/debug/SiteAbbrev/StoryID/output: final output

Thanks!

[Firedancer885]

Originally Posted by JimmXinu

Almost certainly has something to do with weird (or flat out wrong) HTML from the original author. It's going to be a couple days before I have time to look at it in depth.

Thanks.