[GUI Plugin] FanFictionDownLoader

[cryzed]

From what I can tell, the only way that the site allows authors to upload their content is in plain text files. These text files are later displayed within pre-tags on the actual story page to preserve their formatting, including all spaces, newlines etc. If this content wouldn't be contained within these tags (and later aren't since using pre-tags within ebooks would probably cause some readers to display it strangely), normal browsers would (and I'm assuming ebook readers too) collapse repeated spaces into a single space character. Since the only way to really format a plain text file is by using tabs or spaces excessively, I added the option to replace all regular spaces with "&nbsp" HTML entities, which are basically simply spaces that won't be automatically collapsed by browsers.

I might have to improve that somehow, as well as the simple newline replacement which should really wrap paragraphs into p-tags somehow to allow further customization via the CSS file -- that would also allow me to narrow down which space characters to replace instead of replacing all of them blindly (although a RegEx might do). I did something similar a bit better in one of my other projects with this utility function. But this works for now, it's really just an option to force the ebook reader to not collapse space characters (which I'm assuming most would do), if the user really desires this behavior.

[bobthebilder]

Originally Posted by cryzed

@bobthebilder
Attached is a test plugin version with an adapter for http://fictionmania.tv/. See the plugin-defaults.ini accessible via the personal.ini configuration menu within the Calibre plugin to see additional available options (for example additional metadata you want to display on the titlepage). Let me know how it works.

Cryzed, looks perfect. Thank you for this and know that it was much appreciated.

[cryzed]

Glad to hear it ! By the way, if you come across a story with multiple chapters on that site you should take a look at FanFictionDownLoader's anthology feature, available to you after installing the EpubMerge plugin in Calibre. This should allow you to merge seperate stories (or in this case chapters) into a single ebook.

There's unfortunately no guaranteed way to detect if a continuation of a story exists, since there's no real chapter support on the site, so this will have to do.

[rlh3]

The "Get Story URLs from web page" appears to be broken again on Literotica. The error message says that there are "no valid story URLs on the given page."

A random author's page is attached.

http://www.literotica.com/stories/me...ge=submissions

Thanks.

PS I was the one who brought this up back in December, but unfortunately work took me away from the home computer for long enough for me to forget about it, so I never got around to testing the fix that Jimm made.

[Tanjamuse]

@Cryzed

I've come something odd: Whenever it is Older Kids in my edited Rating it writes the following: Older Kidsids, the weird thing is that it is only on FF.net there is a problem.

replace_metadata:
rating=>M=>Adults Only
rating=>T=>Teen
rating=>K\+=>Older Kids - becomes Older Kidsids
rating=>K=>Kids

[cryzed]

Try changing the single "K" in the last line to "K$" .

[Tanjamuse]

That seemed to do the trick Thanks again

[cryzed]

You are welcome .

[AndersW]

I am using Windows 7 Professional. This is really weird. For me, in http://www.fimfiction.net/story/122954/ the "’" disappears if I force it to Windows-1252 encoding, but an apostrophe doesn't take it's place it just disappears. But the only way you see the "’" is when you force the encoding to Windows-1252 , the exact opposite of what I am getting.

We have figured out that it is an encoding problem. How do I fix it?

[davidfor]

Originally Posted by rlh3

The "Get Story URLs from web page" appears to be broken again on Literotica. The error message says that there are "no valid story URLs on the given page."

A random author's page is attached.

http://www.literotica.com/stories/me...ge=submissions

Thanks.

PS I was the one who brought this up back in December, but unfortunately work took me away from the home computer for long enough for me to forget about it, so I never got around to testing the fix that Jimm made.

It looks like the links on the author submission page do not include the "http:" in the links.

JimmXinu: Adding the following near the start of "adapters.getClassFor" will fix this.

    if fixedurl.startswith("//"):
        fixedurl = "http:%s"%url

I think it fits in here as the method is already fixing the URL. Otherwise, it could be done in "geturls.get_urls_from_html" before the call to "adapters.getAdapter".

[JimmXinu]

Originally Posted by davidfor

It looks like the links on the author submission page do not include the "http:" in the links.
...
I think it fits in here as the method is already fixing the URL. Otherwise, it could be done in "geturls.get_urls_from_html" before the call to "adapters.getAdapter".

Well, I've seen worse solutions, so why not.

Attached is a test version with all the recent changes, that change, and a fix to a problem in the literotica adapter that came to light with the internals changes from metadata splitting.

[JimmXinu]

Originally Posted by AndersW

We have figured out that it is an encoding problem. How do I fix it?

Well, that was a surprise.

I hauled out an old Win7 machine to try it and I did indeed get the same mis-coding issue. I haven't used that machine in a while, so it was running calibre 1.27. I updated to the current calibre (1.40), fully expecting that it would correct the problem. But it didn't. Nor did the stack of windows updates it needed.

What does seem to have fixed it, is changing some static strings that get combined with the story description in the fimf adapter to be explicitly unicode.

See if this version works better for you.

[rlh3]

Originally Posted by JimmXinu

Well, I've seen worse solutions, so why not.

Attached is a test version with all the recent changes, that change, and a fix to a problem in the literotica adapter that came to light with the internals changes from metadata splitting.

Seems to work fine. I tested it on several authors and there were no issues.

Thanks to davidfor and Jimm for the fast fix.

rlh3

[davidfor]

Originally Posted by JimmXinu

Well, I've seen worse solutions, so why not.

I think we've all seen worse. And written worse.

Originally Posted by rlh3

Seems to work fine. I tested it on several authors and there were no issues.

Thanks to davidfor and Jimm for the fast fix.

It's working here as well.

On a related note, the literotica pages have tags in the "keywords" of the page. You can see them in the source of the page, or Firefox will show them in the page info (right click on the page and select "View page info). At some time in the past I wrote some code to extract these and use them as tags. If anyone is interested, I can dig this out and add it. With an option to use or not use them.

[AndersW]

Originally Posted by JimmXinu

Well, that was a surprise.

I hauled out an old Win7 machine to try it and I did indeed get the same mis-coding issue. I haven't used that machine in a while, so it was running calibre 1.27. I updated to the current calibre (1.40), fully expecting that it would correct the problem. But it didn't. Nor did the stack of windows updates it needed.

What does seem to have fixed it, is changing some static strings that get combined with the story description in the fimf adapter to be explicitly unicode.

See if this version works better for you.

Works perfectly. Thank you.