[GUI Plugin] FanFictionDownLoader

[scoopaz]

Hi everybody,

Kiwidude: Thanks for the info - I'll go ahead and get that fixed per your link.

Jim: Uninstalled the old plugin completely and installed the updated one - Success! The story downloaded successfully and opened up without any errors.

I suspect I will be abusing this plugin regularly


Lisa

[JimmXinu]

Originally Posted by scoopaz

Jim: Uninstalled the old plugin completely and installed the updated one - Success! The story downloaded successfully and opened up without any errors.

Excellent! Thanks for helping figure this out.

Jim

[JimmXinu]

Originally Posted by JimmXinu

It's a bug in Calibre, not the plugin.

There's a problem in the routine that creates new book entries with "Tags to apply when adding a book" set.

Turns out this was indeed my fault. I have a fix for it now.

I'll be releasing a new version (1.1.2), so it should be available for auto-update soon.

Jim

[JimmXinu]

Somebody on the user group requested an option to keep existing tags on updatemeta.

"Existing tags will be kept and any new tags added.
Completed and In-Progress tags will be still be updated, if known.
Last Updated tags will be updated if lastupdate in include_subject_tags."

I also fixed the Last Update tags feature.

[LMx]

Awesome plugin! thank you very much for your time doing this

my girlfriend is using it often and loves it

however she uses 2 other big websites that are still not supported by this plugin

is there any plans in the future to add support for

http://www.livejournal.com
and/or
http://archiveofourown.org/ ?


also, there were some links that weren't added to the library so i'm trying to help her
here is an example log:

Download FanFiction Book
FanFictionDownLoader v1.1.3
Logfile for book ID None (A Bag Full of Mysteries)
url: (http://www.fanfiction.net/s/6420141/1/) id: (None)
write to c:\users\______\appdata\local\temp\calibre_0.8.34_ tmp__p6hx7\fanfictiondownloader_ayilc3\new-None-dhpynm.epub
Exception: {u'comment': u"'utf8' codec can't decode byte 0xe7 in position 50: invalid continuation byte", u'publisher': u'www.fanfiction.net', u'good': False, u'pubdate': datetime.datetime(2010, 10, 23, 0, 0), u'author': u'tinybee', u'url': u'http://www.fanfiction.net/s/6420141/1/', u'timestamp': datetime.datetime(2012, 1, 12, 22, 15, 44, 429000), u'author_sort': u'tinybee', u'calibre_id': None, u'is_adult': False, u'comments': u'A series of Harry Potter/Crossover one shots. Pairings will be SLASH Harry/? from numerous crossovers. List of what I can do inside. NO LONGER TAKING PROMPTS.', u'username': 'NoneGiven', u'outfile': u'c:\\users\\______\\appdata\\local\\temp\\calibre _0.8.34_tmp__p6hx7\\fanfictiondownloader_ayilc3\\n ew-None-dhpynm.epub', u'added': False, u'title': u'A Bag Full of Mysteries', u'password': '', u'tags': [u'X-overs', u'Harry Potter', u'In-Progress', u'Romance', 'FanFiction', u'Drama'], u'icon': u'dialog_error.png'}:'utf8' codec can't decode byte 0xe7 in position 50: invalid continuation byte
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.jobs" , line 131, in do_download_for_worker
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.writers.base_writer", line 215, in writeStory
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.base_adapter", line 191, in getStory
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.adapter_fanfictionnet", line 208, in getChapterText
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.base_adapter", line 167, in _fetchUrl
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe7 in position 50: invalid continuation byte

if mobi format is selected it also gives an error, although a different "FailedToDownload: Error downloading Chapter: http://www.fanfiction.net/s/6420141/24/! Missing required element!"

txt and html formats were added without problems

is it a bug or a fanfiction.net problem? thanks in advance

[JimmXinu]

Originally Posted by LMx

Awesome plugin! thank you very much for your time doing this

Glad you like it.

Originally Posted by LMx

is there any plans in the future to add support for

http://www.livejournal.com
and/or
http://archiveofourown.org/ ?

livejournal isn't really a fanfic site. From what I remember of it, which admittedly isn't much, it's very free form with little common formatting and story demarcation.

Unless it's changed a lot, it's much more challenging than I, for one, am prepared to tackle for site I don't read myself.

However, if anyone else wants to write an adapter for it, we gladly welcome code contributions.

http://archiveofourown.org has it's own ebook(epub/mobi/etc) downloads. For actual content, they've done a better job than we have--theirs contain images, too. They haven't, however, done as good a job of including metadata for Calibre. I'll think about what I might be able to do there.

Originally Posted by LMx

http://www.fanfiction.net/s/6420141/24/!
txt and html formats were added without problems

is it a bug or a fanfiction.net problem? thanks in advance

This story downloaded fine for me in both epub and mobi. Unless you continue to have problems, I'm going to assume that it was an ffnet glitch that came and went.

Jim

[LMx]

thank you for the quick reply

indeed, it seems to have been a ffnet glitch as i tried it now again and had no problems

good luck with the improvement of the plugin

[davidfor]

Jim,

I'm not a fanfiction reader, but my wife is, so I had a play with your plugin. I tested it with a couple of the links posted here to see how it went and I am very impressed (and amused by the short Dr Who story).

One thing that occurred to me, do you keep the bookmarks that the Calibre viewer puts into the epub? One of the bookmarks is the current reading location, I can see that if the story is being updated with new chapters, if the bookmarks are lost it could get very annoying. If you haven't seen them, the bookmarks are in the file "calibre_bookmarks.txt" in the "META-INF" folder of the epub.

And I just had a look at the story again. On the title page, you have "Categoryq". Is that a typo?

David

[JimmXinu]

Originally Posted by davidfor

I'm not a fanfiction reader, but my wife is, so I had a play with your plugin.

Oddly, you're the third guy I've heard from with a similar comment about using it for an SO. My wife just shakes her head tolerantly at me when I talk about fanfic.

Originally Posted by davidfor

One thing that occurred to me, do you keep the bookmarks that the Calibre viewer puts into the epub? One of the bookmarks is the current reading location, I can see that if the story is being updated with new chapters, if the bookmarks are lost it could get very annoying. If you haven't seen them, the bookmarks are in the file "calibre_bookmarks.txt" in the "META-INF" folder of the epub.

I did not know it did that. I wonder why Calibre puts it there? It has some other, non-invasive method for mobis...

A quick test appears to show that yes, that file is discarded on epub update.

It shouldn't be terribly difficult to retain it when doing update downloads. Overwrites, of course, will still destroy it.

Originally Posted by davidfor

And I just had a look at the story again. On the title page, you have "Categoryq". Is that a typo?

Ah. Oops.

That's a little embarrassing actually--I changed that on purpose to make sure that the correct file was being used and must have forgot to change it back.

Jim

[JimmXinu]

Originally Posted by JimmXinu

It shouldn't be terribly difficult to retain it when doing update downloads. Overwrites, of course, will still destroy it.

Changes to retain calibre_bookmarks.txt through epub updates and fix Categoryq are in 1.1.4. It's uploaded and will start showing up in the automatic plugin list in a while.

Jim

[davidfor]

Originally Posted by JimmXinu

Oddly, you're the third guy I've heard from with a similar comment about using it for an SO. My wife just shakes her head tolerantly at me when I talk about fanfic.

My wife knows that headshake well.

I did not know it did that. I wonder why Calibre puts it there? It has some other, non-invasive method for mobis...

That's interesting. My understanding was that Calibre converted everything to epub to view it. I assumed that it didn't remember the current page if it did this. But, a quick test showed it did.

That's a little embarrassing actually--I changed that on purpose to make sure that the correct file was being used and must have forgot to change it back.

Ah, the deliberate change to help debug something. I know it well. I have had a user see this in my tests and then I spent hours explaining that it wasn't a real change and I was going to remove it.

[JimmXinu]

Originally Posted by davidfor

That's interesting. My understanding was that Calibre converted everything to epub to view it. I assumed that it didn't remember the current page if it did this. But, a quick test showed it did.

I've never really looked into how the book viewer part works. The full-up conversion process generally takes longer than the viewer starting up, though.

I've just glad you knew about the calibre_bookmarks.txt file. I might never have figured out that problem without it. Seems a pretty Evil thing for Calibre to do, quite honestly. I wonder where it remembers it's place in mobis or other formats?

[Jade Aislin]

I love this plugin. The only reason I use calibre is to sort the fanfic I read. I’ve barely scratched the surface of the stories I read at FFN and can't wait to try downloading from TTH.

I do have a question however. How do I get the characters to show up? I have been trying to figure out how to get the characters to show up in the title page, but they never do. I’d prefer the characters to show up as tags, but am unsure if it is possible.

I have characters listed in the titlepage_entries section. The FFN story I am using as a test does have two characters, but I do not see it showing up.

One of the urls I am using is: http://www.fanfiction.net/s/7661640/1/

When I tried putting other items like storyID or siteabbrev on the title page I did not have a problem. It is only the characters.

[JimmXinu]

Originally Posted by Jade Aislin

I do have a question however. How do I get the characters to show up? I have been trying to figure out how to get the characters to show up in the title page, but they never do. I’d prefer the characters to show up as tags, but am unsure if it is possible.

You're doing it right, it sounds like. And when there are characters they will show up as tags by default.

But we only have code for parsing and saving characters for some sites and ffnet isn't one of them. (Currently only FimFiction.net and SIYE do.)

It doesn't look too horrible to add character parsing to ffnet. I'll look into it.

Jim

[Jade Aislin]

You're doing it right, it sounds like. And when there are characters they will show up as tags by default.

But we only have code for parsing and saving characters for some sites and ffnet isn't one of them.

That's good to know. At least I know I'm not doing something wrong.

It doesn't look too horrible to add character parsing to ffnet. I'll look into it.

I hope you are able to get the characters from FFnet. I've been using a complex method for getting my stories into calibre.

1. use fanfictiondownloader to download story list sorted by character.
2. add stories to calibre.
3. manually input custom fields (character, story complete)
4. repeat steps 1-3 for new character
5. find duplicate stories.
6. Merge duplicate stories so that it shows both characters.
7. Repeat for any additional characters.

This is very time consuming. If you can get the characters in the tags, it would eliminate a lot of the work I have to do.


On another note, is it possible to change the font of the comments? I love that the summary is put into the comments, but have trouble reading it on my tablet. I have tried to bulk edit, but could not make any changes as there are no easy html tags to change. Increasing the font in the comments field while the story is downloaded would be a big help.