[GUI Plugin] FanFictionDownLoader

[JimmXinu]

@crackshot91 - I have no idea what bbcode in story descriptions you're talking about. But then I don't read fimfiction.net, either.

Please include a story URL illustrating each of the issues you described.

[crackshot91]

Originally Posted by JimmXinu

@crackshot91 - I have no idea what bbcode in story descriptions you're talking about. But then I don't read fimfiction.net, either.

Please include a story URL illustrating each of the issues you described.

Of course. Here's a good example: http://www.fimfiction.net/story/23865/

Story descriptions on fimfiction.net use bbcode. So, when the discriptions are downloaded into calibre metadata, they need to be converted into html in order to display properly:

http://i.imgur.com/TL9zG.png

As for the character tags issue, I can't seem to get FFDL/Calibre to show the character tags from any fimfiction.net story. Reading this post, it seems I need to use something called the "titlepage_entries section" to enable character tags, but I can't seem to find it. Is it in the personal.ini? Something to do with custom columns? I am so hopeless.

I hope this provides the necessary info. If you need more, please let me know. I'll keep looking around to see if I can find a solution. Thanks!

[JimmXinu]

Well, that's funky.

I'll take a look at it, but I didn't write the adapter for fimfiction.net, one of the other devs did. Back in April they changed it to use an API that fimfic provides to fetch the information in JSON rather than scraping the HTML pages. I expect the provided data hasn't been converted for bbcode. Not sure what's happening with the characters yet.

[crackshot91]

Well, I appreciate your help, JimmXinu. I look forward to getting this resolved!

[JimmXinu]

Okay, I've found a small library for interpreting BBCode to HTML that I can use. You might not like the final effect, though.

The HTML from BBCode ends up in the ebook on the title page, but the plugin always strips HTML from Summary before populating the calibre comments field. So the BBCode essentially gets stripped out. It works that way for all sites because otherwise you end up with some really nasty HTML in the comments from some sites.

As for fimfic characters: somewhere along the way, fimfic stopped putting the character names on the little character icons. You used to be able to hover and see the name--apparently you're just supposed to know now. I've kludged together a way to derive them from the image file names, but it may not be perfect. Let me know if you see any that aren't correct.

Give this version a spin and see what you think.

[kovidgoyal]

@JimmXinu: Have you tried the sanitize_html_comments() function from library/comments.py?

[crackshot91]

Dude, awesome! The characters now appear in both the tags column and the characters custom column. To remove this redundancy, I simply created another custom column for "genre", so now everything looks good there. I have absolutely NO idea what the admins of fimfiction were smoking when they decided to remove the hover-text on the character tags. Madness.

But yeah, the bbcode problem seems tricky. It looks cleaner now, but there's no way to see links because, as you say, all the bbcode is gone. It may be better just to have the bbcode display as normal. I don't mind the missing italics/bolding/colored text, but the links can be important. It looks messy, but at least you can copy paste the link, I suppose. It's unfortunate, and I hope eventually a solution presents itself, but right now, I think the previous way was preferable.

But the character tags are working now! Thank you so much for your time. Really cool of you.

[JimmXinu]

Originally Posted by kovidgoyal

@JimmXinu: Have you tried the sanitize_html_comments() function from library/comments.py?

I haven't before, thanks for the suggestion.

[JimmXinu]

@crackshot91 - Here's a version that uses sanitize_comments_html from calibre.library.comments as Kovid suggested rather than stripping all HTML from the summary.

This version also changes the default value for keep_summary_html to true. (Previously that only effected the summary on the title page inside the ebook. I made a couple minor changes to the AO3, ficbook.net and ficwad.com adapters for better summary HTML, too.)

Hopefully, this one will do what we want.

[crackshot91]

@JimmXinu - Well this is odd... The newest version does work. It updates the summary and all the html appears to be working. Great job!

However, I'm getting this strange error when updating some stories. Some of them don't successfully update. When I look at the log, I see "Status: Bad" and "Comment: 'full_image'":



Here's the url for the one story that updated successfully:

http://www.fimfiction.net/story/22329/

And here's the 5 that didn't:

http://www.fimfiction.net/story/2356/
http://www.fimfiction.net/story/2365/
http://www.fimfiction.net/story/11451/
http://www.fimfiction.net/story/6195/
http://www.fimfiction.net/story/12258/

I'm not sure what's causing that error. I tried un-checking "Update EPUB cover", but I still get that error. I get this for the majority of the stories in my library.

I'm not sure what 'full_image' means. Does it have to do with the cover?

Wait! I just tried something: I un-checked "Include images in EPUBs" and it seems to work. I can update the metadata now. But what does this mean? None of the stories in that list have images in them except for the covers, and even so, I was updating the metadata only, so why does that error come up? I'm not overwriting or updating the epubs. Just the metadata. Weird...

So yeah, this new plugin works great if I disable "Include images in EPUBs". I have a few stories that include images throughout the story, so it'd be great if there was a way to have the summary updating work properly with that option, but you've helped out so much already.

EDIT: Oh, this has to do with keep_summary_html, doesn't it? At least, that's what the tooltip for the "Include images in EPUBs" options says. I don't know much about any of this. I added:

#include_images:true
#make_firstimage_cover:true

To the [epub] section of the personal.ini. I'm assuming this will serve a similar function, keeping images in the EPUB? I just took out "keep_summary_html:true".

EDIT #2: Okay, yeah. Images are no longer included in epubs. Darn. What's weird is you said that this new version changes the default value of "keep_summary_html" to "true", which is the value that "Include images in EPUBs" uses. I don't get it...

[JimmXinu]

This is why we do test versions.

I noticed that the fimfic API returned metadata included 'full_image' and 'image' URLs both for that first story you asked about. Since full_image was a link to a larger version, I assumed it would be better to use that for the cover image. But not all stories have 'full_image', apparently. So now the fimfic adapter falls back to 'image' if there's not 'full_image' given.

Unchecking "Include images in EPUBs" stops FFDL from trying to d/l the image at all, that's why that worked.

The "Include images in EPUBs" checkbox, as it says, is exactly the same as adding:

[epub]
include_images:true
keep_summary_html:true
make_firstimage_cover:true

...to the beginning of your personal.ini

The '#' at the start of an ini line comments it out. You have to remove that for them to actually work.

keep_summary_html:true does not include images. include_images does that. keep_summary_html just lets there be images in the summary, especially for make_firstimage_cover to find.

fimfic is one of only two sites to have official 'cover' images. The other 70+, if they have them, are either in the summary or the first chapter. That's why these options probably seem a bit weird to you.

This version should fix the full_image issue.

[crackshot91]

Gah! Dude, you rock! It's working perfectly now!! HTML displays properly in the summary, covers are updated properly, images display correctly in the epub (and look fine when converted to mobi), and the TOC and chapter marks are working wonderfully when converted from epub to mobi. Also, calibre now displays the character tags, word count, number of chapters, status of the story, and the age rating! This place is awesome!

Thank you!

[JimmXinu]

I'm glad that's sorted it out.

I'll give it a couple days for you, I and who ever else wants to try it, to see what else may crop up before I put those changes out for everyone.

[ilovejedd]

Originally Posted by JimmXinu

Third, not everyone's going to agree that the numbers should be removed or added. So I'm thinking three new configuration options (names subject to change):

• remove_leading_chapter_numbers: true/false
• number_toc_chapters: true/false
• number_chapter_titles: true/false

Question, was this every implemented in FFDL? If so, where do I insert these variables? Thanks!

[JimmXinu]

Originally Posted by ilovejedd

Question, was this every implemented in FFDL? If so, where do I insert these variables? Thanks!

No, it's still on the 'wishlist' of things I may do some day. As I recall, I looked into it once and found that it would require restructuring the code more than I wanted tackle that day. Especially as I'd only ever heard from one user asking for it. Now that it's two users, I'll revisit it sometime.