[GUI Plugin] FanFictionDownLoader

[cryzed]

See if the attached version works for you. I'm not sure if the age verification with the ID card was required before, but you might have to do that -- If you haven't verified your age on the site, downloading adult stories will only work between 23:00 and 04:00 from what I understood (Personally I was not about to give them my personal identification just for verifying my age and found a solution via Google.).

[ILB]

Originally Posted by cryzed

See if the attached version works for you. I'm not sure if the age verification with the ID card was required before, but you might have to do that -- If you haven't verified your age on the site, downloading adult stories will only work between 23:00 and 04:00 from what I understood (Personally I was not about to give them my personal identification just for verifying my age and found a solution via Google.).

Thanks, that works for me.

[JOJO1]

Originally Posted by cryzed

See if the attached version works for you. I'm not sure if the age verification with the ID card was required before, but you might have to do that -- If you haven't verified your age on the site, downloading adult stories will only work between 23:00 and 04:00 from what I understood (Personally I was not about to give them my personal identification just for verifying my age and found a solution via Google.).

It works

[JimmXinu]

Attached is a version with cryzed's changes and also adds changes for potterfics.com(Spanish) to login for higher rated stories.

[Depletion]

Fimfiction just had another massive backend update and currently all stories are failing.

http://www.fimfiction.net/story/1951...the-folk-style
http://www.fimfiction.net/story/224796/barren


Jimm, I also searched a few pages back to see when fimfiction did their last update and noticed post 3386 still had a link to a test version, if you wanted to remove it.

[FaceDeer]

Looks like the Fimfiction api URL no longer works, which FFDL uses for getting some of the metadata for a story. I'll ask Knighty if it's just changed location, should be a simple fix if that's all that happened.

If the API is gone entirely the metadata will have to be parsed out of the story's human-facing page instead, which is a slightly more complicated thing to do but not particularly unusual. Shouldn't be too hard to fix in that case either.

[JimmXinu]

Attached is a new test version with fixes by FaceDeer for the latest FimFiction.net changes.

Please try it out and let us know if you have any problems.

[FaceDeer]

I added a bit of new metadata to the Fimfiction adapter while I was at it; comment count and cover art source URL (for those stories where a source for the art was provided). I love metadata. So meta.

[megme]

Hi Jimm,
FYI
The writer's coffee shop library story urls have changed from http://www.thewriterscoffeeshop.com/...y.php?sid=#### to http://www.twcslibrary.net/viewstory.php?sid=####

The old url is still working but I don't know for how long.

[JimmXinu]

Attached test version contains previous changes plus changes for transition from thewriterscoffeeshop.com to twcslibrary.net.

[BaconButty]

Originally Posted by JimmXinu

Attached test version contains previous changes plus changes for transition from thewriterscoffeeshop.com to twcslibrary.net.

Thanks for the quick action on this. It works well.

A reminder: the personal.ini file will need updating to ensure a permanent login permission to the new website.
I updated mine simply by changing the old TWCS url to the new one.

Thanks again

[megme]

Originally Posted by JimmXinu

Attached test version contains previous changes plus changes for transition from thewriterscoffeeshop.com to twcslibrary.net.

Thank you Jimm, it's perfect

Originally Posted by BaconButty

Thanks for the quick action on this. It works well.

A reminder: the personal.ini file will need updating to ensure a permanent login permission to the new website.
I updated mine simply by changing the old TWCS url to the new one.

Thanks again

Thanks for the reminder

[watani]

While the plugin correctly appends "http:" to the cover image source for FimFiction stories, it isn't doing so for the rest of the images in-line with the text anymore. This means that while the epub can be viewed without problems, converting it to any other format causes calibre to get stuck trying to find an invalid URL for every image (even though the images are cached in the epub).

Here's one story which has a significant number of in-story pictures that illustrates the problem:
http://www.fimfiction.net/story/1218...amlp-crossover

Also, I can only make edits once to the epub generated by the plugin for the story linked above before I can't access file0001.xhtml within the epub. After making edits once to the epub and saving, further attempts to open that particular xhtml results in calibre-parallel.exe (and therefore the book editor) crashing. The remaining contents of the epub (including the other xhtml files) can be opened in the book editor without issue. This problem is not fixed by restarting calibre and the only solution seems to be deleting and re-downloading the story using the plugin.

I haven't tested to see if other stories have this editing issue so I don't know if this is a problem with calibre or with the plugin. I am using calibre 2.7.0 [64bit] for Windows, if that helps.

[FaceDeer]

Hrm. I just dug up an illustrated Fimfiction story I downloaded in December of 2012, http://www.fimfiction.net/story/6007/, and the illustration in chapter 26 has the form

<img alt="" longdesc="http://imageshack.us/a/img109/224/0102d.png" src="../Images/ffdl-7.png" />

But I just had a look at the story you linked downloaded using the most recent working test version that does Fimfiction, and it had the form

<img alt="" class="user_image" longdesc="//fc04.deviantart.net/fs71/f/2014/294/4/4/sneak_attack_by_faith_wolff-d83prhl.png" src="../Images/ffdl-30.png" />

I'd be really quite surprised that the longdesc attribute has any actual effect on anything at all, though. According to the HTML5 specs it's meant to provide a link to a page containing a long description of the image, it shouldn't be relevant to actually displaying the image. The src attribute is for that. So while it does look like it broke at some point, I don't know why it would cause the editor to crash (I can imagine it confusing the converter, though).

I tried making an edit and saving the epub using Sigil, my usual epub editor, and a subsequent attempt to edit the file worked fine. No corruption. I then tried using Calibre's native epub editor and sure enough, the darned thing crashed the second time I opened the file I'd edited with it. It looks like there's some sort of bug with Calibre's native editor that's to blame for that. I reopened the "corrupted" epub in Sigil again and it worked fine, and the image tag was completely unchanged - whatever's causing the crash is somewhere else in this file. I edited and re-saved the "corrupted" chapter file with Sigil, and that fixed things - Calibre's editor was able to edit again. So as an interim measure I suggest downloading Sigil and using that, it seems to handle things correctly and fix whatever Calibre's editor is screwing up.

In short, it looks like this may be a bug with Calibre's epub editor, not FFDL's fault. Though you are correct that the longdesc is no longer in the correct form any more, that probably also needs fixing even though I doubt it's the root cause of this crash. I'll do some further comparisons and see if I can find exactly what Calibre's editor is doing to mangle these files.

Edit: Aha! I just looked in the release notes of version 2.8 of Calibre, which just came out this evening, and one of the bugfixes is:

Edit Book: Fix links that are absolute paths causing crashes in windows

Try updating to 2.8 and see if that fixes your crashing problem. Perhaps the editor was changing some links from relative to absolute paths when it saved the file the first time. I'll try that out after I've done those comparisons.

Edit 2: unfortunately, updating to 2.8 didn't fix the crash for me. And I can't seem to reproduce Sigil "fixing" the epub so that Calibre's editor can successfully load it again, either. Sigh.

It appears that the epub's internal structure when FFDL downloads it is rather different from what it gets changed into when edited by Calibre. In particular, the content.opf and toc.ncx files get moved from the root of the epub down into the OEBPS directory, and the content.opf file gets extensively rewritten to include a bunch of Calibre-specific metadata. This is getting too complicated for me to troubleshoot now, I'm not super familiar with the inner workings of these files or how Calibre handles it. I still blame Calibre's editor for the crashes, though. I just did a test where I took a "clean" original download of the epub and used Sigil to fix all the longdesc attributes, and Calibre's editor still wound up crashing when I tried editing it. So maybe prepare a bug report about the editor's problem to Kovid himself, providing him with a copy of a freshly-downloaded epub so he can see what it's doing when edited by Calibre's editor for the first time, and in the meantime use Sigil for epub editing to avoid the crashes.

There's a Calibre plugin called "Open With" that might make Sigil easier to use, you can use it to open an ebook in any other program you want rather than just the default Calibre viewer or editor.

[watani]

Thanks FaceDeer for the info! Yeah, I was kind of expecting that calibre was the source issue of the crashes, so I'll try to use Sigil for now for editing books.

Actually, the reason I brought up the incorrect image url issue is not because of the crashes, but because I noticed that when I tried to convert the downloaded epub to another format (azw3) it was taking longer than expected. I saw in the job details that this was because during the conversion process calibre kept on trying to pull images from the invalid url and only moved on after a lot of failed attempts. I think the broken url is making calibre think it has to pull the image from another source during conversion even though it's cached in the epub and the epub can be viewed perfectly fine with the images.

Since I use a Kindle, I can't use the epub that the plugin creates and thus would appreciate if the plugin were updated so that the image sources have the "http:" appended allowing me to properly convert between formats.

Sorry for being confusing and putting two separate issues I found together in one comment.