[GUI Plugin] FanFictionDownLoader

[cryzed]

It wasn't dropped, it's still there, I just checked in the code repository. Are you possibly using an outdated or a test version? What makes you say that this is the problem specifically?

[watani]

cryzed, the reason that I thought the fix was dropped is that I previously had a test version which worked, but upon updating to version 2.0.9 the "http:" is no longer being appended to the in-story images URLs. I even tested the most recent test version and found that the URLs are still not being made correctly.

I just deleted the plugin and reinstalled version 2.0.9 from the calibre user plugins page as a double check and sure enough, in a newly downloaded FimFiction story the image URLs are missing the "http:". Could you look into this for me?

[cryzed]

Are you trying to update an existing story? If that's the case it's possible that old chapters downloaded with an earlier version of the plugin won't be overwritten, you will need to completely overwrite it.

I downloaded a story using the link mentioned in this post, and did successfully download the images with the latest version of the plugin.

Again: What makes you so sure that this is the problem? Are the images simply not being displayed, or are you really opening up the e-book file and inspecting various HTML tags for their attributes? Are you maybe simply missing this setting in the plugin options dialogue?

[watani]

I have no problems downloading or viewing the epub with images. The issue has been that within the epub the image HTML tags are not properly made (i.e. longdesc="// instead of longdesc="http://).

The result is that trying to convert the epub to any other format will cause the calibre converter to try to reacquire each image using the broken URL for ~10-15 seconds before moving on to the next image after giving up (a significant problem for an image heavy ebook).

Edit: To answer your questions, yes I've tried deleting the story and redownloading it with the same HTML tag issues. The epub displays the images fine, it is only when you go into the e-book file that you see that the HTML tags are wrong. And yes, the setting in the plugin options dialogue is checked.

[JimmXinu]

Originally Posted by watani

I have no problems downloading or viewing the epub with images. The issue has been that within the epub the image HTML tags are not properly made (i.e. longdesc="// instead of longdesc="http://).
....

I'm testing with story URL http://www.fimfiction.net/story/224372/ due to not having any other FimF story with images in to test with.

After downloading, when I examine the contents of the EPUB html file, I do see longdesc missing the 'http:'. I was certain it was there the first time I tried it, so I'm not sure what happened there.

I frankly didn't pay much attention to the issue before since the other guys seemed to have it in hand. However, on review, I found there was a better way to implement the fix that will also work with other sites should it ever come up.

The attached version implements that changed fix.

FFDL is, in point of fact, abusing longdesc. longdesc was meant to link to a 'long description' page for an image. The attribute longdesc is deprecated and not supported by modern browsers. FFDL uses it to store the original image URL in the EPUB because it was the only way I found at the time to do so without causing errors.

FFDL uses the original URL so it knows when it does and does not need to download new additional images when there's an update and to reuse existing images when the same image is included in the story more than once.

[aleyx]

Originally Posted by JimmXinu

FFDL uses it to store the original image URL in the EPUB because it was the only way I found at the time to do so without causing errors.

I'm not sure about the HTML5 data-* attributes in EPUB; are they valid? A quick search doesn't make me optimistic, but I'm not a EPUB guru.
N.

[JimmXinu]

Originally Posted by aleyx

I'm not sure about the HTML5 data-* attributes in EPUB; are they valid? A quick search doesn't make me optimistic, but I'm not a EPUB guru.
N.

Hmmm. That's idea... [goes away and tries it]...

Epubcheck doesn't like data-origurl as an img attr. It reports:
attribute "data-origurl" not allowed here; expected attribute "dir", "height", "id", "ismap", "lang", "longdesc", "style", "title", "usemap", "width" or "xml:lang"

I'm not dead sure that epubcheck not liking it is a deal killer, but it does point out what the problem is:

EPUB was defined during that period of time between HTML4 and HTML5 when people were honestly expecting XHTML to become the next web standard. So EPUB specifies it's contents should be XHTML. XHTML is a subset of XML and considerably less forgiving that HTML4 or HTML5. In theory, one could add an xmlscheme to extend img to add another attribute. In practice, it would have to be tested on every book reader out there.

So while using longdesc like this is arguably a violation of standard, it is an attribute that is documented and expected to be a URL by the ereaders already in the field.

(And FYI, while longdesc is remove from HTML5, it is still valid in XHTML. Probably because, last I heard, the XHTML working groups had all closed.)

I appreciate the suggestion, but I think it's more practical to keep using longdesc right now.

[ILB]

Originally Posted by JimmXinu

Downloads fine for me, too.

That error suggests that your epub has been corrupted.

FFDL used to have a problem with removing epub if you can FFDL's auto-convert turned on and calibre's default convert output also set to epub. But that was fixed a while ago.

Try downloading as 'overwrite always' or deleting the book and downloading it fresh.

Thanks. I deleted and downloaded it new. Now it´s ok.

[ILB]

Now I´ve got another problem...
If I want do download https://www.fanfiction.net/s/844414/1/Perfect-Potter I get the error message: maximum recursion depth exceeded
Traceback (most recent call last)
Is this something in the story or do I something wrong?

[Firedancer885]

Originally Posted by ILB

Now I´ve got another problem...
If I want do download https://www.fanfiction.net/s/844414/1/Perfect-Potter I get the error message: maximum recursion depth exceeded
Traceback (most recent call last)
Is this something in the story or do I something wrong?

Not just you. I got the same error. "Maximum recursion depth exceeded calling a python object".

[fathertyme]

I'm seeing a weird freeze since the last calibre update.

I've had it happen on two different storiesonline.net stories.

I've restarted the computer, and calibre, and finally ran it in debug mode just to see what it says

one of the stories is: http://storiesonline.net/s/76278/chuck-and-lisa

---- start debug ----

calibre Debug log
calibre 2.10 isfrozen: True is64bit: False
Windows-8-6.2.9200 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '8', '6.2.9200')
Python 2.7.8
Windows: ('8', '6.2.9200', '', 'Multiprocessor Free')
Successfully initialized third party plugins: DeDRM && Fantastic Fiction Adults && FanFictionDownLoader && Barnes & Noble && Generate Cover && SmartEject && Count Pages && Fantastic Fiction && Quality Check && Open With
Starting up...
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.ffdl_plugin(158):Plugin FanFictionDownLoader macmenuhack file_path:C:\Users\fathe_000\AppData\Roaming\calib re\plugins\fanfictiondownloader_macmenuhack.txt
Started up in 76.47 seconds with 762 books
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.ffdl_plugin(734):FanFictionDownLoader v2.0.9
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.ffdl_plugin(770):url:http://storiesonline.net/s/76278
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.adapter_storiesonli nenet(115):URL: http://storiesonline.net/s/76278
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/s/76278:i
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.adapter_storiesonli nenet(97):Will now login to URL (http://storiesonline.net/login.php) as (---login---)
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: ---- login/pass ----
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/s/76278:i
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=0
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=10
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=20
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=30
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=40
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=50
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=60
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=70
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=80
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=90
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=100
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=110
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=120
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=130
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=140
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=150
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=160
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=170
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=180
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=190
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=200
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=210
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=220
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=230
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=240
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=250
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=260

at this point calibre was frozen with the "fetching metadata" window open.. skipping to the end of the log file, I see it kept going:

pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2640
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2650
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2660
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2670
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2680
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2690
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2700
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2710
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2720
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2730
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2740
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2750
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2760
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2770
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2780
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2790
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2800
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2810
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2820
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2830
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_pl ugin.fanficdownloader.adapters.base_adapter(290):# ####################################
pagecache MISS: http://storiesonline.net/a/Dual_Writer&skip=2840

this could be an effect from formatting changes on the website... skip doesn't seem to be used anymore, or at least is a redirect from the format which seems to be:

http://storiesonline.net/a/Dual_Writer
http://storiesonline.net/a/Dual_Writer/2
http://storiesonline.net/a/Dual_Writer/3
http://storiesonline.net/a/Dual_Writer/4
http://storiesonline.net/a/Dual_Writer/5

[firefoxxy]

Hello again!

First, thanks Jimm for the hinto to the hint where my error could be. (Fortunately I only had added a few new expressions. One of them was obviously wrong constructed.) And I hope you can again provide some help with the following:

I wanted to get tags like "a little bit fluff", or "angst and fluff" to become "fluff" but I do not now how to write that. I had used

category=>*.fluff=>fluff

which caused the previous error. I removed this line.

I also tried to get the plugin to rename certain other tags (like Hurt/Comfort into Hurt-Comfort, or "Frigga (Marvel)" to "Frigga") which doesn't work. Next, I noticed that the ship-tags like "Harry Potter/Severus Snape" are not converted to anymore, although it had been working. ((Edit: It does sometimes work, but sometimes not - I have yet to check which stories don't change the tag... Sorry))

I copied the part of my ini-file.

Spoiler:

[archiveofourown.org]
include_in_characters: characters, ships
sort_ships:true
replace_metadata:
 category=>Harry Potter.*=>Harry Potter Works
 category=>The Avengers.*=>The Avengers
 category=>Thor.*=>Thor
 category=>Hurt/Comfort=>Hurt-Comfort
 characters=>Loki.*=>Loki
 characters=>Frigga.*=>Frigga
 characters=>Odin.*=>Odin
 characters=>Thor.*=>Thor
 characters=>([^/]+)/([^/]+)/(.+)=>\1\,\2\,\3
 characters=>([^/]+)/(.+)=>\1\,\2
 characters=>Harry.*=>Harry Potter
 characters=>Severus.*=>Severus Snape
 #characters=>Harry.*=>Harry Potter&&category=>Harry Potter Works
 #characters=>Severus.*=>Severus Snape&&category=>Harry Potter Works
 ships=>Harry Potter/Severus Snape=>Harry/Severus
 ships=>Loki/Thor.*=>Loki/Thor
 ships=>Loki/Tony.*=>Loki/Tony

It seems that "category" is perhaps not what I should use for the "fluff-tag", or is it? And is there a way to include the warnings from AO3? Would I put it in here:

include_in_characters: characters, ships, warnings

I hope you could help me once again? I really appreciate your answers!

Have a nice weekend!

[JimmXinu]

Originally Posted by Firedancer885

Not just you. I got the same error. "Maximum recursion depth exceeded calling a python object".

I'm seeing that problem, too. But it's only with Chapter 27 of that story.

So I suspect there's something wrong in the HTML of that chapter. I will investigate further, but there are a couple other issues to look at this morning, too.

If anyone sees this problem with any other story/chapter, please report here.

[JimmXinu]

Originally Posted by fathertyme

I'm seeing a weird freeze since the last calibre update.

I've had it happen on two different storiesonline.net stories.

I've restarted the computer, and calibre, and finally ran it in debug mode just to see what it says
...
this could be an effect from formatting changes on the website... skip doesn't seem to be used anymore, or at least is a redirect from the format which seems to be:

Yes, this problem is caused by that change to the storiesonline.net site. It's broken until we figure out how to fix it.

[JimmXinu]

Originally Posted by JimmXinu

Yes, this problem is caused by that change to the storiesonline.net site. It's broken until we figure out how to fix it.

Here's a test version with a fix for storiesonline.net, courtesy of Jeff over on the FFDL mail group. This version also turns on page caching for storiesonline.net.