[GUI Plugin] FanFicFare

[JimmXinu]

Originally Posted by DocBrown13

Possible bug, since downloading the latest test version I have been unable to download any stories from http://hpfanficarchive.com.

It just seems to 'hang' at the 'Downloading metadata for stories' part at the start without progressing at all no matter how long i wait and eventually I have to cancel out.

I'm not seeing any problems with the hpfanficarchive stories I've tried.

It could an issue that's specific to the stories you tried (please post story URLs when reporting a problem, even if you think it applies to all stories on a site).

It's also possible that it's due to some extra debug output that was left in that adapter. I've removed it and there will be a new test version today after I've checked a few more things.

[JimmXinu]

Originally Posted by edeniz

I have a problem with tthfanfic.org. It has an annoying habit of putting tags like "xy-Centered" and pairing and genre (such as Romance, Action/Adventure) into "category", so that I have these things in my fandom column after download. Naturally, I'd like to move them from there to "pairings" and "genre" and "character", but try as I might, I could not come up with a working solution. Any ideas on what to do?

TtH's category level system is inconsistent. For example, 'Twilight' is a level 1 category, but 'My Little Pony' is a level 2 under level 1 category 'Cartoons'. And then '*-Centered' and 'Pairing: *' can appear at different levels: 'Television > Roswell > Buffy-Centered' vs 'Stargate > Dawn-Centered > Pairing: Jonas Quinn'. And then there's categories such as 'Pairing: Other' and 'Multiple Pairings'.

Plus, when support for TtH was added, FFF (FFDL at the time) didn't collect characters and ships yet for any site.

I might be convinced to add code to the TtH adapter to use '*-Centered' and 'Pairing: *' to populate characters and ships instead of category. Does anyone see a problem with that?

Originally Posted by edeniz

Another question: In fanficfare, when I'm, for example, renaming a tag like "characters=>Harry P\." I can make it "characters=>Harry P\.=>Harry Potter&&category=>Harry Potter" to make sure that change occurs only in that fandom and nowhere else. So what I'm wondering is why something similar won't work if I were to use, say, genre or pairing instead of category. Like, "rename this tag only if genre is this". (I should add that this is not terribly important, just something I'd been wondering when it didn't work when I tried it out of curiosity on a link where it seemed like it could help. I've since decided to just use replace_metadata to split one list item into multiple list entries.)

What you describe should work, as long as you aren't refer to the same metadata entry both the replacement and conditional parts. Assuming you have the correct keywords, of course. While it's category and genre (singular) for historical reasons, it's characters and ships (plural).

Originally Posted by edeniz

Lastly: Apparently I've screwed up something in my personal.ini without realizing, and now I get tag duplications because I renamed several tags to one. Because I'm using calibre plugin, I'd always looked at the book details browser after the download to check if all is okay, so it's now only that I'm noticing the problem. I've been going spare trying to hunt down the source the last few days, but can't find it. The problem is somewhere in this part here:

[overrides]
...
keep_in_order_genre:true 
keep_in_order_freeformtags:true 
...

The keep_in_order_* feature also disables the dup removal in the current version. I'll add some code to make the lists unique in the next test version.

[JimmXinu]

(Yet another) New Test Version (direct link).

2015-12-11

- Remove 'like' from XenForoForum chapter links in index post.
- Update embedded html5lib from 0.99999 to 0.9999999.
- Remove some debug output (adapter_hpfanficarchivecom.py).
- Make metadata lists unique even when keep_in_order_.
- Change entities to actual utf8 chars in adapter_fanfictionjunkiesde.py.

[edeniz]

Originally Posted by JimmXinu

What you describe should work, as long as you aren't refer to the same metadata entry both the replacement and conditional parts. Assuming you have the correct keywords, of course. While it's category and genre (singular) for historical reasons, it's characters and ships (plural).

Well, that explains it then. I think that was exactly what I did. One example to what I was trying to do is that a story I wanted to download had the genre tag "Episode: xyz" plus the genre tag "Alternate Universe", and I thought I could maybe rename the tag "Alternate Universe" to "Alternate Universe - Canon Divergence". Another was a case where I tried to change a genre tag only when a certain character tag was present, and it didn't work either, so that's how I came to that conclusion. But it's likely that in the second case I simply forgot the -s in "characters" and that's why it didn't work. Thank you for the clarification.

Re tth: Personally, I think that would be great if it is doable.

Re keep_in_order: I've tried the test version, and so far it works great. Thank you.

[eschwartz]

Originally Posted by JimmXinu

- Update embedded html5lib from 0.99999 to 0.9999999.

Well, really that is very unfair to html5lib 0.999999

[DocBrown13]

Originally Posted by JimmXinu

I'm not seeing any problems with the hpfanficarchive stories I've tried.

It could an issue that's specific to the stories you tried (please post story URLs when reporting a problem, even if you think it applies to all stories on a site).

It's also possible that it's due to some extra debug output that was left in that adapter. I've removed it and there will be a new test version today after I've checked a few more things.

I have just downloaded the new test version and tried again and it all seems to be working fine now.

Many Thanks

[edeniz]

Problems with the new test version on AO3 with the html download: It deletes all italics from the main story body and adds them at the end. However, the same problem does not seem to happen in the epub download. Linked stories attached.

Example links:
http://archiveofourown.org/works/4506612
http://archiveofourown.org/works/2132694

[JimmXinu]

Originally Posted by eschwartz

Well, really that is very unfair to html5lib 0.999999

I wish I had something funny to say about it, but I just don't. 0.9x7.

[JimmXinu]

Originally Posted by edeniz

Problems with the new test version on AO3 with the html download: It deletes all italics from the main story body and adds them at the end. However, the same problem does not seem to happen in the epub download. Linked stories attached.

Example links:
http://archiveofourown.org/works/4506612
http://archiveofourown.org/works/2132694

I've run http://archiveofourown.org/works/2132694 with HTML output on the latest code in both plugin and CLI and I don't see that effect.

Your posted HTML has a bunch of extra <body> tags in it. Like in every paragraph. I'm not sure what might cause that.

I see you've customized output_css, have you also customized chapter_start and/or file_start?

[edeniz]

Originally Posted by JimmXinu

I've run http://archiveofourown.org/works/2132694 with HTML output on the latest code in both plugin and CLI and I don't see that effect.

Your posted HTML has a bunch of extra <body> tags in it. Like in every paragraph. I'm not sure what might cause that.

I see you've customized output_css, have you also customized chapter_start and/or file_start?

Yes, actually. Like this:

Spoiler:

# metadata
tocpage_start:
 <a name="TOCTOP"><h2>Table of Contents</h2>
 <p align="center">

# ${number}, ${index}("%04d"%${number}), ${chapter}
tocpage_entry:
 <a href="#section${index}">${chapter}</a><br />

# metadata
tocpage_end:
 </p>

## ${number}, ${index}("%04d"%${number}), ${chapter}
chapter_start:
 <a name="section${index}"><h2>${number}. ${chapter}</h2></a>

## ${number}, ${index}("%04d"%${number}), ${chapter}
chapter_end:


titlepage_start:
 <table class="full">

# ${id}, ${label}, ${value}
titlepage_entry:
 <tr><td width="200px"><b>${label}:</b></td><td>${value}</td></tr>
# metadata
titlepage_end:
 </table>

[JimmXinu]

Originally Posted by edeniz

Yes, actually. Like this:

Aha! Actually, it's a problem with replace_br_with_p. I don't use that regularly, so I tend to forget about it.

But it makes a lot more sense, since that was updated to use BS4.

New Test Version (direct link).

- Fix for replace_br_with_p with BS4.

[edeniz]

Originally Posted by JimmXinu

Aha! Actually, it's a problem with replace_br_with_p. I don't use that regularly, so I tend to forget about it.

But it makes a lot more sense, since that was updated to use BS4.

New Test Version (direct link).

- Fix for replace_br_with_p with BS4.

Thank you! That seems to have done the trick.

[edeniz]

Okay, so it looks like I was a bit premature. The problems the new test version has with AO3 and html are not over. Take a look at the story I attached. The link is this (sorry, it's a long one this time):

http://archiveofourown.org/works/62539 (The Water Grinds the Stone)

It made right-aligned text left-aligned, and somehow put <big> tags in there that made random words huge. (Looks like the problem is that the source text has the font tag "big".) It's unreadable.

To be clear, I don't see the same problem with its epub version. In that one it simply makes right-aligned text left-aligned.

Another example is this one:

http://archiveofourown.org/works/152837 (The Weight of Water)

In this one, there are no huge words, but the text style in the body is messed up, in both html and epub.

In case it helps, I've attached my personal.ini too.

[JimmXinu]

Originally Posted by edeniz

It made right-aligned text left-aligned, and somehow put <big> tags in there that made random words huge. (Looks like the problem is that the source text has the font tag "big".) It's unreadable.

Right aligned vs left is due to FFF removing 'align' attributes on all tags. The 'align' attr is deprecated in HTML5. However, I will add a keep_align_attr setting in the next test version you can turn on.

The 'big' issue is due to replace_br_with_p doing weird things with the weird way the author used <big> tags. (I think. I didn't see the exact problem you did, but I didn't download more than the first couple chapters.)

The author is using <big> as a block tag with <br> tags inside it. It's meant to be like <span>, an inline markup tag.

I could in theory update replace_br_with_p to treat <big> as a block tag, but would that cause problems for other works that use <big> differently? I didn't write that feature and don't use it, so I'm not 100% sure of the finer points. Plus, there's a reason it's in a module named 'htmlheuristics'--it's not going to be perfect.

The <big> tag, BTW, is unsupported (not just deprecated) in HTML5.

For now, I think the answer is to turn off replace_br_with_p before downloading those stories.

BTW, do you actually download epub and html formats separately? Because if it were me, I'd download epub (for the update capability) and convert to html.

[edeniz]

Re align: To be honest, I'd love a "keep_align_attr", if it's not too much of a bother for you. Thank you!

Re "big" tag: It was a surprise to me to see it in this author's stories and be made aware of it being problematic in this manner was unpleasant. Few stories I've seen have ever used such a tag, and all of them were on either LJ or DW. As for "other works that use <big> differently": Hmm. Good question. The only archives I have used to date are ffnet, ao3, tth, wraithbait (and of those I have not downloaded from wraithbait yet), and this is the first time I've seen this tag in action on any of those archives. Personally, I would say that if there isn't a way to get a clean and easy solution to this problem, and if there aren't substantially more people affected, let it be. It'll be a pity, but the epub version is readable, and that's plenty. (Out of curiosity, what about having the option of removing "big" altogether" like with align if it's unsupported anyway? Doable? Not worth the work it'd cause?)

What makes me wonder, though, is this: The author has used this tag in other stories, for example http://archiveofourown.org/works/7606

In it, she has the first letter of every other paragraph big, and yet, this does not pose a problem for the download, be it html or epub. Is it just a case of the tag not being used as often as in the previous example story (The Water Grinds The Stone)?

I'd like to provide more examples for comparison, but usually the stories I download are reasonably formatted (and if not, replace_br_with_p takes care of that wonderfully).

Re conversion of epub: I actually download WIP stories as epub only (because of the update capability you mentioned), and completed ones as html and epub both (because I like html much better - for reading on a desktop or laptop pc, my ereaders like epub best, and the calibre conversion to htmlz somehow never turns out just the way I like it so I gave it up. Maybe I'll try again over the coming holidays.)