[GUI Plugin] FanFictionDownLoader

[JimmXinu]

Originally Posted by JimmXinu

FYI, Twisting the Hellmouth (one of my personal favorite fanfic sites) is in the process of moving servers and domains.

More FYI, looks like TtH is still going to be tthfanfic.org after all. It's mostly back up.

[kovidgoyal]

Originally Posted by JimmXinu

I don't know where calibre gets its initial cert store from, but it could be from the OS.

python's ssl module leaves certificate loading upto the OS. On most linux distributions, that means /etc/ssl/certs

On windows it uses the windows certificate stores.

[Amalthia]

Hi, I'm using the code

replace_br_with_p: false

on this story: http://archiveofourown.org/works/1740875 and it doesn't appear to be working.

Did I need a space before "replace_br_with_p: false" or was I supposed to change "false" to "true?"

Thanks,
Amalthia

[JimmXinu]

Originally Posted by Amalthia

Hi, I'm using the code

replace_br_with_p: false

on this story: http://archiveofourown.org/works/1740875 and it doesn't appear to be working.

Did I need a space before "replace_br_with_p: false" or was I supposed to change "false" to "true?"

The space doesn't matter. However, if your intention is to turn the feature on, yes, you should set it to true instead of false.

As for 'not working', you'll have to be more specific. That's a feature one of the other developers added. I don't use it, so I'm not terribly familiar with it.

[Amalthia]

Originally Posted by JimmXinu

The space doesn't matter. However, if your intention is to turn the feature on, yes, you should set it to true instead of false.

As for 'not working', you'll have to be more specific. That's a feature one of the other developers added. I don't use it, so I'm not terribly familiar with it.

I should have been clearer when I said it wasn't working, what I meant was when I viewed the HTML the <br> tags were still in the story, they were not replaced with <p> tags.

I tried again with "replace_br_with_p: true" and still the same results.

Thanks,
Amalthia

[JimmXinu]

Originally Posted by Amalthia

I should have been clearer when I said it wasn't working, what I meant was when I viewed the HTML the <br> tags were still in the story, they were not replaced with <p> tags.

I've dropped a note to the developer who added that feature asking them to come take a look.

[Feynor]

Originally Posted by Amalthia

I should have been clearer when I said it wasn't working, what I meant was when I viewed the HTML the <br> tags were still in the story, they were not replaced with <p> tags.

I tried again with "replace_br_with_p: true" and still the same results.

Thanks,
Amalthia

br to P tries to determine which number of breaks are to be seen as paragraph breaks, and are to be converted. Mostly that means two consecutive breaks are seen as a paragraph break, and all singles are removed, but if it then run into three or more, it really can't do anything but leave them be.

I'll have a look though.

[Feynor]

Addition, it's been a long time since I wrote that code, and IIRC it won't convert breaks inside a paragraph (<p>) tag.

I'll see if I can add a rule to ignore that rule. in cases where there are lot of the breaks.

Additional: (again)
It is as I suspected, the function shields break tags inside existing block tags (table, blockquote, p etc) as they usually mean that the text in question is pre-formatted with line breaks, rather than paragraph breaks. This could be for songs or poetry, or quotes inside a story.
The trick is to come up with a rule that'll allow me to determine if I should overrule this, and when, without breaking the cases the old rule was meant to shield.

I know where it fails, I just need to figure out how to best fix it, as it is really something AO3 is doing. However others might be doing the same thing.
Heuristics are hard!

P.S.: Jimm, I pushed a modified hpfanficarchive adapter, as I found that it left HTML comments in the story body, which would then become a garbled mess later, I don't know if that was something my br_to_p function did, but the result was that for instance Google Books wouldn't parse the output. So I simply remove all HTML comments. They serve no purpose in a story anyway.

[JimmXinu]

Originally Posted by Feynor

...
I know where it fails, I just need to figure out how to best fix it, as it is really something AO3 is doing. However others might be doing the same thing.
Heuristics are hard!

Are you ready for a test version of your heuristics code to be posted for others to try?

Originally Posted by Feynor

P.S.: Jimm, I pushed a modified hpfanficarchive adapter, as I found that it left HTML comments in the story body, which would then become a garbled mess later, I don't know if that was something my br_to_p function did, but the result was that for instance Google Books wouldn't parse the output. So I simply remove all HTML comments. They serve no purpose in a story anyway.

Please post a story url or two that showed the problem. I've seen some comments issues with other sites and came up with a different solution (using the BS4 parser). And I fear your comment replacement code is too aggressive. Won't it remove all of "<!-- comment --> story <!-- comment -->", not just the comments?

[Feynor]

The ? means the expression is lazy, and will match as few times as possible.
"<!-- comment --> story <!-- comment -->" will leave " story "

Any hpfanficarchive story will do, they all have the comments.
Here's the first story in latest:
http://hpfanficarchive.com/stories/v...=834&chapter=1
the body found by the adapter starts with <!-- STORY START --> and ends with <!-- STORY END -->

The heuristics code only really have one change, an if was changed to a while loop, to remove multiple layers of divs. AO3 uses 2.
That really should be changed to using beautiful soup for safety, as it might be a problem for this structure:
<div>
<div>something</div>
<div>Something else</div>
</div>

I didn't realize that till just now, so it might be a good idea to roll back that change for now..
Edit: I just reverted that change, so there is nothing to test in the heuristics for now. I have to figure out how to solve this.

[JimmXinu]

Originally Posted by Feynor

The ? means the expression is lazy, and will match as few times as possible.
"<!-- comment --> story <!-- comment -->" will leave " story "

I knew that. I just didn't see the '?'. My bad.

Originally Posted by Feynor

Any hpfanficarchive story will do, they all have the comments.
Here's the first story in latest:
http://hpfanficarchive.com/stories/v...=834&chapter=1
the body found by the adapter starts with <!-- STORY START --> and ends with <!-- STORY END -->
...

So wait, it's not that hpfanficarchive has bad comments, the problem is that Google Books doesn't like any comments?

Because if that's true, we should add an option to remove comments for all sites, not just hpfanficarchive.

[Feynor]

Google Books don't mind comments, but somehow the comments from hpfanficarchive becomes messed up, looking like "<!-- <!-- STORY START --> <p>--></p>" if I recall

The nesting of comments is what's illegal, regular browsers just tries to ignore it.

I only noticed it from hpfanficarcive, but it is a good idea to filter out them all, really, but it'll have to be done right after the adapter returns then, before any further processing.

[JimmXinu]

Originally Posted by Feynor

Google Books don't mind comments, but somehow the comments from hpfanficarchive becomes messed up, looking like "<!-- <!-- STORY START --> <p>--></p>" if I recall
...

That problem (comments becoming nested comments) should already be fixed.

We saw it last month when I changed how base_adapter converted soups to unicode so it was the same for BS3 and BS4. I had to make a change to the bundled BeautifulSoup3. See change rev 1537.

Plus, I'm not seeing it the epub I generate from that hpfanficarchive story. Is your code base up to date?

[Feynor]

Originally Posted by JimmXinu

That problem (comments becoming nested comments) should already be fixed.

We saw it last month when I changed how base_adapter converted soups to unicode so it was the same for BS3 and BS4. I had to make a change to the bundled BeautifulSoup3. See change rev 1537.

Plus, I'm not seeing it the epub I generate from that hpfanficarchive story. Is your code base up to date?

It is, I just hadn't loaded a fic from hopfanficarchive for a while, so I hadn't noticed the bug being fixed, just that the comment was removed with my apparently unnecessary addition.
I'm not having a good day it seems :P

[wp709]

Is it possible to include more than the story summary in the "comments" metadata section in Calibre? I'd also like to automatically include ratings and word counts if possible. I've tried reading the wiki but it's all a bit confusing for this beginner.