[GUI Plugin] FanFictionDownLoader

[Depletion]

I was using the commented out info just for examples. And thanks Facedeer, your answer helped me see where I was going wrong. I was going into configure FFDL and the custom column tag when I should have just updated the metadata. Silly mistake in hindsight but I appreciate both your and JimmXinu's help.

[davidfor]

Originally Posted by JimmXinu

Test version with new site: literotica.com adapter mostly by de3sw2aq1. This is very much an adult site.

I had to look

The formatting of the epubs produced has a problem. This is because the story text is wrapped in a single p tag. It uses br tags to create the impression of paragraphs. The adapter passes the text through utf8FromSoup and will use replace_br_with_p if the users options have it set. But, because the text is wrapped in a single set of paragraph tags, nothing is changed. To get it working right, I had to strip the tags off.

At line 200 is:

        story1 = soup1.find('div', 'b-story-body-x')
        storytext = self.utf8FromSoup(url, story1)

I changed this to:

        story1 = soup1.find('div', 'b-story-body-x')
        story1 = str(story1.find('div').p)
        story1 = story1[len('<p>'):len(story1) - len('</p>')]
        story1 = bs.BeautifulSoup(story1)
        storytext = self.utf8FromSoup(url, story1)

In the page loop, line 215 is the same.

This of course doesn't take into account having the "replace_br_with_p" set to false. In that case, the paragraph tag probably should be put back.

[JimmXinu]

Originally Posted by davidfor

The formatting of the epubs produced has a problem. This is because the story text is wrapped in a single p tag. It uses br tags to create the impression of paragraphs. The adapter passes the text through utf8FromSoup and will use replace_br_with_p if the users options have it set. But, because the text is wrapped in a single set of paragraph tags, nothing is changed. To get it working right, I had to strip the tags off.
...

Good catch. Here's a version that addresses the issue.

This version also adds more sleep time between hits to ffnet for metadata. I think that may help people who are getting errors from ffnet; I suspect it's because they are doing too many downloads/updates at once.

However, that also means it takes longer for FFDL to get metadata before pushing downloads into the background. I'm not convinced that this is worth the extra time it will cost people who aren't attempting mass downloads/updates. Let me know what you think.

UPDATE Dec 17, 2013 -- Remove obsolete beta versions
(Look for a newer version in more recent posts.)

[rlh3]

I was doing some random testing of the Literotica adapter and encountered this error.


'NoneType' object has no attribute '__getitem__'
http://www.literotica.com/s/you-dont-choose-love

I have encountered this on several stories, but this is the only one that I remembered to write down.

Also, I cannot seem to get the "are you an adult" dialogue to go away. My personal.ini entry is (username and password deleted):

[Literotica.com]
username:XXXXXX
password:XXXXXX
collect_series: true
is_adult:true

Am I doing something wrong here?

[JimmXinu]

Originally Posted by rlh3

'NoneType' object has no attribute '__getitem__'
http://www.literotica.com/s/you-dont-choose-love

That's a real problem caused by bad parsing when it's part of a 'series' (which are being treated as chapters of a single story) when it's the last series in the list.

Originally Posted by rlh3

Also, I cannot seem to get the "are you an adult" dialogue to go away. My personal.ini entry is (username and password deleted):

[Literotica.com]
username:XXXXXX
password:XXXXXX
collect_series: true
is_adult:true

[literotica.com] -- case matters.

This adapter doesn't use user/pass or the collect_series option.

The decision to treat 'series' as multi-chapter stories was made by the original adapter author. Since they all seem to be name XXX Ch 01, etc, it seems the right call to me.

UPDATE Dec 17, 2013 -- Remove obsolete beta versions
(Look for a newer version in more recent posts.)

[rlh3]

Seems to work just fine now. The case change in the personal.ini fixed that right up. Whatever you did on the other error seems to have solved it. I have only had time for some quick and dirty testing, but I will give it a better test tonight or in the morning, and if anything else comes up I will be sure to mention it.

thanks

[davidfor]

It works, but, for the replace_br_with_p to work, it needs the paragraph tag that wraps the text removed. The conversion to string and then the substring was because I couldn't work out a way to get BeautifulSoup to give me the contents of the paragraph as a BeautifulSoup object that utf8FromSoup would work on.

[JimmXinu]

Originally Posted by davidfor

It works, but, for the replace_br_with_p to work, it needs the paragraph tag that wraps the text removed. The conversion to string and then the substring was because I couldn't work out a way to get BeautifulSoup to give me the contents of the paragraph as a BeautifulSoup object that utf8FromSoup would work on.

Can you provide a story URL and what you think is wrong?

I tested both with and without replace_br_with_p and both looked fine to me.

[davidfor]

I picked a story from the author that rlh3 posted: http://www.literotica.com/s/down-drinking-at-the-bar. The choice was simply that it had a short first paragraph.

When I look at the code your version is producing, at the start I see:

<h3>Down Drinking at the Bar</h3>
<div>
<p>Joe Morrison wasn't particularly happy. His wife, who had been much too fond of drink had developed Korsakov, and she'd been in a care centre for years. He usually visited her in the afternoons.<br />
<br />
Life was lonely at home; it had been lonely for as long as he could remember. There were a few comforts he'd stuck to, all those years: his collection of poetry, string quartets, a dram before bed and the barmaid at the Rose and Crown.<br />
<br />
.
.
.
"Didn't you hear? He drove into tree. Must've been dead at once."<br />
<br />
"Oh no," Polly said. She slumped down with her arms on the bar, put her head on her arms and cried.</p>
</div>

When I strip the paragraph tag, I get:

<h3>Down Drinking at the Bar</h3>
<div>
<p>Joe Morrison wasn't particularly happy. His wife, who had been much too fond of drink had developed Korsakov, and she'd been in a care centre for years. He usually visited her in the afternoons.</p>
<p>Life was lonely at home; it had been lonely for as long as he could remember. There were a few comforts he'd stuck to, all those years: his collection of poetry, string quartets, a dram before bed and the barmaid at the Rose and Crown.</p>
.
.
.
<p>"Didn't you hear? He drove into tree. Must've been dead at once."</p>
<p>"Oh no," Polly said. She slumped down with her arms on the bar, put her head on her arms and cried.</p>
</div>

The first one always has the blank lines when reading. The latter can be styled however you want it.

When I look at the method replace_br_with_p, I found the following comment:

# Find all bexisting blocks with p, pre and blockquote tags, we need to shields break tags inside those.

That's what made me strip the paragraph tag.

[JimmXinu]

Originally Posted by davidfor

I picked a story from the author that rlh3 posted: http://www.literotica.com/s/down-drinking-at-the-bar. The choice was simply that it had a short first paragraph.
...

This version changes the p tag to a div tag and adds another br explicitly to each page. It appears to work for both replace_br_with_p: true and false.

However, I don't have a big enough sample set to know if all literotica stories use <br><br> paragraph breaks.

If this still doesn't satisfy, we'll have to get Asbjørn to look at changing replace_br_with_p--that's his baby and I treat it like a black box. I'm not willing to put conditionals in individual adapters for replace_br_with_p.

UPDATE Jan 4, 2013 - Remove obsolete beta versions

[davidfor]

That looks good. And from a look around, they all seem to be formatted in exactly the same way.

And I can't blame you about replace_br_with_p. I had a quick look and it has far to many and complex regex's for my liking. Luckily the comments gave me an idea of what was happening.

[rlh3]

In continued testing of the Literotica adapter, I have found that the "get story URLs to download from web page" function does not work. I just discovered this function, and thought to try it.

This is the error message I get:

Spoiler:

calibre, version 1.15.0
ERROR: Unhandled exception: <b>TypeError</b>:unbound method getSiteExampleURLs() must be called with LiteroticaSiteAdapter instance as first argument (got nothing instead)

calibre 1.15 isfrozen: True is64bit: False
Windows-8-6.2.9200 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '8', '6.2.9200')
Python 2.7.5
Windows: ('8', '6.2.9200', '', 'Multiprocessor Free')
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 399, in get_urls_from_page_menu
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 415, in get_urls_from_page
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.geturls", line 37, in get_urls_from_page
File "calibre_plugins.fanfictiondownloader_plugin.fanfi cdownloader.adapters.__init__", line 171, in getAdapter
TypeError: unbound method getSiteExampleURLs() must be called with LiteroticaSiteAdapter instance as first argument (got nothing instead)

If this in functionality that is not going to be supported, then no big deal, but I thought that I would mention it.

[JimmXinu]

Originally Posted by rlh3

In continued testing of the Literotica adapter, I have found that the "get story URLs to download from web page" function does not work. I just discovered this function, and thought to try it.

This is the error message I get:
...

Good catch. It will be fixed in the next version.

[cryzed]

Hey there Jimm!
Just letting you know that I still enjoy your plugin and love that you are so actively developing it. I'm really thankful for your continued work.

Greetings!

[JimmXinu]

Originally Posted by cryzed

Just letting you know that I still enjoy your plugin and love that you are so actively developing it. I'm really thankful for your continued work.

Thanks, and you're welcome.