[GUI Plugin] FanFictionDownLoader

[ilovejedd]

Originally Posted by JimmXinu

No, it's still on the 'wishlist' of things I may do some day. As I recall, I looked into it once and found that it would require restructuring the code more than I wanted tackle that day. Especially as I'd only ever heard from one user asking for it. Now that it's two users, I'll revisit it sometime.

If it'll take too much work, forget about it. I actually happen to like the leading chapter numbers and thought the reason I'm not getting it is because you had implemented the above. Apparently, the reason I'm not seeing them is because the site adapter doesn't parse it.

From adapter_dramioneorg.py, the code for getting the chapter url and chapter titles appear to be as follows:

        # Find the chapters:
        for chapter in soup.findAll('a', href=re.compile(r'viewstory.php\?sid='+self.story.getMetadata('storyId')+"&chapter=\d+$")):
            # just in case there's tags, like <i> in chapter titles.
            self.chapterUrls.append((stripHTML(chapter),'http://'+self.host+'/'+chapter['href']+addurl))

The actual HTML code looks like this:

<tr><td>1. <a href="viewstory.php?sid=1331&amp;chapter=1">December 2010: Five Minutes 'til Christmas</a>  <span class="cnotes"><span class="cnotes"><p>My first drabble ever!</p>
<p>Drabbles are supposed to be exactly 100 words long, and often have a prompt word or phrase to use as inspiration. It is a great exercise for new and experienced authors alike.</p>
<p>This particular drabble is rated G, and is quite fluffy, really.</p>
<p>Enjoy!</p>
<p>Prompt: <i>a </i><i>gift</i></p>
</span></span></td><td class="center"><a href="reviews.php?type=ST&amp;item=1331&amp;chapid=4326">1</a> </td><td class="center">90</td><td class="center">412</td></tr>

What changes do I need to make to the adapter code to include the chapter number instead of just getting the title inside the link?

Thanks!

[JimmXinu]

By having only one 'add_chapter_numbers' option instead of one for TOC and one for in-chapter titles, this is simplified a fair bit.

The attached version has new options:

## Don't like the numbers at the start of chapter titles on some
## sites?  You can use strip_chapter_numbers to strip them off.  Just
## want to make them all look the same?  Strip them off, then add them
## back on with add_chapter_numbers.  Don't like the way it strips
## numbers or adds them back?  See chapter_title_strip_pattern and
## chapter_title_add_pattern.
strip_chapter_numbers:true
add_chapter_numbers:true

Add them to your personal.ini (after installing, of course) to give them a try. They will be applied when an existing epub is updated. You can do 'Update Always' to change the chapter titles in existing epubs even without new chapters.

I've included default patterns for the stripping and adding that seem reasonable to me, but they're also configurable:

Spoiler:

## (Two versions of chapter_title_strip_pattern are shown below.  You
## should only have one uncommented.)
## This version will remove the leading number from:
## "1." => ""
## "1. The Beginning" => "The Beginning"
## "1: Start" => "Start"
## "2, Chapter the second" => "Chapter the second"
## etc
chapter_title_strip_pattern:^[0-9]+[\.: -]+

## This version will strip all of the above *plus* remove 'Chapter 1':
## "Chapter 1" => ""
## "1. Chapter 1" => ""
## "1. Chapter 1, Bob's First Clue" => "Bob's First Clue"
## "Chapter 2 - Pirates Place" => "Pirates Place"
## etc
#chapter_title_strip_pattern:^([0-9]+[\.: -]+)?(Chapter *[0-9]+[\.:, -]*)?

## Uses a python template substitution.  The ${index} is the 'chapter'
## number and ${title} is the chapter title, after applying
## chapter_title_strip_pattern.  Those are the only variables available.
## "The Beginning" => "1. The Beginning" 
chapter_title_add_pattern:${index}. ${title}

[ilovejedd]

Originally Posted by JimmXinu

By having only one 'add_chapter_numbers' option instead of one for TOC and one for in-chapter titles, this is simplified a fair bit.

The attached version has new options:

## Don't like the numbers at the start of chapter titles on some
## sites?  You can use strip_chapter_numbers to strip them off.  Just
## want to make them all look the same?  Strip them off, then add them
## back on with add_chapter_numbers.  Don't like the way it strips
## numbers or adds them back?  See chapter_title_strip_pattern and
## chapter_title_add_pattern.
strip_chapter_numbers:true
add_chapter_numbers:true

Add them to your personal.ini (after installing, of course) to give them a try. They will be applied when an existing epub is updated. You can do 'Update Always' to change the chapter titles in existing epubs even without new chapters.

That works perferctly. Thanks!

Regarding "Update Always", if there are no new chapters, does it still re-download all the chapters before updating the epub? Reason I ask is I'd like to update some 1,500 ebooks and I reckon if it downloads all the chapters, I'd play nice and just do it in batches so as not to hammer the site.

[JimmXinu]

Originally Posted by ilovejedd

That works perferctly. Thanks!

Regarding "Update Always", if there are no new chapters, does it still re-download all the chapters before updating the epub? Reason I ask is I'd like to update some 1,500 ebooks and I reckon if it downloads all the chapters, I'd play nice and just do it in batches so as not to hammer the site.

Glad it's working for you.

"Update Always" pulls the first chapter or chapter list, but the whole point behind "Update epub" is to not download chapters it already has.

So an "Update Always" with no new chapters will hit the site servers 2-4 times per story, depending on the site, user/pass, etc.

You might consider is adding this to your personal.ini while doing large updates:

slow_down_sleep_time:2.5

That puts a 2.5 sec sleep before each site request.

[JimmXinu]

Test version including new site merlinfic.dtwins.co.uk.

Assuming no new problems, this is the last test version before the next release.

[Firedancer885]

Originally Posted by JimmXinu

Test version including new site merlinfic.dtwins.co.uk.

Thanks Works fine.

[JimmXinu]

Version 1.6.9 - 14 Sep 2012

• New site merlinfic.dtwins.co.uk (thanks, Ida)
• New option features strip_chapter_numbers and add_chapter_numbers to remove and add numbers to chapter titles.
• Fix BBEncoded summaries for fimfiction.net, improved handling for HTML in summaries in general. Default keep_summary_html:true to now.
• Added code to prevent crashing when include_images accidentally turned on for non-epub formats.

[ilovejedd]

Thanks for the update. Special request, for the next update, can you please add the following additional custom columns?

theme:Theme
timeline:Timeline
cliche:Cliche

I've been playing around with the dramione.org adapter and have added the above info to my local copy. No idea how to make diffs so I'm just including the full adapter_dramioneorg.py file.

Thanks!

[JimmXinu]

Originally Posted by ilovejedd

Thanks for the update. Special request, for the next update, can you please add the following additional custom columns?

theme:Theme
timeline:Timeline
cliche:Cliche

That's unlikely to happen anytime soon.

Each site chooses different metadata to define. Even category and genre vary wildly from site to site. At this point I don't want to add any more new metadata 'columns' that isn't used by a significant number of sites.

(I only reluctantly added Relationships metadata recently after finding that 7 sites actually do provide 'ship data and a further 13 were dedicated to a particular 'ship and could therefore be considered to have implicit 'ship data.)

I have an entry on the 'wishlist' to figure out a better way for individual site-specific metadata, but it hasn't happened yet.

If you like, I am willing to change the dramione.org adapter to add the theme and timeline data to genre and what you've identified as 'cliche' (and dramione.org calls 'Draco' and 'Hermione' categories) to either genre or characters.

Hmmm. And you've found cover images. Good thing I diff'ed the source, since you didn't mention that. I'll include that bit either way. Can you give me URLs to a couple stories there with banner images so I can test it?

[ilovejedd]

Originally Posted by JimmXinu

That's unlikely to happen anytime soon.

Each site chooses different metadata to define. Even category and genre vary wildly from site to site. At this point I don't want to add any more new metadata 'columns' that isn't used by a significant number of sites.

(I only reluctantly added Relationships metadata recently after finding that 7 sites actually do provide 'ship data and a further 13 were dedicated to a particular 'ship and could therefore be considered to have implicit 'ship data.)

I have an entry on the 'wishlist' to figure out a better way for individual site-specific metadata, but it hasn't happened yet.

If you like, I am willing to change the dramione.org adapter to add the theme and timeline data to genre and what you've identified as 'cliche' (and dramione.org calls 'Draco' and 'Hermione' categories) to either genre or characters.

Hmmm. And you've found cover images. Good thing I diff'ed the source, since you didn't mention that. I'll include that bit either way. Can you give me URLs to a couple stories there with banner images so I can test it?

I hear you about the category and genre randomness per site. Heck, I've seen it partially with warnings, too. Right now, I've got a separate Calibre library per site.

Don't bother adding the theme and timeline info to genre. That's what I originally did and it's just not as organized as having the info in its own custom column. Totally messed up my subject tags. Don't really fancy cleaning up my Tags every time I download a new story. I'll just keep doing the edits to my local copy. Comparing the source code is much faster and easier than frequent tag clean up. dramione.org actually has ship data for extra ships. Just didn't bother adding that as I prefer just having the main ship.

I've forgotten about the covers. The cover images are more banners than covers. I actually use Generate Cover for the actual calibre covers but I still do like having the banners inside the epub.

Just looked for random fics with images:

http://dramione.org/viewstory.php?sid=21
http://dramione.org/viewstory.php?sid=1826
http://dramione.org/viewstory.php?sid=1734

[JimmXinu]

Originally Posted by ilovejedd

Don't bother adding the theme and timeline info to genre.

Okay. I'll continue to think about how I might be able to add more site specific metadata to custom columns. Right now my best idea for it would be to do that configuration in personal.ini instead of the GUI config.

Originally Posted by ilovejedd

Just looked for random fics with images:

Thanks, I'll include that cover code in the next release.

[ilovejedd]

Originally Posted by JimmXinu

Okay. I'll continue to think about how I might be able to add more site specific metadata to custom columns. Right now my best idea for it would be to do that configuration in personal.ini instead of the GUI config.

That's probably the best way.

[skb]

I am having trouble downloading a story from ffnet and wasn't sure if it's an issue with the story (the formatting is...weird) or it's too long or...

http://www.fanfiction.net/s/344588/1...ardassian-Mask

FDL gives the message "FFDL found 0 good and 0 bad updates". The log is empty (only contains: "Status Title Author Comment URL").

Using 0.8.68 Calibre and 1.6.8 FFDL

I've said it before, but I'll say it again - this plugin is fantastic. It has changed my whole workflow and made it so much easier to download and read stuff. Thanks again for all the hard work.

[JimmXinu]

Originally Posted by skb

I am having trouble downloading a story from ffnet and wasn't sure if it's an issue with the story (the formatting is...weird) or it's too long or...

http://www.fanfiction.net/s/344588/1...ardassian-Mask

FDL gives the message "FFDL found 0 good and 0 bad updates". The log is empty (only contains: "Status Title Author Comment URL").

That's an ancient story (Jul 2001) with extremely bad HTML. So bad it causes an infinite recursion in FFDL's HTML parser that crashes the background job entirely. (The error output shows up in the Jobs window.)

Given that ffnet wouldn't allow that content now, or indeed, for several years now (see the url and email address in there?), I'm not going to put in the effort and special coding it would take to deal with this one author and their nasty HTML. Sorry.

Originally Posted by skb

I've said it before, but I'll say it again - this plugin is fantastic. It has changed my whole workflow and made it so much easier to download and read stuff. Thanks again for all the hard work.

Glad you like it, sorry I couldn't help you this time.

[skb]

No worries - I thought it might have been the formatting (which is bizarre). I can get it (the story) on PDF but I think FFDL is spoiling me for other formats (and I've hated PDFs since my Palm Pilot days).

Thanks again!