[GUI Plugin] FanFictionDownLoader

[JimmXinu]

Originally Posted by crackshot91

Now, I just have a couple more questions. First: Is there a way to always update the metadata, but only overwrite the epub if it's newer on the site? Does "Always Overwrite" essentially accomplish this?

Not exactly. You can use 'Update Calibre Metadata Only' mode to update metadata without updating/overwriting the epub, but FFDL assumes that metadata has changed only if there are new chapters.

Originally Posted by crackshot91

Also, is there a way to set calibre to automatically convert the epub that FFDL downloads to a mobi file? I found that I need to download the book as an epub and THEN convert to mobi, so that I get the chapter marks and skipping function on my kindle. (Thanks to this thread. )

FFDL, in concert with the Reading List plugin can already do that. Basically, you set FFDL to delete other formats on update/overwrite and add them to a Reading List that will be sent to your device. Then when Reading List syncs your device, it automatically converts them, too.

See Reading List Integration for the full details.

Originally Posted by crackshot91

And lastly, have you considered perhaps adding the ability to download all stories by a single author on a site? It's not urgent, or anything. Just an idea. Right now I use this extension to get multiple urls and it works a treat.

Use FFDL's 'Get Story URLs from Web Page' feature. That's as close as I plan to get.

For one thing, it's more general and can be used more places. For another, I don't want to make it too easy for people to pull large numbers of fics at a time because it's stressful on the site servers. We have had people get (temporarily) locked out of sites before for excessive traffic.

Jim

[crackshot91]

Originally Posted by JimmXinu

Not exactly. You can use 'Update Calibre Metadata Only' mode to update metadata without updating/overwriting the epub, but FFDL assumes that metadata has changed only if there are new chapters.



FFDL, in concert with the Reading List plugin can already do that. Basically, you set FFDL to delete other formats on update/overwrite and add them to a Reading List that will be sent to your device. Then when Reading List syncs your device, it automatically converts them, too.

See Reading List Integration for the full details.



Use FFDL's 'Get Story URLs from Web Page' feature. That's as close as I plan to get.

For one thing, it's more general and can be used more places. For another, I don't want to make it too easy for people to pull large numbers of fics at a time because it's stressful on the site servers. We have had people get (temporarily) locked out of sites before for excessive traffic.

Jim

Awesome reply. Thank you. This plugin is really fun to play with.

The "auto convert" thing is a problem I'll deal with at a later time, but I've downloaded the Reading Lists plugin for later use.

I just realized how silly that overwrite/update question was. Now that I have everything configured, "Overwrite if Newer" is perfect for my needs. I just needed to run "Update metadata only" once, to update my existing library.

And the mass-downloading thing can of course case problems, so I can see why it wouldn't be a great idea. The add url function is perfectly sufficient.

[Amalthia]

Originally Posted by JimmXinu

Can you check that again? Because I just checked to be sure and there is literally no difference between an epub generated by FFDL one at a time vs with a group except the Packaged timestamp.

This is the only symptom that might indicate a problem with FFDL instead of the content from the site.

I tested the multiple download function and it appears to be working.

[JimmXinu]

Originally Posted by Amalthia

I tested the multiple download function and it appears to be working.

Okay, great.

In that case, I'm going to declare this an issue with the Sony PRS-650 and the HTML from those particular stories. Since converting epub->epub fixes it for you, you can just do that for those few.

Or if you want to try and figure it out, I'd start with the smallest story it happens to, and use Tweak epub to remove different sections of the story until you isolate what causes the problem. If you can isolate the exact problem, I can probably add code to FFDL to prevent it in future.

Jim

[evamaria]

Originally Posted by JimmXinu

Please try this version and see how it works. (Mac users only.)

If there are any other Mac users hanging around, I'd love to hear from some that FFDL PI does work for.

Update Sep 6, 2012: You can still try this version, and I'll leave it here in case we need to refer to it, but the changes included have already been included in the v1.6.8 release.

Seems to work perfectly now - thank you so much, this is awesome!

[Dylan Tomorrow]

Originally Posted by JimmXinu

It's working fine for me. I tried it on Ubuntu 12.04.1 LTS with calibre 0.8.68 and FFDL 1.6.8. Although it is a story that requires you to answer that yes, you're an adult.

Giving multiple chapter URLs to the same story won't do anything useful. Either they'll be ignored or they'll all normalize to the same story URL.

You can get more debug output by running calibre from command line as 'calibre-debug -g'.

Thanks! Alright, this is the log:

calibre-debug -g
Starting up...
Key u'G' for shortcut Bücher erwerben is already used by An Reader übertragen, ignoring
Key u'G' for shortcut Bücher erwerben is already used by An Reader übertragen, ignoring
Started up in 14.82 seconds with 283 books
url:(http://www.fictionalley.org/authors/barb/HPATPS.html) begin:(None) end:(None)
FanFictionDownLoader v1.6.8
url:http://www.fictionalley.org/authors/barb/HPATPS.html
from URL(http://www.fictionalley.org/authors/barb/HPATPS.html)
title:Harry Potter and the Psychic Serpent
outfile:/tmp/calibre_0.8.68_tmp_SPpmqW/fanfictiondownloader_F_Nxtn/new-None-Bqbvw4.mobi
Fetched metadata for 1 of 1
Worker Launch took: 2.95985603333
Job: 1 Download FanFiction Book finished
Download FanFiction Book
	FanFictionDownLoader v1.6.8
	Adding jobs for URLs:
	http://www.fictionalley.org/authors/barb/HPATPS.html
	Logfile for book ID None (Harry Potter and the Psychic Serpent)
	url:(http://www.fictionalley.org/authors/barb/HPATPS.html) id:(None)
	write to /tmp/calibre_0.8.68_tmp_SPpmqW/fanfictiondownloader_F_Nxtn/new-None-Bqbvw4.mobi
	Exception: {u'comment': u"Failed to parse image_max_size from personal.ini:['']\nException: invalid literal for int() with base 10: ''", u'status': u'Error', u'author_sort': [u'Barb'], u'pubdate': datetime.datetime(2001, 7, 15, 0, 0, tzinfo=SafeLocalTimeZone()), u'series': None, u'is_adult': False, u'added': False, u'end': None, u'author': [u'Barb'], u'comments': u"In Harry's fifth year he gets a snake with the Sight. Hermione's torn between Ron and Harry, who's torn between her and Ginny, who's torn between him and Draco Malfoy, who's torn between her and loyalty to his father. Plus: a Prophecy, Animagus training, a Dueling Club, Snape's Penseive, kilts, giants, house elf liberation and more!", u'calibre_id': None, u'username': 'NoneGiven', u'begin': None, u'good': False, u'tags': [u'Romance', u'FanFiction', u'Harry Potter'], u'timestamp': datetime.datetime(2012, 9, 9, 18, 27, 51, 363638, tzinfo=SafeLocalTimeZone()), u'outfile': u'/tmp/calibre_0.8.68_tmp_SPpmqW/fanfictiondownloader_F_Nxtn/new-None-Bqbvw4.mobi', u'password': '', u'icon': u'dialog_error.png', u'publisher': u'www.fictionalley.org', u'url': u'http://www.fictionalley.org/authors/barb/HPATPS.html', u'all_metadata': {'rating': u'R', 'numWords': u'341,236', 'site': u'www.fictionalley.org', 'datePublished': u'2001-07-15', 'authorHTML': u"<a class='authorlink' href='http://www.fictionalley.org/authors/barb/'>Barb</a>", 'storyUrl': u'http://www.fictionalley.org/authors/barb/HPATPS.html', 'storyId': u'HPATPS', 'category': u'Harry Potter', 'formatext': u'.mobi', 'author': u'Barb', 'ships': u'', 'formatname': u'mobi', 'version': u'4.4', 'dateUpdated': u'2001-09-04', 'numChapters': u'33', 'description': u"In Harry's fifth year he gets a snake with the Sight. Hermione's torn between Ron and Harry, who's torn between her and Ginny, who's torn between him and Draco Malfoy, who's torn between her and loyalty to his father. Plus: a Prophecy, Animagus training, a Dueling Club, Snape's Penseive, kilts, giants, house elf liberation and more!", 'warnings': u'', 'dateCreated': u'2012-09-09 18:27:51', 'authorId': u'barb', 'authorUrl': u'http://www.fictionalley.org/authors/barb/', 'characters': u'', 'genre': u'Romance', 'title': u'Harry Potter and the Psychic Serpent', 'siteabbrev': u'fa', 'extratags': u'FanFiction'}, u'title': u'Harry Potter and the Psychic Serpent'}:Failed to parse image_max_size from personal.ini:['']
	Exception: invalid literal for int() with base 10: ''
	Traceback (most recent call last):
	  File "calibre_plugins.fanfictiondownloader_plugin.jobs", line 151, in do_download_for_worker
	  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.writers.base_writer", line 230, in writeStory
	  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.base_adapter", line 209, in getStory
	  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.adapter_fictionalleyorg", line 226, in getChapterText
	  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.base_adapter", line 315, in utf8FromSoup
	  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.story", line 389, in addImgUrl
	FailedToDownload: Failed to parse image_max_size from personal.ini:['']
	Exception: invalid literal for int() with base 10: ''
	
	Successfully downloaded:
	
	Unsuccessful:
	Harry Potter and the Psychic Serpent

My personal.ini:

## This is an example of what your personal configuration might look
## like.

[defaults]
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
is_adult:true

## include images from img tags in the body and summary of
## stories. Images will be converted to jpg for size if possible.
include_images:true

## If not set, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
#keep_summary_html:true

## If set, the first image found will be made the cover image. If
## keep_summary_html is true, any images in summary will be before any
## in chapters.
#make_firstimage_cover:true

## Resize images down to width, height, preserving aspect ratio.
## Nook size, with margin.
#image_max_size: 580, 725

## Change image to grayscale, if graphics library allows, to save
## space.
#grayscale_images: false

## Try to collect series name and number of this story in series.
## Some sites (ab)use 'series' for reading lists and personal
## collections. This lets us turn it on and off by site without
## keeping a lengthy titlepage_entries per site and prevents it
## updating in the plugin.
## Turn off in [defaults] or [overrides] to prevent all sites from
## updating series column.
## default is true
#collect_series: false

## Most common, I expect will be using this to save username/passwords
## for different sites.
[www.twilighted.net]
#username:YourPenname
#password:YourPassword
## default is false
#collect_series: true

[www.ficwad.com]
#username:YourUsername
#password:YourPassword

[www.twiwrite.net]
#username:YourName
#password:yourpassword
## default is false
#collect_series: true

[www.adastrafanfic.com]
## Some sites do not require a login, but do require the user to
## confirm they are adult for adult content.
#is_adult:true

[www.archiveofourown.org]
collect_series: true
## added myself


[www.thewriterscoffeeshop.com]
#username:YourName
#password:yourpassword
#is_adult:true
## default is false
#collect_series: true

[www.fictionalley.org]
#is_adult:true

[www.harrypotterfanfiction.com]
#is_adult:true

[www.fimfiction.net]
#is_adult:true
#fail_on_password: false

[www.tthfanfic.org]
#is_adult:true
## tth is a little unusual--it doesn't require user/pass, but the site
## keeps track of which chapters you've read and won't send another
## update until it thinks you're up to date. This way, on download,
## it thinks you're up to date.
#username:YourName
#password:yourpassword


## This section will override anything in the system defaults or other
## sections here.
[overrides]
## default varies by site. Set true here to force all sites to
## collect series.
#collect_series: true

Huh, is there some syntax error I introduced into the personal.ini error I'm missing? I can't see anything wrong with the image_max_size entry. I hope I am not missing the blatantly obvious here .

[Firedancer885]

Originally Posted by Dylan Tomorrow

Huh, is there some syntax error I introduced into the personal.ini error I'm missing? I can't see anything wrong with the image_max_size entry.

You need to uncomment it for it to work delete the # from the beginning of the line.

[JimmXinu]

Originally Posted by Dylan Tomorrow

Huh, is there some syntax error I introduced into the personal.ini error I'm missing? I can't see anything wrong with the image_max_size entry. I hope I am not missing the blatantly obvious here .

It is a configuration issue, but it's more my fault than yours.

In the example personal.ini, I included the (commented out) include_images line under [defaults]. It's really only valid in [epub].

So when you uncommented it, turning it on under [defaults], and then tried to download a mobi, it tried to look up image_max_size, which wasn't defined for mobi.

(BTW, please include output format when reporting problems--I always assume epub otherwise.)

I've made two changes that will be in the next version: 1) The example personal.ini that new users get will correctly put the image options under epub and, 2) the PI will internally force include_images:false when format isn't epub.

For right now, you can add [epub] above your include_images line in your personal.ini and it will work properly.

Jim

[Dylan Tomorrow]

Thank you, that fixed it!

Well, you can see to the left that I only use a Kindle Touch, but also, I simply forgot to mention the format . Will do in the future!

And props for changing the plugin to avoid this happening to others .

[iatheia]

Originally Posted by jjm6383

I'd like to make a site request:

Highlander Fiction:
http://hlfiction.net/index.php

Thanks,
Joyslin

I'm pretty sure it is supported already.

Originally Posted by Firedancer885

Hi,

I'd like to make another site request.
http://merlinfic.dtwins.co.uk/

Bit on the smaller side, not really updated that frequently, but... meh, sure, why not, in a couple of days.

[Firedancer885]

Originally Posted by iatheia

Bit on the smaller side, not really updated that frequently, but... meh, sure, why not, in a couple of days.

Thanks

[Firedancer885]

Hi,

do the legal warnings and access requirements still apply for Adultfanfiction.net? It would be good if FFDL started supporting the site.

[iatheia]

They haven't changed anything on the website. It isn't really something that has an expiration date as long as they have an iron wall of overwarnings.

[JimmXinu]

Originally Posted by iatheia

They haven't changed anything on the website. It isn't really something that has an expiration date as long as they have an iron wall of overwarnings.

I agree. We don't want to add that level of programming complexity or legal obligation.

And I can see their point in limiting their liability. Just today there's a news post on adultfanfiction.net partially about it.

[crackshot91]

I was wondering if there was an easier way to convert bbcode to html for story descriptions downloaded from fimfiction.net?

So far, the easiest way I've found is to use this website to convert the bbcode descriptions into html. Is there any way to do this automatically for each description?

Also, I can't seem to get FFDL to show character tags from fimfiction.net. I may be doing it wrong? I created a custom column for "comma separated text", and assigned that column to "Characters" in the custom column section of the FFDL config. Is that not correct?