[GUI Plugin] FanFictionDownLoader

[JimmXinu]

Version 2.1.03 - 28 Jan 2015

• Add slow_down_sleep_time:2 to AO3 defaults.
• Fix setDescription for adapter_tolkienfanfiction.
• Some improvements to replace_br_with_p heuristics, thanks Asbjorn.
• Fix for Sequels for FimF.
• Updates to Plugin translations from Transifex.

[AndersW]

Originally Posted by JimmXinu

The author didn't use an hr tag there. They faked it with line-through and spaces:

<p><span style="text-decoration: line-through;">&nbsp;[...]&nbsp; </span></p>

If you set keep_style_attr:true in personal.ini, it keeps some of these fake hr's, but not all of them.

It looks like some of them are being lost in processing due to in the author's original HTML.

Unless you can show several examples of this problem in different stories by different authors, I'm inclined to chalk it up to bad HTML by the author and not worry about it.

But I'll look into it more if you can find several instances that show me it's something that commonly happens.

Thanks for looking into it anyway. Something that might be related: In plenty of stories I see italics end after one paragraph, when on the website they go on for several more paragraphs. Is this a result of incorrect tag nesting?
An example is the very beginning of this story. http://www.fimfiction.net/story/145703/

[JimmXinu]

Originally Posted by AndersW

Thanks for looking into it anyway. Something that might be related: In plenty of stories I see italics end after one paragraph, when on the website they go on for several more paragraphs. Is this a result of incorrect tag nesting?
An example is the very beginning of this story. http://www.fimfiction.net/story/145703/

Yes. It appears that somehow FimF stories have a lot of that.

But the latest versions of FFDL handles incorrect nesting better. The first several paragraphs of that story are now all italicized when I download it. You'd have to do an 'overwrite' not an 'update' to fix it in an existing epub.

[McHunter]

Hello...

is there a limit with how many books at once the plugin stops working,
and it is a problem of calibre or the plugin?

I have 2000 books in my library that I try to update at least once a month.

When I selekt all at once and try to use the plugin, than is there a problem when
after downloading the metadata it should starting to download the stories.

If I select less than 1000 (I have had tryed with 800 books) than ist works without a problem.

[aleyx]

Originally Posted by McHunter

Hello...

is there a limit with how many books at once the plugin stops working,
and it is a problem of calibre or the plugin?

I have 2000 books in my library that I try to update at least once a month

The plugin can handle that, and Calibre doesn't care. The websites however, usually really don't like being siphoned like that, and some ban your IP for a little while if you do it too much. That's why the plugin forces a slowdown on the downloads for those sites.

The correct (or at least most polite) way to update stories is to only select those you know are updated, usually by way of mailed notices from the sites. Updating at least once a week instead of once a month can also help if you have many stories updating often.

I have about 500 in-progress stories, from 5 websites. My mail client filters all incoming update notices in a separate folder, and flag them automatically. Every day I look at which stories have a notice, and update my favorites and whichever I feel like reading, and unflag the notices. There's seldom more than three or four, max.

That way I only update those stories I'm reasonably certain I'll read, it takes no time to update, and I know which stories have updated on the site but not in Calibre.

There was a time I thought about writing a plugin to do all of that directly in Calibre, but JimmXinu noted that giving a plugin access to your mail wasn't perhaps the best idea. I haven't thought of a better solution yet that'd work for people that don't have their own SMTP/IMAP servers at home...

N.

[davidfor]

Originally Posted by aleyx

There was a time I thought about writing a plugin to do all of that directly in Calibre, but JimmXinu noted that giving a plugin access to your mail wasn't perhaps the best idea. I haven't thought of a better solution yet that'd work for people that don't have their own SMTP/IMAP servers at home...

I haven't looked, but does FFDL have a way to scan an RSS feed or an updates page and only pick out the stories that are in calibre? If not, that would be one way to handle updating stories.

[aleyx]

Originally Posted by davidfor

I haven't looked, but does FFDL have a way to scan an RSS feed or an updates page and only pick out the stories that are in calibre? If not, that would be one way to handle updating stories.

If it can't, it can be made to do it ^_^

There's the problems of how many FFDL-known sites have RSS feeds, and how to manage high-volume sites (FFNet comes to mind) without missing most of the stories, since we can't really assume that Calibre will run 24/24. But yeah, that sounds like a sound method.

N.

[Tanjamuse]

It would make it a lot easier to keep track of both new stories and updated ones with the RSS feed directly available in Calibre.

[aleyx]

Originally Posted by Tanjamuse

It would make it a lot easier to keep track of both new stories and updated ones with the RSS feed directly available in Calibre.

If your sites had a RSS feed for your account's follows/alerts, then yeah, sure. But a site-wide feed would be too confusing; you'd have a _lot_ more stories that you don't care about than stories you do.

For example, I'm subscribed to SIYE's RSS, which has about two to three updates a day at best these days, but I'm only interested in maybe five a month (aside from the stories I actually follow).

The big sites (FFNet, AO3, even FimF) don't have RSS feeds anyways, aside from site-related news (downtimes, software upgrades, etc) and even then they're migrating to tweeter feeds. As far as I can see, the only way to get notices of story updates and/or new stories across all sites is by mail.

But yeah, aside from that it's an interesting idea. If we could find a way to convert mail alerts to an RSS feed, and use Calibre's RSS infrastructure to pass that feed to FFDL, we'd have something useful for a lot of people, I think.

How to do that without writing a bunch of separate tools is the real question.

N.

[Tanjamuse]

Both AO3 and FF.net has rss feeds

At least I think they're are rss feeds, the little orange icons?

[aleyx]

Originally Posted by Tanjamuse

Both AO3 and FF.net has rss feeds

At least I think they're are rss feeds, the little orange icons?

Ah yes, I see them now in FFNet, when you select a category. My bad.

I've never used that aspect of Calibre though; I'll have to install a recipe and play around, see what's possible to do.

N.

[JimmXinu]

@aleyx explained it pretty well--doing large numbers of updates at once (where 'large' is generally anything more than ~25) is not a polite thing to do. It's a lot of traffic on the site(s) and can get you blocked.

As for RSS feeds, I'm not personally interested in pursuing. But I will accept code contributions if someone else comes up with a clean way to do it.

Something I have been experimenting with is using IMAP to get URLs for updates from the site update notice emails. FFDL on command looks for unread emails in a specified folder/label that contain story URLs and then marks them read before popping up the regular download-from-urls dialog prefilled with the story URLs.

It works with my GMail account, but it requires giving your email address and password to FFDL/calibre. And you have to have IMAP available/turned on.

Are people willing to give FFDL/calibre their address and password?

(I experimented with GMail using OAuth also, but it was considerably more complex and would only work with GMail.)

[aleyx]

Originally Posted by JimmXinu

Something I have been experimenting with is using IMAP to get URLs for updates from the site update notice emails. FFDL on command looks for unread emails in a specified folder/label that contain story URLs and then marks them read before popping up the regular download-from-urls dialog prefilled with the story URLs.

It works with my GMail account, but it requires giving your email address and password to FFDL/calibre. And you have to have IMAP available/turned on.

Are people willing to give FFDL/calibre their address and password?

I'd be very happy to do that. I still think it's the most efficient way, compared to RSS feeds.

Although instead of read/unread, I'd rather go with flag/unflag, which would reduce the "risk" of a user reading notices and forgetting to put the Unread flag back for FFDL.

Another point: What about taking advantage (again) of Reading List? Say that FFDL can ask for a "Stories to update" list, which will be populated by the user and/or by FFDL's mail parser. FFDL would maybe have a new menu item "Update stories from waiting list" (alongside "Update existing"), update the books in the usual way from that list, then flush the list of books with a success return code. That would have the added advantage of allowing the user to easily cherry-pick the stories to update before running the job.

N.

[BookJunkieLI]

Before anyone puts a whole of time into the RSS feeds they won't pull stories that have more than 1 chapter. I tried with both AO3 and FFN because I'd hoped to use it to streamline my own process for pulling new/updated links from the sites. And FFN ignores the filters beyond fandom. I spent a couple of days being very frustrated as I tried to figure out why my lists weren't matching up.

[aleyx]

Originally Posted by BookJunkieLI

And FFN ignores the filters beyond fandom. I spent a couple of days being very frustrated as I tried to figure out why my lists weren't matching up.

I suspected that was the case when the URLs didn't change with filters; I just tried to manually manipulate them (ex. adding "&len=100&c1=2039&c2=2045" to Naruto's feed), but no sugar.

(Yeah, well, I don't smoke, so.)

It wouldn't have been so bad if the feed had nodes for every metadata, but it lacks everything that could be used as a filter (genre, length, characters, language...). There's basically only the dates, the author's URL and the story-id.

Such a waste of what could have been a useful tool...

I swear, the day FFNet presents a useful API, it'll have a lot less difficulties with tools like Calibre. Maybe they should talk with ComicVine or something.

N.