[GUI Plugin] FanFictionDownLoader

[JimmXinu]

Here's a new feature I've implemented that I'd like feedback on: Update logging.

A bit like titlepage, the logpage records configured metadata, but each time an epub story is updated, it adds a new log entry with only the configured metadata that have changed.

Here's an example. The first time, it recorded all the metadata. The next time, it recorded only what was changed.

Update Log

Packaged: 2012-08-03 22:27:39 Published: 2012-06-05 Updated: 2012-07-29 Chapters: 9 Words: 54,470 Status: In-Progress Title: Harry Crow Author: robst Category: Harry Potter Rating: T Summary: The Dursleys are excruciatingly normal. They were just plain ordinary people, trying to live their orderly lives. Why should they take what they consider an abnormal freak into their home? Harry raised by goblins fic.
===================(hr, really)
Packaged: 2012-08-05 09:52:52 Updated: 2012-08-05 Chapters: 10 Words: 61,182

Turning it on is just like titlepage or tocpage: add "include_logpage: true" to the [epub] section of your personal.ini. Use "logpage_entries" to control what your log records and in what order.

[epub]
## include a Update Log page before the story text.  If included, the
## log will be updated each time the epub is an all the metadata
## fields that have changed since the last update (typically
## dateUpdated,numChapters,numWords at a minimum) will be shown.
## Great for tracking when chapters came out and when the description,
## etc changed.
include_logpage: true

## items to include in the log page Empty metadata entries, or those
## that haven't changed since the last update, will *not* appear, even
## if in the list.  You can include extra text or HTML that will be
## included as-is in each log entry. Eg: logpage_entries: ...,<br />,
## summary,<br />,...
logpage_entries: dateCreated,datePublished,dateUpdated,numChapters,numWords,status,title,author,description,category,genre,rating,warnings

Right now, FFDL is putting this right after the title and TOC pages (if present) and does show it in the epub TOC.

FFDL adds new entries to the bottom of the log.

If the logpage is present in the epub, but include_logpage has been turned to false, on the next update, the logpage will be removed and the accumulated log lost even if it's turned back on later.

I think this is handy. When there's an update for an old story I often wonder when the last previous update was. Or some authors change the description a lot, etc.

[JimmXinu]

Version 1.6.3 - 06 Aug 2012

• Optionally keep an Update Log of past updates (epub only). See personal.ini include_logpage option.
• Fix for nha.magical-worlds.us stories with more than 9 chapters.
• 'Get Story URLs from Web Page' now uses AO3 user/pass if saved in personal.ini.
• 'Collection' style AO3 story URLs now work.

[Firedancer885]

Jimm,

when I check Fleschgrade in the Count Pages tab in the configuration window, it get's unchecked automatically. I have both GunningFog and FleschReading checked. Is this a bug or is there a reason for this? I set all three up in the Count Pages plugin. When I use the Count Pages plugin to update the values, all three are updated.
I seem to remember a conversation about this, but coudn't find the relevant pages.

[JimmXinu]

That's a bug. I'll get it fixed in the next version.

[Jade Aislin]

Is anyone else having trouble accessing www.ncisfiction.com? When I tried to update a story from there I get the error message: HTTP Error 503: Service Temporarily Unavailable. I think its a problem with the site itself, but couldn't find any information when I googled it. I get the same error if I try to use the url myself. I was just wondering if anyone else couldn't access the site or if it was just me?

[Aleyst]

Originally Posted by Jade Aislin

Is anyone else having trouble accessing www.ncisfiction.com? When I tried to update a story from there I get the error message: HTTP Error 503: Service Temporarily Unavailable. I think its a problem with the site itself, but couldn't find any information when I googled it. I get the same error if I try to use the url myself. I was just wondering if anyone else couldn't access the site or if it was just me?

Handy site http://www.isup.me, tells you if it is just you (Which btw it isn't).

The site does appear to be down.

[JimmXinu]

www.ncisfiction.com appears up to to me.

If you want me to investigate more than that, post a story link exhibiting the problem.

[Jade Aislin]

I wanted to see if anyone else was having a problem and I couldn't find anywhere else to ask online for the site. I get the same message for whichever story I try.

As it seems that it is just me, I don't think you will be able to help. But, here are a couple urls:

http://www.ncisfiction.com/chapters.php?stid=00982
http://www.ncisfiction.com/chapters.php?stid=00420

Unfortunately, I am at a loss as to what to try next.

[JimmXinu]

The second story URL gave me "Error 503 : Service Temporarily Unavailable" clicking on it and they both do from FFDL.

That's due to load on the server. The server is responding, so FFDL can't wait longer for it.

Nothing for it but to try different times of the day and hope they get less traffic or more capacity.

[Jade Aislin]

I think it might be the new blocker that squidge.org is using. Squidge hosts ncisfiction.com and i was able to find a livejournal account to ask questions on. This is what the response I got when I gave them my IP address.

According to the new "blocker" that we input to protect ourselves from spammer and hackers, your machine seems to be infected by some sort of virus, that's trying to attack the server when you connect up.

Here's the info:

#: 1474 @: Fri, 10 Aug 2012 11:31:13 -0700 Running: 0.4.10a1

Score: 1
Violation count: 1
Why blocked: No known valid python clients or spiders. Infected machine.
Query: stid=03733
Referer:
User Agent: Python-urllib/2.7
Reconstructed URL: http:// www.ncisfiction.com /chapters.php?stid=03733

It sounds like their blocker sees the plugin as a virus. I'm not sure if I should reply to that message as it is in a public area or send a pm. I haven't tried any other squidge site with the plugin to see if the same things happen.

[JimmXinu]

You can try this version which uses a different User-agent, but I still get:

HTTP Error 503: Service Temporarily Unavailable

Since I get the same thing in the browser, I assume their server is overrun.

Until you start getting 400 level HTTP errors instead of 500, this is all I'm prepared to do for this problem.

[JimmXinu]

@Jade Aislin - FYI, I'm able to download those stories from www.ncisfiction.com now without the special User-agent.

How's it working for you?

[Jade Aislin]

I've been talking with the people at the site and they took some of the blockers of a few sections, including story.php, which worked. Unfortunately, I can't get past the home screen, so I can't see the stories or get their urls. I was waiting until that problem was fixed before I posted here. It happens everytime, even the middle of the night, so I doubt its an over use problem.

[yin.]

So, I'm fairly new to this plugin and to calibre features in general and I was wondering if someone here could help me set up some features and options.
I use the plugin mostly for stuff on ffnet and I'd like to use the series information from the site as part of the file name it generates. I haven't completely thought this over yet, but I think I'd prefer my file names formatted as <series> - <author> - <title> and without having them sorted into author sub-folders.
More to the point, I'd like to dump the files into a single directory in my dropbox folder because that makes for very convenient syncing to my phone, where I can then import the entire folder into my reader (I use Moon+ Reader on my Android phone).
Any advice?
Help in this would be greatly appreciated.

[JimmXinu]

@yin - All of that is calibre stuff, not FFDL stuff.

But here's a few hints of where to look that may help you:

This thread explains that you can't change the folder structure of the calibre library.

However, you can use the "Connect to folder" functionality in calibre to point to a folder within your Dropbox and plugboards to control the file names.