[GUI Plugin] FanFicFare

[dakota1979]

I have the same problem, can anybody help or have solution for this ?? I'd like to download stories from fanfiction.net and archiveofourown

Originally Posted by Tanjamuse

Does anyone know what the minimum "slow-down"/delay you can have on FF.net is?

I have slow_down_sleep_time:4 and I'm now wondering if it's changed from 4 seconds to 4 minutes, because it takes me a very long time to download 25 stories.

Does anyone else have this problem?

It went fine on monday but yesterday and today has been very very slow, and I haven't changed any settings on the computer and it was rebooted yesterday.

[McHunter]

Originally Posted by dakota1979

I have the same problem, can anybody help or have solution for this ?? I'd like to download stories from fanfiction.net and archiveofourown

The Problem seems to be by FF.net, some times it takes an eternaty to resond,
I hat the same problem to as I was reading an story online.

[JimmXinu]

I'm not seeing any particular ffnet slow down right now, but I only downloaded one story.

Sounds like there are/were site slow downs.

[Tanjamuse]

For me it still seems to take a lot longer than normal.

I can usually download 25 stories in about 3-4 minutes but now it takes at least 8-15 minutes.

[jennifer_voorhee]

An error occurred today when trying to download a fic (https://www.fanfiction.net/s/691414/1/). The error occurred in both the Calibre plugin as well as on http://fanficfare.appspot.com (which is my go-to if I have trouble downloading in Calibre for some reason).

On appspot, the error given is simply "maximum recursion depth exceeded."

In Calibre, the error is: maximum recursion depth exceeded while calling a Python object.

[JimmXinu]

Originally Posted by jennifer_voorhee

An error occurred today when trying to download a fic (https://www.fanfiction.net/s/691414/1/). The error occurred in both the Calibre plugin as well as on http://fanficfare.appspot.com (which is my go-to if I have trouble downloading in Calibre for some reason).

On appspot, the error given is simply "maximum recursion depth exceeded."

In Calibre, the error is: maximum recursion depth exceeded while calling a Python object.

The story HTML is incorrect. Every story line is of the form:

<p class=MsoNormal><b style='mso-bidi-font-weight:normal'> text</strong></p>

Note that it opens a <b> tag, but closes a </strong> tag. (The same thing happens with strong and </em>. I would speculate that it was a (broken) attempt by ffnet back in 2002 to convert all <b/i> tags to <strong/em> tags that didn't take attributes into account.)

So as far as the HTML parser is concerned, each and every line is nested inside the line above it, and when it gets to chapter 10, there are enough nested lines to trip the maximum recursion depth limit.

I don't immediately see any reasonable way to handle that problem in general, and I assume story HTML that broken is going to be rare enough that it's not worth thinking about any harder.

But since it's pretty trivial for me as a developer to temporarily change the code to fix that specific problem and run the download, here's an epub of it for you.

[JimmXinu]

@version365 raised a question about how merged/anthology books' metadata assigned to numeric columns are handled. I explained the current state and now I'm moving the conversation over here.

So, the basic issue is: FanFicFare assumes that each metadata in an anthology assigned to an int or float custom column should be the summed values from the individual stories. version365 wants to be able have at least one metadata averaged instead.

Actually averaging instead of summing is easy--the complicated part was deciding where in all the different configuration settings it should be controlled from.

After considering several ideas, I decided it needed to be in INI as part of custom_columns_settings. With the latest test version, you can use modes of 'r_anthaver' and 'n_anthaver', which are identical to 'r' and 'n' except for anthologies where they will average the metadata instead of sum.

The average only considers stories that a value for that metadata. So if you have 5 stories in the anthology and the first two have values 1 and 3 while other three don't have that metadata item at all (say they come from a different site), the average will be (1+3)/2, not (1+3+0+0+0)/5.

Spoiler:

## (Plugin Only) - You can also populate calibre custom columns with
## the site specific metadata using custom_columns_settings (but only
## if 'Allow custom_columns_settings from personal.ini' is checked in
## the plugin GUI config.)  There are three parts, the entry name,
## then the label of the calibre custom column, then (optionally) a
## 'mode'.  'r' to Replace any existing values, 'a' to Add to existing
## value (use with tag-like columns), and 'n' for setting on New books
## only.  (Default is 'r'.)

## Literal strings can be set into custom columns using double quotes.
## Each metadata=>column mapping must be on a separate line and each
## needs to have one space at the start of each line.

## 'r_anthaver' and 'n_anthaver' can be used to indicate the same as
## 'r' and 'n' for normal downloads, but to average the metadata for
## the differents story in an anthology before setting in integer and
## float type custom columns.  This can be useful for a averrating
## column, for example.  Default is to sum the values of all stories,
## and numChapters and numWords are always summed.

#custom_columns_settings:
# cliches=>#acolumn
# themes=>#bcolumn,a
# timeline=>#ccolumn,n
# "FanFiction"=>#collection
# averrating=>#averrating,r_anthaver

New test version posted (direct link)

2016-07-12

- Update translations.
- Add n_anthaver and r_anthaver modes to custom_columns_settings for averaging metadata for anthologies before setting in integer and float calibre custom columns.

[snakeling]

Hi Jimm, thanks so much for the plugin, it's completely changed the way I read fanfic.

Would you be willing to add support for snarry.net? As far as I can see, it's a bog standard efiction website.

I have another request, though I'm not sure whether it should go here or in the EpubMerge thread: when I make an anthology epub, the first epub is in the top folder while every other epub is in its own subfolder. Unfortunately my reader (Pocketbook Touch Lux 2) can't cope with that and only shows me the first epub. I get around it by converting the anthology to epub, which flattens it. Would it be possible to add an option to flatten it automatically?

Thanks again

[JimmXinu]

Originally Posted by snakeling

Hi Jimm, thanks so much for the plugin, it's completely changed the way I read fanfic.

Glad you like it.

Originally Posted by snakeling

Would you be willing to add support for snarry.net? As far as I can see, it's a bog standard efiction website.

snarry.net has been requested before. It's very small (279 stories) and it isn't 'bog standard' enough that the eFiction base adapter will work. So I'm not interested in coding support for it myself.

But as usual, I will accept a code contribution if somebody else wants to. The source is at github.

Originally Posted by snakeling

I have another request, though I'm not sure whether it should go here or in the EpubMerge thread: when I make an anthology epub, the first epub is in the top folder while every other epub is in its own subfolder.

Actually, it isn't. For reasons I've never completely understood, the Calibre Edit Book feature chooses (by default) to hide the internal file paths from you as much as it can, which translates to hiding the path of the first merged book.

There's an option in the Edit Book preferences under 'Main Window' > 'Show full file paths in File Browser' that will cause it to show you the true structure; each merged book is in it's own subdirectory.

(Sigil was worse, the last I looked.)

Originally Posted by snakeling

Unfortunately my reader (Pocketbook Touch Lux 2) can't cope with that and only shows me the first epub. I get around it by converting the anthology to epub, which flattens it. Would it be possible to add an option to flatten it automatically?

Flattening the anthology epub would break the un-merge functionality that's used for updating existing anthology epub. And the epub structure is correct according to standard. So as long as you have a work around that works, I'd prefer not to.

[eschwartz]

It makes a lot of sense, actually. calibre doesn't damage the structure at all, and it (rightfully) assumes most people editing their books are less concerned about the filepath and more concerned about finding the correct file to edit. Which is usually easier to find based on filename.
That's what options are for... who cares what the default is, as long as you know how to change it?

[ILB]

If I add some names in column #characters and next use "Update Calibre Metadata from Web Site" my entry disappears and only the downloaded names from the Website remain.
Where in my personal.ini can I specify that downloaded metadata is always added to existing metadata and not overwritten? For every column?
(at tags it works perfect)

[JimmXinu]

Originally Posted by ILB

If I add some names in column #characters and next use "Update Calibre Metadata from Web Site" my entry disappears and only the downloaded names from the Website remain.
Where in my personal.ini can I specify that downloaded metadata is always added to existing metadata and not overwritten? For every column?
(at tags it works perfect)

You'll need to use custom_columns_settings in personal.ini instead of the Custom Columns tab settings and use the 'a' mode.

Spoiler:

## (Plugin Only) - You can also populate calibre custom columns with
## the site specific metadata using custom_columns_settings (but only
## if 'Allow custom_columns_settings from personal.ini' is checked in
## the plugin GUI config.)  There are three parts, the entry name,
## then the label of the calibre custom column, then (optionally) a
## 'mode'.  'r' to Replace any existing values, 'a' to Add to existing
## value (use with tag-like columns), and 'n' for setting on New books
## only.  (Default is 'r'.)

## Literal strings can be set into custom columns using double quotes.
## Each metadata=>column mapping must be on a separate line and each
## needs to have one space at the start of each line.

## 'r_anthaver' and 'n_anthaver' can be used to indicate the same as
## 'r' and 'n' for normal downloads, but to average the metadata for
## the differents story in an anthology before setting in integer and
## float type custom columns.  This can be useful for a averrating
## column, for example.  Default is to sum the values of all stories,
## and numChapters and numWords are always summed.

#custom_columns_settings:
# cliches=>#acolumn
# themes=>#bcolumn,a
# timeline=>#ccolumn,n
# "FanFiction"=>#collection
# averrating=>#averrating,r_anthaver

[ILB]

Originally Posted by JimmXinu

You'll need to use custom_columns_settings in personal.ini instead of the Custom Columns tab settings and use the 'a' mode.

Thanks!

Is there a possibility to automatically add my manual entries to "characters" in the "tags"column - maybe if I update the story?

[JimmXinu]

Originally Posted by ILB

Thanks!

Is there a possibility to automatically add my manual entries to "characters" in the "tags"column - maybe if I update the story?

Nothing that comes to mind that FanFicFare can do about that, but you can do it using Calibre's metadata find and replace. See attached.

(No promises that's exactly correct for your purposes, I only thought about it for a few minutes.)

[readerboy7]

Occasionally, a story thread on SpaceBattles (SB) will link to story posts omakes on SufficientVelocity (SV) or vice versa. When this happens, the omakes in the other forum don't get downloaded, which I find annoying. I can see that, generally speaking, getting FFF to follow every link would be a bad idea, but is it possible to edit my personal.ini so that FFF follows crosslinks to SV from SB and vice versa?