[GUI Plugin] FanFicFare

[rominatrix]

Hi there.

I've been trying to update a fanfic (http://archiveofourown.org/works/414...apters/9356727) that I had on my library but it failed to download. I thought it was something with my internet connection so I deleted it and tried to download it again. However this is the message I'm getting now:

'utf8' codec can't decode byte 0xf3 in position 77: invalid continuation byte

I am using Calibre 2.37.1 and fanficfare is 2.2.12 (it's the last version available according to calibre, cause it didn't show me any new updates and i always keep it updated).

Any ideas?

Thank you.

[Tanjamuse]

Hi Jim. I have a problem with Fanfiction.net.

I have the following lines in my personal.ini but they don't seem to be working very well, is there a specific order for how they should be posted? I have all the correct columns but it doesn't change the ampersand to the §.

join_string_characters:\s&\s
join_string_ships:\s&\s
join_string_category:\s&\s
keep_in_order_ships:true
keep_in_order_category:true
keep_in_order_characters:true
include_subject_tags:genre
add_genre_when_multi_category: Crossover
replace_metadata:
    category=>Dharma & Greg=>Dharma § Greg

Here's a story: https://www.fanfiction.net/s/8909297...e-and-Memories

[Firedancer885]

Hi Jimm,
I tried to download a story off the hpfandom site, and got this error:
'NoneType' object has no attribute 'findParent'
http://www.hpfandom.net/eff/viewstory.php?sid=49212

[JimmXinu]

Originally Posted by McHunter

My Understanding is, that they HAVE made the changes.
If I read that in the Forum correctly, they have done that sometime in July.

I've coded changes that I think address the mediaminer changes. I'm going to take a quick look at the other posted issues before uploading a test version, though.

[JimmXinu]

Originally Posted by Tanjamuse

I have the following lines in my personal.ini but they don't seem to be working very well, is there a specific order for how they should be posted? I have all the correct columns but it doesn't change the ampersand to the §.

Try & instead of &. That should work:

replace_metadata:
    category=>Dharma & Greg=>Dharma § Greg

[Tanjamuse]

That seemed to do the trick Thanks

[JimmXinu]

Originally Posted by Firedancer885

Hi Jimm,
I tried to download a story off the hpfandom site, and got this error:
'NoneType' object has no attribute 'findParent'
http://www.hpfandom.net/eff/viewstory.php?sid=49212

Use the URL to the story index or the first 'chapter' instead of a URL to later chapters. For that story: http://www.hpfandom.net/eff/viewstory.php?sid=42717

Unlike most sites, hpfandom.net doesn't use the 'storyId' in each chapter URL.

I'm looking into how that might be worked around, but there are difficulties.

[JimmXinu]

Originally Posted by rominatrix

I've been trying to update a fanfic (http://archiveofourown.org/works/414...apters/9356727) that I had on my library but it failed to download. I thought it was something with my internet connection so I deleted it and tried to download it again. However this is the message I'm getting now:

'utf8' codec can't decode byte 0xf3 in position 77: invalid continuation byte

What is your OS? We've seen a few people have similar errors on linux OS that we haven't been able to find yet.

Also, if you see the problem in a story that isn't 126 chapters with nearly 500 images, it would make it a lot easier to trouble shoot. I was able to download it successfully, but it's so big I'm not going to try more than once right now.

[JimmXinu]

New test version posted (direct link):

2015-09-08
- Fixes for changes to mediaminer.org site.

[Firedancer885]

Originally Posted by JimmXinu

Use the URL to the story index or the first 'chapter' instead of a URL to later chapters. For that story: http://www.hpfandom.net/eff/viewstory.php?sid=42717

Unlike most sites, hpfandom.net doesn't use the 'storyId' in each chapter URL.

I'm looking into how that might be worked around, but there are difficulties.

Thanks

[Jensus]

Hi Jim,

Today I tried FFF for the first time on www.fanfiction.de.

Example:

Chroniken der letzten Hochkönigin
Marlos-Mehilwa
HTTP Error 403: Forbidden
http://www.fanfiktion.de/s/50b8e0210002b86a06700fa0/1
Error

FFF v2.2.12, Calibre 2.36

After this trial I was banned via flood protection.

The funny thing... earlier today I succeeded with 8 out of 20 Articles.

Is there a way to finetune the time-outs?

Thanks a lot & kind regards from Germany,

Jens

[JimmXinu]

Originally Posted by JimmXinu

Use the URL to the story index or the first 'chapter' instead of a URL to later chapters. For that story: http://www.hpfandom.net/eff/viewstory.php?sid=42717

Unlike most sites, hpfandom.net doesn't use the 'storyId' in each chapter URL.

I'm looking into how that might be worked around, but there are difficulties.

It isn't promising.

Without hitting the server, FFF can't tell that the URL given is for a later chapter, not the start of the story. There's at least three ways that causes problems.

First, I can modify the site adapter to figure that the URL given isn't the 'story' URL. I have this code in hand and it allows downloads using chapter URLs.

Second, the plugin would need to be changed to allow the story URL to be changed after metadata fetch. For virtually all other sites, a 'normalized' story URL can be made without hitting the site and that's what's stored in the calibre DB.

Third, the Reject URL list check as is will not work correctly when you give a chapter URL. The reject list check would have to be later in the processing, changing the behavior and possibly causing other problems.

Perusing the existing code, however,, there are a small number of other site adapters that also set the story URL after fetching data. So I guess I'll have to think about this more deeply and come up with an actual solution.

[JimmXinu]

http://www.fanfiktion.de/s/50b8e0210002b86a06700fa0/1 has 514 chapters. If the others were that long, I'm not surprised you tripped a flood detector.

Originally Posted by Jensus

Is there a way to finetune the time-outs?

Yes, you can set the sleep time between network requests in personal.ini:

[www.fanfiktion.de]
## number of seconds to sleep between calls to the story site.  May by
## useful if pulling large numbers of stories or if the site is slow.
slow_down_sleep_time:2.5

[mehetabelo]

Adding a custom column entry on the title page doesn't seem to be working for me. I am not sure exactly when it quit working correctly, but I noted dates I could find below. I updated to the latest beta release just in case to see if it was something that hadn't been noted and repaired at some point, and that didn't help. Here is the problem and my process I've tried.

Something changed between July 7/18 and 7/25 that caused problems with the entry. I hadn't really noticed it because most of the time I was out of town and didn't spend time trying to troubleshoot it when I noticed the issue. I went back and found the date the last time it worked, (7/18) and the next time I updated something from one of the websites I use it for (7/25).

Next, I went into the threads to find out if something had changed, and it had apparently. Instead of the original command "add_to_titlepage_entries:,calibre_cust_words" words being the #column name (without #) apparently, according to a later post it is: "extra_titlepage_entries: calibre_cust_words". I made the modification to the new option and nothing changed.

When I do an update the last entry on a titlepage is now calibre_cust_words, this is in the line after the summary.
I actually tried both of the options together
"
add_to_titlepage_entries:,calibre_cust_words
extra_titlepage_entries: calibre_cust_words
"
... and get (as the last entry) "calibre_cust_wordscalibre_cust_words" so I'm not sure what's happening, as both options (add_to_titlepage_entries and extra_titlepage_entries) seem to add an entry to the title page, but it's not working with that column name.

I also tried another column instead of words, two separate ones and in each case the had an entry called misc (a misc column I use for notes, and a column called stage) and it did the same thing, showing "calibre_cust_misc" or "calibre_cust_stage" as the last entry on the titlepage, but not actually showing the data in the column. I even tried adding a #in front of the column name just in case. Same situation.

So now, after trying all those different options I'm stuck and I've come here for help. Has something changed that I missed? Or is this specific to me?

add note: hot damn that's a lot of chapters in the above mentioned fanfic.

edit: fixed specifics in first sentences.

[JimmXinu]

Originally Posted by mehetabelo

Adding a custom column entry on the title page doesn't seem to be working for me. ...

The only thing I can think of is that maybe you don't have the checkbox set to send existing calibre columns into FFF update. See attached.

As for extra_titlepage_entries: calibre_cust_words vs add_to_titlepage_entries:,calibre_cust_words, they are slightly different ways to allow additional configuration without replacing lengthy default lines. They were added at different times. You almost certainly should be using one or the other, but not both.

The extra_* params came first and only work on a few ini params. In this case, titlepage_entries. When titlepage_entries is used, both titlepage_entries and extra_titlepage_entries are fetched and concatenated. Both follow the ordinary section precedence rules for FFF ini params.

The ordinary precedence order for sections (using epub from dramione.org as an example) from least to most is:

1. [defaults]
2. [dramione.org]
3. [epub]
4. [dramione.org:epub]
5. [overrides]

A param key word in [defaults] is the most general version and will be 'overwritten' by the same key word in any lower section. The param key word in [overrides], the highest precedence section, beats everything else. (It actually works by looking in each section from [overrides] to [defaults] and stopping at the first one it finds.)

So you can set extra_titlepage_entries in [dramione.org] without overwriting titlepage_entries from [defaults]. But only once. If you set extra_titlepage_entries in [epub], it overwrites the version from [dramione.org]

The 'add_to_' mechanism was added later to both generalize it for all ini params and add more flexibility to allow more specific sections to 'add to' the ini params from more general sections without having to repeat everything. The interesting part is that each section's add_to_s are applied and from most general to most specific:

Here's an example from my own personal.ini:

[defaults]
add_to_replace_metadata:
 genre=>Comedy=>Humor
 category=>StarTrek=>Star Trek
 category=>Warhammer 40,000=>Warhammer 40k

[epub]
add_to_replace_metadata:
 fulldownload=>^.*$=>Yes

[forums.spacebattles.com]
add_to_replace_metadata:
 title=>Wail of the Banshee 3: Boxhead Blues and the Zombocalypse=>Wail of the Banshee

When I download a story from forums.spacebattles.com, my effective replace_metadata is:

replace_metadata:
#from [defaults]
 genre=>Comedy=>Humor
 category=>StarTrek=>Star Trek
 category=>Warhammer 40,000=>Warhammer 40k
#from [forums.spacebattles.com]
 title=>Wail of the Banshee 3: Boxhead Blues and the Zombocalypse=>Wail of the Banshee
#from [epub]
 fulldownload=>^.*$=>Yes

The 'gotcha' about 'add_to_' is that it's a string concatenation, which is why you need the include the leading ',' when it's a list like titlepage_entries instead of replace_metadata.

This really isn't as complex in practice as it is to explain. The code for it is only a few lines.