[GUI Plugin] FanFicFare

[JimmXinu]

Originally Posted by ILB

Is there a possibility to not only replace but to add metadata?

A possibility, yes.

But you're going to have to explain what you want in more detail if you want help.

[ILB]

Thanks!
I thought I ask if it is possible at all before I try to put it into words.
It's already hard to put it in words in German but then also translate into understandable English...

For several fandoms there are different categories (for example based on the book or the movie/series or repeatedly filmed series/movies) :
Sherlock = at fanfiction.net it´s mostly used for the BBC series
Sherlock Holmes = it´s mostly for the Books from A.C. Doyle or maybe some of the older tv series
To make a consistent search possible I even the category by
" category=>^Sherlock$=>Sherlock Holmes(All Media Types)&&category=>^Sherlock$ "
in my personal.ini

I´d like to maintain this new category but also want to keep the original classification and/or to add an (or more) additional category, like "Sherlock Holmes (BBC)".


(Or would it be an easier/better possibility to make a top category and use the original classifications as an subcategory? )

[JimmXinu]

Originally Posted by ILB

For several fandoms there are different categories (for example based on the book or the movie/series or repeatedly filmed series/movies) :
Sherlock = at fanfiction.net it´s mostly used for the BBC series
Sherlock Holmes = it´s mostly for the Books from A.C. Doyle or maybe some of the older tv series
To make a consistent search possible I even the category by
" category=>^Sherlock$=>Sherlock Holmes(All Media Types)&&category=>^Sherlock$ "
in my personal.ini

I´d like to maintain this new category but also want to keep the original classification and/or to add an (or more) additional category, like "Sherlock Holmes (BBC)".

(Or would it be an easier/better possibility to make a top category and use the original classifications as an subcategory? )

 category=>^Sherlock$=>Sherlock Holmes(All Media Types)&&category=>^Sherlock$

You can't use the same metadata key word (category) in both the replacement part (before &&) and the conditional part (after &&) of a replace line. Also, it isn't needed in this case. The replacement part does the same thing the conditional would.

I think you want something more like:

 category=>^(.*Sherlock.*)$=>Sherlock Holmes(All Media Types)\,\1

That says: "Any category that contains 'Sherlock' with anything before and/or anything after it, change to two entries, 'Sherlock Holmes(All Media Types)' and whatever it was before (the '\1').

I don't use hierarchical tags, but from what I understand, you could do this:

 category=>^(.*Sherlock.*)$=>Sherlock Holmes(All Media Types).\1

[ILB]

Originally Posted by JimmXinu

 category=>^Sherlock$=>Sherlock Holmes(All Media Types)&&category=>^Sherlock$

You can't use the same metadata key word (category) in both the replacement part (before &&) and the conditional part (after &&) of a replace line. Also, it isn't needed in this case. The replacement part does the same thing the conditional would.

Oh, I copied it from my "characters" line and only changed it to category. And after it worked I did not think further...

Originally Posted by JimmXinu

I think you want something more like:

 category=>^(.*Sherlock.*)$=>Sherlock Holmes(All Media Types)\,\1

That says: "Any category that contains 'Sherlock' with anything before and/or anything after it, change to two entries, 'Sherlock Holmes(All Media Types)' and whatever it was before (the '\1').

That worked.

Originally Posted by JimmXinu

I don't use hierarchical tags, but from what I understand, you could do this:

 category=>^(.*Sherlock.*)$=>Sherlock Holmes(All Media Types).\1

Since the multiple category works - I don´t think I will use/need hierarchical tags. Anyway thanks for the effort!

[Balthanon]

First, I'd like to say that I've found this add-on quite useful-- it's well done and has replaced the stand alone tool I used to use for saving stories at risk of disappearing. That said, I have a potential enhancement that I would like to suggest.

I haven't read through the whole thread, so this may have been asked in the past, but would it be possible to generate a list of stories to download based upon your follows, favorites, bookmarks, etc... from various websites? I'm personally interested in pulling all my followed stories from fanfiction.net (1200 or so), but most sites have something of that nature. I haven't really found a good method of simply generating all of those as a list of URLs (a couple of unofficial APIs I may try), so I thought I would check to see if you are open to an enhancement.

Ideally, I would just like to put in a user name and not need to bother signing in through the app. This also opens the possibility of finding a user you like and just pulling their list of follows, favorites, or even stories they've written. Lots of possibilities ultimately, though I imagine each adds complexity.

[JimmXinu]

@Balthanon - FanFicFare has a feature "Get Story URLs from Web Page" that you can use to get lists of URLs from author pages or favorite pages.

For ffnet specifically however, it's a little more complicated. Because they use javascript on the author page to switch between author's stories vs author's favorites, you have to use the mobile URLs instead. Change the www to m. Examples:

https://m.fanfiction.net/u/00000/Author-Name
https://m.fanfiction.net/u/00000/Author-Name?a=fs

Because your own followed/alert list is behind a login block, FanFicFare can't read it. (FanFicFare doesn't login to ffnet because a) it doesn't need to to download stories and b) captcha.)

However, you should also be aware that ffnet in particular has a history of blocking users who download 'too many' stories at a time. What exactly is 'too many' isn't known. There's an FAQ about it.

For long term, I recommend using email notices to update your library.

[MCBastos]

Is http://www.ficsite.com/ being considered for support?

[JimmXinu]

Originally Posted by MCBastos

Is http://www.ficsite.com/ being considered for support?

It's on the "Insufficient interest/updates" list.

Then, it had 508 stories and no updates in 6 months. Now it has 506 stories, most recent update appears to be July and it looks buggy (see attached).

I'll accept code if somebody else wants to write support for it, but I'm not interested in doing it. Sorry.

[wildfire142]

getting a

<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

for any storiesonline.net stories I try to download

[davidfor]

Originally Posted by wildfire142

getting a

<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

for any storiesonline.net stories I try to download

It isn't happening here. That error should means a certificate problem. Looking at the certificate, it has been renewed recently. They might have only just installed the new certificate.

[wildfire142]

Originally Posted by davidfor

It isn't happening here. That error should means a certificate problem. Looking at the certificate, it has been renewed recently. They might have only just installed the new certificate.

Just tried again and still getting the error running calibre 2.69 and v 2.4.4 of the plugin.

[JimmXinu]

Originally Posted by wildfire142

<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

for any storiesonline.net stories I try to download

I'm not getting that error from plugin (calibre 2.69 on Win10), the app service, CLI on Win10 or CLI on CentOS 6.8.

So it's either a temporary issue that was on the server site, or a problem with your system's certs. Both of which are outside my control.

I don't know if it's still current, but I posted some advice last time somebody had this problem you might try.

[wildfire142]

thanks will look for other fixes

Found a fixed logged into the site using edge and now the cert is good, normally use Firefox for browsing

[Balthanon]

Originally Posted by JimmXinu

@Balthanon - FanFicFare has a feature "Get Story URLs from Web Page" that you can use to get lists of URLs from author pages or favorite pages.

For ffnet specifically however, it's a little more complicated. Because they use javascript on the author page to switch between author's stories vs author's favorites, you have to use the mobile URLs instead. Change the www to m. Examples:

...

However, you should also be aware that ffnet in particular has a history of blocking users who download 'too many' stories at a time. What exactly is 'too many' isn't known. There's an FAQ about it.

For long term, I recommend using email notices to update your library.

Thanks, that actually was very helpful, because I can work around the limitation for the follows by downloading the HTML for the mobile page. (With a little manipulation to address the relative URLs.) Not sure why I never thought of going to the mobile site for those, since I've done it in the past for some authors with very large favorite lists.

Is there any way to throttle FanFicFare so that you enter a web page and it downloads say 5 stories, waits 15 minutes to a half an hour, downloads 5 more, etc...?

Not sure I want to enter my email password into Calibre, no offense to your plugin. (As you note in the description of the feature, it does open up the password to other plugins as well.)

[JimmXinu]

Originally Posted by Balthanon

Is there any way to throttle FanFicFare so that you enter a web page and it downloads say 5 stories, waits 15 minutes to a half an hour, downloads 5 more, etc...?

There's no batching facility, but you can increase the slow_down_sleep_time parameter in personal.ini. From defaults.ini:

[defaults]
## number of seconds to sleep between calls to the story site.  May be
## useful if pulling large numbers of stories or if the site is slow.
#slow_down_sleep_time:0.5

[www.fanfiction.net]
## fanfiction.net is blocking people more aggressively.  If you
## download fewer stories less often you can likely get by with
## reducing this sleep.
slow_down_sleep_time:4

So you could bump it up even higher than 4. (There are additional hard coded sleeps in the ffnet adapter, so don't be surprised if it takes longer.)

Originally Posted by Balthanon

Not sure I want to enter my email password into Calibre, no offense to your plugin. (As you note in the description of the feature, it does open up the password to other plugins as well.)

I agree. Personally, I created a separate email account that I forward all my update notices to just for calibre to read from.