[GUI Plugin] FanFicFare

[John Maverick]

Since 3 weeks ago I can't update or download stories any from ff.net using FlareSolverr. This is the error shown:

Estado
Título
Autor
Comentario
URL
Error
Unknown
Unknown
Connection to flaresolverr proxy server failed.  Is flaresolverr started?
https://www.fanfiction.net/s/14321954/1/Dragon-Ball-M-Mirai-New-Future

Regardless what the error message said FlareSolverr is running. But I think that is the default failure message.

Is anyone else having the same problem?

My setup is the following:
OS: Linux Mint 21.2 x86_64
Calibre version: 7.4
FanFicFare version: v4.30.0
FlareSolverr version: 3.3.13

Like always thanks in advance.

[JimmXinu]

Originally Posted by John Maverick

Since 3 weeks ago I can't update or download stories any from ff.net using FlareSolverr. This is the error shown:
...

Assuming you are using the default port numbers and running both on the same computer, I would suspect that there's a firewall module or something blocking the connection.

[mmmmmmmm]

Thank you for responding so fast! and for the helpful pointers!!
i'm sorry i wasnt able do respond fast too, but i really do apreciate it!

i'll put my Personal.ini here (thanks for showing me how to do that, its really cool!):

Spoiler:

## This is an example of what your personal configuration might look
## like.  Uncomment options by removing the '#' in front of them.

[defaults]
## [defaults] section applies to all formats and sites but may be
## overridden at several levels.  See
## https://github.com/JimmXinu/FanFicFare/wiki/INI-File for more
## details.
extra_valid_entries:fandoms



## Some sites also require the user to confirm they are adult for
## adult content.  Uncomment by removing '#' in front of is_adult.
is_adult:true

## Don't like the numbers at the start of chapter titles on some
## sites?  You can use strip_chapter_numbers to strip them off.  Just
## want to make them all look the same?  Strip them off, then add them
## back on with add_chapter_numbers.  Don't like the way it strips
## numbers or adds them back?  See chapter_title_strip_pattern and
## chapter_title_add_pattern in defaults.ini.
strip_chapter_numbers:true
add_chapter_numbers:true


[epub]
## Include images from img tags in the body and summary of stories.
## Images will be converted to jpg for size if possible.  Images work
## in epub format only.  To get mobi or other format with images,
## download as epub and use Calibre to convert.
## true by default, uncomment and set false to not include images.
include_images:true

## If set false, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
## true by default, uncomment and set false to not keep summary html.
#keep_summary_html:true

## If set true, and there isn't a specific cover image, the first
## image found in the story will be made the cover image.  If
## keep_summary_html is true, images in the summary will be before any
## in chapters.
## true by default, uncomment and set false to turn off
make_firstimage_cover:false 


## Most common, I expect will be using this to save username/passwords
## for different sites.  Here are a few examples.  See defaults.ini
## for the full list.


[archiveofourown.org]
username:XXXXXXXX
password:XXXXXXXX
is_adult:true


## This section will override anything in the system defaults or other
## sections here.
[overrides]
## default varies by site.  Set true here to force all sites to
## collect series.
collect_series: true

extra_valid_entries:fandoms, freeformtags, freefromtags,  ao3categories, comments, chapterslashtotal, chapterstotal, kudos,  hits, bookmarks, collections, byline, bookmarked, bookmarktags,  bookmarksummary, bookmarkprivate, bookmarkrec, restricted, series00,  series01, series02, series03, series04, series05, series06, series07,  series08, series09, series00Url, series01Url, series02Url, series03Url,   series04Url, series05Url, series06Url, series07Url, series08Url, series90Url,  series00HTML, series01HTML, series02HTML, series03HTML, series04HTML,  series05HTML, series06HTML, series07HTML, series08HTML, series09HTML 
fandoms_label:fandoms 
freeformtags_label:Freeform Tags 
freefromtags_label:Freeform Tags 
ao3categories_label:AO3 Categories 
comments_label:Comments 
chapterslashtotal_label:Chapters/Total Chapters 
chapterstotal_label:Total Chapters 
## kudos_label:Kudos 
## hits_label:Hits 
## collections_label:Collections 
## Count of bookmarks on story by all users bookmarks_label:Bookmarks 
## Tags & Summary from *your* bookmark on the story.  Only collected 
## if always_login:true bookmarked_label:I Bookmarked Story bookmarktags_label:My Bookmark Tags 
## bookmarksummary_label:My Bookmark Summary 
## bookmarkprivate_label:My Bookmark Private 
## bookmarkrec_label:My Bookmark Rec 
## restricted_label:Restricted to Registered Users 
series00HTML_label:Series 
series01HTML_label:Additional Series 
series02HTML_label:Additional Series 
series03HTML_label:Additional Series 
series04HTML_label:Additional Series 
series05HTML_label:Additional Series 
series06HTML_label:Additional Series 
series07HTML_label:Additional Series 
series08HTML_label:Additional Series 
series09HTML_label:Additional Series  
make_linkhtml_entries:series00,series01,series02,series03,series04,series05,series06,series07,series08,series09  
titlepage_entries: seriesHTML,category,genre,language,characters,ships,status,datePublished,dateUpdated,dateCreated,rating,warnings,numChapters,numWords,site,description extra_titlepage_entries: fandoms,freeformtags,ao3categories,comments,chapterslashtotal,chapterstotal,kudos,hits,bookmarks,bookmarked,bookmarktags,bookmarksummary,series01HTML,series02HTML,series03HTML,series04HTML,series05HTML,series06HTML,series07HTML,series08HTML,series09HTML,byline  



custom_columns_settings: 
 fandoms:=>#fandom,a

should i just change everything back to the default and go from there?
----
you mentioned FFF usually calls AO3's Free Form Tags and Categories in genre and Fandoms in category. does that mean i should change the names of my columns?
----
i dont know how to use include_subject_tags but i will check the wiki.
----

You might want to use the GUI Custom columns settings to put Category into your custom column instead of the AO3 specific fandoms. It's up to you.

i dont understand what you mean by this? should i re name my fandom column to 'category'?
-----

AO3 really does not make it easy to wrangle its metadata. The site lets authors put whatever they want into most any field they want, without regard to correct placement, consistency, or even accurate spelling.

i can see how that makes things more complicated!

-------
here is a story ive tried downloading:
https://archiveofourown.org/works/24280306

problems:
- nothing shows up in my fandoms colomn.
- all the tags from all the categories are showing up in my tags column
- nothing shows up in content rating (mature, teen and up, etc. )
- the relationships column sort of works. 'characterA/characterB' tags show up fine, but 'characterA & characterB' tag doesnt really work? like it shows up, but as separate tags? like 'characterA' & 'characterB', so i cant filter by those tags. is there a way to fix that?
- nothing shows up in my category column (M/M, F/F, etc.) but you mentioned FFF calls them genre? should i rename the coulumn? im happy to do that, if it will help


Thank you for your time and effort!

[JimmXinu]

Originally Posted by mmmmmmmm

should i just change everything back to the default and go from there?

Honestly, it would probably be easier. From what I see, this is all you have for real setting changes:

Spoiler:

[defaults]
is_adult:true

strip_chapter_numbers:true
add_chapter_numbers:true

custom_columns_settings: 
 fandoms=>#fandom,a

[epub]
make_firstimage_cover:false

[archiveofourown.org]
username:XXXXXXXX
password:XXXXXXXX

Originally Posted by mmmmmmmm

you mentioned FFF usually calls AO3's Free Form Tags and Categories in genre and Fandoms in category. does that mean i should change the names of my columns?

You can name your custom columns whatever you want. FanFicFare doesn't populate custom columns based on names, only what you configure.

There's two ways to configure FFF to populate a custom column, the Custom Columns tab in FFF Config lists your custom columns and will let you select an FFF standard metadata entry to put in each.

Standard metadata entries are the ones used by all sites/stories.

Site specific metadata entries can only be set to custom columns using custom_columns_settings and you must have the 'Allow custom_columns_settings from personal.ini to override' checkbox enabled (toward bottom of Custom Columns tab).

Unless AO3 is the only site you read, you're probably better served filling your 'Fandoms' column with Category for all sites instead of just site specific fandoms for AO3.

Originally Posted by mmmmmmmm

here is a story ive tried downloading:
https://archiveofourown.org/works/24280306

problems:
- nothing shows up in my fandoms colomn.
- all the tags from all the categories are showing up in my tags column
- nothing shows up in content rating (mature, teen and up, etc. )
- the relationships column sort of works. 'characterA/characterB' tags show up fine, but 'characterA & characterB' tag doesnt really work? like it shows up, but as separate tags? like 'characterA' & 'characterB', so i cant filter by those tags. is there a way to fix that?
- nothing shows up in my category column (M/M, F/F, etc.) but you mentioned FFF calls them genre? should i rename the coulumn? im happy to do that, if it will help

Taken in order:

Fandoms column: Your custom_columns_settings is still broken--need remove the extra colon. It's just => not :=>. Also make sure the 'Allow override' checkbox is checked.

Tags: Unless you change what FFF puts in Tags using the extra_subject_tags setting, that's the default.

Content rating: Not going to be put in any column unless you configure it. FFF labels it just Rating in GUI or lowercase rating for custom_columns_settings. (Calibre's built-in Rating column is completely different--that's a 1-5 stars rating.)

Relationships '/' vs '&' -- As part of each "Comma separated text" custom column definition there's a checkbox for "Contains names". If checked, Calibre automagically splits on '&'.

Category column (M/M, F/F, etc): FFF records that as site specific ao3categories and includes it (and freeformtags) in the standard entry genre.

Saved Metadata Column
FYI, since you are apparently just starting out, I also recommend creating a Long Text type custom column for FanFicFare Metadata and setting FFF to use it (toward bottom of Custom Columns tab).

Then whenever you download or update a story with FFF, it's raw metadata will be saved there and you can use FFF's "Update Calibre Metadata from Saved Metadata Column" mode to update existing books as your library configuration evolves without hitting the servers again.

[Homer55]

Hi!, hope someone can help (if it's even possible).

is there a way to make sure fanficfare only checks for updates after certain chapters? I found some stories that get separated by books each with it's own cover and stuff but the problem is that everything is in the same thread, and what I want to do is separate each book into it's own epub and make sure each book has it's own cover and stuff and leave those alone and only the last book would get updated everytime there's a new chapter but the problem is, if I delete all previous chapters, fanficfare detects that there are less chapters than the total so it downloads them and I just want it to download the new ones. Is there a way to tell it, ignore everything from chapter # and back? and only inject the new ones at the end?

[JimmXinu]

You can set first_chapter:50 (or whatever) under story URL section in personal.ini and FFF will treat it as if you'd used <storyURL>[50-] every time.

This is technically 'undocumented', but it works.

There's also an ignore_chapter_url_list feature.

[John Maverick]

Originally Posted by JimmXinu

Assuming you are using the default port numbers and running both on the same computer, I would suspect that there's a firewall module or something blocking the connection.

I checked the firewall of my PC but didn't find any different. Also tried reinstalling the Docker container of FlareSolverr, but is still giving me the same result.

Also visiting localhost:8191 in the browser shows FlareSolverr running.

I'll try to see what more could be wrong in my setup.

As always thanks for your help

[Homer55]

Originally Posted by JimmXinu

You can set first_chapter:50 (or whatever) under story URL section in personal.ini and FFF will treat it as if you'd used <storyURL>[50-] every time.

This is technically 'undocumented', but it works.

There's also an ignore_chapter_url_list feature.

This worked wonders! thank you so much!

[DaPoke]

Is there a way to rename tags for websites that tend to use abbrievated versions. Like for example rename a tag "abc" => "an actual tag name"

[JimmXinu]

Yes, you can search this thread for replace_metadata, there's also a wiki page ReplaceMetadata.

[DaPoke]

Originally Posted by JimmXinu

Yes, you can search this thread for replace_metadata, there's also a wiki page ReplaceMetadata.

Ah perfect thank you! Putting my answer here for others to see!

I had to actually not to check the genre,category metakeys, but the extratags

Something like this:

replace_metadata:
extratags=>\bsf\b=>science fiction

I had to include the regex to match whole words using `\b` otherwise it would process the next ones and tags would be a weird screwed up mix of replacements.

[mmmmmmmm]

Originally Posted by JimmXinu

Content rating: Not going to be put in any column unless you configure it. FFF labels it just Rating in GUI or lowercase rating for custom_columns_settings. (Calibre's built-in Rating column is completely different--that's a 1-5 stars rating.)

Category column (M/M, F/F, etc): FFF records that as site specific ao3categories and includes it (and freeformtags) in the standard entry genre.

Saved Metadata Column
FYI, since you are apparently just starting out, I also recommend creating a Long Text type custom column for FanFicFare Metadata and setting FFF to use it (toward bottom of Custom Columns tab).

thank you! i've been able to fix most of them now!

im still struggling with Content rating and Category/rating.

i went on to the 'custom columns' section and set that column to 'rating' from the drop down menu. that didnt fix it

i also tried fixing them in the custom_columns_settings:

Spoiler:

custom_columns_settings: 
 fandoms =>#fandom,a
 genre =>#genre,a
 rating =>#rating,a

thanks for letting me know about the Saved Metadata Column, that sounds super helpful and i've set it up now!

[JimmXinu]

I'd double check that you have the custom column identifiers correct for your created columns (#genre vs #genres etc) and that column types are appropriate. You can't put text/tags in an integer field, for example.

[John Maverick]

Originally Posted by JimmXinu

Assuming you are using the default port numbers and running both on the same computer, I would suspect that there's a firewall module or something blocking the connection.

I tested using FlareSolverr from the cli with curl, to see if this is a firewall problem or anything else. It can get a susesful response with webs that don't have Cloudflare challenges, like google.com or php.net.

But when trying to do the same with a story of ff.net like this:

curl -L -X POST 'http://localhost:8191/v1' \
-H 'Content-Type: application/json' \
--data-raw '{
  "cmd": "request.get",
  "url":"https://www.fanfiction.net/s/14321954/1/Dragon-Ball-M-Mirai-New-Future",
  "maxTimeout": 120000
}'

I get this response:

{"status": "error", "message": "Error: Error solving the challenge. Timeout after 120.0 seconds.", "startTimestamp": 1706570463731, "endTimestamp": 1706570584578, "version": "3.3.13"}

It seems more like for some reason FlareSolverr can't resolve the challenge. But is strange that this seems only happening to me. Anyone else is getting similar errors?

[JimmXinu]

Depending on your download volume and habits, and possibly your network neighbors' habits, your IP or network may be on a list for more aggressive blocking.

I use Browser Cache instead of Flaresolverr because I find it more reliable, if less convenient.

I might look at improving the error reporting, however. I thought FFF reported back any error message it got from Flaresolverr.

UPDATE:

Actually, it looks like Flaresolverr doesn't work for me either with that story URL. The arms race continues.