[GUI Plugin] FanFicFare

[JimmXinu]

Originally Posted by julo

I still cannot update/download from SOL and this file from StoryRoom neither.
...
I would appreciate you having a look. Do you need anything more?

Works for me. That's the same error as previously fixed. Try installing the test version again.

[Dood]

Doesn't work for me either.
Installed latest update. (twice)
FFF does nothing for a few minutes and then SOL pops up a login dialog.
personal.ini has the proper settings

[JimmXinu]

I'm going to need to see a debug log then.

[JimmXinu]

New Test Version Posted

2026-10-22
- Reduce debug output.
- Update translations.
- Add QuitDecorator to fetches to allow plugin prog bar quit to work better.

If you cancel the progress bar while getting metadata from sites, it will quit the current story too, not just after it finishes. Before, the current story metadata would finish in the background thread.

Makes the biggest difference on XF sites (SB/SV/etc) that can have a dozen hits just to gather the chapter list.

[julo]

Originally Posted by JimmXinu

Works for me. That's the same error as previously fixed. Try installing the test version again.

Re-install test version seems to do the trick. Thanks.

[Tharos]

If you get that message "FanFiction.Net Message Type 1, Chapter not found. Please check to see you are not using an outdated url."on FFN, you should try shortening the URL.

Example: https://www.fanfiction.net/s/14361429/1/ works better for me than https://www.fanfiction.net/s/14361429/1/One-Thousand-Hands-OC-Senju-SI.

Can I configure FFF to use the shortened URL when I use the cache feature?

[Dood]

Originally Posted by JimmXinu

I'm going to need to see a debug log then.

Hmmm, fired up Calibre in debug mode & downloads worked.
Maybe they tweaked the website.

[whiteowl]

Upon recommendation I started to download a bunch of books. Calibre's job indicator at the bottom of the Calibre's window shows that I have five active jobs right now.
Opening Job window, I see "Download 1 FanFiction Book(s) for forums.spacebattles.com". And I see all five jobs with the very same title...

Is it possible to change that universal title to something like "Downloading from <URL>"? It would be even better to show the book name, it should be available after reading metadata, but a simple URL would also be enough.

[JimmXinu]

Originally Posted by Tharos

If you get that message "FanFiction.Net Message Type 1, Chapter not found. Please check to see you are not using an outdated url."on FFN, you should try shortening the URL.

Example: https://www.fanfiction.net/s/14361429/1/ works better for me than https://www.fanfiction.net/s/1436142...ds-OC-Senju-SI.

Can I configure FFF to use the shortened URL when I use the cache feature?

From defaults.ini:

## ffnet sometimes will fail loading a chapter with the full story
## title in the URL, but work with a partial title.  Technically a
## different URL, but ffnet ignores the story title in URLs.  When
## try_shortened_title_urls is set true, FFF will try up to 4 times,
## shortening the title by one letter off each time.  Note that this
## can cause a lot of each attempts, especially if used with
## check_next_chapter:true and there isn't a 'next' chapter yet.
## See: https://github.com/JimmXinu/FanFicFare/issues/1166
try_shortened_title_urls:false

This isn't new. And whether any particular shortened URL works or not appears to be random. IE, /1/ alone may work sometimes, but other people or times may see /1/whatever or /1/whateve, etc work instead.

Originally Posted by whiteowl

Upon recommendation I started to download a bunch of books. Calibre's job indicator at the bottom of the Calibre's window shows that I have five active jobs right now.
Opening Job window, I see "Download 1 FanFiction Book(s) for forums.spacebattles.com". And I see all five jobs with the very same title...

Is it possible to change that universal title to something like "Downloading from <URL>"? It would be even better to show the book name, it should be available after reading metadata, but a simple URL would also be enough.

Unless something has gone wrong, FFF starts one background job per site (eg forums.spacebattles.com), not one job per story. The idea is to allow parallel download from different sites, but not the same site to trigger blocking less often.

At least, if you've started them all at the same time. If you deliberately started each individually to force parallel download from the same site, you're on your own when it comes to blocking.

But showing the book title when there is only one story download in the job is easy enough. I'll included that in the next test version.

[penumbria]

I use FFF and EpubMerge to download anthologies from AO3. I have been running into two issues lately. First, some of these anthologies are huge, like 100+ entries. And I have to use the browser cache and open pages in browser fixes to get FFF to work almost always lately with AO3.

So, when I go to update the anthology, it takes forever, sometimes literally hours, to update, even when only a single fic or a single chapter of one of the fics in the collection has been added. It opens up every single one in the browser which also stops me from being able to do anything else on it instead of it doing it in the background which is annoying.

Is there a way to keep this from happening and only have it look for new chapters and/or entries in the anthology? Or if not, to shorten how long it takes to open every entry and scan it?

My second issue lately with anthology updating is since I am using the cache and open pages, if the anthology has updated within the past few days, it will often just come back with not seeing the new chapters, even if I open the fic and the chapter index in the browser myself before I tell it to update the anthology. And this is for all anthologies, short or long. Is there a way to force it to see the latest cache?

My personal ini is currently:

is_adult:true
always_login:true
use_archiveofourown_gay:false
use_archive_transformativeworks_org:false
use_browser_cache:true
use_browser_cache_only:true
open_pages_in_browser:true
browser_cache_age_limit:-1

Sometimes I have use_archiveofourown_gay:true, but it sometimes doesn't have all of the latest chapters or it returns an error message. And when I change the age limit to above -1, to a positive number, it tells me it has expired in the cache.

Is there any way to fix these issues?

[penumbria]

Sorry for the double post. Don't know why it happened.

[JimmXinu]

Originally Posted by penumbria

I use FFF and EpubMerge to download anthologies from AO3. I have been running into two issues lately. First, some of these anthologies are huge, like 100+ entries. And I have to use the browser cache and open pages in browser fixes to get FFF to work almost always lately with AO3.
...

Unfortunately, virtually every site now implements some level of blocking for automated scraping. FFF's anthology feature was designed in the days when we could assume reasonably quick and easy scraping.

Originally Posted by penumbria

Is there a way to keep this from happening and only have it look for new chapters and/or entries in the anthology? Or if not, to shorten how long it takes to open every entry and scan it?

I will look into whether it's feasible to change FFF to use the chapter counts / updated dates from the series page to reduce the number of stories updated. That would currently only benefit OTW sites (AO3 & derived), eFiction sites, and TtH as the only sites we collect specific 'series' data for.

Originally Posted by penumbria

My second issue lately with anthology updating is since I am using the cache and open pages, if the anthology has updated within the past few days, it will often just come back with not seeing the new chapters, even if I open the fic and the chapter index in the browser myself before I tell it to update the anthology. And this is for all anthologies, short or long. Is there a way to force it to see the latest cache?

Using Ctrl-R to reload the page in most browsers will force a cache refresh.

Setting browser_cache_age_limit:-1 specifically says to use all old cache entries. Count yourself lucky we can even do that -- other sites set 'no-cache' instead of 'max-age=0' and then the browsers don't cache at all.

What you can do today

The only real alternative right now is to use EpubMerge to manually UnMerge the anthology, update the updated story(s), then use EpubMerge to re-create the anthology (be sure you have not turned off 'Keep UnMerge Data' in EpubMerge's config). Yes, this is a nuisance and introduces issues with it being a new book.

Worst case, as always with AO3, is to download the site's provided EPUBs instead of FFF.

Rant

Downloading / scraping has been getting worse for years now. I honestly think it's going to become impossible within a few years at best. We've already accepted as normal the Browser Cache workaround that would have been unthinkable earlier.

We're basically just bailing the boat at this point.

Scraping that runs in browser as a browser add-on will probably work longer. But I don't think that I have the energy to start over. Especially when WebToEpub already exists.

[JimmXinu]

Originally Posted by penumbria

I use FFF and EpubMerge to download anthologies from AO3. I have been running into two issues lately. First, some of these anthologies are huge, like 100+ entries.

Originally Posted by JimmXinu

I will look into whether it's feasible to change FFF to use the chapter counts / updated dates from the series page to reduce the number of stories updated. That would currently only benefit OTW sites (AO3 & derived), eFiction sites, and TtH as the only sites we collect specific 'series' data for.

penumbria, can you provide some examples(URLs) of these very large series? AO3 apparently paginates large series, but I can't find any off hand.

FYI, FFF will still need to fetch each page of a paginated series to get the full list. Assuming I can get this work.

[penumbria]

Originally Posted by JimmXinu

penumbria, can you provide some examples(URLs) of these very large series? AO3 apparently paginates large series, but I can't find any off hand.

FYI, FFF will still need to fetch each page of a paginated series to get the full list. Assuming I can get this work.

There are 124 in this one: https://archiveofourown.org/series/2540224

This one has 113: https://archiveofourown.org/series/4275145

This is currently 301 fics: https://archiveofourown.org/series/2801707

[ernand]

I am unsure, but how do you use the "update old chapters" function???
I have a long fic where the author updated ch. 1 and 2, and I recall that a new ability to update specific chapters was added recently.

Just unsure how to use it, or if I have to add a new line to my personal.ini. Or if there is a way to run one-time commands without updating the .ini file every time.