[GUI Plugin] FanFicFare

[JimmXinu]

Originally Posted by quilledink

I copied directly from there into the defaults section of my person ini, then adjusted for my username (put ~ in front of /Library for it to automatically fill it in, but it generates as /Users/quilledink/)

As I recall, the ~ for home dir expansion does NOT work in INI files, eg browser_cache_path. defaults.ini shows full absolute path.

And, in fact, your error should be:

"browser_cache_path is not set, or directory does not contain a known browser cache type: 'C:\Users\user\AppData\Local\Mozilla\Firefox\Profi les\xxxxx.default\cache2'"
...except showing your browser_cache_path.

If you run debug mode, the debug log will show you where FFF is generating that error more precisely.

Originally Posted by quilledink

I did see in another thread a while back you suggested something about calibre's "safe_open_url" -- is that something that might work? (I'm not familiar enough with coding in general to know--I more tend to try and logic and research my way through stuff like this).

Unrelated.

Originally Posted by nephtys59

Stopped working this evening with error 428. Damn AO3!

Solved changing this value:
browser_cache_age_limit:-1

Let's hope it lasts.

Be aware that FFF does NOT honor the original cache time out. If you leave browser_cache_age_limit:-1, FFF will happily use hours/days/weeks old cache entries if it finds them.

Keep that in mind if you start seeing FFF not 'noticing' new chapters.

[nephtys59]

Originally Posted by JimmXinu

Be aware that FFF does NOT honor the original cache time out. If you leave browser_cache_age_limit:-1, FFF will happily use hours/days/weeks old cache entries if it finds them.

Keep that in mind if you start seeing FFF not 'noticing' new chapters.

Yes, I'm aware of that. I cleaned the browser cache BEFORE launching the update, so that the cache contained only the fics I wanted to update and I did this even before changing the setting from 4 to -1 but with setting 4 hour I got the error 428 anyway ... the ways of AO3 are mysterious, I believe.
I checked the fics anyway after update and all the chapters were there, even the newer ones. So I consider it a win.

[crashtest]

I'm pretty new to FanFicFare, but I did have it up and working for the past month or so. I'd routinely download and upload AO3 epubs using use_archiveofourown_gay. A few days go my settings suddenly stopped working and I've been trying to get it back online ever since. I've gleaned all the info I can from these threads and I'm still not sure what I'm doing wrong.

Is there something wrong with this personal.ini? When I use it, my browser is open, set to this profile, and logged into AO3. It creates a job, but then the job progress stalls until ~1 minute 20 seconds and fails, giving me error "HTTP Error in FFF 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)'(428) URL" Thanks!

[defaults]
browser_cache_path: /Users/username/Library/Caches/Google/Chrome/Profile 13/Cache/Cache_Data

[archiveofourown.org]
is_adult: true
use_browser_cache:true
use_browser_cache_only:true
open_pages_in_browser:true
browser_cache_age_limit: 4.0

[girlikecupcake]

Originally Posted by crashtest

I'm pretty new to FanFicFare, but I did have it up and working for the past month or so. I'd routinely download and upload AO3 epubs using use_archiveofourown_gay. A few days go my settings suddenly stopped working and I've been trying to get it back online ever since. I've gleaned all the info I can from these threads and I'm still not sure what I'm doing wrong.

Is there something wrong with this personal.ini? When I use it, my browser is open, set to this profile, and logged into AO3. It creates a job, but then the job progress stalls until ~1 minute 20 seconds and fails, giving me error "HTTP Error in FFF 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)'(428) URL" Thanks!

[defaults]
browser_cache_path: /Users/username/Library/Caches/Google/Chrome/Profile 13/Cache/Cache_Data

[archiveofourown.org]
is_adult: true
use_browser_cache:true
use_browser_cache_only:true
open_pages_in_browser:true
browser_cache_age_limit: 4.0

Have you tried other age limits? 4 is a good starting point but about 24 hours ago I had to bump it to 6 even though I knew for a fact things should be fine, and nephtys59 posted a short bit ago about using -1 on error 428 instead to force it to work. I just tested with a story with the cache_age set to 6 and it worked fine - it opened three pages in the browser (chapter index, first chapter, full story) then proceeded to download.

[quilledink]

Ah, thank you so much! I'd gotten errors with other browsers that had shown the full file path, even when using ~, so I'd just assumed that meant it was reading it okay.

And it's working now! Turns out the reason I was getting the "Chrome" error was because I'd messed up the openers_by_scheme changes as well--everything's working now! Thank you again!

Originally Posted by JimmXinu

As I recall, the ~ for home dir expansion does NOT work in INI files, eg browser_cache_path. defaults.ini shows full absolute path.

And, in fact, your error should be:

"browser_cache_path is not set, or directory does not contain a known browser cache type: 'C:\Users\user\AppData\Local\Mozilla\Firefox\Profi les\xxxxx.default\cache2'"
...except showing your browser_cache_path.

If you run debug mode, the debug log will show you where FFF is generating that error more precisely.


Unrelated.


Be aware that FFF does NOT honor the original cache time out. If you leave browser_cache_age_limit:-1, FFF will happily use hours/days/weeks old cache entries if it finds them.

Keep that in mind if you start seeing FFF not 'noticing' new chapters.

[kaitykat]

Writing this up just in case it turns out to be useful to someone else, because it took me a ridiculous amount of time to figure out. I was getting the 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)' error on AO3 even after setting the time out to -1, and I was successfully getting the metadata, but not being able to download any chapters. Turns out FFF was successfully opening the /navigate pages, and the first chapter of multi-chap fics, but failing on getting the "whole work" page in the browser. So if I manually open that page, FFF can grab it from the cache. For single chapter fics, I have to go the navigate page and manually open the first chapter, because for some reason the direct link (without a /chapter/number) isn't enough? I don't know. It's cumbersome, but it's mostly working, so

[nephtys59]

Originally Posted by girlikecupcake

Have you tried other age limits? 4 is a good starting point but about 24 hours ago I had to bump it to 6 even though I knew for a fact things should be fine, and nephtys59 posted a short bit ago about using -1 on error 428 instead to force it to work. I just tested with a story with the cache_age set to 6 and it worked fine - it opened three pages in the browser (chapter index, first chapter, full story) then proceeded to download.

I tried 6 but it didn't work. The -1 works fine. I will empty my browser's cache before every update, so it will work fine.

[wildfire070]

Originally Posted by kaitykat

Writing this up just in case it turns out to be useful to someone else, because it took me a ridiculous amount of time to figure out. I was getting the 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)' error on AO3 even after setting the time out to -1, and I was successfully getting the metadata, but not being able to download any chapters. Turns out FFF was successfully opening the /navigate pages, and the first chapter of multi-chap fics, but failing on getting the "whole work" page in the browser. So if I manually open that page, FFF can grab it from the cache. For single chapter fics, I have to go the navigate page and manually open the first chapter, because for some reason the direct link (without a /chapter/number) isn't enough? I don't know. It's cumbersome, but it's mostly working, so

It's taken some trial and error but this is also what's working for me. For some reason, open_pages_in_browser works for ff.net but doesn't open all the necessary pages in AO3. I've just been manually opening those pages and it downloads successfully.

[Kaigani Scout]

For folks on Windows, here are some settings which have worked for me as of this AM. Context is that I'm running Windows 10: Calibre 9.11; FanFicFare 4.59.6; EpubMerge 3.2.0.

[defaults]
browser_cache_path: \Users\YourUserNameIsHere\AppData\Local\Google\Chr ome\User Data\Default\Cache\Cache_Data

[archiveofourown.org]
username: IfYouLoginItGoesHere
password: TypeItInHereIfYouUseIt
is_adult: true
use_browser_cache:true
use_browser_cache_only:true
open_pages_in_browser:true
browser_cache_age_limit: 4.0

Those settings up there appear to work for Single Works. I have run updates for Single Titles and Multiple Titles ( up to 5 at once ) and things worked with lots of tabs opening and whatnot. Painful for those of us who monitor a lot of WIPs, but there it is.

What that approach does not resolve is how to update Anthology files. I used the settings up there in my Library which is dedicated to anthologies and the program entered a loop which eventually resulted in an escape via error. Using some clues in other posts, I discovered a fairly simple substitution edit to the "cache approach":

[archiveofourown.org]
browser_cache_age_limit: -1

Modifying the Cache Age in this way allows the program/plugins to access, load, and assemble formal Series entries off AO3 ( which are organized by a writer ) into a single, integrated Anthology in a Calibre Library. It also works for Anthologies assembled by a reader from different URL sources, such as ones I create which are thematically-related in some way.

As per normal, once the program has enumerated the contributing source URLs for each Work, a confirmation pop-up will appear. It might appear "behind" the other open windows, so make sure to check for it otherwise you might sit there for awhile waiting for something to happen ( I'm just guessing because that certainly didn't happen to me )... but once you give the green light, the program will cycle through checking each title using the "cache approach". As I was typing this, I ran my last experiment with a 6-part series of 1.2M+ words and it worked.

Now, why this works, I have no idea as I'm not a coder. It does work, though, so kudos and karma and upvotes to all of you nice folks who have contributed by sharing your experiences and findings.

[Elaine T]

Originally Posted by wildfire070

It's taken some trial and error but this is also what's working for me. For some reason, open_pages_in_browser works for ff.net but doesn't open all the necessary pages in AO3. I've just been manually opening those pages and it downloads successfully.

I finally spent the time to get download by browser cache working for me, and for the first time in hours I could update an Ao3 fic.

[JimmXinu]

Originally Posted by wildfire070

It's taken some trial and error but this is also what's working for me. For some reason, open_pages_in_browser works for ff.net but doesn't open all the necessary pages in AO3. I've just been manually opening those pages and it downloads successfully.

If open_pages_in_browser doesn't open the pages, we've seen, on Mac at least, needing to set Calibre's openers_by_scheme tweak.

I have no explanation why it would make a difference on different sites.

Originally Posted by Kaigani Scout

For folks on Windows, here are some settings which have worked for me as of this AM. Context is that I'm running Windows 10: Calibre 9.11; FanFicFare 4.59.6; EpubMerge 3.2.0.

[defaults]
browser_cache_path: \Users\YourUserNameIsHere\AppData\Local\Google\Chr ome\User Data\Default\Cache\Cache_Data

[archiveofourown.org]
username: IfYouLoginItGoesHere
password: TypeItInHereIfYouUseIt
is_adult: true
use_browser_cache:true
use_browser_cache_only:true
open_pages_in_browser:true
browser_cache_age_limit: 4.0

Those settings up there appear to work for Single Works. I have run updates for Single Titles and Multiple Titles ( up to 5 at once ) and things worked with lots of tabs opening and whatnot. Painful for those of us who monitor a lot of WIPs, but there it is.

What that approach does not resolve is how to update Anthology files. I used the settings up there in my Library which is dedicated to anthologies and the program entered a loop which eventually resulted in an escape via error. Using some clues in other posts, I discovered a fairly simple substitution edit to the "cache approach":

[archiveofourown.org]
browser_cache_age_limit: -1

When using browser cache, you don't actually need username/password in personal.ini--you're logging in yourself in browser. And FFF's code for fetching AO3 series will try to login if you have username/password set -- but NOT if you have "open_pages_in_browser" and "use_browser_cache" and "use_browser_cache_only" all set.

You also don't need is_adult. In fact, if you turn that off, it'll be easier to manually load pages because FFF won't ask for urls with "?view_adult=true" added.

As for using browser_cache_age_limit: -1 for series -- that appears to be true? But why?

AO3 apparently serves series pages with the header "cache-control: max-age=0, private, must-revalidate", but that sets the expiry time that FFF ignores...

I thought I had this figured out, but apparently not. I was busy today and I'm thinking tired. And I will be busy a lot of tomorrow. So I'll look at on the weekend.

In the meantime, if it works, it works.

[wildfire070]

Originally Posted by JimmXinu

If open_pages_in_browser doesn't open the pages, we've seen, on Mac at least, needing to set Calibre's openers_by_scheme tweak.

Thanks! I knew there was something I was forgetting!

[jazzhands]

Hopefully this is just a quick question:

I have this in personal.ini because I didn't want to see AO3 tags:

##There isn't any way to stop FFF collecting tags--it's part of the code. However, you can stop FFF from using those tags. [https://bit.ly/3GOUqvE]
[archiveofourown.org]
include_in_genre: ao3categories

BUT I also have a huge list of tags under exclude_metadata_pre:, that looks like:

exclude_metadata_pre:

freeformtags,genre=~^[Aa]ll .*
freeformtags,genre=~^[Aa]ka .*
freeformtags,genre=~^[Aa]uthor.*
freeformtags,genre=~^NOTE.*
...
...
...

I THINK I had that long list because I was trying to eliminate tags from the files (before I found out about include_in_genre: ao3categories).

Question: Do I need the whole list of tags under under exclude_metadata_pre: if I already have the top code snippet (include_in_genre: ao3categories) in the file?

If it's easier for you, a Yes or No answer is just fine. No explanation needed.

[slimcat]

I can't get the browser cache to work on either Firefox or Chrome on Mac, but the failures are different. ¯\_(ツ)_/¯

Chrome fails with 'no such table: resources'. Firefox opens the AO3 page but never finds it in the cache and eventually returns a 428 (debug logs attached).

[booster-t]

Originally Posted by JimmXinu

If open_pages_in_browser doesn't open the pages, we've seen, on Mac at least, needing to set Calibre's openers_by_scheme tweak.

Ummm ... pardon my dumbness, but can someone explain how that tweak works please? Adding it to my personal.ini results in it being red, which means it is incorrect. Help?

Thanks

Stephen