[GUI Plugin] FanFicFare

[steltek]

Originally Posted by Ealdwulf

OK. Finally got a chance to verify this. Cleaned the cache, verified FireFox up to date, restarted the browser.

Current Calibre version: 5.22 [64bit]
Current FanFicFare version: 4.3.11
Current FireFox version: 89.0.2

Opened two stories on FF.net and paged through each chapter, clicked the author URL, and clicked the stories images for each.

confirmed the relevant sections of personal.ini are:

[defaults]
browser_cache_path:C:\Users\bryan\AppData\Local\Mozilla\Firefox\Profiles\4vb4brdx.default-release\cache2

[www.fanfiction.net]
use_cloudscraper:true
use_browser_cache:true
browser_cache_age_limit:4.0
#use_browser_cache_only:false

I have tried both to overwrite if newer (the stories are, one with a single new chapter and the other with four new chapters), and also just updating. Both give same error message in the comment column of the log file for the job:

Directory does not contain a known browser cache type: 'C:\Users\bryan\AppData\Local\Mozilla\Firefox\Profiles\4vb4brdx.default-release'

And looking at about:cache once again confirms the entry:

Storage disk location: 	C:\Users\bryan\AppData\Local\Mozilla\Firefox\Profiles\4vb4brdx.default-release\cache2

I'm not sure what else to do on this, I haven't tried to use WebtoEpub for this but considering the error message I doubt it would allow this to work.

First, you shouldn't need that reference to cloudscraper under [www.fanfiction.net] in [overrides], though I don't know if having it there will actually hurt anything or not. You might try commenting it out for testing purposes just to eliminate a variable.

However, I have a suspicion about something with Firefox that would very well cause the exact problem you are having.

Type about:config in your browser URL bar and hit enter. Search for the following setting: browser.cache.disk.enable
Is it set to true or false? If it is set to false, click on it once to change it to true, then try to page through a fic and run FFF again.

Also, while you are in about:config, check the following additional setting as well:
devtools.cache.disabled
Make sure that this setting is set to false.

Once you have done the two things above, the last thing you need to do is type about:support in the browser URL window and hit enter. Click the button to clear the startup cache under "Try clearing the startup cache". Apparently, for some reason turning on the browser cache doesn't actually seem to take affect until you force-clear the Firefox startup cache using this method. Once you do it, things again start getting written to the cache.

[steltek]

Originally Posted by Philosophize

That's a good point about security, but this Mac is running Catalina, which was released back in 2019. Even the most recent updates haven't introduced all the privacy measures we're reading about. Now that you've mentioned it, I suspect it will eventually become a problem, but I don't think it's my problem here.

I also went ahead and updated Calibre to the latest and the plugin to the version posted a little bit ago today. Errors are exactly the same.

Blaming the OS would be the obvious solution, if it weren't for the fact that it did work for one session already. But I don't have any better ideas - just the speculation that there's an error which isn't being reported correctly.

Philosophize, please try the instructions at the bottom of the post immediately above as well. I was able to reproduce your problem in Firefox and FFF (along with Ealdwulf's) and suspect they are related, so I hope the same instructions might resolve your issue as well.

[Ealdwulf]

Originally Posted by steltek

First, you shouldn't need that reference to cloudscraper under [www.fanfiction.net] in [overrides], though I don't know if having it there will actually hurt anything or not. You might try commenting it out for testing purposes just to eliminate a variable.

However, I have a suspicion about something with Firefox that would very well cause the exact problem you are having.

Type about:config in your browser URL bar and hit enter. Search for the following setting: browser.cache.disk.enable
Is it set to true or false? If it is set to false, click on it once to change it to true, then try to page through a fic and run FFF again.

Also, while you are in about:config, check the following additional setting as well:
devtools.cache.disabled
Make sure that this setting is set to false.

Once you have done the two things above, the last thing you need to do is type about:support in the browser URL window and hit enter. Click the button to clear the startup cache under "Try clearing the startup cache". Apparently, for some reason turning on the browser cache doesn't actually seem to take affect until you force-clear the Firefox startup cache using this method. Once you do it, things again start getting written to the cache.

Well, I checked about:config, Browser.cache.disk.enable was already set to true and devtools.cache.disabled was set to false. I went ahead and clicked on clear startup cache in the about:support, restarted the browser, and still got the same error.

Oh, and I also tried commenting out the cloudscraper reference, no change there either

[steltek]

Originally Posted by Ealdwulf

Well, I checked about:config, Browser.cache.disk.enable was already set to true and devtools.cache.disabled was set to false. I went ahead and clicked on clear startup cache in the about:support, restarted the browser, and still got the same error.

Oh, and I also tried commenting out the cloudscraper reference, no change there either

Clear the cache on your Firefox installation (Tools menu, Settings, Privacy and Security, then under the Cookies and Data section, hit the Clear Data button. You can uncheck the "Cookies and Site Data" checkbox so you don't clear all your cookies, but leave the "Cached Web Content" box checked and hit the Clear button).

Then, open a new tab and page through a few pages of a fic on FFN.net. Open another new tab, type about:cache in the Firefox URL bar and hit enter. At the bottom of the disk section on this page, there should be a blue link titled "List Cache Entries". Click on that link. Are the URLs to the pages of the fic you paged through listed there?

[weirdscx]

I can no longer downloaded from fanfiction.net, it used to work great but now any story I try I get the following error in the log:

cloudscraper reports: "Detected a Cloudflare version 2 Captcha challenge, This feature is not available...."

[steltek]

Originally Posted by weirdscx

I can no longer downloaded from fanfiction.net, it used to work great but now any story I try I get the following error in the log:

cloudscraper reports: "Detected a Cloudflare version 2 Captcha challenge, This feature is not available...."

Cloudflare has placed the Fanfiction.net website in under attack mode for denial of service attack. It has been going on long enough that this is probably a semi-permanent (if not permanent) situation, and there is no possible direct workaround for FFF (or most 3rd party scrapers). If you want more information, you can read back over the last 20 pages or so on this board or take a look at this FAQ item.

If it is only now affecting you, you have been lucky. Based upon how it has affected everyone else, you probably won't be lucky any more.

Your only choice with FFF at this point is to use the browser cache feature as detailed here, or you will need to use another method like the WebToEpub browser extension (which still works at this time but produces Epubs incompatible with FFF).

[JimmXinu]

I apparently missed some posts, sorry.

Unfortunately I don't have much to add to the suggestions.

The only additional thing that comes to mind for people having problems with cache on Windows is that I had to add some windows specific code for file opening/locking to make it work while the browser was still running.

You might try quitting the browser entirely after loading the story in cache and before trying to download it in FFF.

Clearly that isn't a good system for actual use, but it might help us diagnosis the issue.

I may also look again at some code I started once to read from WebToEpub output files instead of cache, since this doesn't seem to be going away.

[Ealdwulf]

Originally Posted by steltek

Clear the cache on your Firefox installation (Tools menu, Settings, Privacy and Security, then under the Cookies and Data section, hit the Clear Data button. You can uncheck the "Cookies and Site Data" checkbox so you don't clear all your cookies, but leave the "Cached Web Content" box checked and hit the Clear button).

Then, open a new tab and page through a few pages of a fic on FFN.net. Open another new tab, type about:cache in the Firefox URL bar and hit enter. At the bottom of the disk section on this page, there should be a blue link titled "List Cache Entries". Click on that link. Are the URLs to the pages of the fic you paged through listed there?

Went through. that, and the pages for the fic are listed.

I'm thinking it has to do with the address listed in the comment:

C:\Users\bryan\AppData\Local\Mozilla\Firefox\Profiles\4vb4brdx.default-release

Notice the example:

C:\Users\YourUser\AppData\Local\Mozilla\Firefox\Profiles\ZjwI7Fo4.default\cache2

for some reason, mine has -release after default and the comment is leaving off the \cache2

Is it possible that there is something in the code that is truncating the line somehow, or the -release is throwing it off?


Oh, and I tried Jim's suggestion of closing FireFox and it did the same thing

[watani]

I've been making use of the personal.ini config file to set certain stories to never update their custom covers from their website source, but have been unable to make the setting work specifically for fanfiction.net stories.

As an example, I've been using the FFF browser cache feature to update the following story:
https://www.fanfiction.net/s/1265889...ds-of-Creation

I have the following rows in the personal.ini configuration:

[https://www.fanfiction.net/s/1265889...s-of-Creation]
never_make_cover:true

Every time I've updated this story (and other fanfiction.net stories that I've applied the "never_make_cover" setting to), the cover still gets replaced with the fanfiction.net cover. I've attached the copied download log text file here of my most recent update (replacing the username directory with '*' for privacy):

FFF_DL_Log.txt

I'm wondering if the browser cache download somehow breaks the normal process to ignore updating the cover. This line occurs in the log and sticks out as unusual:

FFF: WARNING: 2021-07-04 18:38:05,482: adapter_fanfictionnet.py(343): Caught exception in skip_author_cover: cloudscraper reports: "Detected a Cloudflare version 2 Captcha challenge, This feature is not available....".

[JimmXinu]

Originally Posted by Ealdwulf

...

I have no idea anymore. Attached is a test version that enables a bunch of debug output that is normally commented out. A debug log of a run with this version might help.

I don't recommend this version for anyone who's not trying to troubleshoot a cache problem. If you haven't cleared your browser cache, it will generate a huge log and record every URL you've visited.

Originally Posted by watani

Every time I've updated this story (and other fanfiction.net stories that I've applied the "never_make_cover" setting to), the cover still gets replaced with the fanfiction.net cover. I've attached the copied download log text file here of my most recent update (replacing the username directory with '*' for privacy):

When you removed the cover(s), did you remove it just in Calibre, or inside the epub too?

The log shows that FFF is using an existing cover from inside the epub:

FFF: DEBUG: 2021-07-04 18:38:28,910: writer_epub.py(516): writer_epub: no new cover, has old cover, write image.

FFF does that with the intention that you can set never_make_cover but still have your own cover.

[watani]

Originally Posted by JimmXinu

When you removed the cover(s), did you remove it just in Calibre, or inside the epub too?

The log shows that FFF is using an existing cover from inside the epub:

FFF: DEBUG: 2021-07-04 18:38:28,910: writer_epub.py(516): writer_epub: no new cover, has old cover, write image.

FFF does that with the intention that you can set never_make_cover but still have your own cover.

I am replacing the cover both inside Calibre and inside the epub with my own custom cover and I've verified that the cover image inside the epub is actually the custom one before updating. However, the download is still overwriting the cover inside the epub with the source website's cover even with the never_make_cover setting when downloading off of fanfiction.net using the browser cache.

Maybe this observation might help: In the past, when downloading off of other websites and using the never_make_cover setting to preserve the custom cover I've noticed that the "no new cover, has old cover, write image." log message always shows up in two separate rows instead of just once. Here's an example of a download log for a story off of RoyalRoad where the custom cover image is successfully kept over the update: FFF_DL_Log_NoCoverReplace.txt

[JimmXinu]

Originally Posted by watani

...

Still can't duplicate your problem. The never_make_cover feature is implemented at a lower level than the adapters--which site shouldn't make any difference.

How are you removing and replacing the cover? I wonder if it's not updating epub2 metadata?

If I can see your epub, it may help.

[steltek]

Originally Posted by Ealdwulf

Went through. that, and the pages for the fic are listed.

I'm thinking it has to do with the address listed in the comment:

C:\Users\bryan\AppData\Local\Mozilla\Firefox\Profiles\4vb4brdx.default-release

Notice the example:

C:\Users\YourUser\AppData\Local\Mozilla\Firefox\Profiles\ZjwI7Fo4.default\cache2

for some reason, mine has -release after default and the comment is leaving off the \cache2

Is it possible that there is something in the code that is truncating the line somehow, or the -release is throwing it off?


Oh, and I tried Jim's suggestion of closing FireFox and it did the same thing

The '-release' is added to current profile cache path by a new, clean Firefox install. The one I just installed in that Win10 virtual machine has a cache path pretty much identical to yours. Mine on my regular machine doesn't have it simply because it was installed about 7 years ago and has been upgraded successively with each release since then but it kept the old profile folder.

You could try altering the cache path to add/remove the \cache2 to see if it works if you want. Nothing to loose but a few seconds of editing and test time.

Are you running ANY extensions at all? There are extensions that can mess with caching trying to "speed up" or "improve" your browsing. I've found that they generally aren't worth messing with in the long run now (as opposed to the old days when they sometimes made a significant difference). The offer of the special test build from JimmXinu in the above post would also be worth trying.

Also, have you considered trying to load another browser like Chrome, Vivaldi, or Edge? You could also try Waterfox 3rd gen or Waterfox Classic if you want to stick with a Mozilla-based browser. I've successfully used all of these with FFF while simultaneously installed on the same machine (switching between them by just commenting out the different cache path lines in [defaults] in personal.ini.

[watani]

Originally Posted by JimmXinu

Still can't duplicate your problem. The never_make_cover feature is implemented at a lower level than the adapters--which site shouldn't make any difference.

How are you removing and replacing the cover? I wonder if it's not updating epub2 metadata?

If I can see your epub, it may help.

...and of course this time when I update the same story (after deleting the new chapter using Edit Book to get FFF to re-download the chapter) the cover gets kept. I made absolutely no changes to how I normally change to a custom cover nor did I make any changes to the FFF settings. The download log also looks identical to the previous update log that didn't keep the existing cover (with the exception of changes in temporary file locations).

I'm frustrated since it seems like this is an issue I keep coming across and I've never been able to pin down a cause myself or get someone else to reproduce the problem before it randomly fixes itself (I've commented about this issue two or so previous times in this thread). And once the issue is fixed once for a given story, it never re-occurs for all future updates on that story.

Just for completeness, what I do to replace the epub cover is to use Edit Metadata -> Change Cover -> Browse option in Calibre. I have verified (using Edit Book in Calibre) that doing this also changes the cover image inside the epub.

I'll also attach the epub in its present state which is apparently "fixed."

[Ealdwulf]

I ran the debug and it was still not finding anything... so as I was preparing to paste a really long post that had a fairly short debug file, as well as what about:cache reports and, once again, the relevant parts of my personal.ini (defaults and www.fanfiction.net); when I found the blasted problem.

Rather than just the first couple of lines in the defaults section where the browser_cache_path line was located, I was going to put all of it without the instructions and notes, just the settings whether they were commented out or not.

apparently at some point I had tried to work with browser_cache_path in the past (it had to have been on calibre 4.x) and had a second browser_cache_path line... one that was the same, but missing the \cache2 at the end...

so, FFF was a good program that read the configuration file and saw variable browser_cache_path was the correct path but then as it continued reading it saw the same variable and promptly overwrote what it had in place originally.

likely would not have been an issue if the second entry had also had the correct path, but since it didn't...

needless to say, getting rid of the duplicate entry and making sure that the correct path is in place, allowed it to read tthe cache properly. Yay! It didn't seem to think the web site was newer though even though the fic was just updated a week ago or so with a new chapter published and my entry was from more than a moth back. I did get it to update, I just had to select "overwrite always" instead of "overwrite if newer".

so, for funky stuff that is occurring, let this be a lesson. make sure you don't have multiple lines in the personal.ini that duplicate with the last one being wrong