[GUI Plugin] FanFicFare

[steltek]

Originally Posted by ownedbycats

While it may or may not help, I will note that when I used to run into this issue it sometimes helped to clear the cache, close and re-start the browser, and then clear it again before visiting the fanfic to load the chapters.

I've tried that exact process twice. However, it keeps throwing that same error on chapter 69 every time for some reason.

Edit:
It also throws the same error on chapter 69 in the Microsoft Edge browser (also Chrome based) as well. I installed Chrome on my virtual machine setup with no extensions (a clean install) and it threw the exact same error on both chapters 52 and 69. I cleared the Chrome cache, re-browsed the fic, and tried to re-download in FFF. The error went away on chapter 52, but remains on chapter 69. Repeating this results in the chapter 69 error repeating.

Using the Firefox and Waterfox browser caches, I am able to repeatedly download the story on the first try with no errors whatsoever no matter how many times I clear the cache, browse the fic, and run FFF.

There is definitely something wrong here. I suspect this issue may be related to the problems I was having previously as I was using Vivaldi on the affected machine and I vaguely recall seeing those same error lines being in the log I failed to save. It didn't click with me at the time as I was focusing on why it wasn't finding the pages in the cache even though I knew they were there and could actually see them in ChromeCacheViewer.

[JimmXinu]

We have code for two different cache formats for Chrome (and derivatives) use and third different one for Firefox. So it's no surprise that Firefox can behave differently.

I don't see any issues with https://www.fanfiction.net/s/8096183/69/ on Chrome on Win10, nor Chrome on MacOS.

I don't think I can help with this any further without seeing a copy of cache files that are failing. If you're clearing your cache anyway, it should be small enough to zip up and not contain anything private.

[steltek]

Originally Posted by JimmXinu

We have code for two different cache formats for Chrome (and derivatives) use and third different one for Firefox. So it's no surprise that Firefox can behave differently.

I don't see any issues with https://www.fanfiction.net/s/8096183/69/ on Chrome on Win10, nor Chrome on MacOS.

I don't think I can help with this any further without seeing a copy of cache files that are failing. If you're clearing your cache anyway, it should be small enough to zip up and not contain anything private.

I've included the contents of the cache folders for Chrome and Edge in separate WinRAR files attached. I've also attached the job log showing the chapter 69 error as a text file.

EDIT:
I also installed the Calibre 5.23 update, as well as the FFF 4.4.0 updated plugin. No change.

[Nutzer0815]

I installed Calibre 5.23.0 with FanFicFare 4.4.0.
I tried the instructions from https://github.com/JimmXinu/FanFicFa...erCacheFeature.
I updated the cache path for Firefox with my path.
C:\Users\user\AppData\Local\Mozilla\Firefox\Profil es\xvgdeguf.default-1563538979722\cache2
The old error message is gone.
I now get the error message
Directory does not contain a known browser cache type: 'C:\Program Files\Calibre2',
but I cannot locate the error.

[JimmXinu]

New Test Version Posted

2021-07-14
- Change blockfilecache to save uint32 addrs instead of original cache key. Hashing cache key proved unreliable in some cases.
- Allow fictionpress.com with use_browser_cache--user still needs to configure in personal.ini Closes #716

Originally Posted by steltek

I've included the contents of the cache folders for Chrome and Edge in separate WinRAR files attached. I've also attached the job log showing the chapter 69 error as a text file.
...

Okay, once I could see it happen in front of me, I was able to track this down better.

The Chrome Block Cache code was basically hashing the cache key to a binary address via a hash table, but in that particular case, the hash pointed to a null (0s) address in the table.

But during the initial cache table scan (to find the ffnet URLs), the binary address was already known.

So I changed the code to save and use the binary address and skip the hashing/lookup step altogether.

It's now working for me on the examples given and should even give a (minor) performance boost.

Note that this change only affects Chrome derived browsers using Block Cache. Many Chrome browsers use Simple Cache instead--I've no idea which or why.

Originally Posted by Nutzer0815

...
Directory does not contain a known browser cache type: 'C:\Program Files\Calibre2'
...

This error indicates that your browser_cache_path isn't set correctly, since it's trying to use 'C:\Program Files\Calibre2'.

Perhaps you have browser_cache_path appearing more than once?

[steltek]

Originally Posted by JimmXinu

New Test Version Posted

2021-07-14
- Change blockfilecache to save uint32 addrs instead of original cache key. Hashing cache key proved unreliable in some cases.
- Allow fictionpress.com with use_browser_cache--user still needs to configure in personal.ini Closes #716



Okay, once I could see it happen in front of me, I was able to track this down better.

The Chrome Block Cache code was basically hashing the cache key to a binary address via a hash table, but in that particular case, the hash pointed to a null (0s) address in the table.

But during the initial cache table scan (to find the ffnet URLs), the binary address was already known.

So I changed the code to save and use the binary address and skip the hashing/lookup step altogether.

It's now working for me on the examples given and should even give a (minor) performance boost.

Note that this change only affects Chrome derived browsers using Block Cache. Many Chrome browsers use Simple Cache instead--I've no idea which or why.



This error indicates that your browser_cache_path isn't set correctly, since it's trying to use 'C:\Program Files\Calibre2'.

Perhaps you have browser_cache_path appearing more than once?

I can also verify that this fix resolves the problem in both of the test cases I have been using. The speed boost, at least for me, is very significant especially in longer fics. The jobs still don't execute as fast on my machine as they do when using a Firefox-based browser cache, but they the gap is significantly closed.

Originally Posted by Nutzer0815

I installed Calibre 5.23.0 with FanFicFare 4.4.0.
I tried the instructions from https://github.com/JimmXinu/FanFicFa...erCacheFeature.
I updated the cache path for Firefox with my path.
C:\Users\user\AppData\Local\Mozilla\Firefox\Profil es\xvgdeguf.default-1563538979722\cache2
The old error message is gone.
I now get the error message
Directory does not contain a known browser cache type: 'C:\Program Files\Calibre2',
but I cannot locate the error.

When you have personal.ini open for editing, copy/paste C:\Program Files\Calibre2 into the search box at the bottom of the editing window and hit the 'Find' button. This should find any occurrences of that specific text in personal.ini and allow you to locate the problem quickly.

[ownedbycats]

I wasn't having issues but I also noticed a performance increase with the test version.

[kamille3]

I was downloading about 500 stories from literotica. FFF took about 40 min to download the metadata then another 40 mins to finish downloading. Is this amount of time normal? I feel its a bit too long considering the total download size is barely 10MB.

[ownedbycats]

Yes.

Also, don't download that many stories at once! See this in the FAQ.

[Ascello]

How to download only some chapters of a fic?

[DirCat]

On AO3, there's multiple series for some works. I was wondering if it was possible to (a) automatically put the additional series into other custom series columns, and (b) write every series title for the works into a tag column (maybe by using the "column is comprised of other columns" column type).

Any thoughts on it? I've looked over the FAQ and searched for likely terms on here and wasn't able to find any info.

[JimmXinu]

Originally Posted by Ascello

How to download only some chapters of a fic?

Put the desired chapter range after the URL, like:
https://archiveofourown.org/works/32478307[1-2]

Note that this is not saved in the book metadata; so if you later update it without manually entering the chapter range again, it will download all chapters.

Originally Posted by DirCat

On AO3, there's multiple series for some works. I was wondering if it was possible to (a) automatically put the additional series into other custom series columns, and (b) write every series title for the works into a tag column (maybe by using the "column is comprised of other columns" column type).

FFF does collect additional series for AO3, it should work to put them into custom series columns using custom_columns_settings.

From defaults.ini:

## Some adapters collect additional meta information beyond the
## standard ones.  They need to be defined in extra_valid_entries to
## tell the rest of the FanFicFare system about them.  They can be
## used in include_subject_tags, titlepage_entries,
## extra_titlepage_entries, logpage_entries, extra_logpage_entries,
## and include_in_* config items.  You can also add additional entries
## here to build up composite metadata entries.
## archiveofourown.org, for example, fills genre (a standard
## entry) as the composite offreeformtags, ao3categories in
## include_in_genre.  If there's ever more than 4 series, add
## series04,series04Url etc.
extra_valid_entries:fandoms, freeformtags, freefromtags,
 ao3categories, comments, chapterslashtotal, chapterstotal, kudos,
 hits, bookmarks, collections, byline, bookmarked, bookmarktags,
 bookmarksummary, bookmarkprivate, bookmarkrec, restricted, series00,
 series01, series02, series03, series00Url, series01Url, series02Url,
 series03Url, series00HTML, series01HTML, series02HTML, series03HTML

[Philosophize]

It works!

...and I have no idea why.

Sorry I took so long to get back to you, especially after you went to such trouble writing out these instructions, but I just couldn't make the time.

Everything went fine until I got to this part:

Originally Posted by steltek

Then, in the FanFicFare configuration, go all the way over to the right to the "Other" menu. At the bottom of the Other menu, there is an option labeled "View Library Preferences". Click on it. At the bottom of the window that comes up, there are two options, a 'Clear' button and a 'Ok' button. Hit the clear button, answer "yes" when asked if you are sure, and and then hit the ok button. This will reset the FFF options in Personal.ini and within the Calibre library database to their defaults. It should then prompt you to restart Calibre, which you should do.

I did as you instructed, and when I restarted Calibre I opened the personal.ini, only to find that it hadn't changed at all. Everything that had been there before was still there. I was under the impression that I'd get a bare-bones, default ini because your next step was to manually type in the minimal configuration lines to use the cache.

That was discouraging. I was going to post here asking if that was correct or what I might have misunderstood, but first decided to see if it worked. Unsurprisingly, it didn't.

I came here, saw that a new test version had been posted, so decided to install that. Also installed the latest version of Calibre. Skimmed the discussion on proxy servers since it didn't apply to me and I wouldn't be able to offer any advice on it. Switched my configuration lines from Firefox to Chrome (because I ought to test that every so often).

And it worked. I was able add a completely new fic. I was able to update an existing fic. It worked when I manually clicked through chapters and when I used WebToEpub. And it worked when I had lots of fics in the cache (because it's been a month!).

I'm both thrilled that it's working and annoyed that I have no idea why. I assume it's the latest test version, but I got the impression that it was designed to deal with the proxy server issue.

Anyway, I just wanted to thank you again for all your patience and to let you know the latest status. Hopefully I'm not back in the same situation in another week or so....

[Ascello]

Originally Posted by JimmXinu

Put the desired chapter range after the URL, like:
https://archiveofourown.org/works/32478307[1-2]

Note that this is not saved in the book metadata; so if you later update it without manually entering the chapter range again, it will download all chapters.

Thanks! And what if I don't want range, but for example chapter 1 and 3 but not 2?

[JimmXinu]

Download separately or use the ignore_chapter_url_list feature. Search for it in defaults.ini.