[GUI Plugin] FanFicFare

[JimmXinu]

New Test Version Posted

2021-02-17
- Don't require index file in Firefox cache2.

Originally Posted by minorthman

Firefox cache2 directory

C:\Users\Toshiba C55-A5100\AppData\Local\Mozilla\Firefox\Profiles\h1yno njd.default-1611513187773\cache2

contains two subfolders

doomed (empty)
entries (35 items)

...

so it looks like firefox has alot different layout
could not see a index file

That looks correct for a firefox cache2 dir, except for the missing index file. FFF checks for the existence of the index file, but doesn't actually use it.

I tested actually deleting everything in a Firefox cache2 directory and then starting Firefox and it did not create an index file, so apparently Firefox doesn't need it either.

Thanks for your persistence about this. Seeing the contents of your cache2 gave me the clue needed.

FYI, the forum has [code] tags you can wrap around blocks like debug logs to preserve formatting and not get smilies. # button in the post editor.

[ILB]

Originally Posted by JimmXinu

I've written a new wiki document about using the browser cache feature:
https://github.com/JimmXinu/FanFicFa...erCacheFeature

I appreciate some feedback on whether it covers all the necessary details clearly.

I would say that all the necessary details are included.

[minorthman]

Tested the new version with firefox both modes

use_browser_cache:true did a three chapter update

use_browser_cache_only:true did a 19 chapter complete download

both appeared to work ok

have to wait until Cloudflare starts giving errors again

have not had any errors since about 3AM this morning

Thank you

[tricklem]

So, everything seems to be working, but is there a way to tell if things are working because the cache is finally set up correctly or if it is because fanfiction.net is working again?

[JimmXinu]

[www.fanfiction.net]
## browser cache will only be used if use_browser_cache:true and ONLY
## for fanfiction.net right now.  Requires a browser_cache_path set in
## [defaults].
use_browser_cache:true

## use_browser_cache_only:true prevents FFF from falling through to
## request from the server when a page isn't in the browser cache.
## You can turn this on to be sure you're only using the browser
## cache.
use_browser_cache_only:true

[ownedbycats]

Question: Is it normal, when reading from cache, for the dialog window to become "not responding" and turn white? If I leave it alone it'll eventually complete, but if I click on it at all Windows wants me to end the process.

[JimmXinu]

As far as I know, yes. It's off in a tight loop reading the cache. The same thing can happen for other lengthy processing.

Use 'update existing' and background metadata to avoid it.

[ownedbycats]

Honestly, Windows is a bit too quick on the "not responding"—I think it's something like 5 seconds before it whites out the window.

[JimmXinu]

For firefox and chrome simple-cache, reducing browser_cache_age_limit can speed up cache parsing significantly. Doesn't really make any difference on chrome blockfile-cache.

[riothamus]

Originally Posted by ownedbycats

Question: Is it normal, when reading from cache, for the dialog window to become "not responding" and turn white? If I leave it alone it'll eventually complete, but if I click on it at all Windows wants me to end the process.

Happened to me the first time I successfully ran an update with the new cache feature turned on. Subsequent runs were faster (at least those right after the first on the same calibre session).

[steltek]

Originally Posted by JimmXinu

For firefox and chrome simple-cache, reducing browser_cache_age_limit can speed up cache parsing significantly. Doesn't really make any difference on chrome blockfile-cache.

Also, periodically clear the browser cache before paging through the fic in the browser. The less cache FFF has to parse through to find what it needs, the faster it should work.

On my machine, Vivaldi easily collects over 1GB of cached data after only a couple days of use.

[JimmXinu]

New Version Released

Version 4.0.0 - 18 Feb 2021
- Major refactoring/rewrite of network fetching and caching layer using `requests` library.
- use_browser_cache feature added -- Read fanfiction.net stories from your Chrome or FireFox browser cache in addition / instead of the network. Many thanks to Sidney Markowitz for the stellar help with this. See BrowserCacheFeature Wiki Page.
- Add additional_images setting for html & epub formats. Close #648
- Fix for 'Updating' (anonymous?) author in adapter_wuxiaworldsite. For #657
- Allow tweak_fg_sleep etc with any site, add defaults settings for basexenforo, fictionalley and fictionpress.
- Drag/drop of 'emails'--look for story URL in Content-Base header for Thunderbird RSS 'emails'.
- Apply 'email' proc flag when processing dragged .eml files--which are emails.
- Update translations.
- Fix adapter_webnovelcom for some site changes. Closes #644
- Allow tweak_fg_sleep etc with any site, add defaults settings for basexenforo, fictionalley and fictionpress.
- fanficauthors.net doesn't use login anymore. Closes #651
- BG % done by stories AND chapters.
- Restore multi-process background jobs as one process per site.
- Don't count continue_on_chapter_error chapters when checking vs 'new chapters'.
- More properly get msg payload fromemail drag and drop. Closes #645
- Give adapter_archiveofourownorg a get_section_url() for quicker rejects checks from email.
- Convert adapter_spikeluvercom to base_efiction after site changes.
- Remove 'extrasleep' feature--it was ugly--increase slow_down_sleep_time for the couple sites that used it. We can trust the users, right?
- Remove httplib._MAXHEADERS workaround for royalroad.com - not happening anymore.
- Fixes for py2/py3 & a stray INI space.
- Fix for Reject List to equate story URLs with and without titles, for XenForo and ffnet. Otherwise you can end up with dups in Reject List and not recognize new URLs are already being rejected. Also fixes [storyUrl] sections without titles for ffnet.
- Fix fictionpress.net and retain Reject List entries for removed sites.
- Get cloudscraper 1.2.56 changes, but keep py2 compat.
- Fix for chapter error report. Closes #641
- adapter_fanfictionnet: Start keeping story title part of storyUrl.

[tricklem]

Originally Posted by JimmXinu

New Version Released

Version 4.0.0 - 18 Feb 2021
- Major refactoring/rewrite of network fetching and caching layer using `requests` library.
- use_browser_cache feature added -- Read fanfiction.net stories from your Chrome or FireFox browser cache in addition / instead of the network. Many thanks to Sidney Markowitz for the stellar help with this. See BrowserCacheFeature Wiki Page.
- Add additional_images setting for html & epub formats. Close #648
- Fix for 'Updating' (anonymous?) author in adapter_wuxiaworldsite. For #657
- Allow tweak_fg_sleep etc with any site, add defaults settings for basexenforo, fictionalley and fictionpress.
- Drag/drop of 'emails'--look for story URL in Content-Base header for Thunderbird RSS 'emails'.
- Apply 'email' proc flag when processing dragged .eml files--which are emails.
- Update translations.
- Fix adapter_webnovelcom for some site changes. Closes #644
- Allow tweak_fg_sleep etc with any site, add defaults settings for basexenforo, fictionalley and fictionpress.
- fanficauthors.net doesn't use login anymore. Closes #651
- BG % done by stories AND chapters.
- Restore multi-process background jobs as one process per site.
- Don't count continue_on_chapter_error chapters when checking vs 'new chapters'.
- More properly get msg payload fromemail drag and drop. Closes #645
- Give adapter_archiveofourownorg a get_section_url() for quicker rejects checks from email.
- Convert adapter_spikeluvercom to base_efiction after site changes.
- Remove 'extrasleep' feature--it was ugly--increase slow_down_sleep_time for the couple sites that used it. We can trust the users, right?
- Remove httplib._MAXHEADERS workaround for royalroad.com - not happening anymore.
- Fixes for py2/py3 & a stray INI space.
- Fix for Reject List to equate story URLs with and without titles, for XenForo and ffnet. Otherwise you can end up with dups in Reject List and not recognize new URLs are already being rejected. Also fixes [storyUrl] sections without titles for ffnet.
- Fix fictionpress.net and retain Reject List entries for removed sites.
- Get cloudscraper 1.2.56 changes, but keep py2 compat.
- Fix for chapter error report. Closes #641
- adapter_fanfictionnet: Start keeping story title part of storyUrl.

Thanks for all of the hard work you have put in to get this done and thank you to everyone for all the extra help that some of us needed.

[k7oamam0t0]

DLing 3 stories off of SV and SB. the time it takes to DL Stories seems to have significantly Increased with the new version the 3 i'm working on now are 60% with it going for a half an hour.

edit: time to dl fics from these links:
https://forums.sufficientvelocity.co...der-fgo.43852/
https://forums.spacebattles.com/thre...-quest.549311/
https://forums.spacebattles.com/thre...adybug.776197/

55mins 19 secs

[JimmXinu]

Originally Posted by k7oamam0t0

DLing 3 stories off of SV and SB. the time it takes to DL Stories seems to have significantly Increased with the new version the 3 i'm working on now are 60% with it going for a half an hour.

I'm going to assume you are the same person who asked the same question about the same stories on github.

I answered here on github:
https://github.com/JimmXinu/FanFicFare/issues/660