[GUI Plugin] FanFicFare

[Tanjamuse]

I use win 10 but still V4.23 and it works for me, at least it did an hour ago.

[ilovejedd]

Originally Posted by Ytierian

This would be extremely unfortunate. I'm stuck on an older computer (no money to upgrade) so Win-7 is my life now. Since V5 of Calibre won't work on Win-7, I'm stuck at V4.23.

I can work around the issue using the 'Save as eBook' plugin for Firefox, but it's clunky.

Windows 7 can still get free upgrades to Windows 10, I think. Might need a fresh install, though.

[Ytierian]

Originally Posted by ilovejedd

Windows 7 can still get free upgrades to Windows 10, I think. Might need a fresh install, though.

I ran the 'test' - my computer is so old that Win-10 is unsupported.

[JimmXinu]

New Test Version Posted
(Back in the usual place.)

2020-12-23
- Adding cloudscraper and dependencies
- Updates for many included dependencies
- Need to semi-manually raise HTTPError for error codes with cloudscraper/requests.
- Temp: dump ffnet page to debug when metadata parsing fails.
- Clean up cloudscraper import
- Use requests.util.extract_zipped_paths to get browsers.json instead of kludge.
- Don't use mobile User-Agents with cloudscraper--adapter_fanfictionnet doesn't handle mobile pages.
- Document use_cloudscraper better in defaults.ini.

This version includes the module cloudscraper and dependencies and is configured to use it for fanfiction.net and fictionpress.com.

This probably won't help with ffnet in older Calibre versions due to CloudFlare detecting that older versions of Calibre bundle older versions of OpenSSL. If an older version works for you, great, but don't expect it to.

It does still work in older versions for other sites.

I have tested both CLI in Python3 and plugin in Calibre v5.7.2 on Win10, Mac(BigSur) and Linux.

UPDATE: I've written in more detail here, in the github issue.

[ownedbycats]

With the RSS feeds glitching as they are, I ran the test version (edit: 3.26.6 - not sure if the version posted 3 minutes ago is the same) on about 60 in-progress ffnet fics with a slowdown time of about 8 seconds. Got no errors in checking the status, though there were no actual updates.

[ilovejedd]

Originally Posted by Ytierian

I ran the 'test' - my computer is so old that Win-10 is unsupported.

Ah, that's unfortunate.

[ownedbycats]

Originally Posted by JimmXinu

This probably won't help with ffnet in older Calibre versions due to CloudFlare detecting that older versions of Calibre bundle older versions of OpenSSL. If an older version works for you, great, but don't expect it to.

Hmm, wonder if pointing FFF at an external (user-installed) version of OpenSSL would be possible. This stackoverflow thread seems to indicate that it might be difficult at least.

[JimmXinu]

FFF still loads and works back to Calibre 2.85.1. The new cloudscraper code also loads and runs that far back--but because the OpenSSL versions are years out of date, CloudFlare applies higher levels of restriction that cloudscraper can't handle.

While I sympathize with people unable to update their computer, I believe I have already done everything that I reasonably can within the limits of what's available to me.

[Ytierian]

Alright - installed the latest test version - 3.26.7

IT WORKS

Win-7, Calibre 4.23 - was able to update my FanFiction.net stories that had been in limbo since the switch.

[JimmXinu]

New Test Version Posted

2020-12-23
- Changes to allow email chapter update URLs to work in adapter_wattpadcom

And just because it's so important:

Originally Posted by JimmXinu

New Test Version Posted
(Back in the usual place.)

2020-12-23
- Adding cloudscraper and dependencies
- Updates for many included dependencies
- Need to semi-manually raise HTTPError for error codes with cloudscraper/requests.
- Temp: dump ffnet page to debug when metadata parsing fails.
- Clean up cloudscraper import
- Use requests.util.extract_zipped_paths to get browsers.json instead of kludge.
- Don't use mobile User-Agents with cloudscraper--adapter_fanfictionnet doesn't handle mobile pages.
- Document use_cloudscraper better in defaults.ini.

This version includes the module cloudscraper and dependencies and is configured to use it for fanfiction.net and fictionpress.com.

This probably won't help with ffnet in older Calibre versions due to CloudFlare detecting that older versions of Calibre bundle older versions of OpenSSL. If an older version works for you, great, but don't expect it to.

It does still work in older versions for other sites.

I have tested both CLI in Python3 and plugin in Calibre v5.7.2 on Win10, Mac(BigSur) and Linux.

UPDATE: I've written in more detail here, in the github issue.

[vikarti]

Last test version of plugin appears to work for me with 4.23, thanks!
cloudscraper works but logs message about older openssl.

Not sure if it's really matter but:
it's same win10 machine where advice of 'how to fix https errors with one NSFW site in Fanficfare by visiting it in IE/Edge' doesn't work,
also, it's home network, static public IP(i.e. no ISP-level NAT, only my router does NAT), which resolves to reverse DNS name of my choice (gw.mydomain.net), this IP is used by me and other familiy members and not others for years.

[ownedbycats]

A static IP might be problematic if Cloudflare flags it as suspicious.

[Chrystalline]

Originally Posted by Ytierian

This would be extremely unfortunate. I'm stuck on an older computer (no money to upgrade) so Win-7 is my life now. Since V5 of Calibre won't work on Win-7, I'm stuck at V4.23.

Calibre Portable 5.7.2 is working just fine on my Win7 laptop. I had to upgrade to Win10 at work (and I hate it badly enough to start experimenting with Linux), so I go back and forth with my external harddrive with PortableApps. Since I also use DeDRM, I have the older version as well. I just renamed the older version from Calibre-Portable.exe to Calibre-Import.exe before I "installed" it to my external drive. They both show up in my PortableApps list just fine and I've had no trouble switching back and forth and sharing libraries between them.

When Win7 was removed from the supported list, I was having a few glitches in selections (I'd try to select a few stories and it would highlight everything on the page) but now that seems to have been resolved. I recommend trying the current portable version anyway - see if it'll work on yours.

I came by tonight because I was also trying to update O Mine Enemy, and got the 403 Forbidden. Test version works (on my Win7 laptop)

[ackomb]

Hey guys,

I ran into an issue with downloading from fanfiction.net.
For some reason I can no longer download or update anything from their site. AO3 is working fine.
I've ran the whole thing in debug mode (which you guys tought me last time :-) only this time I don't understand how to fix it. Here the debug report, hope you know the answer to my problem.

Spoiler:

Traceback (most recent call last):
File "calibre\customize\ui.py", line 703, in initialize_plugins
File "calibre\customize\ui.py", line 61, in load_plugin
File "calibre\customize\zipplugin.py", line 293, in load
File "importlib\__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "calibre\customize\zipplugin.py", line 192, in exec_module
File "calibre_plugins.fantastic_fiction.__init__", line 10, in <module>
ModuleNotFoundError: No module named 'HTMLParser'
FFF: DEBUG: 2020-12-24 15:29:01,097: calibre_plugins.fanficfare_plugin.fff_plugin(221): Plugin FanFicFare macmenuhack file_path:C:\Users\kombr\AppData\Roaming\calibre\p lugins\fanficfare_macmenuhack.txt
calibre Debug log
calibre 5.8.1 embedded-python: True is64bit: False
Windows-10-10.0.18362 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.18362')
Python 3.8.5
Windows: ('10', '10.0.18362', '', 'Multiprocessor Free')
Interface language: en_GB
Failed to initialize plugin: 'C:\\Users\\kombr\\AppData\\Roaming\\calibre\\plug ins\\Fantastic Fiction.zip'
Successfully initialized third party plugins: Count Pages (1, 11, 0) && FanFicFare (3, 26, 0) && Find Duplicates (1, 8, 3) && Generate Cover (2, 0, 1) && Reading List (1, 7, 1) && User Category (1, 5, 5) && View Manager (1, 6, 0)
calibre 5.8.1 embedded-python: True is64bit: False
Windows-10-10.0.18362 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.18362')
Python 3.8.5
Windows: ('10', '10.0.18362', '', 'Multiprocessor Free')
Interface language: en_GB
Successfully initialized third party plugins: Count Pages (1, 11, 0) && FanFicFare (3, 26, 0) && Find Duplicates (1, 8, 3) && Generate Cover (2, 0, 1) && Reading List (1, 7, 1) && User Category (1, 5, 5) && View Manager (1, 6, 0)
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 92.53889943074003 x 92.67567567567568
Using calibre Qt style: True
[0.00] Starting up...
[0.02] Showing splash screen...
[0.42] splash screen shown
[0.42] Initializing db...
[3.06] db initialized
[3.06] Constructing main UI...
C:\Users\Cynthia\Documents\Calibrebibliotheek is not a valid directory to watch for new ebooks, ignoring
[16.30] main UI initialized...
[16.30] Hiding splash screen
FFF: DEBUG: 2020-12-24 15:29:18,966: calibre_plugins.fanficfare_plugin.fff_plugin(1099) : FanFicFare v3.26.0
FFF: DEBUG: 2020-12-24 15:29:18,987: calibre_plugins.fanficfare_plugin.fff_plugin(1220) : prep_download_loop url:https://www.fanfiction.net/s/13125859/1/
FFF: DEBUG: 2020-12-24 15:29:19,364: calibre_plugins.fanficfare_plugin.fanficfare.confi gurable(1198):
===========
set sleep time 1.0
===========
FFF: DEBUG: 2020-12-24 15:29:19,364: calibre_plugins.fanficfare_plugin.fanficfare.adapt ers.adapter_fanfictionnet(100): URL: https://www.fanfiction.net/s/13125859/1/
FFF: DEBUG: 2020-12-24 15:29:19,364: calibre_plugins.fanficfare_plugin.fanficfare.confi gurable(1154): #####################################
pagecache(GET) MISS: https://www.fanfiction.net/s/13125859/1/
FFF: DEBUG: 2020-12-24 15:29:21,451: calibre_plugins.fanficfare_plugin.fanficfare.confi gurable(1234): Caught an exception reading URL: https://www.fanfiction.net/s/13125859/1/ Exception HTTP Error 403: Forbidden.
FFF: DEBUG: 2020-12-24 15:29:21,451: calibre_plugins.fanficfare_plugin.fanficfare.confi gurable(1247): Giving up on https://www.fanfiction.net/s/13125859/1/
FFF: DEBUG: 2020-12-24 15:29:21,451: calibre_plugins.fanficfare_plugin.fanficfare.confi gurable(1248): HTTP Error 403: Forbidden
NoneType: None
FFF: ERROR: 2020-12-24 15:29:21,451: calibre_plugins.fanficfare_plugin.dialogs(723): Exception: {'title': 'Unknown', 'author_sort': ['Unknown'], 'author': ['Unknown'], 'comments': '', 'good': False, 'status': 'Error', 'showerror': True, 'calibre_id': None, 'begin': None, 'end': None, 'comment': 'HTTP Error 403: Forbidden', 'url': 'https://www.fanfiction.net/s/13125859/1/', 'site': 'www.fanfiction.net', 'series': '', 'added': False, 'pubdate': None, 'publisher': None, 'uniqueurl': 'https://www.fanfiction.net/s/13125859/1/[None-None]', 'listorder': 0, 'collision': 'Update EPUB if New Chapters'}:HTTP Error 403: Forbidden
Traceback (most recent call last):
File "calibre_plugins.fanficfare_plugin.dialogs", line 710, in do_loop
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 1305, in prep_download_loop
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 1185, in get_story_metadata_only
File "calibre_plugins.fanficfare_plugin.fanficfare.adap ters.base_adapter", line 297, in getStoryMetadataOnly
File "calibre_plugins.fanficfare_plugin.fanficfare.adap ters.adapter_fanfictionnet", line 111, in doExtractChapterUrlsAndMetadata
File "calibre_plugins.fanficfare_plugin.fanficfare.adap ters.adapter_fanfictionnet", line 104, in doExtractChapterUrlsAndMetadata
File "calibre_plugins.fanficfare_plugin.fanficfare.adap ters.adapter_fanfictionnet", line 82, in _fetchUrl
File "calibre_plugins.fanficfare_plugin.fanficfare.conf igurable", line 1355, in _fetchUrl
File "calibre_plugins.fanficfare_plugin.fanficfare.conf igurable", line 1344, in _fetchUrlOpened
File "calibre_plugins.fanficfare_plugin.fanficfare.conf igurable", line 1249, in _fetchUrlOpened
File "calibre_plugins.fanficfare_plugin.fanficfare.conf igurable", line 1225, in _fetchUrlOpened
File "calibre_plugins.fanficfare_plugin.fanficfare.conf igurable", line 1186, in _fetchUrlRawOpened
File "urllib\request.py", line 531, in open
File "urllib\request.py", line 640, in http_response
File "urllib\request.py", line 569, in error
File "urllib\request.py", line 502, in _call_chain
File "urllib\request.py", line 649, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden
FFF: DEBUG: 2020-12-24 15:29:25,587: calibre_plugins.fanficfare_plugin.fff_plugin(1813) : Finished Adding/Updating 0 books.
FFF: DEBUG: 2020-12-24 15:29:25,596: calibre_plugins.fanficfare_plugin.fff_plugin(1816) : removed tdir

[Ytierian]

Originally Posted by ackomb

Hey guys,

I ran into an issue with downloading from fanfiction.net.
For some reason I can no longer download or update anything from their site. AO3 is working fine.
I've ran the whole thing in debug mode (which you guys tought me last time :-) only this time I don't understand how to fix it. Here the debug report, hope you know the answer to my problem.

last few pages of this thread are on exactly that.

FFF test version 3.26.7 seems to fix the problem - has for me anyway.