[GUI Plugin] FanFicFare

[kamille3]

Originally Posted by JimmXinu

What does Calibre show for proxies?

Searching for that error suggests that it is a windows/python/https/proxy specific issue when https proxy is set to an 'https://' target that isn't actually https itself.

• https://www.google.com/search?q="Val...rver_hostname"
• https://stackoverflow.com/questions/...sing-fiddler-4
• https://bugs.python.org/issue42627

One answer might be to change the https proxy value in FFF, but I'd like to know what you already have first.

Plus, it can legitimately be https, so I'd probably need to make it a configurable setting...

Calibre correctly detects my Host IP address and port number like every other software on my pc. It shows up like this:

http:192.168.xx.xxx:1234
https:192.168.xx.xxx:1234
ftp:192.168.xx.xxx:1234

[JimmXinu]

New Test Version Posted

2021-07-11
- Adding fix_broken_https_proxy setting for broken Windows https proxy settings.

Originally Posted by kamille3

Calibre correctly detects my Host IP address and port number like every other software on my pc. It shows up like this:

http:192.168.xx.xxx:1234
https:192.168.xx.xxx:1234
ftp:192.168.xx.xxx:1234

Please try the test version after adding fix_broken_https_proxy:true to personal.ini:

[defaults]
fix_broken_https_proxy:true

Run Calibre in debug mode (under Calibre Preferences dropdown) while attempting the download and then post the debug log.

[kamille3]

Originally Posted by JimmXinu

New Test Version Posted

2021-07-11
- Adding fix_broken_https_proxy setting for broken Windows https proxy settings.



Please try the test version after adding fix_broken_https_proxy:true to personal.ini:

[defaults]
fix_broken_https_proxy:true

Run Calibre in debug mode (under Calibre Preferences dropdown) while attempting the download and then post the debug log.

FFF: DEBUG: 2021-07-11 21:38:48,538: calibre_plugins.fanficfare_plugin.fff_plugin(221): Plugin FanFicFare macmenuhack file_path:C:\Users\INDRANIL-pc\AppData\Roaming\calibre\plugins\fanficfare_macmenuhack.txt
calibre Debug log
calibre 5.23 [64bit]  embedded-python: True is64bit: True
Windows-8.1-6.3.9600 Windows ('64bit', 'WindowsPE')
('Windows', '8.1', '6.3.9600')
Python 3.8.5
Windows: ('8.1', '6.3.9600', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: FanFicFare (4, 3, 15)
calibre 5.23 [64bit]  embedded-python: True is64bit: True
Windows-8.1-6.3.9600 Windows ('64bit', 'WindowsPE')
('Windows', '8.1', '6.3.9600')
Python 3.8.5
Windows: ('8.1', '6.3.9600', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: FanFicFare (4, 3, 15)
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 101.6 x 101.6
Using calibre Qt style: True
[0.00] Starting up...
[0.00] Showing splash screen...
[0.33] splash screen shown
[0.33] Initializing db...
[0.33] db initialized
[0.33] Constructing main UI...
Using proxies: {'http': '192.168.42.129:8080', 'https': '192.168.42.129:8080', 'ftp': '192.168.42.129:8080'}
[1.77] main UI initialized...
[1.77] Hiding splash screen
Starting QuickView
FFF: DEBUG: 2021-07-11 21:40:20,880: calibre_plugins.fanficfare_plugin.fff_plugin(675): get_urls_from_page URL:https://www.literotica.com/stories/memberpage.php?uid=989888&page=submissions
FFF: DEBUG: 2021-07-11 21:40:20,929: adapter_literotica.py(40): LiteroticaComAdapter:__init__ - url='http://www.literotica.com/s/story-title'
FFF: DEBUG: 2021-07-11 21:40:20,931: configurable.py(1010): use_browser_cache:
FFF: DEBUG: 2021-07-11 21:40:20,931: configurable.py(1024): use_basic_cache:true
FFF: DEBUG: 2021-07-11 21:40:20,932: fetcher.py(234): 
========== MISS (GET) BasicCache
https://www.literotica.com/stories/memberpage.php?uid=989888&page=submissions
FFF: DEBUG: 2021-07-11 21:40:20,932: fetcher.py(463): 
---------- REQ (GET) RequestsFetcher
https://www.literotica.com/stories/memberpage.php?uid=989888&page=submissions
FFF: DEBUG: 2021-07-11 21:40:20,932: fetcher.py(423): Session Proxies Before:{}
FFF: DEBUG: 2021-07-11 21:40:20,932: fetcher.py(426): Attempting to collect proxy settings through Calibre
FFF: DEBUG: 2021-07-11 21:40:20,932: fetcher.py(430): Calibre Proxies:{'http': '192.168.42.129:8080', 'https': '192.168.42.129:8080', 'ftp': '192.168.42.129:8080'}
FFF: DEBUG: 2021-07-11 21:40:20,932: fetcher.py(444): Session Proxies After:{'http': '192.168.42.129:8080', 'https': '192.168.42.129:8080', 'ftp': '192.168.42.129:8080'}

[JimmXinu]

Originally Posted by kamille3

...

So that shows FFF taking proxy settings from Calibre and setting them, but not the incorrect 'https://' in https proxy setting I thought was happening.

The question is, did it work?

[Hortensia]

Hello,

Is there a possibility to manually adjust the content of a custom column after the personal.ini file has been run and before the fanfic is downloaded or the library is updated? (I am not sure which of these actions generate the default book cover.)

The reason why I am trying to do so is that the content of this custom column is too random to be treated in a completely automatic way, I need some manual input. Then, once all my metadata is correct, I generate a new book cover (Edit metadata > Generate cover). I have modified the default settings (Edit metadata > Long click on Generate cover > Customize the styles and colors of the generated cover) to include the metadata I have in this manually completed custom column.

In short, I would like to manually validate all the metadata before the automatic generation of a default cover when the ebook is added to the library.

Thank you!

[kamille3]

Originally Posted by JimmXinu

So that shows FFF taking proxy settings from Calibre and setting them, but not the incorrect 'https://' in https proxy setting I thought was happening.

The question is, did it work?

Sadly it did not. I keep getting this...

calibre, version 5.23.0
ERROR: Unhandled exception: <b>ValueError</b>:check_hostname requires server_hostname

calibre 5.23 [64bit]  embedded-python: True is64bit: True
Windows-8.1-6.3.9600 Windows ('64bit', 'WindowsPE')
('Windows', '8.1', '6.3.9600')
Python 3.8.5
Windows: ('8.1', '6.3.9600', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: FanFicFare (4, 3, 15)
Traceback (most recent call last):
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 654, in get_urls_from_page_menu
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 677, in get_urls_from_page
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\geturls.py", line 45, in get_urls_from_page
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\adapters\base_adapter.py", line 443, in get_urls_from_page
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\requestable.py", line 114, in get_request
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\requestable.py", line 106, in get_request_redirected
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\fetcher.py", line 390, in get_request_redirected
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\fetcher.py", line 240, in fetcher_do_request
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\fetcher.py", line 136, in fetcher_do_request
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\fetcher.py", line 363, in do_request
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\fanficfare\fetcher.py", line 465, in request
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\requests\sessions.py", line 542, in request
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\requests\sessions.py", line 655, in send
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\requests\adapters.py", line 439, in send
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\urllib3\connectionpool.py", line 696, in urlopen
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\urllib3\connectionpool.py", line 964, in _prepare_proxy
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\urllib3\connection.py", line 359, in connect
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\urllib3\connection.py", line 500, in _connect_tls_proxy
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\urllib3\util\ssl_.py", line 453, in ssl_wrap_socket
  File "C:\Users\MY-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\urllib3\util\ssl_.py", line 495, in _ssl_wrap_socket_impl
  File "ssl.py", line 500, in wrap_socket
  File "ssl.py", line 997, in _create
ValueError: check_hostname requires server_hostname

[JimmXinu]

New Test Version Posted

2021-07-12
- Add http_proxy and https_proxy setting, remove fix_broken_https_proxy setting.

Originally Posted by kamille3

Sadly it did not. I keep getting this...
...

Okay, last try--if this doesn't work, then I'm declaring it a fundamental incompatibility between your proxy and urllib3/python libs.

Install test version and put your proxy in your personal.ini but explicitly with http:// like so:

[defaults]
https_proxy:http://192.168.42.129:8080

[JimmXinu]

Originally Posted by Hortensia

...In short, I would like to manually validate all the metadata before the automatic generation of a default cover when the ebook is added to the library.

Not as part of FFF. Many users who are that particular tend to keep an 'incoming' library and a separate 'cleaned' library for the books that have been fixed up.

You might also look at the QuarantineAndScrub plugin.

[kamille3]

Originally Posted by JimmXinu

New Test Version Posted

2021-07-12
- Add http_proxy and https_proxy setting, remove fix_broken_https_proxy setting.

Okay, last try--if this doesn't work, then I'm declaring it a fundamental incompatibility between your proxy and urllib3/python libs.

Install test version and put your proxy in your personal.ini but explicitly with http:// like so:

[defaults]
https_proxy:http://192.168.42.129:8080

Tried it but still getting the same error. Anyways thank you for dedicating so much time to this issue.

[Hortensia]

Originally Posted by JimmXinu

Not as part of FFF. Many users who are that particular tend to keep an 'incoming' library and a separate 'cleaned' library for the books that have been fixed up.

You might also look at the QuarantineAndScrub plugin.

Thank you, I will look at these options!

[JimmXinu]

New Version Released

Version 4.4.0 - 2021-07-13
- Add http_proxy and https_proxy settings. Should be unnecessary, also taking from Calibre.
- Fix for XF2 stories using author img as cover when absolute URL.
- XF: data-url or data-src for lazyload images. Closes #713
- adapter_royalroadcom: Collect numWords. #712
- XenForo2 sites (SB&SV) collect estimatedWords. #712
- Update packaged urllib3 and certifi modules.
- Pre-v5 Calibre fix for nsapa_proxy.
- Record error message for new downloads too since Failed Chapters are now a thing.
- adapter_wwwnovelallcom - Updates for site changes. 'translator' no longer available?
- New Site: worldofx.de (German language X-files eFiction)
- Fixes to nsapa_proxy from nsapa https://github.com/JimmXinu/FanFicFare/pull/711
- Add link to wiki/BrowserCacheFeature in Cloudflare error. #708
- Treat img url contains '.svg?' same as endswith('.svg')--Calibre image processing chokes hard on SVG. #709
- Implement use_browser_cache for ficbook.net #704
- Forgot to import traceback in browser cache changes.
- Skip Failures to read cache entries for Chrome Simple Cache and Firefox Cache2 instead of error.
- Fix date reading for XenForo forums sites
- Change defaults.ini to correctly show default values for remove_spoilers, legend_spoilers
- Collect 'fandoms' for adapter_scribblehubcom.
- PI: Only update author link (AKA authorUrl) in Calibre if changed. Only really affects older versions of Calibre. Calibre does the same starting around Sept 2020.

[dysphere]

When trying to download a specific fic on fanfiction.net (Harry Potter and the Natural 20) and using the browser cache feature (I've definitely manually visited every chapter of the fic and I have it set up so all cached files will be used), I keep getting an error for one of the chapters.

Chapter URL:
https://www.fanfiction.net/s/8096183/69/

Error:
Traceback (most recent call last):
File "C:\Users\my-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\ fanficfare\adapters\base_adapter.py", line 240, in getStory
File "C:\Users\my-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\ fanficfare\adapters\base_adapter.py", line 419, in getChapterTextNum
File "C:\Users\my-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\ fanficfare\adapters\adapter_fanfictionnet.py", line 374, in getChapterText
File "C:\Users\my-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\ fanficfare\requestable.py", line 114, in get_request
File "C:\Users\my-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\ fanficfare\requestable.py", line 106, in get_request_redirected
File "C:\Users\my-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\ fanficfare\fetcher.py", line 390, in get_request_redirected
File "C:\Users\my-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\ fanficfare\fetcher.py", line 240, in fetcher_do_request
File "C:\Users\my-pc\AppData\Roaming\calibre\plugins\FanFicFare.zip\ fanficfare\fetcher.py", line 278, in fetcher_do_request
fanficfare.exceptions.HTTPErrorFFF: HTTP Error in FFF 'Page not found or expired in Browser Cache (see FFF setting browser_cache_age_limit)'(428) URL:'https://www.fanfiction.net/s/809

[steltek]

Originally Posted by dysphere

When trying to download a specific fic on fanfiction.net (Harry Potter and the Natural 20) and using the browser cache feature (I've definitely manually visited every chapter of the fic and I have it set up so all cached files will be used), I keep getting an error for one of the chapters.

Out of curiosity, what browser are you using? I got the exact same error on the same chapter as you did in Vivaldi (which is a Chrome-based browser), and it repeated again on that same chapter (chapter 69) when I cleared the cache and walked through it a second time. However, it downloads just fine in Firefox on the first try.

EDIT:
I had another fic that was doing the exact same thing in Vivaldi, with the same error. Clearing the cache and re-walking through the fic resulted in the same error, but it kept randomly occurring at a different chapter each time.

I then edited the epub in Calibre, deleted the chapter pages of the epub from the error chapter to the end, saved the epub. I switched over to Firefox, browsed the missing chapters into the Firefox cache, and was able to complete the update of the fic. Have no idea what is going on with this, maybe an issue with the Vivaldi/Chrome cache that isn't occurring with Firefox?

[ownedbycats]

While it may or may not help, I will note that when I used to run into this issue it sometimes helped to clear the cache, close and re-start the browser, and then clear it again before visiting the fanfic to load the chapters.

[dysphere]

I am using Chrome, yes. It does appear to be a problem with the Vivaldi/Chrome cache.