MobileRead Forums - View Single Post

JimmXinu · 12-23-2020, 11:59 AM

New Test Version Posted
(Back in the usual place.)

2020-12-23
- Adding cloudscraper and dependencies
- Updates for many included dependencies
- Need to semi-manually raise HTTPError for error codes with cloudscraper/requests.
- Temp: dump ffnet page to debug when metadata parsing fails.
- Clean up cloudscraper import
- Use requests.util.extract_zipped_paths to get browsers.json instead of kludge.
- Don't use mobile User-Agents with cloudscraper--adapter_fanfictionnet doesn't handle mobile pages.
- Document use_cloudscraper better in defaults.ini.

This version includes the module cloudscraper and dependencies and is configured to use it for fanfiction.net and fictionpress.com.

This probably won't help with ffnet in older Calibre versions due to CloudFlare detecting that older versions of Calibre bundle older versions of OpenSSL. If an older version works for you, great, but don't expect it to.

It does still work in older versions for other sites.

I have tested both CLI in Python3 and plugin in Calibre v5.7.2 on Win10, Mac(BigSur) and Linux.

UPDATE: I've written in more detail here, in the github issue.

12-23-2020, 11:59 AM	#5029
JimmXinu Plugin Developer Posts: 7,009 Karma: 4604635 Join Date: Dec 2011 Location: Midwest USA Device: Kobo Clara Colour running KOReader	New Test Version Posted (Back in the usual place.) 2020-12-23 - Adding cloudscraper and dependencies - Updates for many included dependencies - Need to semi-manually raise HTTPError for error codes with cloudscraper/requests. - Temp: dump ffnet page to debug when metadata parsing fails. - Clean up cloudscraper import - Use requests.util.extract_zipped_paths to get browsers.json instead of kludge. - Don't use mobile User-Agents with cloudscraper--adapter_fanfictionnet doesn't handle mobile pages. - Document use_cloudscraper better in defaults.ini. This version includes the module cloudscraper and dependencies and is configured to use it for fanfiction.net and fictionpress.com. This probably won't help with ffnet in older Calibre versions due to CloudFlare detecting that older versions of Calibre bundle older versions of OpenSSL. If an older version works for you, great, but don't expect it to. It does still work in older versions for other sites. I have tested both CLI in Python3 and plugin in Calibre v5.7.2 on Win10, Mac(BigSur) and Linux. UPDATE: I've written in more detail here, in the github issue.