[GUI Plugin] FanFicFare

[ownedbycats]

quick question: If I edit an existing chapter (e.g. removing excess linebreaks), the edits should remain unless I use one of the 'overwrite' options, right? I can't quite remember

[Atherton25]

Originally Posted by ownedbycats

quick question: If I edit an existing chapter (e.g. removing excess linebreaks), the edits should remain unless I use one of the 'overwrite' options, right? I can't quite remember

That has always been my understanding, and I've yet to see different. FFF only adds chapters unless you have a overwrite option selected

[chrisric]

Hi - Has scribblehub.com started blocking FFF? Or maybe I'm doing something wrong?

Debug:

Spoiler:

EpubMerge: DEBUG: 2022-12-26 09:46:48,804: calibre_plugins.epubmerge.epubmerge_plugin(155): macmenuhack file_path:/Users/chrisric/Library/Preferences/calibre/plugins/fanficfare_macmenuhack.txt
FFF: DEBUG: 2022-12-26 09:46:48,824: calibre_plugins.fanficfare_plugin.fff_plugin(218): Plugin FanFicFare macmenuhack file_path:/Users/chrisric/Library/Preferences/calibre/plugins/fanficfare_macmenuhack.txt
QLayout::addChildLayout: layout "" already has a parent
calibre Debug log
calibre 5.25 embedded-python: True is64bit: True
macOS-10.15.7-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '19.6.0', 'Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64')
Python 3.8.5
OSX: ('10.15.7', ('', '', ''), 'x86_64')
Interface language: None
Successfully initialized third party plugins: Count Pages (1, 11, 2) && EpubCheck (0, 2, 4) && EpubMerge (2, 13, 0) && EpubSplit (3, 3, 0) && FanFicFare (4, 13, 5) && Generate Cover (2, 1, 0) && Goodreads (1, 5, 3) && Modify ePub (1, 7, 3)
calibre 5.25 embedded-python: True is64bit: True
macOS-10.15.7-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '19.6.0', 'Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64')
Python 3.8.5
OSX: ('10.15.7', ('', '', ''), 'x86_64')
Interface language: None
Successfully initialized third party plugins: Count Pages (1, 11, 2) && EpubCheck (0, 2, 4) && EpubMerge (2, 13, 0) && EpubSplit (3, 3, 0) && FanFicFare (4, 13, 5) && Generate Cover (2, 1, 0) && Goodreads (1, 5, 3) && Modify ePub (1, 7, 3)
devicePixelRatio: 2.0
logicalDpi: 72.0 x 72.0
physicalDpi: 98.00000147181235 x 98.00000147181234
Using calibre Qt style: True
[0.00] Starting up...
[0.02] Showing splash screen...
[0.30] splash screen shown
[0.30] Initializing db...
[0.41] db initialized
[0.41] Constructing main UI...
[2.49] main UI initialized...
[2.49] Hiding splash screen
Starting QuickView
FFF: DEBUG: 2022-12-26 09:47:13,181: calibre_plugins.fanficfare_plugin.fff_plugin(1129) : FanFicFare v4.13.5
FFF: DEBUG: 2022-12-26 09:47:13,327: configurable.py(1045): use_browser_cache:
FFF: DEBUG: 2022-12-26 09:47:13,327: configurable.py(1059): use_basic_cache:true
FFF: DEBUG: 2022-12-26 09:47:13,328: adapter_scribblehubcom.py(118): URL: https://www.scribblehub.com/series/2...-up-the-world/
FFF: DEBUG: 2022-12-26 09:47:13,328: fetcher.py(234):
========== MISS (GET) BasicCache
https://www.scribblehub.com/series/2...-up-the-world/
FFF: DEBUG: 2022-12-26 09:47:13,328: fetcher.py(469):
---------- REQ (GET) RequestsFetcher
https://www.scribblehub.com/series/2...-up-the-world/
FFF: DEBUG: 2022-12-26 09:47:13,457: fetcher.py(482): response code:403
FFF: ERROR: 2022-12-26 09:47:13,457: calibre_plugins.fanficfare_plugin.dialogs(714): Exception: {'title': 'Leveling up the World', 'author_sort': 'Lise', 'author': ['Lise'], 'comments': '', 'good': False, 'status': 'Error', 'showerror': True, 'calibre_id': 778, 'begin': None, 'end': None, 'comment': "HTTP Error in FFF '403 Client Error: Forbidden for url: https://www.scribblehub.com/series/231006/leveling-up-the-world/'(403)", 'url': 'https://www.scribblehub.com/series/231006/leveling-up-the-world/', 'site': 'www.scribblehub.com', 'series': '', 'added': False, 'pubdate': None, 'publisher': 'www.scribblehub.com', 'listorder': 0, 'collision': 'Update EPUB if New Chapters'}:HTTP Error in FFF '403 Client Error: Forbidden for url: https://www.scribblehub.com/series/2...orld/'(403)
Traceback (most recent call last):
File "/Users/chrisric/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/fetcher.py", line 483, in request
resp.raise_for_status() # raises RequestsHTTPError if error code.
File "/Users/chrisric/Library/Preferences/calibre/plugins/FanFicFare.zip/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://www.scribblehub.com/series/2...-up-the-world/

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "calibre_plugins.fanficfare_plugin.dialogs", line 701, in do_loop
self.foreach_function(book)
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 1343, in prep_download_loop
story = self.get_story_metadata_only(adapter)
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 1215, in get_story_metadata_only
adapter.getStoryMetadataOnly(get_cover=False)
File "/Users/chrisric/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/adapters/base_adapter.py", line 308, in getStoryMetadataOnly
self.doExtractChapterUrlsAndMetadata(get_cover=get _cover)
File "/Users/chrisric/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/adapters/base_adapter.py", line 412, in doExtractChapterUrlsAndMetadata
return self.extractChapterUrlsAndMetadata()
File "/Users/chrisric/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/adapters/adapter_scribblehubcom.py", line 120, in extractChapterUrlsAndMetadata
data = self.get_request(url)
File "/Users/chrisric/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/requestable.py", line 114, in get_request
return self.get_request_redirected(url,
File "/Users/chrisric/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/requestable.py", line 106, in get_request_redirected
(data,rurl) = self.configuration.get_fetcher().get_request_redir ected(
File "/Users/chrisric/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/fetcher.py", line 398, in get_request_redirected
fetchresp = self.do_request('GET',
File "/Users/chrisric/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/fetcher.py", line 240, in fetcher_do_request
fetchresp = chainfn(
File "/Users/chrisric/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/fetcher.py", line 136, in fetcher_do_request
fetchresp = chainfn(
File "/Users/chrisric/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/fetcher.py", line 371, in do_request
fetchresp = self.request(method,url,
File "/Users/chrisric/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/fetcher.py", line 504, in request
raise exceptions.HTTPErrorFFF(
fanficfare.exceptions.HTTPErrorFFF: HTTP Error in FFF '403 Client Error: Forbidden for url: https://www.scribblehub.com/series/2...orld/'(403)
[16.53] splash screen hidden
[16.53] Started up in 16.53 seconds with 813 books

Thanks.

-Chris

[Atherton25]

Originally Posted by chrisric

Hi - Has scribblehub.com started blocking FFF? Or maybe I'm doing something wrong?

Just tried it. Granted, I had a little over 600 chapters of that fic already downloaded, but the 20 chapters I downloaded for that fic downloaded fine.
To check some things, ry using the specific chapter count IE https://www.royalroad.com/fiction/39026[1] for just the first chapter, or https://www.royalroad.com/fiction/39026[1,30] (for chap 1-30) and see if it works.

If that works, and you were downloading all (or most) of the fic perhaps downloading the entire thing may have caused it the website to reject your requests as 640 chapters is quite a bit. You may have gotten a temp ban.

In that case, you have 2 options. I think you could do https://www.royalroad.com/fiction/39026[31-60] after the first 30 chapters progressively building up to the 640. I'm guessing 50 or even 100 chapters at a time wouldn't be a big deal. The other option is change/up the sleep times for scribblehub.
IE:

[www.scribblehub.com]
slow_down_sleep_time:0.5
tweak_fg_sleep:true
min_fg_sleep:1.0
max_fg_sleep:3.0
max_fg_sleep_at_downloads: 10
connect_timeout:60.0

Anyway, good luck. Oh and Merry Christmas.

[chrisric]

Originally Posted by Atherton25

Just tried it. Granted, I had a little over 600 chapters of that fic already downloaded, but the 20 chapters I downloaded for that fic downloaded fine...

Thanks for the help Atherton25.

Must be an issue on my end if it works for you, I'll upgrade Calibre and FFF to the latest versions and try again (not likely to be the issue though)

I used a VPN to test and it comes back with a 503 error which is weird. The 403 error I get without VPN comes up before it even tries to pull chapters, I think my IP has been banned/blocked for some reason.

Cheeers.

-Chris

[ownedbycats]

Depending on your ISP, unplugging your modem (if you have access) for a few minutes may give you a new IP address.

[cryosaur]

I too have been having the same problem for the past few weeks: adding/updating anything from scribblehub.com fails with 403 error. The site is perfectly visible and functional in a web browser on the same computer--immediately before giving a link to FFF and after it fails--but it seems that FFF cannot get a valid page back from any address in the domain.

Tried changing the user_agent string, but still got the 403 error. Also tried messing with the sleep settings like Atherton25 suggested, but this also had no effect.

[WovenTales]

It might have been nice if I had thought of this before downloading over five hundred fics, but I've just realized I can put the `description` text into a custom column and leave Calibre's "Comments" open for more personal notes... or, at least the first half of that. I can't figure out how to disable the comments population itself. The preferences panel does list it on the "Standard Columns" tab, but only as a "New only" toggle and not a complete disable, and I didn't see (but could easily have missed) anything in the INI. Is that something which can be done in the current version of FFF, or is it a feature request?

[criscros]

I have a couple of webnovels that have been stubbed, so now FFF won't update because the chapters are different. If I do overwrite always, I lose the missing chapters.
Is there a way to update and not lose chapters?

[ownedbycats]

I'd suggest just renaming to FanficTitle (Old Version) and downloading new; often removed chapters are indicative of big rewrites.

[criscros]

Originally Posted by ownedbycats

I'd suggest just renaming to FanficTitle (Old Version) and downloading new; often removed chapters are indicative of big rewrites.

Makes sense. Even though I probably wouldn't reread the rewrite, I'd like the old chapters to be able to back reference things.
Thanks!

[chrisric]

Originally Posted by chrisric

Hi - Has scribblehub.com started blocking FFF? Or maybe I'm doing something wrong?

FYI for closure... I tried this again today, same error. I updated Calibre and FFF to latest versions and now it works fine.

No idea why, I didn't see any scribblehub fixes in the notes.

Anyway, Thanks for the help.

-Chris

[JimmXinu]

Back from trip Multi-reply:

Originally Posted by LadyLight

I have the same question.

add_to_replace_metadata:
 ..
 ships=>(.*)[&](.*)=>\1 and \2

almost works. but then the output looks like:

<CharacterA> and amp; <CharacterB>

For various reasons, & chars are stored internally as html entities, so you need to match &amp; instead of just &:

add_to_replace_metadata:
 ..
 ships=>(.*)&(.*)=>\1 and \2

Originally Posted by WovenTales

It might have been nice if I had thought of this before downloading over five hundred fics, but I've just realized I can put the `description` text into a custom column and leave Calibre's "Comments" open for more personal notes... or, at least the first half of that. I can't figure out how to disable the comments population itself. ...

The Comments field New Only flag is as close as it gets for that. I frankly don't consider it a common enough use case to add code for.

You could create a copy of description with include_in_newentry:description.NOREPL, set that to your custom column and clear the contents of description with replace_metadata or the include/exclude settings, but you'd also have to change the title page definition.

Originally Posted by chrisric

FYI for closure... I tried this again today, same error. I updated Calibre and FFF to latest versions and now it works fine.

No idea why, I didn't see any scribblehub fixes in the notes.

This is almost certainly you (meaning your IP, IP range, or possibly ISP) being flagged on the site or their service provider for extra screening for a period of time that has now expired.

[JimmXinu]

New Test Version Posted

2022-12-30
- Fixes for config base_xenforo options, closes #902

[WovenTales]

Originally Posted by JimmXinu

Back from trip Multi-reply:
The Comments field New Only flag is as close as it gets for that. I frankly don't consider it a common enough use case to add code for.

You could create a copy of description with include_in_newentry:description.NOREPL, set that to your custom column and clear the contents of description with replace_metadata or the include/exclude settings, but you'd also have to change the title page definition.

That's fair. Thanks for the suggestion; I'll look into that, along with finally trying to figure out Action Chains, since I was wanting to do something with them for a "Dormant" status anyway.

And thanks for the fast response to the treadmarks, as well! Everything's working as advertised.