[GUI Plugin] FanFicFare

[JimmXinu]

pjm13,

Those numbers should not be high enough to be an issue.

The debug log should list a lot more output and have timestamps indicating when each line generated. When running in debug mode, Calibre automatically shows you the debug log txt file when you quit.

It's vaguely possible that there's a problem with one of the specific emails in question. I would try manually marking all but one read and trying it again, then marking each of the problematic emails unread one at a time and trying it to see if the problem can be narrowed down to one email.

[pjm13]

Originally Posted by JimmXinu

pjm13,

Those numbers should not be high enough to be an issue.

The debug log should list a lot more output and have timestamps indicating when each line generated. When running in debug mode, Calibre automatically shows you the debug log txt file when you quit.

It's vaguely possible that there's a problem with one of the specific emails in question. I would try manually marking all but one read and trying it again, then marking each of the problematic emails unread one at a time and trying it to see if the problem can be narrowed down to one email.

Oh, I didn't know about the .txt file. I closed calibre and got this now:

FFF: DEBUG: 2022-01-12 12:39:10,808: calibre_plugins.fanficfare_plugin.fff_plugin(221): Plugin FanFicFare macmenuhack file_path:C:\Users\mine\AppData\Roaming\calibre\pl ugins\fanficfare_macmenuhack.txt
calibre Debug log
calibre 5.34 embedded-python: True is64bit: False
Windows-10-10.0.22000 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.22000')
Python 3.8.5
Windows: ('10', '10.0.22000', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Count Pages (1, 11, 1) && FanFicFare (4, 8, 0) && Generate Cover (2, 0, 2)
calibre 5.34 embedded-python: True is64bit: False
Windows-10-10.0.22000 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.22000')
Python 3.8.5
Windows: ('10', '10.0.22000', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Count Pages (1, 11, 1) && FanFicFare (4, 8, 0) && Generate Cover (2, 0, 2)
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 157.8252427184466 x 157.65517241379308
Using calibre Qt style: True
[0.00] Starting up...
[0.00] Showing splash screen...
[0.33] splash screen shown
[0.33] Initializing db...
[0.49] db initialized
[0.49] Constructing main UI...
[1.80] main UI initialized...
[1.80] Hiding splash screen
Starting QuickView
calibre 5.34 embedded-python: True is64bit: False
Windows-10-10.0.22000 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.22000')
Python 3.8.5
Windows: ('10', '10.0.22000', '', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Count Pages (1, 11, 1) && FanFicFare (4, 8, 0) && Generate Cover (2, 0, 2)
Traceback (most recent call last):
File "calibre_plugins.fanficfare_plugin.fff_plugin" , line 572, in get_urls_from_imap_menu
File "C:\Users\ramon\AppData\Roaming\calibre\plugins\Fa nFicFare.zip\fanficfare\geturls.py", line 233, in get_urls_from_imap
File "imaplib.py", line 881, in uid
File "imaplib.py", line 1205, in _simple_command
File "imaplib.py", line 1022, in _command_complete
File "imaplib.py", line 1148, in _get_tagged_response
File "imaplib.py", line 1050, in _get_response
File "imaplib.py", line 1158, in _get_line
File "imaplib.py", line 314, in readline
File "socket.py", line 669, in readinto
File "ssl.py", line 1241, in recv_into
File "ssl.py", line 1099, in read
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

[13.09] splash screen hidden
[13.09] Started up in 13.09 seconds with 1855 books

Hope this is what you meant?

I will anyways do what you suggested next (to find out if one specific email is the culprit)! Thank you!

[chrisric]

Originally Posted by JimmXinu

Before the next release. Wiki page(s) will need to change, too.

novelfull.com needs FlareSolverr now. I have mostly given on trying to deal with CloudFlare directly. See this wiki page.

Hi JimmXinu - Just FYI as you tried helping me with FlareSolverr a few days back. That wiki entry fixed my issue.. I had no idea we had to add the personal.ini config.

I figured there was no FlareSolverr specific config as it's not listed in the defaults.ini. I was just pointing my Mac's proxy settings to FlareSolverr which didn't work of course.

Thanks for the hard work, life would be terrible without FFF.

-Chris

[JimmXinu]

New Test Version Posted

- Changes for upcoming Qt6 Calibre

This version contains changes necessary for the upcoming Qt6 Calibre version.

I haven't tested against every version, but this should still be backward compatible to v2.85.1 and newer.

[JimmXinu]

Originally Posted by chrisric

I figured there was no FlareSolverr specific config as it's not listed in the defaults.ini. I was just pointing my Mac's proxy settings to FlareSolverr which didn't work of course.

Fair point. I've added to defaults.ini for the next version:

## As a (second) work around for certain sites blocking automated
## downloads, FFF offers the ability to request pages through nsapa's
## fanfictionnet_ff_proxy and FlareSolverr proxy servers.  See
## https://github.com/JimmXinu/FanFicFare/wiki/ProxyFeatures for more
## details.

## FlareSolverr (https://github.com/FlareSolverr/FlareSolverr) is a
## generic proxy that works with several otherwise blocked sites.
## Note that FFF cannot fetch images through FlareSolverr.
## It's recommended to only set use_flaresolverr_proxy:true for specific sites.
#[www.fanfiction.net]
#use_flaresolverr_proxy:true
## option settings, these are the defaults:
#flaresolverr_proxy_address:localhost
#flaresolverr_proxy_port:8191
#flaresolverr_proxy_protocol:http

[Ian.M]

Since upgrading to FanFicFare version 4.9.0, I'm getting 'Unmatched group' errors on all SufficientVeocity and SpaceBattles fics I try to update. Here are two examples:

Status
Title
Author
Comment
URL
Error
Abaddon Born
PublicLee S.
unmatched group
https://forums.sufficientvelocity.com/threads/abaddon-born-e-worm-cyoa.46367/
Error
Ring-Maker
LithosMaitreya
unmatched group
https://forums.spacebattles.com/threads/ring-maker-worm-lord-of-the-rings-alt-power.517894/

I don't have any personal.ini sections for [forums.spacebattles.com], [forums.sufficientvelocity.com], or [base_xenforoforum].

I've attached a Calibre debug log of an attempt to update one fic from SufficientVeocity using version 4.9.0. The same fic updates normally after rolling back FanFicFare to version 4.8.0.

[pjm13]

Update on the E-Mail fetching problem from earlier (in case other people face the same issue):
I did as you said (marked one E-Mail at a time as unread) and was able to fetch all emails like this. I don't know what exactly was the problem because ALL emails worked if fetched individually, so I don't know if one of them made problems for the other ones.
I also now remove them from the fetching folder as soon as the fetching was successful to hopefully avoid this problem in the future.

[JimmXinu]

Originally Posted by Ian.M

Since upgrading to FanFicFare version 4.9.0, I'm getting 'Unmatched group' errors on all SufficientVeocity and SpaceBattles fics I try to update. Here are two examples:

I'm glad you included the debug log. It showed you are using Calibre 4.23 and this is a py2 vs py3 issue with regular expression handling.

A recent change to better handle 'tags' in thread titles on XenForo sites works in py3, but not py2.

Here's a stackoverflow question about it.

I've opened an issue and it will be addressed, but I want to give the hseg a chance at it first.

In the meantime, you (Ian.M) could:

1. Continue to use the previous FFF version
2. Switch to Calibre 5 and the latest FFF
3. Reinstate the previous tagstitle handling in your personal.ini. Spoiler below.

Since you said you don't already have a [base_xenforoforum], it should be pretty easy to just put this in your personal.ini.

Spoiler:

[base_xenforoforum]
add_to_replace_metadata:
# 'detects' if there are 'tags'. That is, [( in title.  Replaces
# previous version's include_metadata_pre Can't do on tagsfromtitle
# because that's applied to each part after split.
 tagsfromtitledetect=>^[^\]\)]+$=>
# for QuestionableQuesting NSFW subforum.
## tagsfromtitle=>^\[NSFW\].*?(\[([^\]]+)\]|\(([^\)]+)\)).*?$=>NSFW\,\2\,\3
 tagsfromtitle=>^\[NSFW\].*?([\(\[]([^\]\)]+)[\)\]]).*?$=>NSFW,\2
# remove anything outside () or []
## tagsfromtitle=>^.*?(\[([^\]]+)\]|\(([^\)]+)\)).*?$=>\2\,\3
 tagsfromtitle=>^.*?([\(\[]([^\]\)]+)[\)\]]).*?$=>\2
# remove () []
# tagsfromtitle=>[\(\)\[\]]=>
# shield these html entities from the ';' pattern below
 tagsfromtitle=>&(amp|lt|gt);=>&\1FFF_ENT_MARKER
# change (spaces)slash(or semicolon, pipe or comma)(spaces) to comma
 tagsfromtitle=> *[/;|,] *=>,
# revert html entities
 tagsfromtitle=>&(amp|lt|gt)FFF_ENT_MARKER=>&\1;
 tagsfromtitle=> [xX] =>,
# split commas into individual list items.
# these will work on up to 14 splits, should be plenty.
 tagsfromtitle=>([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),(.+)=>\1\,\2\,\3,\4\,\5\,\6
 tagsfromtitle=>([^,]+),([^,]+),([^,]+),([^,]+),(.+)=>\1\,\2\,\3,\4\,\5
 tagsfromtitle=>([^,]+),([^,]+),([^,]+),(.+)=>\1\,\2\,\3,\4
 tagsfromtitle=>([^,]+),([^,]+),(.+)=>\1\,\2\,\3
 tagsfromtitle=>([^,]+),(.+)=>\1\,\2

# remove [] or () blocks and leading/trailing spaces/dashes/colons
## title=>[-: ]*[\(\[]([^\]\)]+)[\)\]][-: ]*=>
 title=>[-: ]*(\[([^\]]+)\]|\(([^\)]+)\))[-: ]*=>
# remove 'Thread' and the next word, usually "Thread 2", "Thread
# four", "Thread iv", "Story Thread", etc
 title,tagsfromtitle=>[-: ,]*(Story *)?[Tt]hread [^ ]+[-: ]*=>

# Normalize 'fanfiction/fanfic/fan-fiction' a little.
 forumtags=>[Ff]an-?[Ff]ic(tion)?=>FanFiction

[Ian.M]

I'm still using Calibre 4.23 because I've got another PC with Calibre 2.0 that I cant upgrade (its on XP and the hardware isn't suitable for any newer Windows OS) that I need to sync the Calibre library with. When Calibre 5.0 came out, I read https://calibre-ebook.com/new-in/fourteen :

Backwards incompatibilities
There are a few minor backwards incompatibilities to be aware of in calibre 5
...
* Because of the support for library wide annotations, the database format was slightly changed. While calibre 4.23 is able to open and operate with calibre 5 databases, older versions of calibre may have trouble.

Until I bite the bullet and put another solution together for reading my ebooks on a big screen in that room, so I can retire Calibre 2.0 and scrap the XP PC, I'm not likely to upgrade from 4.23.

However, many thanks for the suggested [base_xenforoforum] tagstyle rollback via personal.ini. I'll probably give it a shot next time I need to update a SufficientVeocity or SpaceBattles fic.

[ownedbycats]

Question: How would I preserve the right-align on this story? I'm not too worried about the other styling (though it'd be nice to keep it), but it's hard to read it when it's all left-aligned. I have this:

[archiveofourown.org]
## Some attributes cause problems for EBook readers.  By default,
## FanFicFare will remove all attributes except the ones specified
## from all tags.  (The only exception is that <img> tags will also
## keep src, alt and longdesc attributes.)
## Example: To add 'style', 'title' and 'align' to the list to keep,
## in your personal.ini [defaults] put:
add_to_keep_html_attrs:,title,align,blockquote,div

But it's not working here. Inspect element shows that the author is using some custom classes (wait, that's possible?) but I'm not sure how to handle that -- even if I knew how to add the "message sent" class to the config it wouldn't be attached to the right-align CSS.

Downloading the ePub (or even the HTML) from AO3 directly also has the same issue and I got a network error with WebToEpub okay WebToEpub doesn't work with "view full work," but I still got no styling with it.

[JimmXinu]

Yes, AO3 has a feature for authors to make custom 'work skins'; basically including custom CSS for their stories. AO3 also lets you create & use custom skins for the site itself, but that's distinct and for the user vs the author.

I know that I looked at it in detail before (and even used it for a story), but elected not to include an FFF feature for it. For... reasons?

I probably judged it too much effort for too little reward. I think most authors generate their HTML with style attrs instead of custom CSS. At least, I don't remember seeing very many stories that use it.

FFF's customization features should allow you to apply it for a given story anyway with relatively low effort.

Search the story source HTML for "workskin" and you'll find a style tag with the work skin CSS you can copy.

Depending on what's in the CSS and how it's applied, you probably need to either remove the #workskin selectors from the CSS or add the id="workskin" attr to your chapter HTML appropriately. That second could be tricky? Oh, and escape any % to %% for INI.

This appears to be sufficient for that story, copy paste, find/replace, escape %, done:

[https://archiveofourown.org/works/36423070]
add_to_output_css:
 p{font-family:'Helvetica',sans-serif;margin-top:7px;margin-bottom:7px;margin-right:15px;margin-left:15px}.chat{width:350px;margin:auto}.leftlabel{color:#9f9fa0;font-size:11px;font-weight:bold;margin-bottom:0;float:left;margin-left:0}.rightlabel{color:#9f9fa0;font-size:11px;font-weight:bold;margin-bottom:0;float:right;margin-right:0}.centerlabel{color:#9f9fa0;font-size:11px;text-align:center}.line{display:inline-block;width:100%%}.message{font-size:16px;max-width:300;overflow-wrap:break-word;margin-bottom:10px;display:block}.sent{border-radius:30px 30px 5px 30px;background:#0b82ff;color:#fff;float:right}.received{border-radius:30px 30px 30px 5px;background:#e5e4ea;color:#000;float:left}

[ownedbycats]

That works, thank you.

[chrisric]

Is there a way to save partial chapter downloads of books/novels? I can't see the option, I might be blind though.

I thought I seen a "save failed downloads" option (timeout/not all chapters downloaded etc) once but now I can't see it. Might be wishful thinking.

Thanks.

-Chris

[ownedbycats]

## Because some adapters can pull chapter URLs from human posts, the
## odds of errors in the chapter URLs can be higher for some
## sites/stories.  You can set continue_on_chapter_error:true to
## continue on after failing to download a chapter and instead record
## an error message in the ebook for that chapter.
continue_on_chapter_error:false

[Emerald_Lilies]

First off, thank you so much for all your work!

I've been going through the forum (and slowly teaching myself regex) and I haven't been able to figure this out. I have my personal.ini set up with onlyromships to only include pairings with a "/". When I get a fic that only has an "&" gen pairing in ships, it won't remove that pairing.

https://archiveofourown.org/works/36...pters/90879328

The relevant (I think?) bit of my personal.ini (and the whole thing attached):

Spoiler:

[archiveofourown.org]
## adds to include_subject_tags instead of replacing it.
extra_subject_tags:fandoms,freeformtags,ao3categories

include_in_onlyromships:ships

add_to_include_metadata_post:
 onlyromships=~[ ]*/[ ]*

#add_to_exclude_metadata_post:
#onlyromships=~&

add_to_extra_valid_entries:,ships2chars

include_in_ships2chars:ships.NOREPL
# NOREPL prevents any ships patterns from being applied to ships2chars.

add_to_replace_metadata:
 ships2chars=>[ ]*/[ ]*=>/
 ships2chars=>[ ]*&[ ]*=>/
 ships2chars=>([^/]+)/([^/]+)/([^/]+)/(.+)=>\1\,\2\,\3\,\4
 ships2chars=>([^/]+)/([^/]+)/(.+)=>\1\,\2\,\3
 ships2chars=>([^/]+)/(.+)=>\1\,\2

include_in_characters:characters,ships2chars

I've tried it with the add_to_include_metadata_post and the add_to_exclude_metadata_post bits both included and individually. Not sure where to go from here.