[GUI Plugin] FanFicFare

[JimmXinu]

New Test Version Posted

2021-01-28
- Plugin: Only run one BG download at time.
- Get cloudscraper 1.2.56 changes, but keep py2 compat.

Some users over on the mailing list pointed out running parallel downloads on the same site is more likely to trip blocking. This version prevents that by only running one download at a time.

I said over there:

So, I've been researching this a bit more.

First, FFF plugin launches a background job in Calibre. As V1cT1T10uS pointed out, Calibre has a setting for how many BG jobs to run at a time--often set (or limited) to the number of CPUs.

The weird part is that FFF has *also* been launching that many parallel process again from its BG process. So if you launched FFF with a bunch of downloads *and* started a bunch of converts or something, Calibre could technically be running twice the configured number of background processes (but not jobs in the job list).

Way back when, I lifted the BG processing code for FFF plugin from the ISBN plugin and I honestly didn't understand exactly what was going on at the time. The inherited comment even says that it launched a process per book to work around a memory leak issue in ISBN.

I've posted a test version plugin that only does one download at a time:
https://www.mobileread.com/forums/sh...25&postcount=2

Currently, it still launches a separate process for each download, but I will be experimenting with what can be done--all in one process (like CLI does lists), maybe process per site.

[ownedbycats]

Currently, it still launches a separate process for each download, but I will be experimenting with what can be done--all in one process (like CLI does lists), maybe process per site.

For what it matters, before I switched to email drag-and-drop I used to run two parallel update jobs - one for ffnet fics and one for ao3 fics. It also had the advantage of running a bit faster.

[ownedbycats]

I did a bit of investigating into the eml issue. It's due to the presence of two Unicode Latin-1 supplement characters (in this case, the chapter title is "Déjà-vu") in the message body. Honestly something that didn't occur to me at first until I went mucking around in the geturls.py code.

I was able to confirm this by saving a copy of the .eml and modifying it to remove the characters. Also emailing myself two messages: one containing just the url, and the other one containing the URL and chapter title. Former worked as expected, latter pasted the filepath.

ffnet uses Content-Type: text/plain; charset="utf-8" for its messages.

Haven't tested whether the issue is limited to Latin-1 supplement. I did notice this in geturls.py:

def get_urls_from_text(data,configuration=None,normalize=False,email=False):
    urls = collections.OrderedDict()
    try:
        # py3 can have issues with extended chars in txt emails
        data = ensure_str(data,errors='replace')
    except UnicodeDecodeError:
        data = data.decode('utf8') ## for when called outside calibre.

I hope this helps a bit.

[JimmXinu]

Originally Posted by JimmXinu

Calibre could technically be running twice the configured number of background processes (but not jobs in the job list)

Originally Posted by ownedbycats

For what it matters, before I switched to email drag-and-drop I used to run two parallel update jobs - one for ffnet fics and one for ao3 fics. It also had the advantage of running a bit faster.

Congratulations! You managed to (potentially) triple your configured Calibre processes instead of double them.


Although automatically launching one BG job per site is an idea I considered. I'm not sure the complications are worth it for most users. Separate 'Proceed with updating you library' dialogs per site/BG job--or a lot more complicated update code.

There are some other ideas, but there are tradeoffs. I will come back to it in a while.

Originally Posted by ownedbycats

I did a bit of investigating into the eml issue. It's due to the presence of two Unicode Latin-1 supplement characters (in this case, the chapter title is "Déjà-vu") in the message body. Honestly something that didn't occur to me at first.

If you want to figure out a fix and make a pull request in github, I'll look at it.

But it's a pretty obscure corner case (and possibly OS dependent?) and I'm busy else where in the system right now. I'm only looking at the BG jobs issue because it's tangentially related to what I'm working on.

[ownedbycats]

Yeah its pretty low priority (I guess it might be a bigger problem for users who read non-English fanfics). Just something to look at again if you need to change that bit of code again at some point.

EDIT: I did mess around with it a bit with some things I googled. Changing errors='strict' didn't work, neither did adding encoding='utf-8'. Commenting the line out to see what would happen caused FFF to fail to load at all. This is why I am not a programmer.

ANOTHER EDIT: Although I couldnt fix it myself, I did file the bug on the Github mostly for documentation purposes.

[ackomb]

Originally Posted by JimmXinu

Sorry, I knew that but failed communicate it to you. (I was testing in an empty library with just one book, so clearing the column did remove it from all stories.)

I believe you might also be able to change the 'canonical' spelling Calibre uses in Calibre's tag browser?

I, too, am not aware of anyway to change Calibre behavior about this.

No worries, I'm just happy to have a solution for my issue .
At least it wasn't the code I'd written. It's a bit of work to adjust it, but once it's done the issue should be gone for future downloads.
Thanks again.

[thehoboking]

Hello,

Just wanted to report an error I'm getting when trying to download novels from Webnovel.com

Status: Error
Title: Scholar's Advanced Technological System
Author: Morning Star LL
Comment: HTTP Error 404: Not Found
URL: https://www.webnovel.com/book/12583970906002305

Thanks

[JimmXinu]

Originally Posted by thehoboking

Hello,

Just wanted to report an error I'm getting when trying to download novels from Webnovel.com

Status: Error
Title: Scholar's Advanced Technological System
Author: Morning Star LL
Comment: HTTP Error 404: Not Found
URL: https://www.webnovel.com/book/12583970906002305

Thanks

This has also been reported in Issue #644.

I replied:

Something in the site appears to have changed.

I didn't implement this adapter, don't use it, and don't know how it works. Getting it fixed is not high priority for me right now.

Perhaps someone who does use it wants to look into it?

[JimmXinu]

New Test Version Posted

2021-01-30
- Fix for Reject List to equate story URLs with and without titles, for XenForo and ffnet. Otherwise you can end up with dups in Reject List and not recognize new URLs are already being rejected. Also fixes [storyUrl] sections without titles for ffnet.
- Fix fictionpress.net and retain Reject List entries for removed sites.

UPDATE: I found a couple bugs immediately after uploading (as you do), fixed and re-uploaded.

[JimmXinu]

New Test Version Posted

2021-01-31
- Major refactoring/rewrite of network fetching and caching layer using requests library.
- Convert adapter_spikeluvercom to base_efiction after site changes.
- Remove 'extrasleep' feature--it was ugly--increase slow_down_sleep_time for the couple sites that used it. We can trust the users, right?
- Remove httplib._MAXHEADERS workaround for royalroad.com - not happening anymore.
- Bump Test Version 3.99.X

It's been working for me, but this is a major update that I've been working on for the last week. It touches most of the files in the system.

Approach with caution.

And I just realized I haven't tested other than Windows yet. Double caution.

UPDATE: And already found one bug with py2 Calibre 2.85.1

[ownedbycats]

Is the extrasleep part of why some updates of ffnet fics seemed to take a silly long amount of time? I initially thought it was solely due to the removal of the extra BG process.

[JimmXinu]

Newer Test Version Posted

2021-01-31 Take 2
- Major refactoring/rewrite of network fetching and caching layer using requests library.
- Convert adapter_spikeluvercom to base_efiction after site changes.
- Remove 'extrasleep' feature--it was ugly--increase slow_down_sleep_time for the couple sites that used it. We can trust the users, right?
- Remove httplib._MAXHEADERS workaround for royalroad.com - not happening anymore.
- Bump Test Version 3.99.X
- Fixes for py2/py3 & a stray INI space.

Basic testing done on Win10Cal2.85.1 & Cal5, macos Cal5 and linux Cal5.


'extrasleep' was an internal thing that was added to hardcode extra sleep time into requests for certain sites. In practice, it was only ever used with base_xenforo (SB, SV, QQ, etc) and ffnet.

It complicated the 'fetcher' layer while--in theory--being redundant with just increasing slow_down_sleep_time. Its advantage was users couldn't easily turn it down or off.


ffnet download times are slower now both because of serializing BG downloads and increased slow_down_sleep_time for ffnet.

I plan to revisit parallel BG downloads, but with an eye to having parallel queues for different sites, so each site only has one download at a time, but different sites can run in parallel.

[ownedbycats]

Question (more of a theoretical thing for the moment):

Can any metadata entry be used as a conditionalkey, or just category?

e.g.

category=>generalcategory=>morespecificcategory&&characters=>(Characters|That|Only|Appear|In|One|Entry|Of|A|Series)

[darth1]

My apology's if this has been answered but has anyone else had this error? Because I normally have to wait 5 hrs at the bare minimum before fanficfare will work and am using 3.29.2


Status
Title
Author
Comment
URL
Error
Diligence of the Shield Hero
Allen Blaster
cloudscraper reports: "Cloudflare IUAM possibility malformed, issue extracing delay value."
https://www.fanfiction.net/s/1335028...he-Shield-Hero

the cloudflare thing comes up a "lot" lately

[ownedbycats]

Someone else had the issue on Github, I think it cleared up for them after a day or two. The test version in post #2 might help as it contains a Cloudscraper update.