Register Guidelines E-Books Today's Posts Search

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 01-28-2021, 09:11 PM   #5296
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 6,974
Karma: 4604635
Join Date: Dec 2011
Location: Midwest USA
Device: Kobo Clara Colour running KOReader
New Test Version Posted

2021-01-28
- Plugin: Only run one BG download at time.
- Get cloudscraper 1.2.56 changes, but keep py2 compat.

Some users over on the mailing list pointed out running parallel downloads on the same site is more likely to trip blocking. This version prevents that by only running one download at a time.

I said over there:
Quote:
So, I've been researching this a bit more.

First, FFF plugin launches a background job in Calibre. As V1cT1T10uS pointed out, Calibre has a setting for how many BG jobs to run at a time--often set (or limited) to the number of CPUs.

The weird part is that FFF has *also* been launching that many parallel process again from its BG process. So if you launched FFF with a bunch of downloads *and* started a bunch of converts or something, Calibre could technically be running twice the configured number of background processes (but not jobs in the job list).

Way back when, I lifted the BG processing code for FFF plugin from the ISBN plugin and I honestly didn't understand exactly what was going on at the time. The inherited comment even says that it launched a process per book to work around a memory leak issue in ISBN.

I've posted a test version plugin that only does one download at a time:
https://www.mobileread.com/forums/sh...25&postcount=2

Currently, it still launches a separate process for each download, but I will be experimenting with what can be done--all in one process (like CLI does lists), maybe process per site.
JimmXinu is offline   Reply With Quote
Old 01-28-2021, 09:35 PM   #5297
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 10,994
Karma: 75337983
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Quote:
Currently, it still launches a separate process for each download, but I will be experimenting with what can be done--all in one process (like CLI does lists), maybe process per site.
For what it matters, before I switched to email drag-and-drop I used to run two parallel update jobs - one for ffnet fics and one for ao3 fics. It also had the advantage of running a bit faster.

Last edited by ownedbycats; 01-28-2021 at 10:06 PM.
ownedbycats is offline   Reply With Quote
Advert
Old 01-28-2021, 10:06 PM   #5298
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 10,994
Karma: 75337983
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
I did a bit of investigating into the eml issue. It's due to the presence of two Unicode Latin-1 supplement characters (in this case, the chapter title is "Déjà-vu") in the message body. Honestly something that didn't occur to me at first until I went mucking around in the geturls.py code.

I was able to confirm this by saving a copy of the .eml and modifying it to remove the characters. Also emailing myself two messages: one containing just the url, and the other one containing the URL and chapter title. Former worked as expected, latter pasted the filepath.

ffnet uses Content-Type: text/plain; charset="utf-8" for its messages.

Haven't tested whether the issue is limited to Latin-1 supplement. I did notice this in geturls.py:

Code:
def get_urls_from_text(data,configuration=None,normalize=False,email=False):
    urls = collections.OrderedDict()
    try:
        # py3 can have issues with extended chars in txt emails
        data = ensure_str(data,errors='replace')
    except UnicodeDecodeError:
        data = data.decode('utf8') ## for when called outside calibre.
I hope this helps a bit.

Last edited by ownedbycats; 01-28-2021 at 10:22 PM.
ownedbycats is offline   Reply With Quote
Old 01-28-2021, 10:37 PM   #5299
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 6,974
Karma: 4604635
Join Date: Dec 2011
Location: Midwest USA
Device: Kobo Clara Colour running KOReader
Quote:
Originally Posted by JimmXinu View Post
Calibre could technically be running twice the configured number of background processes (but not jobs in the job list)
Quote:
Originally Posted by ownedbycats View Post
For what it matters, before I switched to email drag-and-drop I used to run two parallel update jobs - one for ffnet fics and one for ao3 fics. It also had the advantage of running a bit faster.
Congratulations! You managed to (potentially) triple your configured Calibre processes instead of double them.


Although automatically launching one BG job per site is an idea I considered. I'm not sure the complications are worth it for most users. Separate 'Proceed with updating you library' dialogs per site/BG job--or a lot more complicated update code.

There are some other ideas, but there are tradeoffs. I will come back to it in a while.

Quote:
Originally Posted by ownedbycats View Post
I did a bit of investigating into the eml issue. It's due to the presence of two Unicode Latin-1 supplement characters (in this case, the chapter title is "Déjà-vu") in the message body. Honestly something that didn't occur to me at first.
If you want to figure out a fix and make a pull request in github, I'll look at it.

But it's a pretty obscure corner case (and possibly OS dependent?) and I'm busy else where in the system right now. I'm only looking at the BG jobs issue because it's tangentially related to what I'm working on.
JimmXinu is offline   Reply With Quote
Old 01-28-2021, 10:58 PM   #5300
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 10,994
Karma: 75337983
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Yeah its pretty low priority (I guess it might be a bigger problem for users who read non-English fanfics). Just something to look at again if you need to change that bit of code again at some point.

EDIT: I did mess around with it a bit with some things I googled. Changing errors='strict' didn't work, neither did adding encoding='utf-8'. Commenting the line out to see what would happen caused FFF to fail to load at all. This is why I am not a programmer.

ANOTHER EDIT: Although I couldnt fix it myself, I did file the bug on the Github mostly for documentation purposes.

Last edited by ownedbycats; 01-29-2021 at 03:25 PM.
ownedbycats is offline   Reply With Quote
Advert
Old 01-29-2021, 02:32 AM   #5301
ackomb
Zealot
ackomb has a spectacular aura aboutackomb has a spectacular aura aboutackomb has a spectacular aura aboutackomb has a spectacular aura aboutackomb has a spectacular aura aboutackomb has a spectacular aura aboutackomb has a spectacular aura aboutackomb has a spectacular aura aboutackomb has a spectacular aura aboutackomb has a spectacular aura aboutackomb has a spectacular aura about
 
Posts: 106
Karma: 4486
Join Date: Mar 2020
Location: Netherlands
Device: i-pad
Quote:
Originally Posted by JimmXinu View Post
Sorry, I knew that but failed communicate it to you. (I was testing in an empty library with just one book, so clearing the column did remove it from all stories.)

I believe you might also be able to change the 'canonical' spelling Calibre uses in Calibre's tag browser?

I, too, am not aware of anyway to change Calibre behavior about this.
No worries, I'm just happy to have a solution for my issue .
At least it wasn't the code I'd written. It's a bit of work to adjust it, but once it's done the issue should be gone for future downloads.
Thanks again.
ackomb is offline   Reply With Quote
Old 01-30-2021, 10:10 AM   #5302
thehoboking
Junior Member
thehoboking began at the beginning.
 
Posts: 1
Karma: 10
Join Date: Jan 2021
Device: PC
Webnovel.com Error

Hello,

Just wanted to report an error I'm getting when trying to download novels from Webnovel.com

Status: Error
Title: Scholar's Advanced Technological System
Author: Morning Star LL
Comment: HTTP Error 404: Not Found
URL: https://www.webnovel.com/book/12583970906002305

Thanks
thehoboking is offline   Reply With Quote
Old 01-30-2021, 10:27 AM   #5303
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 6,974
Karma: 4604635
Join Date: Dec 2011
Location: Midwest USA
Device: Kobo Clara Colour running KOReader
Quote:
Originally Posted by thehoboking View Post
Hello,

Just wanted to report an error I'm getting when trying to download novels from Webnovel.com

Status: Error
Title: Scholar's Advanced Technological System
Author: Morning Star LL
Comment: HTTP Error 404: Not Found
URL: https://www.webnovel.com/book/12583970906002305

Thanks
This has also been reported in Issue #644.

I replied:
Quote:
Something in the site appears to have changed.

I didn't implement this adapter, don't use it, and don't know how it works. Getting it fixed is not high priority for me right now.

Perhaps someone who does use it wants to look into it?
JimmXinu is offline   Reply With Quote
Old 01-30-2021, 05:26 PM   #5304
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 6,974
Karma: 4604635
Join Date: Dec 2011
Location: Midwest USA
Device: Kobo Clara Colour running KOReader
New Test Version Posted

2021-01-30
- Fix for Reject List to equate story URLs with and without titles, for XenForo and ffnet. Otherwise you can end up with dups in Reject List and not recognize new URLs are already being rejected. Also fixes [storyUrl] sections without titles for ffnet.
- Fix fictionpress.net and retain Reject List entries for removed sites.

UPDATE: I found a couple bugs immediately after uploading (as you do), fixed and re-uploaded.

Last edited by JimmXinu; 01-30-2021 at 06:27 PM. Reason: Bug fixes
JimmXinu is offline   Reply With Quote
Old 01-31-2021, 03:57 PM   #5305
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 6,974
Karma: 4604635
Join Date: Dec 2011
Location: Midwest USA
Device: Kobo Clara Colour running KOReader
New Test Version Posted

2021-01-31
- Major refactoring/rewrite of network fetching and caching layer using requests library.
- Convert adapter_spikeluvercom to base_efiction after site changes.
- Remove 'extrasleep' feature--it was ugly--increase slow_down_sleep_time for the couple sites that used it. We can trust the users, right?
- Remove httplib._MAXHEADERS workaround for royalroad.com - not happening anymore.
- Bump Test Version 3.99.X

It's been working for me, but this is a major update that I've been working on for the last week. It touches most of the files in the system.

Approach with caution.

And I just realized I haven't tested other than Windows yet. Double caution.

UPDATE: And already found one bug with py2 Calibre 2.85.1

Last edited by JimmXinu; 01-31-2021 at 04:09 PM.
JimmXinu is offline   Reply With Quote
Old 01-31-2021, 04:10 PM   #5306
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 10,994
Karma: 75337983
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Is the extrasleep part of why some updates of ffnet fics seemed to take a silly long amount of time? I initially thought it was solely due to the removal of the extra BG process.

Last edited by ownedbycats; 01-31-2021 at 04:31 PM.
ownedbycats is offline   Reply With Quote
Old 01-31-2021, 04:48 PM   #5307
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 6,974
Karma: 4604635
Join Date: Dec 2011
Location: Midwest USA
Device: Kobo Clara Colour running KOReader
Newer Test Version Posted

2021-01-31 Take 2
- Major refactoring/rewrite of network fetching and caching layer using requests library.
- Convert adapter_spikeluvercom to base_efiction after site changes.
- Remove 'extrasleep' feature--it was ugly--increase slow_down_sleep_time for the couple sites that used it. We can trust the users, right?
- Remove httplib._MAXHEADERS workaround for royalroad.com - not happening anymore.
- Bump Test Version 3.99.X
- Fixes for py2/py3 & a stray INI space.

Basic testing done on Win10Cal2.85.1 & Cal5, macos Cal5 and linux Cal5.


'extrasleep' was an internal thing that was added to hardcode extra sleep time into requests for certain sites. In practice, it was only ever used with base_xenforo (SB, SV, QQ, etc) and ffnet.

It complicated the 'fetcher' layer while--in theory--being redundant with just increasing slow_down_sleep_time. Its advantage was users couldn't easily turn it down or off.


ffnet download times are slower now both because of serializing BG downloads and increased slow_down_sleep_time for ffnet.

I plan to revisit parallel BG downloads, but with an eye to having parallel queues for different sites, so each site only has one download at a time, but different sites can run in parallel.
JimmXinu is offline   Reply With Quote
Old 02-02-2021, 01:55 AM   #5308
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 10,994
Karma: 75337983
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Question (more of a theoretical thing for the moment):

Can any metadata entry be used as a conditionalkey, or just category?

e.g.
Code:
category=>generalcategory=>morespecificcategory&&characters=>(Characters|That|Only|Appear|In|One|Entry|Of|A|Series)
ownedbycats is offline   Reply With Quote
Old 02-02-2021, 02:18 AM   #5309
darth1
Member
darth1 began at the beginning.
 
Posts: 19
Karma: 10
Join Date: Mar 2018
Device: none
My apology's if this has been answered but has anyone else had this error? Because I normally have to wait 5 hrs at the bare minimum before fanficfare will work and am using 3.29.2


Status
Title
Author
Comment
URL
Error
Diligence of the Shield Hero
Allen Blaster
cloudscraper reports: "Cloudflare IUAM possibility malformed, issue extracing delay value."
https://www.fanfiction.net/s/1335028...he-Shield-Hero

the cloudflare thing comes up a "lot" lately
darth1 is offline   Reply With Quote
Old 02-02-2021, 02:24 AM   #5310
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 10,994
Karma: 75337983
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Someone else had the issue on Github, I think it cleared up for them after a day or two. The test version in post #2 might help as it contains a Cloudscraper update.
ownedbycats is offline   Reply With Quote
Reply

Tags
fanfiction


Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] KindleUnpack - The Plugin DiapDealer Plugins 523 07-15-2025 06:45 PM
[GUI Plugin] Open With kiwidude Plugins 404 02-21-2025 05:42 AM
[GUI Plugin] Marvin XD Philantrop Plugins 126 01-29-2017 12:48 PM
[GUI Plugin] KiNotes -axel- Plugins 0 07-14-2013 06:39 PM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 12:27 PM


All times are GMT -4. The time now is 10:49 PM.


MobileRead.com is a privately owned, operated and funded community.