Autoadd problems on Windows 10 1903

[DaltonST]

Since I used my Job Spy plugin's tweak to auto-add from a particular path based on a particular Library, and since that has quit working on my desktop, I have been looking at this.

I have a Windows 10 Home 8" tablet build 1809 with Calibre64 3.44, and auto-add works perfectly. Even with Job Spy's tweak. Even with the auto-add directory on my LAN.

I have been trying to get auto-add to work on my Dell AIO desktop Windows 10 Home build 1903 (a.k.a. 18362.19h1_release.190318-1202) with Calibre64 3.44 for a few days now. All of my auto-add directories are on my SSD.

Running "calibre --ignore-plugins" does not solve it, ever.

Running "calibre-debug -g" shows nothing, ever.

Grasping at straws, and although it certainly should not matter (and it did not), I added the auto-add directories I tested with to the "exclusion" list for both Malwarebytes and Defender. That would have been far too easy.

I wonder if anyone else with the identical build of Windows 10 Home that I have, 18362.19h1_release.190318-1202, has the same problems with auto-add.


To be continued...

[DaltonST]

I ran Calibre 3.44 from source after making some changes to auto_add.py:

    def auto_add(self):
        from calibre.utils.ipc.simple_worker import fork_job, WorkerError
        from calibre.ebooks.metadata.opf2 import metadata_to_opf
        from calibre.ebooks.metadata.meta import metadata_from_filename

        files = [x for x in os.listdir(self.path) if
                    # Must not be in the process of being added to the db
                    x not in self.staging and
                    # Firefox creates 0 byte placeholder files when downloading
                    os.stat(os.path.join(self.path, x)).st_size > 0 and
                    # Must be a file
                    os.path.isfile(os.path.join(self.path, x)) and
                    # Must have read and write permissions
                    os.access(os.path.join(self.path, x), os.R_OK|os.W_OK) and
                    # Must be a known ebook file type
                    self.is_filename_allowed(x)
                ]
        #~ ===========================================================

        #~ zzzzzzzzzzzz RESULTS USING ORIGINAL zzzzzzzzzzzzzzzzzzzzzzzzzz
        if DEBUG:
            n = len(files)
            print("zzzzzzzzzzzzzzzzzzzz   ORIGINAL:  Number of files in os.listdir accepted: ", bytes(n))
            for f in files:
                print("zzzzzzzzzzzzzzzzzzzz   os.listdir: ", f)
        #~ zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

        files = [x for x in os.listdir(self.path) if
                    # Must not be in the process of being added to the db
                    x not in self.staging and
                    # Firefox creates 0 byte placeholder files when downloading
                    os.stat(os.path.join(self.path, x)).st_size > 0 and
                    # Must be a file
                    os.path.isfile(os.path.join(self.path, x)) and
                                        # Must have read and write permissions
                                        #~ os.access(os.path.join(self.path, x), os.R_OK|os.W_OK) and  
                    # Must be a known ebook file type
                    self.is_filename_allowed(x)
                ]

        #~ zzzzzzzzzzzz RESULTS USING REVISED zzzzzzzzzzzzzzzzzzzzzzzzzz
        from calibre.constants import DEBUG
        if DEBUG:
            n = len(files)
            print("zzzzzzzzzzzzzzzzzzzz   REVISED:  Number of files in os.listdir accepted: ", bytes(n))
            for f in files:
                print("zzzzzzzzzzzzzzzzzzzz   os.listdir: ", f)
        #~ zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

The debug results:

[7.28] main UI initialized...
[7.28] Hiding splash screen
Job Spy: Quality Fixes have been activated as of:  2019-07-04 15:36:00
[8.23] splash screen hidden
[8.25] Started up in 8.25 seconds with 1130 books
zzzzzzzzzzzzzzzzzzzz   ORIGINAL:  Number of files in os.listdir accepted:  0
zzzzzzzzzzzzzzzzzzzz   REVISED:  Number of files in os.listdir accepted:  2
zzzzzzzzzzzzzzzzzzzz   os.listdir:  Collapsing Empire, The - John Scalzi.epub
zzzzzzzzzzzzzzzzzzzz   os.listdir:  The Consuming Fire (The Interdependency, Book 2) by John Scalzi.epub
Traceback (most recent call last):
  File "Y:\calibre_source_v3_44_0\src\calibre\gui2\auto_add.py", line 163, in auto_add
  File "Y:\calibre_source_v3_44_0\src\calibre\utils\ipc\simple_worker.py", line 225, in fork_job
  File "Y:\calibre_source_v3_44_0\src\calibre\utils\ipc\simple_worker.py", line 139, in create_worker
  File "Y:\calibre_source_v3_44_0\src\calibre\utils\ipc\launch.py", line 237, in __call__
  File "subprocess.py", line 394, in __init__
  File "subprocess.py", line 644, in _execute_child
WindowsError: [Error 740] The requested operation requires elevation

I am the Administrator of the PC, I have UAC set to "do nothing", and I execute Calibre "As Administrator".

As previously mentioned, the problem exists even if --ignore-plugins is specified.


ADDED/CHANGED: resetting the auto-add folder permissions properties to "Full-Control" for me, Administrators, Everyone, etc. did NOT remove "Error 740", but it DID show that the ORIGINAL code was considering the epubs as being readable and writable. So,it appears that it is necessary but not sufficient that the permissions of the auto-add folder be minimal/loose.


ADDED:
This is what Microsoft says about the above UAC error shown in the attached image:

 This this is the purpose of UAC, it strips admin privileges from normally-launched processes.
This encourages you to not use dangerous admin powers where you don't need to. You may try to modify the UAC on Local Security Policy for test:
 1 Login with Administrator
  2 Type in secpol.msc into the Start menu search box and hit enter
 3 browse  down to Local Policies \ Security Options\ User Account Control:  Behavior of the elevation prompt for administrators in Admin Approval  Mode” and double-click on it.
 4 Change the setting to “Elevate without prompting”.

Great. But secpol.msc and the group policy editor, gpedit, do not exist in Windows 10 Home. I do not want to have to upgrade to Windows 10 Pro for the single purpose of setting UAC to “Elevate without prompting”.


I wonder what version of Windows 10 is Calibre auto-add tested on. Home or Pro? And what Version(s)?





DaltonST

[DaltonST]

Continuing my prior posts, I reinstalled Windows 10 1903 by downloading the Media Creation Tool for 1903. I kept all of my files and apps, but let it replace Windows 10 itself as it wished. Result: still get:

Traceback (most recent call last):
  File "Y:\calibre_source_v3_44_0\src\calibre\gui2\auto_add.py", line 165, in auto_add
  File "Y:\calibre_source_v3_44_0\src\calibre\utils\ipc\simple_worker.py", line 225, in fork_job
  File "Y:\calibre_source_v3_44_0\src\calibre\utils\ipc\simple_worker.py", line 139, in create_worker
  File "Y:\calibre_source_v3_44_0\src\calibre\utils\ipc\launch.py", line 237, in __call__
  File "subprocess.py", line 394, in __init__
  File "subprocess.py", line 644, in _execute_child
WindowsError: [Error 740] The requested operation requires elevation

[DaltonST]

After I reinstalled Windows 10 Home 1903, and auto-add was still broken, I continued searching and found 2 workarounds:


[1] https://www.winhelponline.com/blog/r...he-uac-prompt/



I created an on-demand Task that runs Calibre at the "highest level of security". I then created a .bat file that runs schtasks.exe specifying the name of the Task to run. Task name: "Run Calibre with Highest Security"

It worked perfectly the very first time I ran this task manually from the Task Scheduler. I was still running from my personal 3.44 source code, however.

I run Calibre from shortcuts that use the --with-library option anyway, so this is a trivial difference.

I am using this workaround, and not the one below.

schtasks.exe /Run /I /TN "Run Calibre with Highest Security"

Caution: you still need to loosen the Security permissions for your auto-add directories as described in a previous post (see: "necessary but not sufficient").






[2] https://www.winhelponline.com/blog/u...administrator/



For Windows 10 Home, the Registry Hack option for Users works, but it has a very high security risk associated with it.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Curr entVersion\Policies\System

Changing the 3 (prompt) to a 0 (no prompt) for ConsentPromptBehaviorUser is dangerous. My ConsentPromptBehaviorAdmin was already 0 (no prompt).

Obviously the process that Calibre spawns for auto-add is being treated as a User in my system, and not an Administrator (as it should be, since Calibre is being run as an Administrator).





DaltonST

[BetterRed]

@DaltonST - I broke out your posts from this thread as we don't yet know what OS the originator of that thread is using. Might not even be Windows.

I left a reference to this thread.

BR

[BetterRed]

@DaltonST - can you confirm this issue only exists because you run calibre as Admin, or is it the same when calibre is run as User? IIRC there are other things that go awry when its run as Admin.

And that the issue exists on 1903 Pro? I realise the 'workaround' would be to use secpol or gpedit rather than a registry hack.

BR

[Divingduck]

Is Ransomware protection activated?

[DaltonST]

can you confirm this issue only exists because you run calibre as Admin, or is it the same when calibre is run as User?

[*] Running as the default "Run as User", nothing happens at all since the auto-add worker process has no security permissions even to "read" the auto-add directory, much less to "write" (to delete).


[*] Running as "Run as Administrator" fixes the auto-add import, but the "write" (to delete) fails with a lack of security permissions. So, it auto-adds, but does not delete the auto-added files.


[*] The auto-add spawned worker process source code uses:

# Must have read and write permissions
 ===>>> os.access(os.path.join(self.path, x), os.R_OK|os.W_OK) and

[*] Running as User fails even os.R_OK.


[*] Running as Administrator passes os.R_OK but fails os.W_OK.

IIRC there are other things that go awry when its run as Admin.

Never has been for me. Running as Admin never has caused any problems whatsoever for me since 2013. Also, Windows 10 1903 security permissions are a far cry from Windows 10 0001 and Windows 7 security permissions. Microsoft is strengthening them every month, and changing how it updates existing Windows 10 installations every month. Comparing an "old" Windows 10 installation to a "newly upgraded" Windows 10 1903 installation to a "fresh from the factory 1903 OEM installation" is comparing apples and oranges. Every installation is unique.

And that the issue exists on 1903 Pro?

[*] I have Home, not Pro, so I can confirm nothing about Pro.

Is Ransomware protection activated?

[*] For Defender, I do not use the "Controlled Folder Access", so, no. Also, Defender real-time protection is turned off automatically by Windows 10 because I use another anti-virus application, Malwarebytes. Defender just runs "periodic scans" by itself.

[*] For Malwarebytes, yes. However, turning off Malwarebytes ransomware protection changes nothing.



I would love to know what version of Windows 10 (Home or Pro or both) and what build (1809 or 1903) Calibre is tested (and built) with, and whether or not Windows Update is active and allowed to run normally whenever it wishes to. Also, whether Calibre is tested using "Run as Administrator", "Run as User", or both.





DaltonST

[kovidgoyal]

You should never run calibre as administrator, that is completely unsupported. And something on your computer is blocking calibre from running worker processes for auto-add. Find out whatever that is and fix it and you will be fine.

[DaltonST]

I reset all Win 10 Home local policy permissions to their defaults using "secedit /configure /cfg %windir%\inf\defltbase.inf /db defltbase.sdb /verbose", which solved nothing.

I used the Winaero Tweaks tool for mass-resetting permissions on all top directories, subdirectories and files on my C: drive. Also for the drives with the auto-add directories. Nothing changed.

I manually changed permissions of certain directories to ensure they were loose enough.

Win 10 Event Viewer continues to show the same UAC error for the Calibre auto-add worker process as described and discussed in previous posts.

The only way to get auto-add to add is to run Calibre from source that I changed (as shown in a prior post), and to run Calibre via a Task at the "highest level of security" (as described in a prior post). It still cannot delete the original books just auto-added. UAC is stopping it.

UAC is the problem. No other software. The debug logs and the Event Viewer all show the same UAC error and reason.


The only option remaining (other than turning off auto-add so it does not run and fail every few seconds endlessly in the background as shown by the debug log and the Event Viewer) is to install Windows 10 Pro and mess with the security policy editors that come with Pro (not Home).





DaltonST

[BetterRed]

FTR: I am running 1809 Pro.

Originally Posted by DaltonST

[*] Running as the default "Run as User", nothing happens at all since the auto-add worker process has no security permissions even to "read" the auto-add directory, much less to "write" (to delete).
[*] Running as "Run as Administrator" fixes the auto-add import, but the "write" (to delete) fails with a lack of security permissions. So, it auto-adds, but does not delete the auto-added files.
[*] The auto-add spawned worker process source code uses:

# Must have read and write permissions
 ===>>> os.access(os.path.join(self.path, x), os.R_OK|os.W_OK) and

[*] Running as User fails even os.R_OK.
[*] Running as Administrator passes os.R_OK but fails os.W_OK.

Never has been for me. Running as Admin never has caused any problems whatsoever for me since 2013. Also, Windows 10 1903 security permissions are a far cry from Windows 10 0001 and Windows 7 security permissions. Microsoft is strengthening them every month, and changing how it updates existing Windows 10 installations every month. Comparing an "old" Windows 10 installation to a "newly upgraded" Windows 10 1903 installation to a "fresh from the factory 1903 OEM installation" is comparing apples and oranges. Every installation is unique.

Sounds like a can of worms, it would be interesting to know if any other Windows 1903 users are experiencing similar problems.

Curious: what happens if you put a shortcut to calibre.exe into shell:sendto and use it to add books ?

Originally Posted by DaltonST

[*] I have Home, not Pro, so I can confirm nothing about Pro.

I wasn't sure if you might have sneaked in a 'test' of Pro when you used the MCT to create a fresh install of Windows

Originally Posted by DaltonST

[*] For Defender, I do not use the "Controlled Folder Access", so, no. Also, Defender real-time protection is turned off automatically by Windows 10 because I use another anti-virus application, Malwarebytes. Defender just runs "periodic scans" by itself.

Interesting: is that 'turning off automatically' a feature introduced in 1903, or is it a feature of Home. I run Defender and MWB and all options in both are on - and I only exclude one folder from either - my sweather tray app, which MWB thinks is a trojan.

Originally Posted by DaltonST

[*] I would love to know what version of Windows 10 (Home or Pro or both) and what build (1809 or 1903) Calibre is tested (and built) with, and whether or not Windows Update is active and allowed to run normally whenever it wishes to. Also, whether Calibre is tested using "Run as Administrator", "Run as User", or both.

FWIW - as a regular calibre user I run it under Pro. Windows Updates are allowed to run normally. Upgrades are deferred until 60 days after they're designated as Enterprise ready - so I don't expect to see 1903 until October/November.

BR

[Divingduck]

I've running calibre with Windows Pro and Home since 08.06.2019 with build 1903 (Build 18362.207)and in VM's too. On the Pro machine I add last week about 1.5k new documents via autoadd on one of my bigger libraries. This is a task thats is running twice a month. Ok so fare with no problems.

Had you already check wether a new admin user account make a difference? Guess, if that wont' help you need to setup your system from scratch as you have already done most repair actions, but maybe it is worth it to check windows Fix-It-Tools before.
An additional point is maybe to check the windows update log via powershell (Get-WindowsUpdateLog) wether there was something going wrong during the update process.

Good luck...

[stumped]

a quick and reversible test:
move the entire calibre library out of any users area i.e. completely out of documents, point calibre at the new location, then see if auto add works ?
i was going to offer to test on my set up ( windows 10- home) but I see i am still at 1809 - when did 1903 actually roll out - I recall it being delayed due to glitches....

ah - on the windows update page I see that 1903 is availble but I have to say yes please- did they change the mothod of dorcing it upon us

I will get that done I think - for better or worse, so if you are in need of other testers, just say.
I don't use auto-add normally but i can easily set up a test. I have defender replaced with panda AV, so the anti-ransomware stuff shoudl be OFF. my calibre folder is in My documents , still

[stumped]

i can report that it is all good here. updated to 1903 ( i'd forgotten how tedious that is - even with SSD the install phase took about an hour..
one that was done I told calibre to auto add from a test folder & restarted calibre, then dropped a book into that folder. it was auto added.
win 10 home 1903. not running defender. calibre library in My documents. calibre 3.42 - have not done the 3.44 update yet
UAC is On.
I am an admin level user

[DaltonST]

My PC was originally Windows 7 just when 10 came out, then I added RAM, an SSD and then free Windows 10 upgrades from 0001 thru 1903. I have always been an Administrator.

Apparently the update from 1809 to 1903 disabled security permissions inheritance for random top-level folders, including but not limited to c:\users\.

It also messed with "ownership".

I took "ownership" of, and enabled inheritance within, key folders on all of my logical drives.

UAC, and hence Auto-add, works perfectly now.

Enabling inheritance is easy once you figure out it is broken. In Explorer, right-click a folder, then click properties, security, advanced, then click the inheritance button in the lower left corner.

Taking ownership is easy if you run the following in an elevated command box:

To take ownership and grant full permission on a folder:

    takeown /f foldername /r /d y


    and then:

 
    icacls foldername /grant administrators:F /t

DaltonST