Python can't find standard library in subprocess on Fedora

[xxyzz]

A user reported this error: https://github.com/xxyzz/WordDumb/issues/54

I'm able to fix the error by setting `PYTHONHOME` in `subprocess.run()`'s `env` parameter but I want to know why this error only occurs on Fedora.

[kovidgoyal]

presumably because fedora just moved to python 3.10

[xxyzz]

But the same code runs fine on Fedora's python.

import subprocess

r = subprocess.run(["python3", "-m", "pip", "--version"], check=True, text=True, capture_output=True)

this would fail in calibre-debug,

`r.stderr`:

Could not find platform independent libraries <prefix>\n
Could not find platform dependent libraries <exec_prefix>\n
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]\n
Python path configuration:\n  
PYTHONHOME = (not set)\n  
PYTHONPATH = (not set)\n  
program name = 'python3'\n  
isolated = 0\n  
environment = 1\n  
user site = 1\n  
import site = 1\n 
sys._base_executable = '/usr/bin/python3'\n  
sys.base_prefix = '/tmp/t/python-n51y8gtd'\n  
sys.base_exec_prefix = '/tmp/t/python-n51y8gtd'\n  
sys.platlibdir = 'lib'\n  
sys.executable = '/usr/bin/python3'\n  
sys.prefix = '/tmp/t/python-n51y8gtd'\n  
sys.exec_prefix = '/tmp/t/python-n51y8gtd'\n  
sys.path = [\n
    '/tmp/t/python-n51y8gtd/lib/python310.zip',\n
    '/tmp/t/python-n51y8gtd/lib/python3.10',\n
    '/tmp/t/python-n51y8gtd/lib/lib-dynload',\n
]\n
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding\n
Python runtime state: core initialized\n
ModuleNotFoundError: No module named 'encodings'\n\n

Current thread 0x00007f824f149740 (most recent call first):\n
<no Python frame>\n"

[kovidgoyal]

Originally Posted by xxyzz

But the same code runs fine on Fedora's python.

import subprocess

subprocess.run(["python3", "-m", "pip", "--version"], check=True, text=True, capture_output=True)

this would fail in calibre-debug

You would need to ask whoever maintains the fedora calibre package or tell your user to use the official binary.

[xxyzz]

I forget to mention this is the official calibre binary, Fedora 36's package is still calibre 5.

[kovidgoyal]

The tracebackk in your bug report includes lines from the system pythons subprocess module. The only way that can happen is if you added the system python's path to sys.path, since calibre has its own python. Dont do that.

[xxyzz]

No I didn't add anything to `sys.path`...

That one line of code(`subprocess.run(["python3", "-m", "pip", "--version"], text=True, capture_output=True)`) alone will get this error.

[kovidgoyal]

aaah you are running external python3 as a subprocess. You need to use the sanitize_env_vars() context manager in that case.

[xxyzz]

Thank you so much!