Could calibre-debug ignore unknow option if the first argument is a .py file?

[xxyzz]

Hi Kovid, I use calibre-debug to run pip install command like this:

$ calibre-debug -e /usr/lib/python3.10/site-packages/pip/__pip-runner__.py -- install pkg

But some dependency packages may use pip arguments unsupported by calibre-debug. For example `--egg-base` in this issue: https://github.com/xxyzz/WordDumb/is...sue-1561774062

Could these lines https://github.com/kovidgoyal/calibr...g.py#L306-L307 be moved above `option_parser().parse_args()`?

[kovidgoyal]

Use --

as described here:

https://manual.calibre-ebook.com/dev...all-of-calibre

[xxyzz]

I already add the `--` argument but pip won't add it.

Here is a command to reproduce the error: "calibre-debug -e /usr/lib/python3.10/site-packages/pip/__pip-runner__.py -- install -t temp sudachidict-core"

and here is the error message:

Collecting sudachidict-core
  Using cached SudachiDict-core-20230110.tar.gz (9.0 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 2
  ╰─> [24 lines of output]
      Usage: calibre-debug [options]

      Various command line interfaces useful for debugging calibre. With no options,
      this command starts an embedded Python interpreter. You can also run the main
      calibre GUI, the calibre E-book viewer and the calibre editor in debug mode.

      It also contains interfaces to various bits of calibre that do not have
      dedicated command line tools, such as font subsetting, the E-book diff tool and so
      on.

      You can also use calibre-debug to run standalone scripts. To do that use it like this:

          calibre-debug -e myscript.py -- --option1 --option2 file1 file2 ...

      Everything after the -- is passed to the script. You can also use calibre-debug
      as a shebang in scripts, like this:

          #!/usr/bin/env -S calibre-debug -e -- --


      Whenever you pass arguments to calibre-debug that have spaces in them, enclose the arguments in quotation marks. For example: "/some path/with spaces"

      calibre-debug: error: no such option: --egg-base
      ['calibre-debug', '-c', '\nexec(compile(\'\'\'\n# This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py\n#\n# - It imports setuptools before invoking setup.py, to enable projects that directly\n#   import from `distutils.core` to work with newer packaging standards.\n# - It provides a clear error message when setuptools is not installed.\n# - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so\n#   setuptools doesn\'t think the script is `-c`. This avoids the following warning:\n#     manifest_maker: standard file \'-c\' not found".\n# - It generates a shim setup.py, for handling setup.cfg-only projects.\nimport os, sys, tokenize\n\ntry:\n    import setuptools\nexcept ImportError as error:\n    print(\n        "ERROR: Can not execute `setup.py` since setuptools is not available in "\n        "the build environment.",\n        file=sys.stderr,\n    )\n    sys.exit(1)\n\n__file__ = %r\nsys.argv[0] = __file__\n\nif os.path.exists(__file__):\n    filename = __file__\n    with tokenize.open(__file__) as f:\n        setup_py_code = f.read()\nelse:\n    filename = "<auto-generated setuptools caller>"\n    setup_py_code = "from setuptools import setup; setup()"\n\nexec(compile(setup_py_code, filename, "exec"))\n\'\'\' % (\'/private/var/folders/05/x6p94j1j60v7g4jqv7ty70vc0000gn/T/pip-install-wbo8cn49/sudachidict-core_96b2a6d901534e538c891974b5ba2848/setup.py\',), "<pip-setuptools-caller>", "exec"))', 'egg_info', '--egg-base', '/private/var/folders/05/x6p94j1j60v7g4jqv7ty70vc0000gn/T/pip-pip-egg-info-xz1txqv5']
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[kovidgoyal]

I ran that command and it worked fine for me.

calibre-debug -e /usr/lib/python3.10/site-packages/pip/__pip-runner__.py -- install -t temp sudachidict-core        
Collecting sudachidict-core
  Downloading SudachiDict-core-20230110.tar.gz (9.0 kB)
  Preparing metadata (setup.py) ... done
Collecting SudachiPy<0.7,>=0.5
  Downloading SudachiPy-0.6.6-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 4.7 MB/s eta 0:00:00
Building wheels for collected packages: sudachidict-core
  Building wheel for sudachidict-core (setup.py) ... done
  Created wheel for sudachidict-core: filename=SudachiDict_core-20230110-py3-none-any.whl size=71665379 sha256=cd6452b0c199c5a0241e79b0418c5833c3dc4193026eb7e311d333682156c963
  Stored in directory: /home/kovid/.cache/pip/wheels/32/d9/23/a5865d2c5ccf74dd7c8fd98fddad58c4262602597fac43d608
Successfully built sudachidict-core
Installing collected packages: SudachiPy, sudachidict-core
Successfully installed SudachiPy-0.6.6 sudachidict-core-20230110

calibre-debug does not need to know any specific arguments. The point of -- is that all arguments after it are ignored by calibre-debug and passed to the script being executed.

[xxyzz]

Maybe because you're using Arch Linux which calibre-debug actually is the same as the python command, but that command would fail on macOS and Windows. The error is raised from the "optparse.OptionParser" library.

It there a way to use calibre's Python interpreter as normal Python command? That would be much easier to install packages via pip.

[kovidgoyal]

That would imply that pip or a setup script in the package is re-executing the process using calibre-debug as the interpreter.

No, there is no way to use the python interpreter as a normal interpreter. It's designed to run in embedded, isolated mode.

You could create your own wrapper script that accepts the same arguments as python and translates them into equivalent calibre-debug invocations.

[xxyzz]

Originally Posted by kovidgoyal

You could create your own wrapper script that accepts the same arguments as python and translates them into equivalent calibre-debug invocations.

You mean create a bash script like this one?

#!/usr/bin/env bash

FIRST_ARG="$1"
shift
calibre-debug -e "$FIRST_ARG" -- "$@"

but I run the command "python3 -m pip --python ./calibre-python.sh install -t temp sudachidict-core" and still get the same error.

[kovidgoyal]

Yeah I guess that wont work since pip or whatever is doing that will use the program that ran it not its wrapper.

What you can do is have an env var that causes calibre-debug to interpret commadn line arguments like the python interpeter. Feel free to send a PR implementing that. But be warned that this only soves the problem of command line argument parsing, calibre-debug is not designed to behave like a python interpeter in many ways beyond just parsing command line arguments.

[xxyzz]

I think another safer option is to run all the code in a normal Python interpreter, so the dependency packages don't need to compatible with calibre's Python. And this should require less maintenance effort when pip or calibre updates. And thanks for your help!