ewoks install: fresh env by default with the pip-venv package manager

[woutdenolf]

In #65 we changed the graph attributes

"requirements": ["ewokscore==5.1.0", "networkx==3.4.2"]

to

"requirements": {
    "python": {"version": "3.12.11", "implementation": "CPython", "...": "..."},
    "system": {"system": "Linux", "machine": "x86_64", "...": "..."},
    "distributions": [
        {"name": "ewokscore", "version": "5.1.0", "installer": "pip"},
        {"name": "networkx", "version": "3.4.2", "installer": "pip"}
    ],
    "manager": {
        "name": "pip",   # Renamed in this PR to "pip-venv"
        "version": "25.0.1",
        "files": {"requirements.txt": "ewokscore==5.1.0\nnetworkx==3.4.2\n"}
    }
}

The final goal is to support more package managers than pip (uv, poetry, pixi and conda).

This PR is another step:

• Refactor the Pip manager to support environment creation and use it by default (rename it to pip-venv).
• In-place installation is opt-in.
• Remove the non-Pip managers to be handled in follow-up PRs.

See docs on how this PR affects ewoks install and ewoks convert

• doc/howtoguides/requirements.rst
• doc/tutorials/install.rst
• doc/tutorials/install/pip_venv.rst

This was cherry-picked from #330 to make the review easier. Still a large PR, sorry. The other PRs already adds uv, pixi, conda and poetry. This PR already lays the groundwork for those in terms of core, test and doc infrastructure.

FYI python -m ewoks._requirements.__init__ does a pip gathering in your current environment. Might be useful for the reviewing.

[codecov]

Codecov Report

❌ Patch coverage is 81.20805% with 112 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/ewoks/_requirements/utils/detect.py	51.11%	22 Missing ⚠️
src/ewoks/_requirements/__init__.py	75.90%	20 Missing ⚠️
src/ewoks/__main__.py	62.79%	16 Missing ⚠️
src/ewoks/_requirements/utils/base_manager.py	84.52%	13 Missing ⚠️
src/ewoks/bindings.py	43.75%	9 Missing ⚠️
src/ewoks/_requirements/utils/files.py	72.00%	7 Missing ⚠️
src/ewoks/_requirements/utils/requirements_txt.py	93.02%	6 Missing ⚠️
src/ewoks/_requirements/utils/process.py	79.16%	5 Missing ⚠️
...quirements/utils/metadata/from_requirements_txt.py	60.00%	4 Missing ⚠️
.../ewoks/_requirements/utils/metadata/last_resort.py	20.00%	4 Missing ⚠️
... and 3 more

📢 Thoughts on this report? Let us know!

[payno]

Thanks for the detailed documentation.

To be honest, it's hard for to be very critical about the PR. It would require a deeper knowledge of this system and/or package managers. And this would require quiet some time for me. I think Loïc will give a better review, but feel free to merge if needed.

My main concern is trying to reduce long-term maintenance. I'm very worried about the effort needed to handle all the use cases. IMO, forcing "reproducers" to use the same package manager isn't a big requirement, compared to the time/energy we'd need to spend making sure we don't end up with corner cases — and I'd advocate for this, at least.

[payno]

Hard to be very accurate at the moment because I don't know this part of the code yet.

But having this or in Path(root or self.environments_root()).expanduser() / name sounds a bit weird.

Makes me think that the environment location might not be static. From what I get from claude he/she says this will happen when user override the default location. But in this case I would expect environments_root to also be redefined.

[woutdenolf]

This function takes the root (optional coming from the user) or the environment root, expands ~ and return the child name. Not sure what the problem is.

[payno]

in environment_location we use the term name (and scalar). I guess you want to open the API to handle any location. But this might make thinks complex to maintain.

[woutdenolf]

I don't get this comment. The name of an environment and the location (directory provided by the user: an environment prefix or a project) are two very different things.

[payno]

Not simple but should we consider this case ? the original manager not being here ?
To be honest I am a bit afraid that trying to bridge the gap between the different package manager is a pain we might want to avoid (at least to start. Because once we engage ourself this way we must continue).

[woutdenolf]

Requirements are always stored in two ways:

• python's distribution info extracted from importlib.metadata
• package manager specific (requirements.txt for pip for example)

We already need to fallback to importlib.metadata, especially for pip, since the result of pip freeze is not always installable (e.g. points to local paths).

Since we already need this fallback mechanism even when the producer and installer both use the same package manager, cross-manager support comes for free (using the importlib.metadata fallback).

[payno]

I think there's a corner case worth highlighting.

Right now, if I'm not mistaken, nothing prevents a user from converting a workflow and ending up with a Python environment that fully supports running it — which is very convenient.

However, if that user then shares it with a third party, the installation becomes useless.

[woutdenolf]

However, if that user then shares it with a third party, the installation becomes useless.

Don't get it. Shares what with a third party?

• user1 generates a JSON file (ewoks convert) and share it with user2.
• user2 receives an JSON file and installs (ewoks install) and executes (ewoks execute).

We are not sharing the environment, we are sharing the workflow (JSON or whatever).