diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 6d20fb11bfd..e3585e55f60 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -617,6 +617,7 @@ peps/pep-0735.rst @brettcannon peps/pep-0736.rst @Rosuav peps/pep-0737.rst @vstinner peps/pep-0738.rst @encukou +# peps/pep-0739.rst peps/pep-0740.rst @dstufft peps/pep-0741.rst @vstinner peps/pep-0742.rst @JelleZijlstra @@ -680,13 +681,14 @@ peps/pep-0801.rst @warsaw peps/pep-0802.rst @AA-Turner peps/pep-0803.rst @encukou peps/pep-0804.rst @pradyunsg -# ... +# peps/pep-0805.rst peps/pep-0806.rst @JelleZijlstra peps/pep-0807.rst @dstufft peps/pep-0808.rst @FFY00 peps/pep-0809.rst @zooba peps/pep-0810.rst @pablogsal @DinoV @Yhg1s peps/pep-0811.rst @sethmlarson @gpshead +# peps/pep-0812.rst peps/pep-0813.rst @warsaw @ericvsmith peps/pep-0814.rst @vstinner @corona10 peps/pep-0815.rst @emmatyping @@ -705,6 +707,7 @@ peps/pep-0828.rst @ZeroIntensity peps/pep-0829.rst @warsaw peps/pep-0830.rst @gpshead peps/pep-0831.rst @pablogsal @Fidget-Spinner @savannahostrowski +peps/pep-0832.rst @brettcannon # ... peps/pep-2026.rst @hugovk # ... @@ -788,5 +791,4 @@ peps/pep-8015.rst @vstinner peps/pep-8016.rst @njsmith @dstufft # ... peps/pep-8100.rst @njsmith -# peps/pep-8101.rst -# peps/pep-8102.rst +# 81** PEPs are collectively owned, so no individual assignees. diff --git a/peps/pep-0832.rst b/peps/pep-0832.rst new file mode 100644 index 00000000000..cceda0a100e --- /dev/null +++ b/peps/pep-0832.rst @@ -0,0 +1,385 @@ +PEP: 832 +Title: Virtual environment discovery +Author: Brett Cannon +Discussions-To: Pending +Status: Draft +Type: Standards Track +Created: 19-Jan-2026 +Python-Version: 3.15 +Post-History: Pending + + +Abstract +======== + +This PEP sets out to help make the discovery of a project's virtual environment +easier for tools by providing a default location as well as a way for a project +to point to its preferred virtual environment when it differs from the default +location. + + +Motivation +========== + +Typically, when someone is working on a project larger than a single file, a +long-lived virtual environment is desirable (the single file case is covered by +:pep:`723` and ephemeral virtual environments). As such, tools working on the +user's behalf may want to create and/or use the same virtual environment. These +tools could be custom scripts in the project like running the test script or +third-party tools like package installers. + +Unfortunately, there's no guidance on where tools should put a virtual +environment or how to find where one is ultimately put. There's somewhat of a +convention in the community to put a virtual environment locally at the root of +the project in a directory named :file:`.venv`, but being a convention means it +isn't consistently followed. As well, there is no mechanism to point to a +virtual environment regardless of its location. + +This lack of guidance on where tools can find a virtual environment makes the +developer experience less smooth than it could be. If you rely on shell +activation to use the proper virtual environment, then you have to make sure to +do that (either manually or by configuring some automatic shell integration) as +well as not to accidentally reuse that activated shell with another project (or +have set up shell automation to handle the deactivation). Otherwise tools need +to guess or have custom logic per tool that creates virtual environments or to +ask users to manually specify where the virtual environment is. + +For virtual environment creation, it leads to different instructions per +project on what to do. And those instructions can be critical if scripts and +project configuration rely on the virtual environment being in a certain place. +This can also be an issue when a tool creates a virtual environment +automatically but it isn't obvious where the environment was placed. + + +Rationale +========= + +There are three aspects to where a virtual environment is placed. The first is +whether the virtual environment is local to the project or stored globally with +other virtual environments. Keeping the virtual environment local means that it +is isolated and unique to the project. As well, it means that if you delete the +project you also delete the virtual environment. If you store the virtual +environment globally then you can share it among multiple projects and delete +all virtual environments at once by deleting the directory that contains them +all. Keeping virtual environments global also means it won't be backed up +automatically if a project is stored e.g. in a directory automatically backed +up to remote storage where you pay based on how much storage you use. + +Another aspect is the directory name used for the virtual environment +(although this really only affects local virtual environments). If one views +virtual environments as more of an implementation detail, a directory name +starting with :file:`.` seemingly makes sense to mark it hidden or de-emphasized +in various tools such as shells and code editors. But hiding it can make +accessing the directory harder via tools that don't expose paths starting with +a ``.``. + +Lastly, there's whether you have one virtual environment at a time or many. +Having only one can minimize disk space for some tools and keeps it simple by +not trying to manage multiple virtual environments. Having multiple virtual +environments, though, means not having to constantly recreate virtual +environments when e.g. needing to test against multiple Python versions. + +This PEP takes a two-pronged approach to making virtual environments easily +discoverable while supporting all aspects mentioned above. First, by default, +the virtual environment for the project could be put in the :file:`.venv` +directory of the project (this can be a hardlink, symlink, etc.). This name +has been chosen due to preexisting tool support: +`Poetry `__ +will detect a virtual environment in such a location, +`PDM `__ +and `uv `__ +create their environments there by default already ( +`Hatch can support `__ +a virtual environment there). `VS Code `__ +will select it automatically while you can configure +`PyCharm `__ +to use it. The default :file:`.gitignore` file for +`GitHub `__, +`GitLab `__, +and `Codeberg `__ +already ignore the path. + +But for various reasons (from personal preference to preexisting tool defaults), +the :file:`.venv` directory in the project root may not work. In those cases, a +:file:`.venv` **file** which points to the virtual environment by default +should be provided in the project's root directory (i.e. the same location as +specified above for the :file:`.venv` directory). This file should point to the +virtual environment to use by default; there can be other virtual environments +for the project, but the :file:`.venv` file should point to the virtual +environment to be used if no preference is specified. While a hardlink, +symlink, etc. for :file:`.venv` could serve the same purpose, not all file +systems support such links. As well, situations like the automatic backup case +mentioned previously require a level of indirection so that backup tools don't +implicitly follow into a virtual environment and back it up. + +The :file:`.venv` file is meant to represent the virtual environment a workflow +tool is expected to use that is external to the one that wrote the +:file:`.venv` file (e.g. Hatch wrote the file and VS Code is going to read it). +This means that a workflow tool shouldn't update the :file:`.venv` file as it +runs a test suite through multiple versions of Python. But if the workflow tool +has a command to control what virtual environment is used when running Python, +then the file should be updated as necessary to match what environment the +workflow tool would use (e.g. :file:`.venv` should always match what virtual +environment `'hatch run' `__ +would use). This is not expected to cause a "noisy neighbour" problem as it's +not expected to change that rapidly. + + +Specification +============= + +The virtual environment for a project SHOULD be in a directory named +:file:`.venv` (i.e. :file:`.venv/pyvenv.cfg` will exist which can be used to +detect the existence of a virtual environment) in the root of the project +(typically next to the :file:`pyproject.toml` file) if it makes sense for it +to reside there (e.g. the tool creating the virtual environment doesn't already +use a different location by default or the user didn't specify a location). +This can be a hardlink or symlink to a virtual environment. + +In all other situations where placing a virtual environment at the project root +in a :file:`.venv` directory is not possible or desirable, a :file:`.venv` +**file** SHOULD be written in the project root instead. The file's contents +MUST be a single line containing the path to the directory containing the +virtual environment (i.e. the directory containing :file:`pyvenv.cfg`). The +file MUST be encoded in UTF-8. A single trailing newline in the file is +acceptable (either ``\r\n`` or ``\n``), and so code reading the file should +strip off any trailing newline. The path in the file is expected to be specific +to the machine it's on, so there are no requirements on path formatting (i.e. a +POSIX path is not required). The path MAY be relative to the :file:`.venv` file +to ease creation of the file by hand. Tools MUST verify that the directory the +file points to exists before using it to prevent tools from being tricked into +e.g. blindly passing the file contents into +``subprocess.run(..., shell=True)``. + +Tools looking for a virtual environment SHOULD look for the :file:`.venv` +directory or file and handle them appropriately. Tools SHOULD NOT prefer one +format over another when searching for a virtual environment (e.g. if a tool +looks up through parent directories for a virtual environment, it shouldn't +look for a directory first and then a file; the first thing found with the +:file:`.venv` path name should be chosen). Sharing the same path name for both +the directory and file means there is no precedence issue within the same +directory. + +This PEP proposes some changes to the :mod:`venv` module to go along with the +above recommendations: + +#. A ``DEFAULT_NAME: str`` global that's set to ``".venv"``. +#. Create a + ``read_redirect_file(project_root: os.PathLike|str) -> pathlib.Path[str]`` + function for getting the path from a redirect file in + ``project_root / DEFAULT_NAME``. Raises an exception if the location + recorded in the redirect file does not exist. +#. :class:`venv.EnvBuilder` gains + ``write_redirect_file(project_root: os.PathLike, env_dir: os.PathLike) -> None`` + and an equivalent ``write_redirect_file()`` function for the module. The + function and method will create a redirect file at + ``project_root / DEFAULT_NAME`` that points to *env_dir*. +#. :meth:`venv.EnvBuilder.create` and :func:`venv.create` gain a keyword-only + ``project_root: os.PathLike | None`` parameter that will write out a + :file:`.venv` file to that directory via + ``EnvBuilder.write_redirect_file()``. If the value for *env_dir* ends in + ``DEFAULT_NAME`` and *project_root* points to the parent directory of + *env_dir* then ``write_redirect_file()`` will not be called. +#. The *env_dir* parameter for :meth:`venv.EnvBuilder.create` and + :func:`venv.create` get a default value of ``DEFAULT_NAME``. +#. The ``-m venv`` CLI will gain a default value for its *ENV_DIR* argument of + ``DEFAULT_NAME`` (it's currently an error not to provide the argument). +#. The ``-m venv`` CLI will gain a ``--project-root`` option that mirrors the + new parameter to :meth:`venv.EnvBuilder.create`. It will be an error to use + the option when multiple *ENV_DIR* arguments are provided. +#. A function named + ``executable(dir: os.PathLike, *, traverse: bool = False) -> pathlib.Path`` + will be added; it will look for a virtual environment in *dir* at + ``DEFAULT_NAME`` (directory or redirect file) and return the path to the + ``python`` executable for the virtual environment, raising + an exception if the path to a virtual environment is not found or the + virtual environment is somehow corrupted. If *traverse* is true, then + traversal through the parent directories of *dir* to look for + ``DEFAULT_NAME`` as a file or directory will be done and will stop at the + first ``DEFAULT_NAME`` found closest to *dir*. + +With regard to committing a :file:`.venv` file to version control, it MAY be +done when the location of the virtual environment is considered static to a +project once it is set up. For instance, some projects that use tox_ have a +"dev" environment defined in their configuration that ends up at ``.tox/dev``. +Setting a :file:`.venv` file to point to that virtual environment and checking +in the file is reasonable. The same goes for a project that is only worked on +within a container where the location of the virtual environment is controlled +and thus static on the file system. The guidance of NOT committing your actual +virtual environment to version control is unchanged by this PEP. + + +Project Support for this PEP +============================ + +Speaking to various tool maintainers about this PEP: + +- Supports + + 1. PDM (Frost Ming) + 2. Poetry (Randy Döring) + 3. venv (Vinay Sajip) + 4. Virtualenv (Bernát Gábor) + 5. Tox (Bernát Gábor) + 6. Hatch (Cary Hawkins) + +- Lukewarm + + 1. uv (Zanie Blue) + +- Opposes + + 1. Hatch (Ofek Lev) + + +Backwards Compatibility +======================= + +For the virtual environment location aspect of this PEP, the backwards +compatibility concern would be over some alternative use of :file:`.venv`. But +due to the current usage already in the community, the likelihood of an +alternative usage is probably small. This will likely lead to tools showing an +error message when a ``.venv`` file is used, though. While the error message +would likely be around ``.venv`` being a file and thus not explaining *why* +there's a file, it just prevents any tool from overlooking the ``.venv`` file and +blindly creating another virtual environment. + +The other possible backwards compatibility concern is the new default value for +the ``-m venv`` CLI. But since it's currently an error not to specify the +directory, the impact should be minimal. + + +Security Implications +===================== + +Not checking the contents of a potentially malicious :file:`.venv` file and +passing it to a shell process (e.g. ``subprocess.run(..., shell=True)``) would +be a serious security concern. This is why this PEP says tools MUST make sure +the path is valid before using it. + +Setting a :file:`.venv` file to a path that isn't a virtual environment is only +a concern if the arguments the user provided to the executable were also a +concern. That would require the user to craft appropriate arguments +on top of using the malicious :file:`.venv` file. + + +How to Teach This +================= + +For new users, they can be told that ``python -m venv`` creates a virtual +environment in :file:`.venv`, and that any other tool that creates a virtual +environment on their behalf. + +For experienced users, they should be taught the default location for a +project's virtual environment is at the root of the project in :file:`.venv`. +If the currently active virtual environment lives elsewhere, a :file:`.venv` +file will be there to tell them where to find the virtual environment. + + +Reference Implementation +======================== + +The proposed code changes to :mod:`venv` can be found at +https://github.com/brettcannon/cpython/tree/venv-location-pep. A diff showing +the changes can be seen at +https://github.com/brettcannon/cpython/compare/main...brettcannon:cpython:venv-location-pep. + + +Rejected Ideas +============== + +Use a name other than ``.venv`` +------------------------------- + +Some people either don't like that ``.venv`` is hidden by some tools by +default thanks to the leading ``.``, or don't like ``venv`` as an +abbreviation. Since there doesn't seem to be a clear consensus on an +alternative, a different name doesn't fundamentally change any semantics, +existing tools seem to already support ``.venv``, one can still use a different +name for an environment thanks to redirect file support as proposed by this +PEP, and the author of this PEP prefers the name, ``.venv`` was chosen. +Discussing alternative names was viewed as bikeshedding. + + +Open Issues +=========== + +An API for discovering environments +----------------------------------- + +While what the PEP currently proposes is simple, there has been some concern +that it's *too* simple. As such, there's concern that an API to allow +for broader environment discovery is necessary. + +A possible way to provide such an API is to introduce a ``[workflow]`` table to +:file:`pyproject.toml`. That could have an ``environments`` key that holds a +table specifying how to run a discovery app to get environment details in +JSON. For example: + +.. code-block:: toml + + [workflow] + environments = {tool = "...", command = ["..."]} + +The command could output JSON that lists any and all known environments. For +flexibility, the output could include more than just virtual environments, such as +global installs of Python. The output could also specify the environment to use +by default if a tool doesn't have a reason or way to make a choice of which +environment to use. + +.. code-block:: json + + { + "version": "1.0", // ... or whatever to version the JSON schema. + "default": 0, // Optional; index into + "environments": [ + { + "type": "virtual", + "path": ".venv", + "python_version": "...", // Optional + "name": "..." // Optional + }, + ] + } + +The discovery tool could also create a virtual environment as a side-effect +of being called. As well, it could write out a ``.venv`` file or directory if +it makes sense. Having ``.venv`` checked for prior to calling the discovery +tool would help avoid the overhead of calling the discovery tool. + +But this expands the complexity of this PEP. It also isn't necessary to +be a part of this PEP as it could be added later on. + + +"MAY" instead of "SHOULD" +------------------------- + +Some have suggested removing the "SHOULD" for ``.venv`` in either the directory +or file case and make it a "MAY". That would weaken the PEP, but that is +what some want in order to not feel like a specific workflow is being forced +upon tool authors. + + +Acknowledgements +================ + +Thanks to everyone who participated in an earlier discussion on this topic at +https://discuss.python.org/t/22922/. Thanks to Cary Hawkins of Hatch, +Randy Döring of Poetry, Frost Ming of PDM, Bernát Gábor of virtualenv & tox, +Vinay Sajip of venv, and Zanie Blue of uv for feedback on the initial draft of +this PEP. + +Change History +============== + +N/A + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. + + +.. _tox: https://tox.wiki/