The Python tooling, especially when it comes to package management, can be incredibly annoying to deal with. It's been one of my primary gripes with Python development for years. Recently I have been working on a relatively major Python project in my spare time and have really tried to setup a proper working environment for myself.

Often times you don't want to be on the latest Python available. There's a variety of reasons for this but usually the main one is due to package requirements. There are different ways to manage multiple Python versions. The AUR contains a few older versions of python. Or if you prefer, Nix and Guix can be pinned to a commit containing a specific Python version. But perhaps the easiest method is to use pyenv.

In addition to being able to install various versions of Python, pyenv has shims that allow your shell to determine which version to use depending on which project you're in. This is a perfectly acceptable way of doing it but I only use pyenv to install different versions. For actually choosing which version of Python to use based on which project I'm in I use direnv which I will cover a little later.

Modern Python (3.3+) has a built-in way of creating virtual envrionments. For older Pythons you'll need to use Virtualenv. In addition there are tools that look to improve the workflow surrounding virtual environment development in Python such as Pipenv and Poetry. And as always you can leverage the power of Nix or Guix to bypass using Python-specific virtual environments entirely. My personal preference these days is using Poetry.

direnv is a phenomenal tool that I have been aware of for a while but never actually used until recently. Basically it can automatically enable/disable environment variables depending on the current directory (and subdirectories). This allows you to very quickly have your environment set up per project just by ~cd~ing into a directory.

This is why I'm not using the pyenv shim, direnv handles that for me so it's 1 less command and setup to worry about and in addition direnv is language-agnostic. I can use it for almost anything.

There is a global configuration file for direnv where we'll define how we want it to interface with our virtual environment. This example is designed for using with poetry, but it's fairly straightforward to use with other setups. In fact there is an extensive Community Wiki that covers almost every setup you could want. And creating your own is relatively simple once you understand how it works. Modern direnv supports pyenv out of the box so we don't even need to set anything up for that.

In $XDG_CONFIG_HOME/direnv/direnvrc add the following code:

layout_poetry() {
  if [[ ! -f pyproject.toml ]]; then
    log_error 'No pyproject.toml found. Use `poetry new` or `poetry init` to create one first.'
    exit 2
  fi

  # create venv if it doesn't exist
  poetry run true

  export VIRTUAL_ENV=$(poetry env info --path)
  export POETRY_ACTIVE=1
  PATH_add "$VIRTUAL_ENV/bin"
}

This is taken straight from the Community Wiki and since it's a basic bash function, it should be easy enough to follow. Essentially we first confirm that this is poetry based project after which we set the appropriate environment variables and add the location of the virtual environment that poetry created in our PATH.

Now that direnv knows how to interface properly with poetry we can set up our individual projects. Go to the root directory of a poetry based project and create a file called .envrc. In this file add the following 2 lines:

layout pyenv 3.9.6
layout poetry

The first line uses the built-in pyenv integration to set Python 3.9.6 as the version of Python for our project. Make sure you had previously installed that version using pyenv install 3.9.6.

The second line utilizes the custom function we defined earlier to setup our environment to use the virtual environment that poetry is managing.

That's it. Two simple files that make life so easy.

Of course it's all well and good that our shell fully understands how our Python project is laid out. The next step is to have our editor understand it as well. This is a big part of why I went with direnv. There is really good integration with Emacs. There are Emacs extensions for interacting with poetry and pyenv but that's already 2 more extensions, not to mention I would need to find other well working extensions for other languages and ecosystems if I wasn't relying on direnv.

So far I'm really loving this system. Not just for Python but the combination of direnv and lsp works beautifully with practically every programming language and environment without needing to install and configure a lot of very language-specific Emacs extensions. At most you'll just need to install the appropriate language server and then (sometimes) the specific lsp library for that language server like we did for pyright.