A Bug With Caching Python Virtualenvs on Self-Hosted GitHub Runners

If you are at a company which owns their own hardware to run GitHub self-hosted runners, you know this can save your company a significant amount of money. However, there are a fair amount of nuances to watch out for. In this post, one thing to watch out for when restoring a Python virtualenv from cache.

Context

For context, the specific setup I’m talking about is running multiple self-hosted GitHub runners directly on a single host, not isolated from each other in any way besides being in their own runner directories (e.g., no VM per runner). This setup can be a nice performance win because it makes it easier to share cache and data.

As a brief sidebar, many people running this type of setup don’t actually use explicit caching at all. For example, where usage of actions/cache is common for GitHub-hosted runners, it is glacially slow for self-hosted runners. So much so that there are alternatives, e.g., buildjet/cache or whywaita/action-cache-s3, which do provide good network performance.

This story is about actually using one of those caches.

We turned on buildjet/cache and were impressed with the performance. But, then found all sorts of odd bugs in our workflows, that felt like environment issues (e.g., ‘module not found’ on imports that should work).

The Issue

The root issue is that when tools (e.g., pytest) are installed by poetry/pip, they are prefixed with a shebang that has a full path to Python. More specifically, suppose you store your virtualenv in .venv, then every script in .venv/bin/ will have this shebang (including activate, meaning this will mess up poetry shell if you use Poetry!). For example:

./.venv/bin/pytest
1:#!/data/github/actions-runner-6/_work/repo/.venv/bin/python

We see here that it was actions-runner-6 which wrote the cache. Then, when a different runner, e.g., actions-runner-1 comes along and hits the cache to restore its virtualenv, when it tries to invoke pytest, things will likely fail, since that binary is calling python in a different environment.

A Hacky Solution

There might be a better approach (and I’d love to hear it!), but one hacky solution is to simply add a step in your workflow to rewrite those shebangs if the cache was hit. One way to do so (in a more readable way than awk/sed/etc.) is to use ripgrep and rep.

For example:

...
    - name: Load cached venv
      id: cached-poetry-deps
      uses: buildjet/cache@v4
      with:
        path: .venv
        key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }
    - name: Correct the .venv/bin paths
      if: steps.cached-poetry-deps.outputs.cache-hit == 'true'
      run: |
        rg "#!" .venv/bin/ --no-ignore -n | rep "#!.+\$" "#!$(pwd)/.venv/bin/python" -w
...

Posts from blogs I follow

Ugo Conti's Whistle-Controlled Synthesizer

Whistling is very nearly a pure sound wave, and a high one at that, which can sound thin and obnoxious. A few years ago I got excited about making whistling sound better by bringing it down a few octaves and synthesizing overtones, and built something t…

via Jeff Kaufman's Writing March 01, 2024

Logo: Clj-Reload

Clj-Reload is smarter way to reload Clojure code during development.

via tonsky.me March 01, 2024

Today's only half of the leap year fun

It's that time again, when code written in the past four years shows up in our lives and breaks something. Still, while you're enjoying the clown show of game companies telling people to manually set the clocks on their consoles and people not being ab…

via Writing - rachelbythebay February 29, 2024

Generated by openring-rs