Skip to content

Free-threaded CPython: concurrent new_threadstate() / PyGILState_Ensure() from N≥~8 OS threads deadlocks in _PyEval_StopTheWorld #157666

Description

@CypherGrue

Bug report

Bug description:

In Python/pystate.c, new_threadstate() → _Py_qsbr_reserve() → (when QSBR freelist is empty) _PyEval_StopTheWorld(interp) (see qsbr.c: _Py_qsbr_reserve, and 3.14.0's pystate.c STW path). When ≥~8 OS threads enter PyGILState_Ensure simultaneously from fresh threads, the first one to hit an empty QSBR freelist requests an STW; the other ≥7 threads are concurrently in (or about to enter) tstate_wait_attach/tstate_try_attach/new_threadstate. They cannot proceed because the STW is not yet complete, and the STW requester cannot complete because the attach path is blocked — a circular stall (futex_do_wait, ~0 CPU).

Distinct from #151518 (and not fixed by the first pull request there).

Short reproducer:

/* N pthreads, each: PyGILState_Ensure -> loop of 5,000x PyEval_SaveThread/RestoreThread -> PyGILState_Release.
   Trigger: >=8 threads (deterministic at 8, 12, 16, 20 on every version tested);
   6 and below: passes. 

cc -O2 -pthread -o repro repro_ft.c \
   -I/tmp/<prefix>/include/python3.1Xt \
   /tmp/<prefix>/lib/libpython3.1Xt.a -lpthread -lm -lutil -ldl

timeout 45 ./repro 8   # -> "launched worker 8" then HANG (exit 124)
*/
#include <Python.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>

static volatile long long counter = 0;
static volatile int g_stop = 0;
static int g_iters = 5000;
static int g_mode = 1;  /* 1=churn, 0=once */

static void *worker(void *arg) {
    PyGILState_STATE st = PyGILState_Ensure();
    long long n = 0;
    for (long long i = 0; i < g_iters && !g_stop; i++) {
        if (g_mode == 0) { /* once: already ensured once, loop is Save/Restore only */
        }
        PyThreadState *ts = PyEval_SaveThread();
        for (volatile int k = 0; k < 100; k++) {}
        PyEval_RestoreThread(ts);
        n++;
    }
    PyGILState_Release(st);
    __sync_fetch_and_add(&counter, n);
    return NULL;
}

int main(int argc, char **argv) {
    int n = argc > 1 ? atoi(argv[1]) : 8;
    g_mode = argc > 2 && !strcmp(argv[2], "once") ? 0 : 1;
    g_iters = argc > 3 ? (int)atol(argv[3]) : 5000;
    Py_Initialize();
    fprintf(stderr, "Py_VERSION=%s\n", Py_GetVersion());
    pthread_t *t = malloc(sizeof(pthread_t) * n);
    for (int i = 0; i < n; i++) { pthread_create(&t[i], NULL, worker, NULL); fprintf(stderr, "launched worker %d\n", i); }
    for (int i = 0; i < n; i++) pthread_join(t[i], NULL);
    fprintf(stderr, "ALL WORKERS COMPLETED (PASS) counter=%lld\n", counter);
    Py_Finalize();
    return 0;
}

Environment / repro notes

  • Tested on Ryzen 7700X (8 physical cores, 16 logical) (my B200 is undergoing repairs, honestly)
  • Ubuntu 25.04
  • Linux kernel 7.2.2
  • GNU C (cc) 14.2.0
  • GIL-disabled builds only (Py_GIL_DISABLED=1); GIL-enabled builds do not exercise the Py_GILState_* STW path in this code.
  • LD_PRELOAD none; no ASAN/UBSAN at build time.

Results

Threads 3.14.0 (installed, FT) 3.15.0rc2 (unpatched, FT) 3.15.0rc2 + PR #152826 (FT)
4 PASS PASS PASS
6 PASS PASS PASS
8 HANG (exit 124) HANG (exit 124) HANG (exit 124)
12 HANG (exit 124) HANG (exit 124) HANG (exit 124)
20 HANG (exit 124)

Checkbox (per CPython "Reporting bugs" convention)

  • I can reproduce this on the current stable (3.14.0) and current development/RC (3.15.0rc2) free-threading builds.
  • I built from source (3.15.0rc2) and I can provide the exact --disable-gil build recipe.
  • I can reproduce this on the Windows or macOS free-threading builds — not tested (Linux x86_64 only so far).
  • I can reproduce it on Python < 3.14 — not tested (free-threading is 3.13+/3.14+/3.15+; I expect the same stall on 3.13 free-threading but have not measured it).

CPython versions tested on:

3.15

Operating systems tested on:

Linux

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions