Skip to content

Support an explicit environment mapping for multiprocessing spawn #157613

Description

@K0u1e

Bug report

Bug description:

"""Linux spawn/environment race. Use --no-environment-updates as a control."""

from collections import Counter
from concurrent.futures import ThreadPoolExecutor
import multiprocessing as mp
import os
import sys
import threading
import time

def child():
    pass


def update_environment(stop):
    keys = [f"SPAWN_REPRO_{i}" for i in range(256)]
    while not stop.is_set():
        for key in keys:
            os.environ[key] = "test"
        for key in keys:
            os.environ.pop(key, None)
        time.sleep(0)


def start_process(_):
    process = mp.get_context("spawn").Process(target=child)
    try:
        process.start()
    except Exception as error:
        process.close()
        return error
    return process


if __name__ == "__main__":
    print(sys.version, flush=True)
    # Start the resource tracker before racing, using only the public API.
    warmup = mp.get_context("spawn").Process(target=child)
    warmup.start()
    warmup.join()
    warmup.close()

    stop = threading.Event()
    updater = threading.Thread(target=update_environment, args=(stop,))
    if "--no-environment-updates" not in sys.argv:
        updater.start()
    try:
        with ThreadPoolExecutor(32) as executor:
            for round_number in range(20):
                # Finish all starts before joining: avoid concurrent reaping.
                results = list(executor.map(start_process, range(32)))
                counts = Counter()
                for result in results:
                    if isinstance(result, Exception):
                        counts[repr(result)] += 1
                    else:
                        result.join()
                        counts[result.exitcode] += 1
                        result.close()
                print(f"round {round_number + 1}: {dict(counts)}", flush=True)
                if counts[0] != 32:
                    raise SystemExit(1)
    finally:
        stop.set()
        if updater.ident is not None:
            updater.join()

if you run python test.py, you will get :

round 1: {0: 32}
round 2: {0: 32}
round 3: {0: 32}
round 4: {0: 31, "BrokenPipeError(32, 'Broken pipe')": 1}

and if you run with --no-environment-updates, the subprocess create successfully.

round 1: {0: 32}
round 2: {0: 32}
round 3: {0: 32}
round 4: {0: 32}
round 5: {0: 32}
round 6: {0: 32}
round 7: {0: 32}
round 8: {0: 32}
round 9: {0: 32}
round 10: {0: 32}
round 11: {0: 32}
round 12: {0: 32}
round 13: {0: 32}
round 14: {0: 32}
round 15: {0: 32}
round 16: {0: 32}
round 17: {0: 32}
round 18: {0: 32}
round 19: {0: 32}
round 20: {0: 32}

In my tests on Linux with CPython 3.14, the multiprocessing "spawn" start method follows a vfork() + execv() path. The child shares the parent’s address space until it successfully executes a new program or terminates. During this interval, other threads in the parent can continue modifying the process environment.

With another thread concurrently modifying os.environ, I observed execv() returning EFAULT, followed by the child exiting with status 255. At the Python level, this can surface as a BrokenPipeError from Process.start(). These observations are consistent with a race between the kernel reading the environment passed to execve() and another thread modifying the underlying global environ array.

The C posix_spawn() API accepts an explicit envp argument, and Python’s subprocess.Popen(env=...) similarly allows callers to supply an environment mapping. Could we consider adding an env parameter to multiprocessing.Process for the "spawn" start method, for example mp.get_context("spawn").Process(target=child, env=child_env)? This would allow callers to provide a stable environment for the child interpreter before exec, without temporarily modifying the process-global environment.

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Linked PRs

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions