Skip to content

ProcessPoolExecutor.map may hang forever when the submitted iterable is far greater than max_tasks_per_child #111498

Description

@alirzaev

Bug report

Bug description:

Summary

ProcessPoolExecutor.map may hang forever when the submitted iterable is far greater than max_tasks_per_child

Internally ProcessPoolExecutor.map submits to the executor all items from the iterable (via submit). The worker process exits when the first max_tasks_per_child work items are processed but a new worker process isn't created because ProcessPoolExecutor._adjust_process_count can be invoked only in submit

Reproducible example

import time
from concurrent.futures import ProcessPoolExecutor


def work(i):
    time.sleep(1)

    return i * i


def main():
    with ProcessPoolExecutor(max_workers=1) as executor:
        _ = list(executor.map(work, range(100)))
        # completes

    with ProcessPoolExecutor(max_workers=1, max_tasks_per_child=5) as executor:
        _ = list(executor.map(work, range(100)))
        # hangs forever


if __name__ == '__main__':
    main()

Expected behavior

Program exits

Actual behavor

Program hangs forever

CPython versions tested on:

3.11

Operating systems tested on:

Windows

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions