8000 `concurrent.futures.Executor.map` temporarily exceeds its `buffersize` while collecting the next result · Issue #131466 · python/cpython · GitHub
[go: up one dir, main page]

Skip to content
concurrent.futures.Executor.map temporarily exceeds its buffersize while collecting the next result #131466
Open
@ebonnal

Description

@ebonnal

Bug report

Bug description:

Potential follow up after the merge of #125663 (introduce buffersize param in Executor.map).

"Bug": concurrent.futures.Executor.map temporarily exceeds its buffersize while collecting the next result:

Let's define:

results: Iterator = executor.map(fn, iterable, buffersize=buffersize)

What happens when calling next(results):

  1. fetch the next arg from interable and put a task for fn(arg) in the buffer
  2. wait for next result to be available
  3. yield the collected result

So during step 2. the buffer's size is actually buffersize + 1.

If we want a strict guarantee that there is at most buffersize concurrent tasks even during a next, we have to swap step 1. and step 2.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0