CANCEL_NEW concurrency strategy does not cancel Late runs, causing queue buildup #21060
Description
Bug summary
Description
When using ConcurrencyLimitConfig(limit=1, collision_strategy=ConcurrencyLimitStrategy.CANCEL_NEW),
runs that enter Late state are never cancelled.
CANCEL_NEW only acts on the Scheduled → Pending transition. If the work queue is fully occupied,
runs never reach that transition — they stay in Late state and accumulate indefinitely.
Reproduction
- Create a deployment with
concurrency_limit=1andCANCEL_NEWstrategy - Fill the work queue to 100% capacity so no new runs can be picked up
- Trigger several runs of the same flow (or let the cron schedule create them)
- All runs enter
Latestate instead of being cancelled - Once the queue frees up, all accumulated runs start executing one by one —
only then doesCANCEL_NEWkick in and cancel the extras
Expected behavior
Runs should be cancelled immediately when the concurrency limit is reached,
regardless of whether they are in Late or Pending state.
Workaround
We query the Prefect API at the start of each flow to find and cancel
other Late/Scheduled runs for the same deployment.
Version
Prefect 3.4.25, Python 3.12
Version info
Prefect 3.4.25, Python 3.12
Additional context
No response