Jobs dynamically change AllocatedRunspaceCount #25539
Comments
PowerShell is the best engine, and I use it to make the best cars.Welcome to the "Kasini3000 Project", Open source, free, cross platform, based on PowerShell rather than Ansible over YAML It has multi-process concurrency and multi-threaded concurrency. You can dynamically set the EACH load job number of multiple processes/threads from sqlite db by PowerShell global variable$global:kcommit每次执行任务数 = 100 # To-be-run records read from the database each time Keep total concurrency by PowerShell global variable$global:kcommit最大并发进程数 = 99 #max count limit for run simultaneously job by k-commit.ps1 Only adjust batch tasks with specific commentsEach task has a comment field on SQLite, and as long as you define comments when submitting a batch of tasks at the kasini3000 command line, |
Summary of the new feature / enhancement
For better adaption to dynamic load it would be great if jobs could be dynamically updated with the amount of parallel childjobs it processes.
Such a feature is especially desirable when the limit in question does not arise from local resources like cpu, or memory but from something remote and where an API as it is way more difficult to impossible to get a good limit without dialing it in (aka slowly increasing the load)
E.g. all jobs get scheduled via:
$bgJobs = ... | ForEach-Object -ThrottleLimit 32 -AsJob -Parallel {Do-Some -LongRunningBackgroundJob}. Then something at runtime happens and the limit needs to be updated (or alternatively simply because the system is either under or over utilized by this task).Proposed technical implementation details (optional)
Add an "UpdateThrottleLimit" method to System.Management.Automation.PSTasks.PSTaskJob that when called will influence how many pending jobs get scheduled moving forward
$bgJobs.UpdateThrottleLimit(20).Or the
AllocatedRunspaceCountproperty is extended with an additional setter$bgJobs.AllocatedRunspaceCount = 20E.g. there are 2000 jobs in the queue the current limit is 32 and therefore 32 are currently running. When the limit is lowered (
$bgJobs.UpdateThrottleLimit(5)) no additional job is started until the number of currently running jobs falls below it. If the limit is increased ($bgJobs.UpdateThrottleLimit(64)) additional child jobs are started until it is reached.The text was updated successfully, but these errors were encountered: