Cluster overwhelm countermeasures #13108
Conversation
… connection statistics
This counts ongoing RestHandler instances coming from the low prio queue.
Some additional debugging output.
|
Note that this currently does compile on Windows with some stupid initializer list error. Will fix next week. |
There was a problem hiding this comment.
I have left some inline comments, which I would like to be considered.
For the "production" code comments, none of them justifies to block this PR, so I am fine with the comments not being addressed.
But I am missing CHANGELOG entries here, which would like to sort out before accepting.
| if ((::queueWarningTick++ & 0xFFu) == 0) { | ||
| auto const& now = std::chrono::steady_clock::now(); | ||
| if (::conditionQueueFullSince == time_point{}) { | ||
| logQueueWarningEveryNowAndThen(::queueWarningTick, _maxFifoSize, approxQueueLength); | ||
| logQueueWarningEveryNowAndThen(::queueWarningTick, _maxFifoSizes[3], approxQueueLength); |
There was a problem hiding this comment.
Do we want to add this warning now to prio 2 queue as well?
Having 4 lanes now there is a chance for lane 2 and 3 to starve, while lane 0 and 1 are busy. (0 should be fine, but 1 certainly has the right to do heavy load (e.g. replication => funny RocksDB stalls.)
| // We limit the number of ongoing low priority jobs to prevent a cluster | ||
| // from getting overwhelmed | ||
| std::size_t const ongoing = _ongoingLowPriorityGauge.load(); | ||
| if (ongoing >= _ongoingLowPriorityLimit) { |
There was a problem hiding this comment.
For now this is good.
For the future:
As soon as we have higher parallelization in AQL, this is not enough anymore. (I am think about multiple diamonds in AQL where the upper diamond can work parallel to the lower diamond. as soon as we have this the amount of fan-out requests in AQL is not bounded by the number of DBServers, but by number diamonds * DBServers..
| @@ -490,12 +490,12 @@ bool CommTask::handleRequestSync(std::shared_ptr<RestHandler> handler) { | |||
| // and there is currently only a single client handled by the IoContext | |||
| auto cb = [self = shared_from_this(), handler = std::move(handler), prio]() mutable { | |||
| if (prio == RequestPriority::LOW) { | |||
| SchedulerFeature::SCHEDULER->ongoingLowPriorityTasks() += 1; | |||
| SchedulerFeature::SCHEDULER->trackBeginOngoingLowPriorityTask(); | |||
| Gauge<uint64_t>& SupervisedScheduler::ongoingLowPriorityTasks() { | ||
| return _ongoingLowPriorityGauge; | ||
| void SupervisedScheduler::trackBeginOngoingLowPriorityTask() { | ||
| if (_server.isStopping()) { |
This PR combines several measures. It tries to implement a whole lot
of countermeasures for bad behaviour of the cluster during high load
situations. The document to discuss these countermeasures is here:
This started as a document to collect evidence and problems, but
also ideas for counter measures. Now, it serves as a design
document for the changes actually done. A lot of discussions and
testing will still be required.
Enterprise PR: https://github.com/arangodb/enterprise/pull/600
Scope & Purpose
(Please describe the changes in this PR for reviewers - mandatory)
Backports:
We need to discuss how much of this is needed in 3.7 and 3.6, since the behaviour of clusters under load is pretty bad in all versions.
Related Information
(Please reference tickets / specification / other PRs etc)
Testing & Verification
(Please pick either of the following options)
http://172.16.10.101:8080/view/PR/job/arangodb-matrix-pr/13136/