more tests for replication consistency #13507
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…l/replication-key-tests
…l/replication-key-tests
…l/replication-key-tests
…l/replication-key-tests
…l/replication-key-tests
…odb/arangodb into experimental/replication-key-tests
neunhoef
approved these changes
Feb 9, 2021
| Result res; | ||
|
|
||
| // non-unique indexes will not cause any constraint violation | ||
| if (_unique) { |
There was a problem hiding this comment.
Let's do:
if (!_unique) {
return res;
}
to get fewer indentations.
|
One thing is still on my list: There is a possible assertion failure in maintainer mode if a thread tries to add the first document of a collection at the same time and another thread tries to remove it immediately. In this case the remove can "overtake" the insert and move the document count below zero for a brief moment, enough to trigger the assertion. Do we want to address this in this PR, too? |
Co-authored-by: Max Neunhöffer <max@arangodb.com>
Co-authored-by: Max Neunhöffer <max@arangodb.com>
…l/replication-key-tests
…odb/arangodb into experimental/replication-key-tests
…/arangodb/arangodb into experimental/replication-key-tests
…/arangodb/arangodb into experimental/replication-key-tests
…odb/arangodb into experimental/replication-key-tests
jsteemann
added a commit
that referenced
this pull request
Feb 10, 2021
This is a preparation PR for backporting some further replication tests and bugfixes from devel to 3.7. It isolates some of the changes done in #13507 into a small PR to facilitate reviewing. A further PR with the added replication tests and a bugfix for 3.7 will follow up. This PR extends the length of some internal byte buffers from 11 to 21, so that they cannot overflow, regardless of what input is coming in. The overflow was theoretically possible before, but only with manipulated timestamp input. The PR also fixes an issue in a RocksDB heap helper class that does a potentially unsafe self-move assignment. This issue was raised when running tests with a debug STL build (`_GLIBCXX_DEBUG`).
9 tasks
|
Tests blue |
10 tasks
KVS85
added a commit
that referenced
this pull request
Feb 11, 2021
* backport some changes from #13507 This is a preparation PR for backporting some further replication tests and bugfixes from devel to 3.7. It isolates some of the changes done in #13507 into a small PR to facilitate reviewing. A further PR with the added replication tests and a bugfix for 3.7 will follow up. This PR extends the length of some internal byte buffers from 11 to 21, so that they cannot overflow, regardless of what input is coming in. The overflow was theoretically possible before, but only with manipulated timestamp input. The PR also fixes an issue in a RocksDB heap helper class that does a potentially unsafe self-move assignment. This issue was raised when running tests with a debug STL build (`_GLIBCXX_DEBUG`). * Update CHANGELOG Co-authored-by: Vadim <vadim@arangodb.com>
elfringham
pushed a commit
to elfringham/arangodb
that referenced
this pull request
Apr 20, 2021
Co-authored-by: Max Neunhöffer <max@arangodb.com> Co-authored-by: Max Neunhoeffer <max@arangodb.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Scope & Purpose
Fixes replication bug which leads to data corruption. In the Merkle sync protocol there is the following problem: When there are large amounts of differences between source and destination, then the revisions in the source are shipped in batches to the destination. In that case, it is possible that a smaller (=earlier) revision is inserted in the destination before a larger (=later) revision with the same primary key is removed. As such this would not be a problem and is ultimately inevitable. However, some performance optimizations lead to the fact that this condition is not detected and so the data structures are corrupted. Specifically, the operations flag
ignoreUniqueConstraintso far prevented any unique conflict detection in the primary index or any other unique constraint index. Furthermore, even if the conflict would be detected, another optimization (DisableIndexing) prevented the transaction from seeing its own writes, so the delete of the wrong revision was not detected and would lead to another unique constraint violation.Finally, the case of multiple unique constraints being violated in this case was not considered. In the process of hunting down these problems some more minor problems (like a buffer overrun in
ridBuffers), a bug in RocksDB which leads to an assertion failure and a bug in document count accounting which can lead to an assertion failure were found and are fixed in this PR.Adds more tests for incremental sync.
Additional tests can be started as follows:
Backports:
Testing & Verification
(Please pick either of the following options)
Link to Jenkins PR run:
http://172.16.10.101:8080/view/PR/job/arangodb-matrix-pr/13892/