-
Notifications
You must be signed in to change notification settings - Fork 853
Introducing Replicated Logs #14486
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Introducing Replicated Logs #14486
Changes from 1 commit
Commits
Show all changes
569 commits
Select commit
Hold shift + click to select a range
86ed1f0
Fixing some bugs in appendEntries. Retry with delay if reason is not …
e5afc69
Fixed a few paths
goedderz d23c280
Use StaticStrings in AgencyLogSpecification
goedderz e1bd089
Used StaticStrings and AgencyPaths in RestLogHandler
goedderz 39ff6aa
Use AgencyPaths in Maintenance and Supervision
goedderz 296fa10
Add a string constructor for LogPayload, so the tests compile again
goedderz 641b0ce
Added log status on coordinator.
3bbf672
Fixing deserialization.
da3711e
Pass replicated logs to dropCollectionCoordinator
goedderz 5f5d851
Added a TODO note
goedderz 76040b7
Fixed c++ tests.
7afafb4
Fixing tests for real.
4222682
Create agency methods and simplified RestLogHandler.cpp
a8592ef
Set waitForSync for replicated logs
goedderz 5386a21
Fixed c&p error
goedderz 1700471
More static strings.
32e2ef9
Set waitForSync for replicated logs
goedderz b3236a1
Merge branch 'devel' of https://github.com/arangodb/arangodb into fea…
goedderz f9a9ae6
Added tailing to coordinator.
4d6c0f8
Reverted changes
goedderz 7153327
Merge branch 'feature/replication-2.0' of github.com:arangodb/arangod…
f9c66bd
Resign on shutdown.
238a64a
When dropping a collection, drop replicated logs as well.
goedderz fdb5fc3
Added getFirstIndexOfTerm / getLastIndexOfTerm
goedderz b057c15
Fixed compile errors in new maintenance tests
goedderz 1cf4052
Fixed duplicate logIds, and related log messages
goedderz 9848080
Select random follower.
3117394
Merge branch 'feature/replication-2.0' of github.com:arangodb/arangod…
6fc3a1b
Hide the InMemoryLog::_log member
goedderz f811d54
Added some assertions to a test
goedderz bf3550b
Adapted test to new result structure
goedderz 397e554
Fixed JS test
goedderz cd65d56
Fixed compile error in c++ tests
goedderz 89f85d3
Refactored Supervision code to make election testable.
4bedfb9
Added tests for checkReplicatedLog in agency.
6c927ad
Added test with wc = 2.
d8d2736
Fixing wc=2 test.
7975dd5
Fixed logIds
goedderz ddd7956
Make supervision report in Current why no leader can be elected.
5cd4314
Merge remote-tracking branch 'origin/devel' into feature/replication-2.0
f101230
Added AsyncMockLog for lonelyLeader test.
8604e52
A little cleanup
goedderz 25faa0c
Added tests for database creation / deletion
goedderz 9eb6967
Disabled FailedFollower and FailedLeader for new replication databases.
c537b71
Added AgencyCollectionSpecification.
1a57315
Stop async mock logs
goedderz 7991960
Added detectConflict algorithm that is used on the follower to detect…
a7463c1
Enabled optimization for conflict detection.
d89c0cd
Added more logging.
1b6f63a
Moved log create/drop logic from vocbase to log manager, and added a …
goedderz d33c293
Added the collection name to the replicated log LogContext
goedderz 38e332e
Minor changes
goedderz 0dc517c
Fix compile errors with tsan
goedderz a2cffaf
Added UnshackledMutex
goedderz d5a1eaf
Use UnshackledMutex in LogCore for _operationMutex
goedderz 15950e4
Make sure _stopping is initialized before the thread is started
goedderz 932483a
Fixed problems in concurrent test
goedderz b096c09
Use a different immer memory policy, to work around a concurrency iss…
goedderz d43338d
Use transparent functor
goedderz 359e1dc
Added missing dependency
goedderz 9a24222
Fixed an assertion
goedderz c529f18
No longer call `insertAsync` for empty AppendEntriesRequests, and hol…
goedderz 63f3d3a
Added another assertion in the AsyncMockLog
goedderz 19ffb17
Cleanup rest handler.
790a115
Added CollectionGroups to ClusterInfo.
cf447d7
Avoid jslint error
goedderz a0311c4
Made collection name optional in ensureReplicatedLog.
21250d0
Merge remote-tracking branch 'origin/devel' into feature/replication-2.0
54023ce
Replace lastAckedIndex with lastAckedEntry having a meaningful term.
add04ec
Merge branch 'devel' of github.com:arangodb/arangodb into feature/gua…
goedderz c6668d3
Minor changes
goedderz 3dc12ed
Merge branch 'devel' of github.com:arangodb/arangodb into feature/rep…
goedderz 159a458
Merge branch 'feature/guarded-mutex' of github.com:arangodb/arangodb …
goedderz 4dbb050
Cleaned up some TODOs.
15c0c93
More TODO cleanup.
b63e404
noexcept in finally.
857acbb
Made finally exception safe.
9587f44
Require admin access to database for log api. Only enabled in maintai…
ea89666
Added missing tests
goedderz e2dfe36
Added a feature flag for Replication2
goedderz a06fbfc
Replaced leaderId by optional because leader is optional.
9490a3d
Added maintenance tests.
1150d86
More tests for maintenanceMore tests for maintenance..
7d0da23
Fixed maintenance code.
a3bbd09
Extracted action code into algorithms to make it unit testable.
8401ad2
More tests for maintenance action code.
33ce3fd
Replaced references for replicated logs by shared ptrs.
7a989c0
Disable functional tests when replication2 is disabled
goedderz 3398cfd
Cleanup some code paths.
ce98e73
More code cleanup.
ec1e8a5
Moved FeatureFlag.cpp/.h
goedderz edc7cf9
Fixed a few compile errors
goedderz 70d462c
Fixed compile errors in the tests
goedderz d4be0ec
More code cleanup in LogFollower.
56e831f
Fixed a warning
goedderz 5f51904
More code clean up of follower.
6113d06
Fixed bug in messageId handling.
7c50d7d
Cleanup more code.
e391089
Renamed messages.h to NetworkMessages.h
16856c0
Fixing wrong assertion.
4175c15
Move MeasureTimeGuard to common header.
9e43467
Removed TermData and use LogConfig instead. Clean up code at places.
25b06f4
Build wrapper around LogStatus.
95585ca
Fixed tests and compilation.
a736dd3
Allow loading a replication 2 db, even if replication 2 is disabled
goedderz f473421
Add replication 2 tests to arangodbtests binary, and make building ar…
goedderz b86e5bb
Moved Status Objects into LogStatus.h
3f9314f
Added more forward declarations to include the api headers less.
a2cb01c
PR clean up.
5817591
Refactored some headers to optimize includes.
1c922e0
Fixing StorageEngineMock.
af4768c
Added some mis ing declarations
goedderz bbf6dfe
Started writing a test for UnshackledMutex
goedderz 040a4b0
Optimized DBServerAgencySync.h
5a463c9
Added peek() on LogIterators
goedderz 419780d
Revert "Added peek() on LogIterators"
goedderz d81df02
Merge remote-tracking branch 'origin/devel' into feature/replication-2.0
aa2245b
Minor changes
goedderz c610400
Got rid of unnecessarily passed weak_ptr parentLogs
goedderz 91940ad
Removed unused includes.
731e8ad
Collected some changes from replication 2.0 PR.
b3ae831
Merge remote-tracking branch 'origin/feature/replicated-log-companion…
18b6567
Added exception header.
12a307c
Merge remote-tracking branch 'origin/feature/replicated-log-companion…
f0f925b
More code that is not related to replicated logs.
43313c4
Made LogIterator peek()able
goedderz 96b89da
Added readEnd() to PersistedLog and RocksDBLog
goedderz 5aa99be
Reverted accidental change
goedderz 7cc75e5
Revert "Made LogIterator peek()able"
goedderz fa0af53
Revert "Added readEnd() to PersistedLog and RocksDBLog"
goedderz 57c36d9
Merge remote-tracking branch 'origin/devel' into feature/replication-2.0
b4ff84c
Merge remote-tracking branch 'origin/devel' into feature/replicated-l…
16ae7f2
Merge remote-tracking branch 'origin/feature/replicated-log-companion…
56d8eaf
Clean up LogicalCollection changes.
14b438d
Made a comment out of a TODO note
goedderz 6b847ae
Revert "Run gtest binary arangodbtests_replication2"
15f6f57
Added modifications on metrics feature api.
f310f05
Merge remote-tracking branch 'origin/feature/replicated-log-companion…
2348a56
Reverted Collection changes.
358dccd
Reverted collection code.
7ea920d
Added path components.
5dcc68f
Merge remote-tracking branch 'origin/feature/replicated-log-companion…
8fe6c1d
Merge remote-tracking branch 'origin/feature/replicated-log-companion…
9b42045
Fixing clusterinfo.
b0568ed
Reverted ClusterInfo changes.
3ee74a6
Merge remote-tracking branch 'origin/feature/replicated-log-companion…
db4c458
More reverts in logical collection.
f99e800
Merge remote-tracking branch 'origin/feature/replicated-log-companion…
3457738
Merge remote-tracking branch 'origin/feature/replication-2.0' into fe…
2e03ee9
Reverted more of ClusterInfo.
09f0db3
Fixing Maintenance Tests.
da00636
Merge remote-tracking branch 'origin/feature/replicated-log-companion…
4fe1fa7
Cleanup in Maintenance and ClusterInfo.
8f22481
Merge remote-tracking branch 'origin/feature/replicated-log-companion…
13bef2c
Merge remote-tracking branch 'origin/feature/replication-2.0' into fe…
4bfd458
Revert IRSearch Tests.
52b76b3
Minor changes from review
goedderz d0ebb4c
Remove code duplication as suggested in the review
goedderz 8e17cc0
Made FATAL_ERROR_(EXIT|ABORT) exception safe and replaced the macros …
goedderz 5d08a49
Fixed some compile errors
goedderz fcfffe7
Completed UnshackledMutexTest
goedderz 1ed681f
Merge branch 'feature/replicated-log-companion-pr' into feature/repli…
goedderz 7b75ac2
Merge branch 'feature/guarded-mutex' of https://github.com/arangodb/a…
goedderz 45e466c
Merge branch 'feature/replicated-log-companion-pr' of https://github.…
goedderz d511074
Create the AppendEntriesRequest after the optional delay, so it can u…
goedderz bab222f
Merge branch 'devel' of https://github.com/arangodb/arangodb into fea…
goedderz 2213793
Merge branch 'feature/replicated-log-companion-pr' of https://github.…
goedderz cec1f26
Fixed compile and linker errors
goedderz 49cf5a0
Resolve ambiguous call
goedderz 335880d
Resolve ambiguous call
goedderz 24a7b55
Replace unlikely
goedderz 2711672
🤦
goedderz 0afdf78
Added assertions to clarify a constructor's mode of operation
goedderz 8d27093
Added assertions to clarify a constructor's mode of operation
goedderz f6e7453
Added retry to SchedulerExecutor for RocksDBLogPersistor. Implemented…
968854c
Changed and added an assertion to clarify the most important property
goedderz 99e2b2f
Added some comments
goedderz 179cc0c
Merge branch 'feature/replicated-log-companion-pr' of github.com:aran…
goedderz c63e834
Fix test compilation.
658603b
Renamed metrics as suggested by promtool
goedderz 0bb98db
Fixed compile error in tests
goedderz 0ea5fc4
Merge branch 'feature/replication-2.0-pr' of github.com:arangodb/aran…
goedderz 13eb2c7
Made queue code exception safe.
9c04337
Added exponential backoff.
638bf83
Fixed copyright header
goedderz 95901b4
Merge branch 'feature/replicated-log-companion-pr' of github.com:aran…
goedderz 339b747
Fixed copyright headers
goedderz 28ac084
Merge branch 'feature/replication-2.0-pr' of github.com:arangodb/aran…
goedderz d38819c
Renamed RocksDBLog to RocksDBPersistedLog.
72dd8ba
Replaced prevLogIndex/prevLogTerm by prevLogEntry :: TermIndexPair. R…
c879bcb
Resolved a TODO
goedderz 5f23f0b
Merge branch 'feature/replication-2.0-pr' of github.com:arangodb/aran…
goedderz 35b0043
Fixing failover.
4f7cad3
A little cleanup around RocksDBLogPersistor
goedderz 28520ee
Merge branch 'feature/replication-2.0-pr' of https://github.com/arang…
goedderz f4580aa
Merge branch 'devel' of https://github.com/arangodb/arangodb into fea…
goedderz 11b0abe
Make updateReplicatedLog noexcept.
723ce02
Removed a superfluous try-catch block, and resolved a TODO
goedderz 98783e3
Merge remote-tracking branch 'origin/devel' into feature/replication-…
ed74d4e
Reverted changes in ShardingInfo that are not yet necessary
goedderz ff8127a
Removed DelayedLogLeader
goedderz fe69d8f
Some minor changes
goedderz eb17e17
Rename runAsyncStep to triggerAsyncReplication
goedderz 6291151
LogLeader::insert now calls triggerAsyncReplication by default
goedderz 3cca504
Added test that checks for correct commit behavior for entries from p…
f2a6e14
Fixing tests. Added ChangeStream test.
fa36999
Fixing off-by-one error in InMemoryLog. Fixing waitForIterator return…
6c752ba
Move PrevTermCommitTest to the correct sources in cmake
goedderz 76a022f
Do not commit log entries from previous terms
goedderz 2d72668
Increase Replication2's default log level to INFO
goedderz c537b3f
Change the test so it fails with the usual gtest assertions, instead …
goedderz 2f05f8b
Wrapped LogEntry into InMemoryLogEntry
goedderz 9bd22e6
Insert an empty LogEntry in LogLeader::construct
goedderz 2a3456d
[WIP] Separated internal and user-facing log entries and iterators
goedderz 9229c63
Apply suggestions from code review
dd07391
More changes from code review.
6396c12
Added code for loading PersistingLogEntries from disk.
1ec2db9
Added limit to rest api. Skip private log entries and wait for first …
68fa272
Fixed issues from code review.
7569596
Apply suggestions from code review
d54d36b
Added and fixed tests. Applied suggestions from code review.
1a3b80a
Apply suggestions from code review
4998603
Moved extractors to header files and use them exclusively.
a23cc59
Merge remote-tracking branch 'origin/devel' into feature/replication-…
25ef544
Fixing compile issue introduced by github suggestion.
f59d3a4
Remove useless test.
e4488c3
Fixing checkLogIds.py by ignoring the correct file.
8d89b0f
Renamed LogParticipantI to ILogParticipant.
ad0efd0
Apply suggestions from code review
3339c31
Merge remote-tracking branch 'origin/devel' into feature/replication-…
5ddb06d
Implemented suggestions from code review.
b59503a
Fix typo.
5e0efb8
Added comment on implement_compare. Added logging to checkReplicatedLog.
2a3027a
Merge remote-tracking bra
8000
nch 'origin/devel' into feature/replication-…
672afd5
Added commit index as static string.
3719217
Renamed splice to slice. Fixed saturatedDecrement.
7df53d6
Simplify log thread allocation.
c47404a
Review: Wrote out microseconds in metric documentation
goedderz 3bfa5d4
Review: Removed useless return values
goedderz b35e708
Merge branch 'devel' of github.com:arangodb/arangodb into feature/rep…
goedderz 88b4d1e
Fixing bugs found during code review. Use UnshackledMutex in more pla…
6a2f111
Fixing RestLogHandler. Returned only quorum data that contains the wr…
b4c9a85
Small changes from code review.
0f33c96
Applied suggestions from code review.
ddd19ce
Apply suggestions from code review
2714e12
Applied more suggestions. Code clean up. LogTerm/LogIndex always by-v…
48e1cc9
Apply suggestions from code review
2a92412
Added forward declaration of friend operators.
1a2d1c4
Removed dependency on application server.
72d3dba
Remove wrong static assertion.
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Resign on shutdown.
- Loading branch information
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -93,6 +93,43 @@ | |
using namespace arangodb; | ||
using namespace arangodb::basics; | ||
|
||
struct arangodb::VocBaseLogManager { | ||
explicit VocBaseLogManager(arangodb::application_features::ApplicationServer& server) {} | ||
|
||
auto getReplicatedLogById(arangodb::replication2::LogId id) const | ||
-> arangodb::replication2::replicated_log::ReplicatedLog const& { | ||
if (auto iter = _logs.find(id); iter != _logs.end()) { | ||
return iter->second; | ||
} | ||
// THROW_ARANGO_EXCEPTION_PARAMS(TRI_ERROR_REPLICATION_REPLICATED_LOG_NOT_FOUND, id.id()); | ||
maierlars marked this conversation as resolved.
Show resolved
Hide resolved
|
||
THROW_ARANGO_EXCEPTION_FORMAT(TRI_ERROR_REPLICATION_REPLICATED_LOG_NOT_FOUND, | ||
"replicated log %" PRIu64 " not found", id.id()); | ||
} | ||
|
||
auto getReplicatedLogById(arangodb::replication2::LogId id) | ||
-> arangodb::replication2::replicated_log::ReplicatedLog& { | ||
if (auto iter = _logs.find(id); iter != _logs.end()) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we need to protect There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed |
||
return iter->second; | ||
} | ||
// THROW_ARANGO_EXCEPTION_PARAMS(TRI_ERROR_REPLICATION_REPLICATED_LOG_NOT_FOUND, id.id()); | ||
maierlars marked this conversation as resolved.
Show resolved
Hide resolved
|
||
THROW_ARANGO_EXCEPTION_FORMAT(TRI_ERROR_REPLICATION_REPLICATED_LOG_NOT_FOUND, | ||
"replicated log %" PRIu64 " not found", id.id()); | ||
} | ||
|
||
auto resignAll() { | ||
std::unique_lock guard(_mutex); | ||
for (auto&& [id, log] : _logs) { | ||
auto core = log.drop(); | ||
core.release(); | ||
} | ||
_logs.clear(); | ||
} | ||
|
||
// Needs to be a map for stable pointers | ||
std::map<arangodb::replication2::LogId, arangodb::replication2::replicated_log::ReplicatedLog> _logs; | ||
std::mutex _mutex; | ||
}; | ||
|
||
/// @brief increase the reference counter for a database | ||
bool TRI_vocbase_t::use() { | ||
auto expected = _refCount.load(std::memory_order_relaxed); | ||
|
@@ -789,6 +826,7 @@ void TRI_vocbase_t::shutdown() { | |
} | ||
|
||
_collections.clear(); | ||
_logManager->resignAll(); | ||
} | ||
|
||
/// @brief returns names of all known (document) collections | ||
|
@@ -1592,33 +1630,6 @@ arangodb::Result TRI_vocbase_t::dropView(DataSourceId cid, bool allowDropSystem) | |
return TRI_ERROR_NO_ERROR; | ||
} | ||
|
||
struct arangodb::VocBaseLogManager { | ||
explicit VocBaseLogManager(arangodb::application_features::ApplicationServer& server) {} | ||
|
||
auto getReplicatedLogById(arangodb::replication2::LogId id) const | ||
-> arangodb::replication2::replicated_log::ReplicatedLog const& { | ||
if (auto iter = _logs.find(id); iter != _logs.end()) { | ||
return iter->second; | ||
} | ||
// THROW_ARANGO_EXCEPTION_PARAMS(TRI_ERROR_REPLICATION_REPLICATED_LOG_NOT_FOUND, id.id()); | ||
THROW_ARANGO_EXCEPTION_FORMAT(TRI_ERROR_REPLICATION_REPLICATED_LOG_NOT_FOUND, | ||
"replicated log %" PRIu64 " not found", id.id()); | ||
} | ||
|
||
auto getReplicatedLogById(arangodb::replication2::LogId id) | ||
-> arangodb::replication2::replicated_log::ReplicatedLog& { | ||
if (auto iter = _logs.find(id); iter != _logs.end()) { | ||
return iter->second; | ||
} | ||
// THROW_ARANGO_EXCEPTION_PARAMS(TRI_ERROR_REPLICATION_REPLICATED_LOG_NOT_FOUND, id.id()); | ||
THROW_ARANGO_EXCEPTION_FORMAT(TRI_ERROR_REPLICATION_REPLICATED_LOG_NOT_FOUND, | ||
"replicated log %" PRIu64 " not found", id.id()); | ||
} | ||
|
||
// Needs to be a map for stable pointers | ||
std::map<arangodb::replication2::LogId, arangodb::replication2::replicated_log::ReplicatedLog> _logs; | ||
std::mutex _mutex; | ||
}; | ||
|
||
TRI_vocbase_t::TRI_vocbase_t(TRI_vocbase_type_e type, arangodb::CreateDatabaseInfo&& info) | ||
: _server(info.server()), | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this acquire the
_mutex
when accessing_logs
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed