8000 one shard db - devel (#9395) · archerli/arangodb@0b8c75c · GitHub
[go: up one dir, main page]
Skip to content

Commit 0b8c75c

Browse files
ObiWahnjsteemann
ObiWahn
authored and
jsteemann
committed
one shard db - devel (arangodb#9395)
1 parent fbcb4b7 commit 0b8c75c

File tree

218 files changed

+12296
-8686
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

218 files changed

+12296
-8686
lines changed

‎CHANGELOG

Lines changed: 26 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,20 @@
11
devel
22
-----
3+
* Add ReplicationFactor, MinReplicationFactor and sharding strategy to database creation
4+
dialogue in web UI. Preselect database default values for collection creation in web UI.
5+
6+
* Add new JavaScipt function `db._properties()` and REST route `GET /_api/database/properties`
7+
that provides information about the current database's properties.
8+
9+
* Add new options `sharding` and `replicationFactor` for database creation methods. The
10+
specified values will provide the defaults for all collections created in a database.
11+
12+
Valid values for `sharding` are `""`, "flexible", "single". The first 2 values are
13+
treated equally. Values for `replicationFactor` are natural numbers or the string
14+
`satellite`.
15+
16+
* Add new server option `--cluster.default-replication-factor` that allows to set the
17+
default replication factor for non-system collections (default: 1).
318

419
* Fix a problem with AQL constrained sort in the cluster, which might abort
520
queries.
@@ -43,7 +58,7 @@ devel
4358

4459
* Allowing inconsistent rather than forcing hot backups.
4560

46-
* Fixed adding an orphan collections as the first collection in a SmartGraph.
61+
* Fixed adding an orphan collections as the first collection in a SmartGraph.
4762

4863
* Fixed issue #9862: ServerException: RestHandler/RestCursorHandler.cpp:279
4964

@@ -153,22 +168,22 @@ v3.5.0-rc.6 (2019-07-29)
153168

154169
* Added startup error for bad temporary directory setting.
155170

156-
If the temporary directory (--temp.path) setting is identical to the database
157-
directory (`--database.directory`) this can eventually lead to data loss, as
158-
temporary files may be created inside the temporary directory, causing overwrites of
171+
If the temporary directory (--temp.path) setting is identical to the database
172+
directory (`--database.directory`) this can eventually lead to data loss, as
173+
temporary files may be created inside the temporary directory, causing overwrites of
159174
existing database files/directories with the same names.
160-
Additionally the temporary directory may be cleaned at some point, and this would lead
175+
Additionally the temporary directory may be cleaned at some point, and this would lead
161176
to an unintended cleanup of the database files/directories as well.
162-
Now, if the database directory and temporary directory are set to the same path, there
163-
will be a startup warning about potential data loss (though in ArangoDB 3.4 allowing to
177+
Now, if the database directory and temporary directory are set to the same path, there
178+
will be a startup warning about potential data loss (though in ArangoDB 3.4 allowing to
164179
continue the startup - in 3.5 and higher we will abort the startup).
165180

166181
* Make TTL indexes behave like other indexes on creation.
167-
182+
168183
If a TTL index is already present on a collection, the previous behavior
169184
was to make subsequent calls to `ensureIndex` fail unconditionally with
170185
the error "there can only be one ttl index per collection".
171-
186+
172187
Now we are comparing the attributes of the to-be-created index with the
173188
attributes of the existing TTL index and make it only fail when the
174189
attributes differ. If the attributes are identical, the `ensureIndex`
@@ -184,15 +199,15 @@ v3.5.0-rc.5 (2019-07-22)
184199
many insync followers. With this mechanism users can avoid to have collections diverge too much in case of failure scenarios.
185200
minReplicationFactor can have the values: 1 <= minReplicationFactor <= replicationFactor.
186201
Having minReplicationFactor == 1 ArangoDB behaves the same way as in any previous version.
187-
202+
188203
* Fixed a query abort error with smart joins if both collections were restricted to a
189204
single shard using the "restrict-to-single-shard" optimizer rule.
190205

191206
* Fixed a performance regression of COLLECT WITH COUNT INTO.
192207

193208
* Fixed some races in cluster collection creation, which allowed collections with the
194209
same name to be created in parallel under some rare conditions.
195-
210+
196211
* arangoimport would not stop, much less report, communications errors. Add CSV reporting
197212
of line numbers that are impacted during such errors
198213

‎Documentation/DocuBlocks/Rest/Database/get_api_database_current.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11

22
@startDocuBlock get_api_database_current
3-
@brief retrieves information about the current database
3+
@brief retrieves information about the current database (alias /_api/database/properties)
44

55
@RESTHEADER{GET /_api/database/current, Information of the database, getDatabases:current}
66

@@ -17,6 +17,11 @@ The response is a JSON object with the following attributes:
1717

1818
- *isSystem*: whether or not the curre 10000 nt database is the *_system* database
1919

20+
- *sharding*: information about the default sharding method for collections created in this database
21+
22+
- *replicationFactor*: the default replication factor for collections in this database
23+
24+
2025
@RESTRETURNCODES
2126

2227
@RESTRETURNCODE{200}

‎Documentation/DocuBlocks/Rest/Database/get_api_database_new.md

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,18 @@
77
@RESTBODYPARAM{name,string,required,string}
88
Has to contain a valid database name.
99

10+
@RESTBODYPARAM{options,object,optional,get_api_database_new_USERS}
11+
Optional object which can contain the following attributes:
12+
13+
@RESTSTRUCT{sharding,get_api_database_new_USERS,string,optional,string}
14+
The sharding method to use for new collections in this database. Valid values
15+
are: "", "flexible", or "single". The first two are equivalent.
16+
17+
@RESTSTRUCT{replicationFactor,get_api_database_new_USERS,string,optional,number}
18+
Default replication factor for new collections created in this database.
19+
Special values include "satellite", which will replicate the collection to
20+
every DB-server, and 1, which disables replication.
21+
1022
@RESTBODYPARAM{users,array,optional,get_api_database_new_USERS}
1123
Has to be an array of user objects to initially create for the new database.
1224
User information will not be changed for users that already exist.
@@ -66,7 +78,11 @@ Creating a database named *example*.
6678
}
6779

6880
var data = {
69-
name: name
81+
name: name,
82+
options: {
83+
sharding: "flexible",
84+
replicationFactor: 3
85+
}
7086
};
7187
var response = logCurlRequest('POST', url, data);
7288

@@ -76,7 +92,9 @@ Creating a database named *example*.
7692
logJsonResponse(response);
7793
@END_EXAMPLE_ARANGOSH_RUN
7894

79-
Creating a database named *mydb* with two users.
95+
Creating a database named *mydb* with two users, flexible sharding and
96+
default replication factor of 3 for collections that will be part of
97+
the newly created database.
8098

8199
@EXAMPLE_ARANGOSH_RUN{RestDatabaseCreateUsers}
82100
var url = "/_api/database";
@@ -91,13 +109,13 @@ Creating a database named *mydb* with two users.
91109
name: name,
92110
users: [
93111
{
94-
username : "admin",
95-
passwd : "secret",
112+
username: "admin",
113+
passwd: "secret",
96114
active: true
97115
},
98116
{
99-
username : "tester",
100-
passwd : "test001",
117+
username: "tester",
118+
passwd: "test001",
101119
active: false
102120
}
103121
]

‎arangod/Agency/StoreCallback.cpp

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,12 +34,12 @@ StoreCallback::StoreCallback(
3434
bool StoreCallback::operator()(arangodb::ClusterCommResult* res) {
3535

3636
if (res->status == CL_COMM_ERROR) {
37-
LOG_TOPIC("9sdbf0", TRACE, Logger::AGENCY)
37+
LOG_TOPIC("9dbf0", TRACE, Logger::AGENCY)
3838
<< _url << "(" << res->status << ", " << res->errorMessage
3939
<< "): " << _body->toJson();
4040

4141
if (res->result->getHttpReturnCode() == 404 && _agent != nullptr) {
42-
LOG_TOPIC("9sdbf0", DEBUG, Logger::AGENCY) << "dropping dead callback at " << _url;
42+
LOG_TOPIC("9dbfa", DEBUG, Logger::AGENCY) << "dropping dead callback at " << _url;
4343
_agent->trashStoreCallback(_url, _body);
4444
}
4545
}

‎arangod/Aql/BlocksWithClients.cpp

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -63,14 +63,18 @@ using VelocyPackHelper = arangodb::basics::VelocyPackHelper;
6363
using StringBuffer = arangodb::basics::StringBuffer;
6464

6565
BlocksWithClients::BlocksWithClients(ExecutionEngine* engine, ExecutionNode const* ep,
66-
std::vector<std::string> const& shardIds)
66+
std::vector<std::string> const& shardIds)
6767
: ExecutionBlock(engine, ep),
6868
_nrClients(shardIds.size()),
69+
_type(ScatterNode::ScatterType::SHARD),
6970
_wasShutdown(false) {
7071
_shardIdMap.reserve(_nrClients);
7172
for (size_t i = 0; i < _nrClients; i++) {
7273
_shardIdMap.emplace(std::make_pair(shardIds[i], i));
7374
}
75+
auto scatter = ExecutionNode::castTo<ScatterNode const*>(ep);
76+
TRI_ASSERT(scatter != nullptr);
77+
_type = scatter->getScatterType();
7478
}
7579

7680
std::pair<ExecutionState, bool> BlocksWithClients::getBlock(size_t atMost) {
@@ -112,16 +116,16 @@ std::pair<ExecutionState, Result> BlocksWithClients::shutdown(int errorCode) {
112116
/// corresponding to <shardId>
113117
size_t BlocksWithClients::getClientId(std::string const& shardId) const {
114118
if (shardId.empty()) {
115-
THROW_ARANGO_EXCEPTION_MESSAGE(TRI_ERROR_INTERNAL, "got empty shard id");
119+
THROW_ARANGO_EXCEPTION_MESSAGE(TRI_ERROR_INTERNAL,
120+
"got empty distribution id");
116121
}
117-
118122
auto it = _shardIdMap.find(shardId);
119123
if (it == _shardIdMap.end()) {
120-
std::string message("AQL: unknown shard id ");
124+
std::string message("AQL: unknown distribution id ");
121125
message.append(shardId);
122126
THROW_ARANGO_EXCEPTION_MESSAGE(TRI_ERROR_INTERNAL, message);
123127
}
124-
return ((*it).second);
128+
return it->second;
125129
}
126130

127131
void BlocksWithClients::throwIfKilled() {

‎arangod/Aql/BlocksWithClients.h

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ class ExecutionEngine;
5555
class BlocksWithClients : public ExecutionBlock {
5656
public:
5757
BlocksWithClients(ExecutionEngine* engine, ExecutionNode const* ep,
58-
std::vector<std::string> const& shardIds);
58+
std::vector<std::string> const& shardIds);
5959

6060
~BlocksWithClients() override = default;
6161

@@ -99,6 +99,9 @@ class BlocksWithClients : public ExecutionBlock {
9999
/// @brief _nrClients: total number of clients
100100
size_t _nrClients;
101101

102+
/// @brief type of distribution that this nodes follows.
103+
ScatterNode::ScatterType _type;
104+
102105
private:
103106
bool _wasShutdown;
104107
};

‎arangod/Aql/ClusterNodes.cpp

Lines changed: 44 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -40,11 +40,13 @@
4040
#include "Aql/ScatterExecutor.h"
4141
#include "Aql/SingleRemoteModificationExecutor.h"
4242
#include "Aql/SortingGatherExecutor.h"
43+
#include "Cluster/ServerState.h"
4344

4445
#include "Transaction/Methods.h"
4546

4647
#include <type_traits>
4748

49+
using namespace arangodb;
4850
using namespace arangodb::basics;
4951
using namespace arangodb::aql;
5052

@@ -90,13 +92,17 @@ arangodb::velocypack::StringRef toString(GatherNode::SortMode mode) noexcept {
9092

9193
/// @brief constructor for RemoteNode
9294
RemoteNode::RemoteNode(ExecutionPlan* plan, arangodb::velocypack::Slice const& base)
93-
: ExecutionNode(plan, base),
95+
: DistributeConsumerNode(plan, base),
9496
_vocbase(&(plan->getAst()->query()->vocbase())),
9597
_server(base.get("server").copyString()),
96-
_ownName(base.get("ownName").copyString()),
97-
_queryId(base.get("queryId").copyString()),
98-
_isResponsibleForInitializeCursor(
99-
base.get("isResponsibleForInitializeCursor").getBoolean()) {}
98+
_queryId(base.get("queryId").copyString()) {
99+
// Backwards compatibility (3.4.x)(3.5.0) and earlier, coordinator might send ownName.
100+
arangodb::velocypack::StringRef tmpId(getDistributeId());
101+
tmpId = VelocyPackHelper::getStringRef(base, "ownName", tmpId);
102+
if (tmpId != getDistributeId()) {
103+
setDistributeId(tmpId.toString());
104+
}
105+
}
100106

101107
/// @brief creates corresponding ExecutionBlock
102108
std::unique_ptr<ExecutionBlock> RemoteNode::createBlock(
@@ -131,21 +137,19 @@ std::unique_ptr<ExecutionBlock> RemoteNode::createBlock(
131137
ExecutorInfos infos({}, {}, nrInRegs, nrOutRegs, std::move(regsToClear),
132138
std::move(regsToKeep));
133139

134-
return std::make_unique<ExecutionBlockImpl<RemoteExecutor>>(&engine, this,
135-
std::move(infos), server(),
136-
ownName(), queryId());
140+
return std::make_unique<ExecutionBlockImpl<RemoteExecutor>>(
141+
&engine, this, std::move(infos), server(), getDistributeId(), queryId());
137142
}
138143

139144
/// @brief toVelocyPack, for RemoteNode
140-
void RemoteNode::toVelocyPackHelper(VPackBuilder& nodes, unsigned flags) const {
145+
void RemoteNode::toVelocyPackHelper(VPackBuilder& nodes, unsigned flags,
146+
std::unordered_set<ExecutionNode const*>& seen) const {
141147
// call base class method
142-
ExecutionNode::toVelocyPackHelperGeneric(nodes, flags);
148+
DistributeConsumerNode::toVelocyPackHelperInternal(nodes, flags, seen);
143149

144150
nodes.add("database", VPackValue(_vocbase->name()));
145151
nodes.add("server", VPackValue(_server));
146-
nodes.add("ownName", VPackValue(_ownName));
147152
nodes.add("queryId", VPackValue(_queryId));
148-
nodes.add("isResponsibleForInitializeCursor", VPackValue(_isResponsibleForInitializeCursor));
149153

150154
// And close it:
151155
nodes.close();
@@ -191,13 +195,13 @@ std::unique_ptr<ExecutionBlock> ScatterNode::createBlock(
191195
}
192196

193197
/// @brief toVelocyPack, for ScatterNode
194-
void ScatterNode::toVelocyPackHelper(VPackBuilder& nodes, unsigned flags) const {
198+
void ScatterNode::toVelocyPackHelper(VPackBuilder& nodes, unsigned flags,
199+
std::unordered_set<ExecutionNode const*>& seen) const {
195200
// call base class method
196-
ExecutionNode::toVelocyPackHelperGeneric(nodes, flags);
201+
ExecutionNode::toVelocyPackHelperGeneric(nodes, flags, seen);
197202

198203
// serialize clients
199204
writeClientsToVelocyPack(nodes);
200-
201205
// And close it:
202206
nodes.close();
203207
}
@@ -228,10 +232,14 @@ bool ScatterNode::readClientsFromVelocyPack(VPackSlice base) {
228232
++pos;
229233
}
230234

235+
_type = static_cast<ScatterNode::ScatterType>(
236+
basics::VelocyPackHelper::getNumericValue<uint64_t>(base, "scatterType", 0));
237+
231238
return true;
232239
}
233240

234241
void ScatterNode::writeClientsToVelocyPack(VPackBuilder& builder) const {
242+
builder.add("scatterType", VPackValue(static_cast<uint64_t>(getScatterType())));
235243
VPackArrayBuilder arrayScope(&builder, "clients");
236244
for (auto const& client : _clients) {
237245
builder.add(VPackValue(client));
@@ -314,9 +322,10 @@ std::unique_ptr<ExecutionBlock> DistributeNode::createBlock(
314322
}
315323

316324
/// @brief toVelocyPack, for DistributedNode
317-
void DistributeNode::toVelocyPackHelper(VPackBuilder& builder, unsigned flags) const {
325+
void DistributeNode::toVelocyPackHelper(VPackBuilder& builder, unsigned flags,
326+
std::unordered_set<ExecutionNode const*>& seen) const {
318327
// call base class method
319-
ExecutionNode::toVelocyPackHelperGeneric(builder, flags);
328+
ExecutionNode::toVelocyPackHelperGeneric(builder, flags, seen);
320329

321330
// add collection information
322331
CollectionAccessingNode::toVelocyPack(builder);
@@ -396,9 +405,10 @@ GatherNode::GatherNode(ExecutionPlan* plan, size_t id, SortMode sortMode) noexce
396405
: ExecutionNode(plan, id), _sortmode(sortMode) {}
397406

398407
/// @brief toVelocyPack, for GatherNode
399-
void GatherNode::toVelocyPackHelper(VPackBuilder& nodes, unsigned flags) const {
408+
void GatherNode::toVelocyPackHelper(VPackBuilder& nodes, unsigned flags,
409+
std::unordered_set<ExecutionNode const*>& seen) const {
400410
// call base class method
401-
ExecutionNode::toVelocyPackHelperGeneric(nodes, flags);
411+
ExecutionNode::toVelocyPackHelperGeneric(nodes, flags, seen);
402412

403413
if (_elements.empty()) {
404414
nodes.add("sortmode", VPackValue(SortModeUnset.data()));
@@ -438,8 +448,18 @@ std::unique_ptr<ExecutionBlock> GatherNode::createBlock(
438448
getRegisterPlan()->nrRegs[getDepth()]);
439449
IdExecutorInfos infos(getRegisterPlan()->nrRegs[getDepth()],
440450
calcRegsToKeep(), getRegsToClear());
441-
return std::make_unique<ExecutionBlockImpl<IdExecutor<SingleRowFetcher<true>>>>(
442-
&engine, this, std::move(infos));
451+
if (ServerState::instance()->isCoordinator()) {
452+
// In the coordinator case the GatherBlock will fetch from RemoteBlocks.
453+
// We want to immediately move the block on and not wait for additional requests here (hence passthrough)
454+
return std::make_unique<ExecutionBlockImpl<IdExecutor<true, SingleRowFetcher<true>>>>(
455+
&engine, this, std::move(infos));
456+
} else {
457+
// In the DBServer case the GatherBlock will merge local results and then expose them (directly or indirectly)
458+
// To the RemoteBlock on coordinator. We want to trigger as few requests as possible, so we invest the little
459+
// memory inefficiency that we have here in favor of a better grouping of requests.
460+
return std::make_unique<ExecutionBlockImpl<IdExecutor<false, SingleRowFetcher<false>>>>(
461+
&engine, this, std::move(infos));
462+
}
443463
}
444464
std::vector<SortRegister> sortRegister;
445465
SortRegister::fill(*plan(), *getRegisterPlan(), _elements, sortRegister);
@@ -542,9 +562,10 @@ std::unique_ptr<ExecutionBlock> SingleRemoteOperationNode::createBlock(
542562
}
543563

544564
/// @brief toVelocyPack, for SingleRemoteOperationNode
545-
void SingleRemoteOperationNode::toVelocyPackHelper(VPackBuilder& nodes, unsigned flags) const {
565+
void SingleRemoteOperationNode::toVelocyPackHelper(VPackBuilder& nodes, unsigned flags,
566+
std::unordered_set<ExecutionNode const*>& seen) const {
546567
// call base class method
547-
ExecutionNode::toVelocyPackHelperGeneric(nodes, flags);
568+
ExecutionNode::toVelocyPackHelperGeneric(nodes, flags, seen);
548569
CollectionAccessingNode::toVelocyPackHelperPrimaryIndex(nodes);
549570

550571
// add collection information

0 commit comments

Comments
 (0)
0