8000 Feature/satellite graphs by hkernbach · Pull Request #11015 · arangodb/arangodb · GitHub
[go: up one dir, main page]

Skip to content

Feature/satellite graphs #11015

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 61 commits into from
Mar 15, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
425c005
added replicationFactor satellite
hkernbach Jan 6, 2020
555ca78
added method to check if graph exists only out of satellite collections
hkernbach Jan 6, 2020
8d1bbfb
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sat…
hkernbach Jan 27, 2020
bf150c0
Added SatelliteTraversalNode
goedderz Jan 28, 2020
872721b
added scatterSatelliteGraphRule optimizer rule, still wip
hkernbach Jan 28, 2020
9e2ee40
Merge branch 'feature/satellite-graphs' of github.com:arangodb/arango…
hkernbach Jan 28, 2020
116662d
protected to private, reformat
hkernbach Jan 28, 2020
6c24691
moved optimizer rule logic out of header file
hkernbach Jan 28, 2020
35e222e
added isEligibleAsSatelliteTraversal method to traversal node
hkernbach Jan 28, 2020
c604333
added isEligibleAsSatelliteTraversal method to traversal node
hkernbach Jan 28, 2020
f80b013
added method ensureSatelliteCollectionSharding
hkernbach Jan 28, 2020
7122f1f
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sat…
hkernbach Jan 28, 2020
16cfd74
preparation for satellites
hkernbach Jan 29, 2020
cfd855c
Replaced the SatelliteTraversalNode move constructor with a copy cons…
goedderz Jan 29, 2020
ae32ada
format, remove graphmanager ee switch
hkernbach Jan 29, 2020
bf98f28
Merge branch 'feature/satellite-graphs' of github.com:arangodb/arango…
hkernbach Jan 29, 2020
74df9c2
Moved SatelliteTraversalNode to the enterprise repo
goedderz Jan 29, 2020
e8072c4
Bugfix in scatter rule
goedderz Jan 29, 2020
9761b4d
added repl factor sat check, needs tidy up later
hkernbach Jan 29, 2020
f082be8
added indicator to satellite graph tile
hkernbach Jan 29, 2020
f939bf3
added modal entry point for satellite graphs - ui
hkernbach Jan 29, 2020
ece8b72
Several bugfixes
goedderz Jan 29, 2020
adfe02c
8000 ui - more satellite graph preparation, additional cleanup
hkernbach Jan 29, 2020
648f822
Merge branch 'feature/satellite-graphs' of github.com:arangodb/arango…
hkernbach Jan 29, 2020
0243278
Fixed query snippets
goedderz Jan 30, 2020
e1e61d4
Fix non-maintainer build
goedderz Jan 30, 2020
0aa6acd
Two bugfixes
goedderz Jan 30, 2020
afefc0c
Implemented shortest path
goedderz Jan 31, 2020
a42af7b
merge devel into feature branch
hkernbach Mar 3, 2020
50dfcf0
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sat…
hkernbach Mar 3, 2020
b23d31a
optimized graph satellite ui
hkernbach Mar 3, 2020
10ccf45
static strings
hkernbach Mar 3, 2020
541c383
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sat…
hkernbach Mar 3, 2020
af1f28e
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sat…
hkernbach Mar 4, 2020
f4e03b7
properly add replication factor to options
hkernbach Mar 4, 2020
4a5c8a0
rm wrong cast
hkernbach Mar 4, 2020
de0d9f5
only persist cluster values when actually a cluster is used
hkernbach Mar 6, 2020
db397ec
prepare sat graph testsuite
hkernbach Mar 6, 2020
2d27cd4
other isSmart
hkernbach Mar 6, 2020
10b4305
move some logic we will use later to a helper
hkernbach Mar 6, 2020
655b697
remove log
hkernbach Mar 6, 2020
cfe831d
sat check
hkernbach Mar 9, 2020
3334ee3
add isSatellite
hkernbach Mar 10, 2020
d19763a
do NOT set distributeShardsLike to empty in satellite collection case
hkernbach Mar 10, 2020
6241f80
Feature/satellite graphs kshortestpathnode (#11253)
goedderz Mar 11, 2020
49b646f
jslint
hkernbach Mar 11, 2020
79ebd90
Merge branch 'feature/satellite-graphs' of github.com:arangodb/arango…
hkernbach Mar 11, 2020
fcc8d15
Implemented common ancestor SatelliteGraphNode
goedderz Mar 11, 2020
46aaa75
Merge branch 'feature/satellite-graphs' of github.com:arangodb/arango…
goedderz Mar 11, 2020
fc32950
Make castTo work for GraphNodes again
goedderz Mar 11, 2020
ae50a9f
fixed a failing test
hkernbach Mar 11, 2020
6cb629e
Merge branch 'feature/satellite-graphs-implement-common-satellitegrap…
Mar 12, 2020
7be51fe
Added SatelliteGraphNode::graph() (#11256)
goedderz Mar 12, 2020
8e71d94
merge devel into feature branch
hkernbach Mar 12, 2020
f85153b
fix http gharial test
hkernbach Mar 12, 2020
5251d2d
Added exception check for replicationFactor == 0; made isSatellite mo…
Mar 12, 2020
43e047b
Make OneShard not set distributeShardsLike for satellite collections
goedderz Mar 13, 2020
996defe
Bugfix when removing multiple satellite joins
goedderz Mar 13, 2020
1611c4b
Fixed clone() for all SatelliteNodes
goedderz Mar 14, 2020
54a2a75
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sat…
goedderz Mar 14, 2020
59f3653
Fixed non-enterprise compile error
goedderz Mar 14, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions arangod/Aql/EngineInfoContainerDBServerServerBased.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ EngineInfoContainerDBServerServerBased::EngineInfoContainerDBServerServerBased(Q
// NOTE: We need to start with _lastSnippetID > 0. 0 is reserved for GraphNodes
}

void EngineInfoContainerDBServerServerBased::injectVertexColletions(GraphNode* graphNode) {
void EngineInfoContainerDBServerServerBased::injectVertexCollections(GraphNode* graphNode) {
auto const& vCols = graphNode->vertexColls();
if (vCols.empty()) {
std::map<std::string, Collection*> const* allCollections =
Expand Down Expand Up @@ -226,7 +226,9 @@ void EngineInfoContainerDBServerServerBased::addNode(ExecutionNode* node) {
case ExecutionNode::TRAVERSAL:
case ExecutionNode::SHORTEST_PATH:
case ExecutionNode::K_SHORTEST_PATHS: {
injectVertexColletions(static_cast<GraphNode*>(node));
auto* const graphNode = ExecutionNode::castTo<GraphNode*>(node);
graphNode->prepareOptions();
injectVertexCollections(graphNode);
break;
}
default:
Expand Down Expand Up @@ -516,7 +518,7 @@ void EngineInfoContainerDBServerServerBased::cleanupEngines(
// the DBServers. The GraphNode itself will retain on the coordinator.
void EngineInfoContainerDBServerServerBased::addGraphNode(GraphNode* node) {
node->prepareOptions();
injectVertexColletions(node);
injectVertexCollections(node);
// SnippetID does not matter on GraphNodes
_shardLocking.addNode(node, 0);
_graphNodes.emplace_back(node);
Expand Down
2 changes: 1 addition & 1 deletion arangod/Aql/EngineInfoContainerDBServerServerBased.h
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ class EngineInfoContainerDBServerServerBased {
ServerID const& server, std::string const& serverDest,
std::vector<bool> const& didCreateEngine) const;

void injectVertexColletions(GraphNode* node);
void injectVertexCollections(GraphNode* node);
private:
std::stack<std::shared_ptr<QuerySnippet>, std::vector<std::shared_ptr<QuerySnippet>>> _snippetStack;

Expand Down
61 changes: 35 additions & 26 deletions arangod/Aql/ExecutionNode.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -463,6 +463,12 @@ ExecutionNode::ExecutionNode(ExecutionPlan* plan, VPackSlice const& slice)
VelocyPackHelper::getBooleanValue(slice, "isInSplicedSubquery", false);
}

ExecutionNode::ExecutionNode(ExecutionPlan& plan, ExecutionNode const& other)
: ExecutionNode(&plan, other.id()) {
TRI_ASSERT(&plan == other.plan());
other.cloneWithoutRegisteringAndDependencies(plan, *this, false);
}

/// @brief toVelocyPack, export an ExecutionNode to VelocyPack
void ExecutionNode::toVelocyPack(VPackBuilder& builder, unsigned flags,
bool keepTopLevelOpen) const {
Expand All @@ -485,59 +491,64 @@ ExecutionNode* ExecutionNode::cloneHelper(std::unique_ptr<ExecutionNode> other,
bool withDependencies, bool withProperties) const {
ExecutionPlan* plan = other->plan();

if (plan == _plan) {
cloneWithoutRegisteringAndDependencies(*plan, *other, withProperties);

auto* registeredNode = plan->registerNode(std::move(other));

if (withDependencies) {
cloneDependencies(plan, registeredNode, withProperties);
}

return registeredNode;
}

void ExecutionNode::cloneWithoutRegisteringAndDependencies(ExecutionPlan& plan, ExecutionNode& other, bool withProperties) const {

if (&plan == _plan) {
// same execution plan for source and target
// now assign a new id to the cloned node, otherwise it will fail
// upon node registration and/or its meaning is ambiguous
other->setId(plan->nextId());
other.setId(plan.nextId());

// cloning with properties will only work if we clone a node into
// a different plan
TRI_ASSERT(!withProperties);
}

other->_regsToClear = _regsToClear;
other->_depth = _depth;
other->_varUsageValid = _varUsageValid;
other->_isInSplicedSubquery = _isInSplicedSubquery;
other._regsToClear = _regsToClear;
other._depth = _depth;
other._varUsageValid = _varUsageValid;
other._isInSplicedSubquery = _isInSplicedSubquery;

if (withProperties) {
auto allVars = plan->getAst()->variables();
auto allVars = plan.getAst()->variables();
// Create new structures on the new AST...
other->_varsUsedLater.reserve(_varsUsedLater.size());
other._varsUsedLater.reserve(_varsUsedLater.size());
for (auto const& orgVar : _varsUsedLater) {
auto var = allVars->getVariable(orgVar->id);
TRI_ASSERT(var != nullptr);
other->_varsUsedLater.insert(var);
other._varsUsedLater.insert(var);
}

other->_varsValid.reserve(_varsValid.size());
other._varsValid.reserve(_varsValid.size());

for (auto const& orgVar : _varsValid) {
auto var = allVars->getVariable(orgVar->id);
TRI_ASSERT(var != nullptr);
other->_varsValid.insert(var);
other._varsValid.insert(var);
}

if (_registerPlan.get() != nullptr) {
auto otherRegisterPlan =
std::shared_ptr<RegisterPlan>(_registerPlan->clone(plan, _plan));
other->_registerPlan = otherRegisterPlan;
std::shared_ptr<RegisterPlan>(_registerPlan->clone(&plan, _plan));
other._registerPlan = otherRegisterPlan;
}
} else {
// point to current AST -> don't do deep copies.
other->_varsUsedLater = _varsUsedLater;
other->_varsValid = _varsValid;
other->_registerPlan = _registerPlan;
other._varsUsedLater = _varsUsedLater;
other._varsValid = _varsValid;
other._registerPlan = _registerPlan;
}

auto* registeredNode = plan->registerNode(std::move(other));

if (withDependencies) {
cloneDependencies(plan, registeredNode, withProperties);
}

return registeredNode;
}

/// @brief helper for cloning, use virtual clone methods for dependencies
Expand Down Expand Up @@ -1067,8 +1078,6 @@ RegisterId ExecutionNode::getNrOutputRegisters() const {
ExecutionNode::ExecutionNode(ExecutionPlan* plan, size_t id)
: _id(id), _depth(0), _varUsageValid(false), _plan(plan), _isInSplicedSubquery(false) {}

ExecutionNode::~ExecutionNode() = default;

size_t ExecutionNode::id() const { return _id; }

void ExecutionN 10000 ode::swapFirstDependency(ExecutionNode* node) {
Expand Down
22 changes: 20 additions & 2 deletions arangod/Aql/ExecutionNode.h
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@
#include "Aql/WalkerWorker.h"
#include "Aql/types.h"
#include "Basics/Common.h"
#include "Basics/TypeTraits.h"
#include "Containers/HashSet.h"

namespace arangodb {
Expand Down Expand Up @@ -162,6 +163,14 @@ class ExecutionNode {
ExecutionNode() = delete;
ExecutionNode(ExecutionNode const&) = delete;
ExecutionNode& operator=(ExecutionNode const&) = delete;
protected:
/// @brief Clone constructor, used for constructors of derived classes.
/// Does not clone recursively, does not clone properties (`other.plan()` is
/// expected to be the same as `plan)`, and does not register this node in the
/// plan.
ExecutionNode(ExecutionPlan& plan, ExecutionNode const& other);

public:

/// @brief constructor using an id
ExecutionNode(ExecutionPlan* plan, size_t id);
Expand All @@ -170,7 +179,7 @@ class ExecutionNode {
ExecutionNode(ExecutionPlan* plan, arangodb::velocypack::Slice const& slice);

/// @brief destructor, free dependencies
virtual ~ExecutionNode();
virtual ~ExecutionNode() = default;

public:
/// @brief factory from JSON
Expand All @@ -195,7 +204,13 @@ class ExecutionNode {
TRI_ASSERT(result != nullptr);
return result;
#else
return static_cast<T>(node);
// At least GraphNode is virtually inherited by its subclasses. We have to
// use dynamic_cast for these types.
if constexpr (can_static_cast_v<FromType, T>) {
return static_cast<T>(node);
} else {
return dynamic_cast<T>(node);
}
#endif
}

Expand Down Expand Up @@ -303,6 +318,9 @@ class ExecutionNode {
ExecutionNode* cloneHelper(std::unique_ptr<ExecutionNode> Other,
bool withDependencies, bool withProperties) const;

void cloneWithoutRegisteringAndDependencies(ExecutionPlan& plan, ExecutionNode& other,
bool withProperties) const;

/// @brief helper for cloning, use virtual clone methods for dependencies
void cloneDependencies(ExecutionPlan* plan, ExecutionNode* theClone, bool withProperties) const;

Expand Down
23 changes: 23 additions & 0 deletions arangod/Aql/ExecutionPlan.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2443,6 +2443,29 @@ bool ExecutionPlan::fullCount() const noexcept {
return lastLimitNode != nullptr && lastLimitNode->fullCount();
}

void ExecutionPlan::prepareTraversalOptions() {
::arangodb::containers::SmallVector<ExecutionNode*>::allocator_type::arena_type a;
::arangodb::containers::SmallVector<ExecutionNode*> nodes{a};
findNodesOfType(nodes,
{arangodb::aql::ExecutionNode::TRAVERSAL,
arangodb::aql::ExecutionNode::SHORTEST_PATH,
arangodb::aql::ExecutionNode::K_SHORTEST_PATHS},
true);
for (auto& node : nodes) {
switch (node->getType()) {
case ExecutionNode::TRAVERSAL:
case ExecutionNode::SHORTEST_PATH:
case ExecutionNode::K_SHORTEST_PATHS: {
auto* graphNode = ExecutionNode::castTo<GraphNode*>(node);
graphNode->prepareOptions();
} break;
default:
TRI_ASSERT(false);
THROW_ARANGO_EXCEPTION(TRI_ERROR_INTERNAL_AQL);
}
}
}

#ifdef ARANGODB_ENABLE_MAINTAINER_MODE
#include <iostream>

Expand Down
2 changes: 2 additions & 0 deletions arangod/Aql/ExecutionPlan.h
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,8 @@ class ExecutionPlan {
/// @brief static analysis
void planRegisters() { _root->planRegisters(); }

void prepareTraversalOptions();

/// @brief unlinkNodes, note that this does not delete the removed
/// nodes and that one cannot remove the root node of the plan.
void unlinkNodes(std::unordered_set<ExecutionNode*> const& toUnlink);
Expand Down
Loading
0