10000 Feature/single server smart graph by hkernbach · Pull Request #14821 · arangodb/arangodb · GitHub
[go: up one dir, main page]

Skip to content

Feature/single server smart graph #14821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 173 commits into from
Nov 17, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
173 commits
Select commit Hold shift + click to select a range
80e813b
add comments, make interfaces clearer
hkernbach Sep 15, 2021
e420e18
another comment
hkernbach Sep 15, 2021
f9f9e40
renamed _gmngr to _graphManager
hkernbach Sep 15, 2021
f7f8990
remove isCluster for smart graphs check
hkernbach Sep 15, 2021
9063d73
next position we need to enable in non-cluster mode
hkernbach Sep 15, 2021
1630844
a todo, an improved error message
romanatarango Sep 16, 2021
c1d357b
Merge branch 'feature/single-server-smart-graph' of github.com:arango…
romanatarango Sep 16, 2021
46e194c
rm log devel
hkernbach Sep 16, 2021
f306018
Merge branch 'feature/single-server-smart-graph' of github.com:arango…
romanatarango Sep 16, 2021
2f7cb05
Merge branch 'devel' into feature/single-server-smart-graph
Sep 17, 2021
58b4a73
Added tests for single server smart graphs.
Sep 17, 2021
1f97c3b
Added tests for single server disjoint smart graphs and single server…
Sep 17, 2021
1bd7524
Merge branch 'feature/single-server-smart-graph' of github.com:arango…
romanatarango Sep 20, 2021
0ed0129
isSatellite is now also being allowed to be set in single server mode
hkernbach Sep 21, 2021
46fbe58
added const qualifiers, todo
romanatarango Sep 21, 2021
80fd936
adapted to work with SSSG
romanatarango Sep 21, 2021
4952adb
Merge branch 'feature/single-server-smart-graph' of github.com:arango…
romanatarango Sep 21, 2021
7157829
refactoring and edited comments
romanatarango Sep 23, 2021
8000
c82f44b
temporarily added temporary files
romanatarango Sep 24, 2021
581663b
refactored a temporary file to the class CollectionValidator
romanatarango Sep 24, 2021
4a164c6
added comments to our codebase
hkernbach Sep 24, 2021
467d84d
added todo
hkernbach Sep 24, 2021
9c940f2
added VocBase/Methods/CollectionValidator.cpp to CMake
romanatarango Sep 24, 2021
f748203
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
romanatarango Sep 24, 2021
0768d88
Merge branch 'feature/single-server-smart-graph' of github.com:arango…
romanatarango Sep 24, 2021
52c00c1
compilation bug fix
romanatarango Sep 24, 2021
56fb22c
use devel vocbase.cpp - as sth went wrong during merge
hkernbach Sep 24, 2021
d3134bc
Merge branch 'feature/single-server-smart-graph' of github.com:arango…
romanatarango Sep 24, 2021
98ec561
attempt to make use of single server sg collection creation
hkernbach Sep 24, 2021
bea5b61
refactored and finished CollectionValidator
romanatarango Sep 24, 2021
936ce29
Merge branch 'feature/single-server-smart-graph' of github.com:arango…
romanatarango Sep 24, 2021
8b661fe
refactored getBuilder
romanatarango Sep 24, 2021
ff9d851
always use isIgnoredHiddenEnterpriseCollection
hkernbach Sep 27, 2021
fb5c7b1
cleanup CollectionValidator, remove unused var
hkernbach Sep 27, 2021
2c69ad6
added disclaimer and pragma once
romanatarango Sep 27, 2021
dd27d15
Merge branch 'feature/single-server-smart-graph' of github.com:arango…
romanatarango Sep 27, 2021
9a39337
Merge branch 'devel' into feature/single-server-smart-graph
Sep 27, 2021
0e227f0
added missing helper init (incl. revs), format, fixed assert
hkernbach Sep 27, 2021
1479f98
slightly refactored drop collections standalone / cluster
hkernbach Sep 27, 2021
d696ee2
added missing parameters for new collections
hkernbach Sep 28, 2021
5d7e94f
format
hkernbach Sep 29, 2021
0f7a712
format
hkernbach Sep 29, 2021
526931c
added some comments
romanatarango Sep 29, 2021
db580c5
Merge remote-tracking branch 'origin' into feature/single-server-smar…
romanatarango Sep 29, 2021
f4e2bdb
forbid physical collections for rocksdb serialization
hkernbach Sep 29, 2021
f1d6316
Added SSSG flag to other 'create' method.
Sep 30, 2021
c78a820
Merge branch 'feature/single-server-smart-graph' of github.com:arango…
hkernbach Sep 30, 2021
44c90e6
Merge branch 'feature/single-server-smart-graph' of github.com:arango…
romanatarango Sep 30, 2021
80bfe9a
Merge branch 'feature/single-server-smart-graph' of github.com:arango…
hkernbach Sep 30, 2021
e081e91
Splitted VirtualCollection into VirtualSmartEdgeCollection and Virtua…
romanatarango Oct 1, 2021
7a7d1c4
Feature/single server smart graph dump restore tests (#14828)
adityamukho Oct 4, 2021
2f99262
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 4, 2021
30d6ba5
parse and verify isSystem during collection creation
hkernbach Oct 4, 2021
6801ee4
pass boolean sssg property correctly
hkernbach Oct 5, 2021
03d00ed
rm unused parameter
hkernbach Oct 5, 2021
7d1c417
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 6, 2021
ec8b3c1
added assert, as code comment says that this method is only allowed i…
hkernbach Oct 6, 2021
f8349fb
typo
hkernbach Oct 6, 2021
83454ab
also expose smart properties in case we do have a smart collection in…
hkernbach Oct 6, 2021
1eb0056
only set numberOfShards to zero in case of cluster
hkernbach Oct 6, 2021
3fd8900
added todo, changed getBuilder to createCollectionProperties
hkernbach Oct 6, 2021
dd09566
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 6, 2021
bbc1a50
add static strings
hkernbach Oct 7, 2021
f17df56
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 7, 2021
b71effd
verifyEdge inside the one document worker
hkernbach Oct 8, 2021
f628c29
merge conflict
hkernbach Oct 8, 2021
3cce4c2
typo
hkernbach Oct 8, 2021
711419e
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 11, 2021
016f5c4
make logical statement readable, fix statement, add comment, format
hkernbach Oct 11, 2021
b8f4ec5
SmartVertex insert support
hkernbach Oct 11, 2021
520a800
make createKey and createSmartToSatKey const
hkernbach Oct 11, 2021
6cc4030
make createKey const
hkernbach Oct 11, 2021
c82863f
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 11, 2021
c66257f
fix logical issue
hkernbach Oct 11, 2021
377dd03
added missing cluster subquery insert verification check
hkernbach Oct 12, 2021
6864918
validator fixes
hkernbach Oct 13, 2021
bc65ddb
extended validation
hkernbach Oct 13, 2021
366e0b2
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 13, 2021
a148b22
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 14, 2021
98ce5f2
tmp parameter for decoratoree
hkernbach Oct 15, 2021
96c1e36
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 15, 2021
74817f2
add _distributeShardsLike to properties in case we are smart or satel…
hkernbach Oct 15, 2021
a6eb562
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 18, 2021
5cebb7b
more readable graph test names
hkernbach Oct 18, 2021
b2913a0
make more graph variants executable on single server generic suite
hkernbach Oct 18, 2021
3384ab8
add forgotten const statement
hkernbach Oct 18, 2021
e25a950
add forgotten bool parameter
hkernbach Oct 18, 2021
2ccdb30
fix isCoordinator check
hkernbach Oct 18, 2021
fd3430f
add sssg test to community dump test to blacklist
hkernbach Oct 18, 2021
2be844b
blacklist hint
hkernbach Oct 18, 2021
511b791
add sssg test to community dump_authentication test to blacklist
hkernbach Oct 18, 2021
235a04a
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 18, 2021
8d2770c
only require sgm module when enterprise is available
hkernbach Oct 19, 2021
2925c67
expand assert
hkernbach Oct 19, 2021
f637653
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 19, 2021
56be21c
do not expose all properties for a general graph in a singleserver
hkernbach Oct 19, 2021
ea36617
let instead of const
hkernbach Oct 19, 2021
9e941e8
exclude sgm tests from dump_modified
hkernbach Oct 19, 2021
9c3ba3c
Feature/single server smart graph satellite (#14921)
romanatarango Oct 19, 2021
af9adb0
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 19, 2021
b39898d
more test added to cluster dump blacklist
hkernbach Oct 20, 2021
4b78fd8
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 20, 2021
d8f90fb
fix test
hkernbach Oct 20, 2021
1add3e9
added assert, fixed isSatellite method
hkernbach Oct 20, 2021
ed393d8
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 20, 2021
e51701f
only run ee tests when ee is active..
hkernbach Oct 20, 2021
0e23973
jslint
hkernbach Oct 20, 2021
3cd7f93
assert too strict
hkernbach Oct 20, 2021
380eb47
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 20, 2021
04f670c
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 21, 2021
863ae69
fix boolean logical error
hkernbach Oct 21, 2021
1ddb80f
fix graph validation after restore
hkernbach Oct 21, 2021
1cf913e
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 21, 2021
4de7733
enable creation of enterprise sharded graphs also in the UI (#14936)
hkernbach Oct 21, 2021
b05fe49
added orphan checks to generic graph suite (#14953)
hkernbach Oct 26, 2021
06b5eb3
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 27, 2021
013d0e5
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Oct 29, 2021
eaafd94
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Nov 2, 2021
1823a31
pls make gitignore changes in a different repo
hkernbach Nov 2, 2021
a5095de
Update arangod/Cluster/ClusterHelpers.h
hkernbach Nov 2, 2021
7b9554d
Update arangod/Cluster/ClusterInfo.cpp
hkernbach Nov 2, 2021
c4edd18
removed todo
hkernbach Nov 2, 2021
a51c0cf
Merge branch 'feature/single-server-smart-graph' of github.com:arango…
hkernbach Nov 2, 2021
dc446b4
resolved todo - added static double const var
hkernbach Nov 2, 2021
8e5c389
removed todo
hkernbach Nov 2, 2021
21b957f
removed obsolete todo
hkernbach Nov 2, 2021
b665176
removed obsolete todo
hkernbach Nov 2, 2021
fb5fc6a
removed obsolete todo - validation takes place in constructor of logi…
hkernbach Nov 2, 2021
e8fd414
Feature/single server smart graph satellite (#14945)
romanatarango Nov 2, 2021
78db38e
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Nov 2, 2021
4804261
Merge branch 'devel' of github.com:arangodb/arangodb into feature/sin…
hkernbach Nov 4, 2021
f05b15b
Changelog added
hkernbach Nov 4, 2021
0dab8e3
check if parameter can be removed
hkernbach Nov 4, 2021
b04aff9
unused code
hkernbach Nov 5, 2021
95f4e12
fix analyzer test in dump_setup_mixed testsuite
hkernbach Nov 5, 2021
5f7350d
renamed createCollectionOnCoordinator to createCollectionsOnCoordinator
romanatarango Nov 8, 2021
cbce417
merge devel into feature branch - changelog conflict
hkernbach Nov 8, 2021
74627da
renamed sssg to isSingleServerEnterpriseCollection
hkernbach Nov 8, 2021
11555de
more use of velocypack helper methods
hkernbach Nov 8, 2021
9e121b0
Update arangod/Sharding/ShardingInfo.cpp
hkernbach Nov 8, 2021
afc7b14
removed old code
hkernbach Nov 8, 2021
a9e79a8
more use of velocypack helper methods
hkernbach Nov 8, 2021
8000
cd4f874
refactor, makeSatellite instead of markAsSatellite
hkernbach Nov 8, 2021
311fe8d
Schema is now taking full responsibility of edges
hkernbach Nov 9, 2021
7dff9ef
Feature/single server smart graph fix dump mixed tests (#15056)
hkernbach Nov 9, 2021
0751013
merge devel into feature branch - changelog conflict
hkernbach Nov 9, 2021
1d5edb1
rename one more createCollectionsOnCoordinator ..
hkernbach Nov 10, 2021
384cb8e
merge devel into feature branch
hkernbach Nov 10, 2021
059d7e2
merge devel into feature branch
hkernbach Nov 10, 2021
137838b
forgot to return error
hkernbach Nov 10, 2021
33b098c
added isRestore flag to collection creation method
hkernbach Nov 11, 2021
f507379
merge devel into feature branch
hkernbach Nov 11, 2021
098da76
also create graph of all types in dump setup ee, remove isCluster con…
hkernbach Nov 11, 2021
9979cc1
Update CHANGELOG
hkernbach Nov 12, 2021
e17dd55
Update arangod/VocBase/Methods/Collections.cpp
hkernbach Nov 12, 2021
f321307
moved back comment to proper place, renamed isSingleServerEnterpriseC…
hkernbach Nov 12, 2021
686dd65
Merge branch 'feature/single-server-smart-graph' of github.com:arango…
hkernbach Nov 12, 2021
88f7d83
implemented suggested change
hkernbach Nov 12, 2021
f459b87
add comment to section
hkernbach Nov 12, 2021
5b5d863
renamed createLogicalCollections to prepareLogicalCollectionStubs
hkernbach Nov 12, 2021
7993dd4
remove const declaration from createKey
hkernbach Nov 12, 2021
8df47db
remove const declaration from createKey
hkernbach Nov 12, 2021
947353a
remove isEnterprise variants from ce testcode
hkernbach Nov 12, 2021
0a17c22
remove uneccessary types
hkernbach Nov 12, 2021
a75cd20
remove const declaration
hkernbach Nov 12, 2021
92f45cf
rm onst
hkernbach Nov 15, 2021
6c42853
added reminder todo
hkernbach Nov 15, 2021
b5826c5
removed unusual doxygen format
hkernbach Nov 15, 2021
39d3339
make code more readable
hkernbach Nov 15, 2021
2a3c9da
removed CollectionValidator class
hkernbach Nov 15, 2021
d577117
removed CollectionValidator class
hkernbach Nov 15, 2021
92a33ec
removed not needed parameter
hkernbach Nov 16, 2021
e5d5f31
merge devel into feature branch
hkernbach Nov 16, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added tests for single server disjoint smart graphs and single server…
… satellite graphs (broken).
  • Loading b 8000 ranch information
Aditya Mukhopadhyay committed Sep 17, 2021
commit 1f97c3b9b31be8c98bdaff20b034fc9e11c20bf9
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,9 @@ const TestVariants = Object.freeze({
SmartGraph: 3,
SatelliteGraph: 4,
DisjointSmartGraph: 5,
SmartGraphSingleServer: 6
SmartGraphSingleServer: 6,
DisjointSmartGraphSingleServer: 7,
SatelliteGraphSingleServer: 8,
});

const graphWeightAttribute = 'distance';
Expand Down Expand Up @@ -86,6 +88,7 @@ class TestGraph {
sgm._create(this.name(), [this.eRel], [], options);
break;
}
case TestVariants.DisjointSmartGraphSingleServer:
case TestVariants.DisjointSmartGraph: {
const options = {
numberOfShards: this.numberOfShards,
Expand All @@ -96,6 +99,7 @@ class TestGraph {
sgm._create(this.name(), [this.eRel], [], options);
break;
}
case TestVariants.SatelliteGraphSingleServer:
case TestVariants.SatelliteGraph: {
const options = {
replicationFactor: 'satellite'
Expand All @@ -105,7 +109,12 @@ class TestGraph {
}
}

if (this.testVariant === TestVariants.SingleServer || this.testVariant === TestVariants.SmartGraphSingleServer) {
if (
[
TestVariants.SingleServer, TestVariants.SmartGraphSingleServer, TestVariants.DisjointSmartGraphSingleServer,
TestVariants.SatelliteGraphSingleServer
].includes(this.testVariant)
) {
this.verticesByName = TestGraph._fillGraph(this.graphName, this.edges, db[this.vn], db[this.en], this.unconnectedVertices);
} else {
const shardAttrsByShardIndex = this._shardAttrPerShard(db[this.vn]);
Expand Down Expand Up @@ -133,7 +142,9 @@ class TestGraph {
switch (this.testVariant) {
case TestVariants.SingleServer:
case TestVariants.SatelliteGraph:
case TestVariants.SatelliteGraphSingleServer:
case TestVariants.SmartGraphSingleServer:
case TestVariants.DisjointSmartGraphSingleServer:
case TestVariants.GeneralGraph: {
return `${this.vn}/nonExistingVertex`;
}
Expand Down Expand Up @@ -273,7 +284,7 @@ class ProtoGraph {
prepareSmartGraphs(variant = TestVariants.SmartGraph) {
return this.smartShardings.map((sharding, idx) => {
const {numberOfShards, vertexSharding} = sharding;
const suffix = ProtoGraph._buildSmartSuffix(sharding, idx);
const suffix = ProtoGraph._buildSmartSuffix(sharding, idx, variant);

const vn = this.protoGraphName + '_Vertex' + suffix;
const en = this.protoGraphName + '_Edge' + suffix;
Expand All @@ -285,10 +296,10 @@ class ProtoGraph {
});
}

prepareDisjointSmartGraphs() {
prepareDisjointSmartGraphs(variant = TestVariants.DisjointSmartGraph) {
return this.smartShardings.map((sharding, idx) => {
const {numberOfShards, vertexSharding} = sharding;
const suffix = ProtoGraph._buildSmartSuffix(sharding, idx);
const suffix = ProtoGraph._buildSmartSuffix(sharding, idx, variant);

// All tests are based on fully connected graphs.
// So just place all vertices on the same shard, no matter what.
Expand All @@ -302,26 +313,26 @@ class ProtoGraph {

const eRel = sgm._relation(en, vn, vn);

return new TestGraph(gn, this.edges, eRel, vn, en, vertexSharding, TestVariants.DisjointSmartGraph, numberOfShards, this.unconnectedVertices);
return new TestGraph(gn, this.edges, eRel, vn, en, vertexSharding, variant, numberOfShards, this.unconnectedVertices);
});
}

prepareSatelliteGraphs() {
prepareSatelliteGraphs(variant = TestVariants.SatelliteGraph) {
// We're not able to test multiple shards in a SatelliteGraph as a SatelliteGraph has only one shard by default
const suffix = '_satellite';
const suffix = `_satellite_${variant}`;
const numberOfShards = 1;
const vn = this.protoGraphName + '_Vertex' + suffix;
const en = this.protoGraphName + '_Edge' + suffix;
const gn = this.protoGraphName + '_Graph' + suffix;

const eRel = cgm._relation(en, vn, vn);

return [new TestGraph(gn, this.edges, eRel, vn, en, [], TestVariants.SatelliteGraph, numberOfShards, this.unconnectedVertices)];
return [new TestGraph(gn, this.edges, eRel, vn, en, [], variant, numberOfShards, this.unconnectedVertices)];
}

static _buildSmartSuffix({numberOfShards, vertexSharding, name}, shardingIndex) {
static _buildSmartSuffix({numberOfShards, vertexSharding, name}, shardingIndex, variant) {
if (name) {
return `_${name}`;
return `_${name}_${variant}`;
}

// vertexSharding is an array of pairs, each pair holding a vertex (string)
Expand All @@ -340,7 +351,7 @@ class ProtoGraph {
suffix = `_shardingNr${shardingIndex}`;
}

return suffix;
return `${suffix}_${variant}`;
}


E C319 xpand Down
10 changes: 8 additions & 2 deletions tests/js/server/aql/aql-graph-traversal-generic-noncluster.js
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,12 @@ function graphTraversalGenericGeneralGraphStandaloneSuite() {
_.each(protoGraph.prepareSmartGraphs(TestVariants.SmartGraphSingleServer), function (testGraph) {
testGraphs[protoGraph.name()][`${testGraph.name()}-smart`] = testGraph;
});
_.each(protoGraph.prepareDisjointSmartGraphs(TestVariants.DisjointSmartGraphSingleServer), function (testGraph) {
testGraphs[protoGraph.name()][`${testGraph.name()}-disjointSmart`] = testGraph;
});
_.each(protoGraph.prepareSatelliteGraphs(TestVariants.SatelliteGraphSingleServer), function (testGraph) {
testGraphs[protoGraph.name()][`${testGraph.name()}-satellite`] = testGraph;
});
});

const suite = {
Expand Down Expand Up @@ -81,8 +87,8 @@ function graphTraversalGenericGeneralGraphStandaloneSuite() {
_.each(testsByGraph, function (localTests, graphName) {
let graphs = testGraphs[graphName];
_.each(localTests, function (test, testName) {
_.each(graphs, function (graph){
suite[testName + '_' + graph.name()] = function () {
_.each(graphs, function (graph, name){
suite[`${testName}_${name}`] = function () {
test(graph);
};
});
Expand Down
0