8000 Feature/abort traversals by hkernbach · Pull Request #21765 · arangodb/arangodb · GitHub
[go: up one dir, main page]

Skip to content

Feature/abort traversals #21765

8000
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 34 commits into
base: devel
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
532d00b
arangodb release 3.12.4
KVS85 Jan 23, 2025
af6afa9
first draft, added cancel of traversal in one sided enumerator, and t…
hkernbach Apr 28, 2025
4cfc171
added bidirectionalCircle as new graph in graph test suite
hkernbach May 6, 2025
db7f093
added hugeCompleteGraph
hkernbach May 6, 2025
a03dc03
added query cancel observer to twosided enumerator as well, so kpaths…
hkernbach May 13, 2025
a250d68
merge main into feature branch
hkernbach May 14, 2025
7692de9
only print debug creation info if set manually in constructor
hkernbach May 14, 2025
7ef978f
revert changelog merge issue
hkernbach May 14, 2025
67f8176
added additional check for cancelation of query in the twosidedenumer…
hkernbach May 14, 2025
8e05284
Merge remote-tracking branch 'origin/devel' into feature/abort-traver…
hkernbach May 14, 2025
2de1512
eslint
hkernbach May 14, 2025
aa29f4e
eslint
hkernbach May 16, 2025
6f21c2f
clang format
hkernbach May 16, 2025
af166b8
clang format
hkernbach May 16, 2025
f35a566
Merge remote-tracking branch 'origin/devel' into feature/abort-traver…
hkernbach May 16, 2025
e22c7c3
batchwise verted and edge insertion during fillGraph (integration tests)
hkernbach May 19, 2025
48c95dc
Merge remote-tracking branch 'origin/devel' into feature/abort-traver…
hkernbach May 19, 2025
033326a
try to help garbage collection v8
hkernbach May 19, 2025
2b79028
slightly increased timeout to 10s in clsuter, 5s in single server
hkernbach May 19, 2025
7a04512
added early exit to weighted two sided enumerator, yens still missing
hkernbach May 19, 2025
cc7991d
debug flag on
hkernbach May 19, 2025
5c4cb55
no need to call clear manually. will be called in destructor automat…
hkernbach May 20, 2025
7ff7f4f
format
hkernbach May 20, 2025
75804b8
improved wait & kill mechanism - should now be way more stable
hkernbach Jul 10, 2025
0c94492
Merge remote-tracking branch 'origin/devel' into feature/abort-traver…
hkernbach Jul 10, 2025
83aae9d
adaptive kill query
hkernbach Jul 10, 2025
e7a1026
increase debug logging. Some are optional, some are mandatory. If our…
hkernbach Jul 10, 2025
31b64e0
eslint
hkernbach Jul 10, 2025
c36118d
more test output
hkernbach Jul 11, 2025
a192771
Change test suite params
jbajic Jul 11, 2025
3ebbdee
Merge remote-tracking branch 'origin/devel' into feature/abort-traver…
hkernbach Jul 14, 2025
030d3d4
refactored generic graph tests based on Jures suggestion
hkernbach Jul 14, 2025
45d34de
incr. completegraph to consist out of 1k nodes to make computation mo…
hkernbach Jul 14, 2025
4ec116f
Merge remote-tracking branch 'origin/devel' into feature/abort-traver…
hkernbach Jul 23, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
clang format
  • Loading branch information
hkernbach committed May 16, 2025
commit 6f21c2fdfa06ed01a503d7e4b9c86db74af205c7
13 changes: 5 additions & 8 deletions arangod/Aql/ExecutionNode/EnumeratePathsNode.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -483,8 +483,7 @@ std::unique_ptr<ExecutionBlock> EnumeratePathsNode::createBlock(
opts->tmpVar(), std::move(reversedUsedIndexes),
opts->getExpressionCtx(), {}, opts->collectionToShard(),
opts->getVertexProjections(), opts->getEdgeProjections(),
opts->produceVertices(), opts->useCache(),
opts->query());
opts->produceVertices(), opts->useCache(), opts->query());

using Provider = SingleServerProvider<SingleServerProviderStep>;
if (opts->query().queryOptions().getTraversalProfileLevel() ==
Expand Down Expand Up @@ -680,13 +679,11 @@ std::unique_ptr<ExecutionBlock> EnumeratePathsNode::createBlock(
} else { // Cluster case (on coordinator)
auto cache = std::make_shared<RefactoredClusterTraverserCache>(
opts->query().resourceMonitor());
ClusterBaseProviderOptions forwardProviderOptions(cache, engines(), false,
opts->produceVertices(),
opts->query());
ClusterBaseProviderOptions forwardProviderOptions(
cache, engines(), false, opts->produceVertices(), opts->query());
forwardProviderOptions.setClearEdgeCacheOnClear(false);
ClusterBaseProviderOptions backwardProviderOptions(cache, engines(), true,
opts->produceVertices(),
opts->query());
ClusterBaseProviderOptions backwardProviderOptions(
cache, engines(), true, opts->produceVertices(), opts->query());
backwardProviderOptions.setClearEdgeCacheOnClear(false);
// A comment is in order here: For all cases covered here
// (k-shortest-paths, all shortest paths, k-paths) we do not need to
Expand Down
10 changes: 4 additions & 6 deletions arangod/Aql/ExecutionNode/ShortestPathNode.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -511,12 +511,10 @@ std::unique_ptr<ExecutionBlock> ShortestPathNode::createBlock(
using ClusterProvider = ClusterProvider<ClusterProviderStep>;
auto cache = std::make_shared<RefactoredClusterTraverserCache>(
opts->query().resourceMonitor());
ClusterBaseProviderOptions forwardProviderOptions(cache, engines(), false,
opts->produceVertices(),
opts->query());
ClusterBaseProviderOptions backwardProviderOptions(cache, engines(), true,
opts->produceVertices(),
opts->query());
ClusterBaseProviderOptions forwardProviderOptions(
cache, engines(), false, opts->produceVertices(), opts->query());
ClusterBaseProviderOptions backwardProviderOptions(
cache, engines(), true, opts->produceVertices(), opts->query());

auto usesWeight =
checkWeight(forwardProviderOptions, backwardProviderOptions);
Expand Down
5 changes: 2 additions & 3 deletions arangod/Aql/ExecutionNode/TraversalNode.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -817,9 +817,8 @@ std::unique_ptr<ExecutionBlock> TraversalNode::createBlock(
bool isSmart) const {
TraverserOptions* opts = this->options();

arangodb::graph::OneSidedEnumeratorOptions options{opts->minDepth,
opts->maxDepth,
opts->query()};
arangodb::graph::OneSidedEnumeratorOptions options{
opts->minDepth, opts->maxDepth, opts->query()};
/*
* PathValidator Disjoint Helper (TODO [GraphRefactor]: Copy from createBlock)
* Clean this up as soon we clean up the whole TraversalNode as well.
Expand Down
3 changes: 2 additions & 1 deletion arangod/Graph/Enumerators/OneSidedEnumerator.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,8 @@ void OneSidedEnumerator<Configuration>::clearProvider() {
template<class Configuration>
void OneSidedEnumerator<Configuration>::computeNeighbourhoodOfNextVertex() {
if (_options.isKilled()) {
// Clear false may sounds misleading, but this means we do not want to keep the path store
// Clear false may sounds misleading, but this means we do not want to keep
// the path store
clear(false);
THROW_ARANGO_EXCEPTION(TRI_ERROR_QUERY_KILLED);
}
Expand Down
4 changes: 3 additions & 1 deletion arangod/Graph/Options/OneSidedEnumeratorOptions.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,9 @@
using namespace arangodb;
using namespace arangodb::graph;

OneSidedEnumeratorOptions::OneSidedEnumeratorOptions(size_t minDepth, size_t maxDepth, aql::QueryContext& query)
OneSidedEnumeratorOptions::OneSidedEnumeratorOptions(size_t minDepth,
size_t maxDepth,
aql::QueryContext& query)
: _minDepth(minDepth), _maxDepth(maxDepth), _observer(query) {}

OneSidedEnumeratorOptions::~OneSidedEnumeratorOptions() = default;
Expand Down
3 changes: 2 additions & 1 deletion arangod/Graph/Options/OneSidedEnumeratorOptions.h
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@ namespace arangodb::graph {

struct OneSidedEnumeratorOptions {
public:
OneSidedEnumeratorOptions(size_t minDepth, size_t maxDepth, aql::QueryContext& query);
OneSidedEnumeratorOptions(size_t minDepth, size_t maxDepth,
aql::QueryContext& query);
~OneSidedEnumeratorOptions();

[[nodiscard]] size_t getMinDepth() const noexcept;
Expand Down
18 changes: 9 additions & 9 deletions arangod/Graph/Options/QueryContextObserver.h
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,14 @@

#include "Aql/QueryContext.h"

// This class serves as a wrapper around QueryContext to explicitly track where query killing
// is being used in the graph traversal code. It provides a single point of access to check
// if a query has been killed, making it easier to maintain and modify the query killing
// behavior if needed.
// This class serves as a wrapper around QueryContext to explicitly track where
// query killing is being used in the graph traversal code. It provides a single
// point of access to check if a query has been killed, making it easier to
// maintain and modify the query killing behavior if needed.
//
// While this adds a small layer of indirection, it helps with code clarity and maintainability.
// If profiling shows this wrapper causes significant overhead, we can remove it and use
// QueryContext directly.
// While this adds a small layer of indirection, it helps with code clarity and
// maintainability. If profiling shows this wrapper causes significant overhead,
// we can remove it and use QueryContext directly.
//
// We can change this or discuss if this approach is not liked.

Expand All @@ -42,11 +42,11 @@ namespace arangodb::graph {
class QueryContextObserver {
public:
explicit QueryContextObserver(aql::QueryContext& query) : _query(query) {}

[[nodiscard]] bool isKilled() const { return _query.killed(); }

private:
aql::QueryContext& _query;
};

} // namespace arangodb::graph
} // namespace arangodb::graph
5 changes: 4 additions & 1 deletion arangod/Graph/Options/TwoSidedEnumeratorOptions.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,10 @@ TwoSidedEnumeratorOptions::TwoSidedEnumeratorOptions(size_t minDepth,
size_t maxDepth,
PathType::Type pathType,
aql::QueryContext& query)
: _minDepth(minDepth), _maxDepth(maxDepth), _pathType(pathType), _observer(query) {
: _minDepth(minDepth),
_maxDepth(maxDepth),
_pathType(pathType),
_observer(query) {
if (getPathType() == PathType::Type::AllShortestPaths) {
setStopAtFirstDepth(true);
} else if (getPathType() == PathType::Type::ShortestPath) {
Expand Down
6 changes: 2 additions & 4 deletions arangod/Graph/Providers/BaseProviderOptions.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,7 @@ SingleServerBaseProviderOptions::SingleServerBaseProviderOptions(
MonitoredCollectionToShardMap const& collectionToShardMap,
aql::Projections const& vertexProjections,
aql::Projections const& edgeProjections, bool produceVertices,
bool useCache,
aql::QueryContext& query)
bool useCache, aql::QueryContext& query)
: _temporaryVariable(tmpVar),
_indexInformation(std::move(indexInfo)),
_expressionContext(expressionContext),
Expand Down Expand Up @@ -171,8 +170,7 @@ void SingleServerBaseProviderOptions::unPrepareContext() {
ClusterBaseProviderOptions::ClusterBaseProviderOptions(
std::shared_ptr<RefactoredClusterTraverserCache> cache,
std::unordered_map<ServerID, aql::EngineId> const* engines, bool backward,
bool produceVertices,
aql::QueryContext& query)
bool produceVertices, aql::QueryContext& query)
: _cache(std::move(cache)),
_engines(engines),
_backward(backward),
Expand Down
17 changes: 6 additions & 11 deletions arangod/Graph/Providers/BaseProviderOptions.h
F436
Original file line number Diff line number Diff line change
Expand Up @@ -100,10 +100,10 @@ struct SingleServerBaseProviderOptions {
MonitoredCollectionToShardMap const& collectionToShardMap,
aql::Projections const& vertexProjections,
aql::Projections const& edgeProjections, bool produceVertices,
bool useCache,
aql::QueryContext& query);
bool useCache, aql::QueryContext& query);

SingleServerBaseProviderOptions(SingleServerBaseProviderOptions const&) = delete;
SingleServerBaseProviderOptions(SingleServerBaseProviderOptions const&) =
delete;
SingleServerBaseProviderOptions(SingleServerBaseProviderOptions&&) = default;

aql::Variable const* tmpVar() const;
Expand Down Expand Up @@ -133,9 +133,7 @@ struct SingleServerBaseProviderOptions {

aql::Projections const& getEdgeProjections() const;

bool isKilled() const noexcept {
return _queryObserver.isKilled();
}
bool isKilled() const noexcept { return _queryObserver.isKilled(); }

private:
// The temporary Variable used in the Indexes
Expand Down Expand Up @@ -188,8 +186,7 @@ struct ClusterBaseProviderOptions {
ClusterBaseProviderOptions(
std::shared_ptr<RefactoredClusterTraverserCache> cache,
std::unordered_map<ServerID, aql::EngineId> const* engines, bool backward,
bool produceVertices,
aql::QueryContext& query);
bool produceVertices, aql::QueryContext& query);

ClusterBaseProviderOptions(
std::shared_ptr<RefactoredClusterTraverserCache> cache,
Expand Down Expand Up @@ -236,9 +233,7 @@ struct ClusterBaseProviderOptions {
_clearEdgeCacheOnClear = flag;
}

bool isKilled() const noexcept {
return _queryObserver.isKilled();
}
bool isKilled() const noexcept { return _queryObserver.isKilled(); }

private:
std::shared_ptr<RefactoredClusterTraverserCache> _cache;
Expand Down
0