Feature/aql subquery execution block impl execute implementation #10606
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ExecutionBlockImpl (not complete, only skip path) everything compiles, but is not tested
…nd with variants everywhere.
…pose a DataRange, this might not be the correct one every where
…y and SingleRowFetcher. Both untested and yet incomplete, they will only work for MVP.
…SubqueryOperationsStack
…Rows function + test" This reverts commit 2368252.
…/arangodb/ArangoDB into feature/AqlSubqueryOperationsStack
…ough. This state is broken in the sense that a server does not boot with it and catch tests are broken
…/arangodb/ArangoDB into feature/AqlSubqueryOperationsStack
…e locally green, however there is at least one query still red (on ShadowRows)
This reverts commit aa275b4.
…/arangodb/ArangoDB into feature/AqlSubqueryOperationsStack
…re/AqlSubqueryExecutionBlockImplExecuteImplementation
…ndle ShadowRows properly
* Move ModificationExecutors to new interface * added inputRange method to inputMatrix class, added switch between DataRange Input <-> Matrix for ModificationExecutors * forgot to enable allrows modification executor in fastForwardType * removed obsolete comment * removed obsolete comment Co-authored-by: Heiko <hkernbach@users.noreply.github.com> Co-authored-by: Michael Hackstein <michael@arangodb.com>
* removed obsolete comment, added produceRows and skipRowsRange, enabled MaterializeExecutor as newStyleExecutor * forgot to use advanceRow, fetch input during skipRowsRange * removed old implementations * removed comment, fixed a logical loop error * skip could be simplified as well * temp workaround Co-authored-by: Michael Hackstein <michael@arangodb.com>
* Clarified comment * Updated comment * Added two static strings * Avoid unintentional streaming operator of string_view as a container * Added parser for execute rest call * Fixed ambiguous overload in ResultT * Added execute and executeForClient to ExecutionEngine * Implemented executeForClient in RestAqlHandler * Add default value * Updated comment * Changed error attribute * Added API toggle to RemoteExecutor * Fixed compile error * Fixed duplicate log id * Try to make MSVC work * Moved AqlExecuteResult to a separate file * Moved AqlExecuteResult to a separate file * Implemtented execute in Remote; except for (de)serialization, which is still missing * Tried to fix MSVC compile error * Revert "Tried to fix MSVC compile error" This reverts commit f6f43b2. * Next try to fix MSVC * Implemented (de)serialization for RemoteExecutor::execute * Fixes * Added minimal AqlCall (de)serialization tests * Bugfix * Added minimal (de)serialization tests for AqlCallStack and AqlExecuteResult * Fixed test * Changed Serialization format a little * Bugfix * Initialize SingletonBlock immediately with an empty row * Revert "Next try to fix MSVC" This reverts commit 528c4c7. * Revert "Try to make MSVC work" This reverts commit ba7d9c0. * Work around MSVC shortcoming * Work around MSVC shortcoming * Attempt to fix windows compile issue Co-authored-by: Michael Hackstein <michael@arangodb.com>
…uery executor (#11196) * Intermediate commit, does not compile. * New style for SubqueryExecutor. * Removed bad assert. * Reset subsqueryInitialised in skip. Co-authored-by: Michael Hackstein <michael@arangodb.com>
…t collect (#11172) * Move SubqueryStartExecutor to new interface * Diddle * Tests pass for SubqueryStartExecutor * Delete most of the SubqueryEndExecutor tests They are irrelevant now as they only tested shadowrow behaviour. Shadow Rows are now handled by ExecutionBlockImpl. * Skeleton tests * Junk * Cleanup and make work * Tweak ExecutorTestHelper to test pipelines * SubqueryStart/SubqueryEnd * Blafummel. * Modifications * Fixup tests * Address some comments * Fix. * Fix tests * Various fixes. * Made the branch compile on Mac * Intermediate commit, does not compile, required for merge * Fixed one of the SplicedSubqueryTests * Seperated ShadowRow handling for SubqueryStart/End executors to get a better overview and understanding on how it should behave. * Allow to use a fullCallStack in ExecutorTestHelper. * Added asserts on SubqueryEnd that is not allowed to be called with Skipped from upstream * Use AqlTestCase in SubqueryStartExecutorTest. Transformed first test to TestHelper pipeline * Let ExecutorTestHelper also test for shadowRows * Adapted SingleInput=>Data+Shadow test to testing Helepr and make it pass * Transformed existing SubqueryStart tests to Helper, fixed a test where there was no place for the ShadowRow to be written * Fixed Skip in SubqueryStartExecutor * Allow the Pipeline to add Consumer and dependency without fiddling with the Deque from outside * Added Skip adn ShadowRowForwarding tests to SubqueryStartExecutor * Fixed fullCount of SubqueryStart to be called properly. * Renamed isModificationSubquery -> isModificatioNode again * Consume more than one subquery en-block * Fixed debugReporting in SplicedSubqueryTest. Disabled the skip in subquery tests for now * Removed AQL debug log level from testing config * First version of Subquery Skip bypassing. Including 1 passing test. * Added disabled tests for skip, and proposed to do it later * Fixed reporting of FILTERED in fullCount of EnumerateCollectionExecutor * Enable calculationExecutor execute interface * Fixed compile issues * To Velocypack the modification Subquery value * Fixed illegal assert and reactivated failure tests * Fixed compile issue * Deactivated Calculation again * Added a testSuite for countCollectExecutor * Fixed Subquery Test for CountCollect * Added CountCollectImplementation * Added Compatibility Mode to AqlCallStack * Removed IdExecutor<void> the only feature it has can be moved into IDexecutor in general. This still has to be done * When falling back in BlocksWithClients use compatibility stack * Fixed EnumerateCollectionExecutor * Fixed fullCountBehaviour of EnumerateList. Also removed a unlimited,fullCount combination in IndexExecutor * Fixed endless loop while skipAl in OperationCursor * Fixed View Executor, which sends a no-limit no-offset upstream. * Fixed forwarding in NoResultsExecutor Co-authored-by: Markus Pfeiffer <markuspf@users.noreply.github.com>
… bugfixes (#11201) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode
…instead of cloneValueInto (#11213) * Fix memory leak in traversal * Use move instead of clone for AqlValues
…er executors (#11202) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded
* Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather Co-authored-by: Michael Hackstein <michael@arangodb.com>
* Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Fixed return state of GatherNode * Added an assertion before accessing a vectir unbounded * Fixed uninitialized member in DistributeExecutor * Fixed use before vector initialization in SortingGather * Fixed uninitialized dependencies in MultiDepRowFetcher * First step towards parallel Aql * Fixed an assertion * Fixed upstream skipping in ParallelUnsortedGather * [WIP] Changed Api for MultiDepExecutors in ExecBlockImpl (not yet in the actual executors) * Moved AqlCallSet into a separate file * Changed SortingGather to use the new API * Changed ParallelUnsortedGather to use the new API * Changed UnsortedGather to use the new API * Moved AqlCall operator<< into .cpp file * Implement operator<< for AqlCallSet * Fix boolean mix-up * Fixed state machine: go to UPSTREAM when the AqlCallSet is not empty * Fixed assertion * Bugfix * SortingGather bugfixes * Added init() method to fix an assertion and cleanup * Removed unused variable * Fixed constrained sort * Fixed constrained sort #2 * Fix boolean mix-up * Remove old interface * Use call parameter for upstream request in produceRows * Remove more old interface code * Add skip methods to MultiAqlItemBlockInputRange * Skip in UnsortedGather * skip for UnsortedGather * Fix skip and upstream calls for UnsortedGather * skipRowsRange change * Remove useless comments * Moved multi-dep related code from ExeBlockImpl to MultiFetcher * Cleanup in SortingGather, implemented parallel fullCount * Try to fix a windows compile error * Simplify and extend skipRowsRange for UnsortedGatherExecutor * Made ParallelUnsortedGather actually parallel * Removed erroneous assertion * Undid erroneous change * Fixed MacOs compile. Also disabled tests for non-relevant AqlCallStacks. They will be removed * Fixed initialize Cursor for multi dependency blocks * Fixed fullCount case in parallel unsorted gather * Fixed fullCount upstream call of ParallelUnsortedGatherExecutor * Fixed fullCount in SortingGather * Windows \o/ if you cannot work properly with constexpr and static asserts, we do not let you do it! * Do not advance in Unsorted gather if there are still rows to skip * Add more comparison operators for AqlCall limits * Send clientCall limits to upstream in SortingGather * Improved fullCount in SortingGatherExectur * Disabled a cluster profile test. We now ask the RemoteNode more often if it already has data. It is a bit unclear to me if this is now better performance wise (<< i think so) or triggers undesired side effects * Helpless attempt to work around issues in stonage Visual Studio Compiler we are using. * Clearly adding an operator on a well defined type causes ambigousness on random basic types using the operator Co-authored-by: Michael Hackstein <michael@arangodb.com> Co-authored-by: Markus Pfeiffer <markus@arangodb.com>
…ss skip (#11203) * Fixed range-handling for Modification Executors * DataRange handling in ModificationExecutor * Honor batch-size defined by UpstreamExecutor * Fixed compile issue * More fixes in modification * Remvoed log devel * Fixed profiler Test. for NoResults node we cahnge the behaviour * Activated getSome failure tests in ExecuteRestHandler * Fixed skipping in Index * Let the MultiDependencySingleROwFetcher return the correct states. * Fixed non-maintainer compilation * Attempt to fix windows compile issue * Fixed the non-maintainer compile ina different way * Added API in MultiAqlItemBlockInputRange to get Number of dependencies * Comments * Savepoint commit, does not compile, but no harm is done. Will start 57AE breaking things now * Another savepoint commit. does not compile, yet. * First draft of new Style SortingGather not yet implemented: Parallelism this needs to be handled in ExecutionBlockImpl now. * Allow waiting within old-style subquery * Fixed invalid skipRwos in unsorted gather * First draft of ParallelUnsortedGatherExecutor * Removed unused local variables * Added some Assertions in MultiAqlItemBlockInputRange * Initialize dependdencies of MultiDependencyFetcher * Fixed skipRows loop in UnsortingGatherNode * Added an implementation for a SkipResult, in order to simplify exchange of it. * Moved production API -> SkipResult * Made tests compile with new SkipResult * Added a test using skip and limit on subqueries * Prepared to use subqueries in SkipResult * Let subqueries modify the SkipResult subquery stack * Fixed return state of GatherNode * Activate all spliced subquery tests \o/ * Let SubqueryEnd honor the client request * Added a Maintainer only test for the stack, if it is 36 compatible * Added first part of side-effect executors. They now send upstream a fetch-all stack. Need to fix downstream reporting, and call the executor with hardLimit 0 * Add a fake FASTFORWARD call into a subquery-skipped ModificationExecutor. * Added helper shadow row function for SideEffect executors * Let the Modification Executor also produce data, even if no FullCount is used. * Revert "Let the Modification Executor also produce data, even if no FullCount is used." This reverts commit b1c6af5. * Revert "Revert "Let the Modification Executor also produce data, even if no FullCount is used."" This reverts commit ac94ae3. * Implemented proper fastForwarding and skipReporting in ExecutorsWithSideEffects. * Removed unreachable code, somehow the G++ in our tests tries to comile it and fails now. * noexcept -> throw is nono good. Thank you compiler for helping me here \o/ * Implment copy on SkipResult * Adapted SubqueryStartTest to allow easy testing for Skipping on outer levels. * Fixed koenig lookup of SkipResult ostream operator * Removed special case of SubqueryStartExecutor and include it on the handling for SideEffect Executors * Sorry needed to make the _operations vector mutual because of 3.6 compatibility * Attempt to fix windows compile issue * Fixed behvaiour of SubqueryEndExecutor * Another windows attempt * Fixed modify test, which would actually iterate over too many documents if the LIMIT does not LIMIT the executed modification operations anymore. * Fixed tests that assert on existence of SubqueryNode, now there will be SubqueryStartNode! * Consider a hardLimitFastForward inside the Callstack like a needToSkipSubquery. But do not report the skipping of it. * Fixed all tests that are related to subqueries, which now need to assert spliced queries. * Fixed jslint * Fixed the callstack that has been seperated from the clientCall. In some places it was not handled correctly. * Fixed skip result forwarding in Scatter/Gather * Fixed assertion if the ConstFetcher gets a block with subquery level skip injected * Moved merging of SubquerySkips in MultiDependencies into the Fetcher * Removed dead code and fixed overproduction of Rows in Subquery Executor * Fixed bypassing of skip in SideEffect executors if they trigger waiting at some point * Refactored old SubqueryExecutor, there has been some issue with WAITING within the cluster. * Removed debug logging in test * Fixed empty subquery executor * Added an assertion in the AqlResult that no invalid block tries to be serialized * Added clientId into profile tracing. Fixed return of invalid blocks in blocks with client * Removed invalid AqlExecuteResult from Test * Update tests/Aql/SubqueryStartExecutorTest.cpp Co-Authored-By: Tobias Gödderz <tobias@arangodb.com> * Fixed comment, thanks to reviewer Co-authored-by: Tobias Gödderz <tobias@arangodb.com>
* Add some FilterExecutor tests * Calculation executor tests * Some more testcode * Add NoneEntry for EntryBuilder * Add some tests for CalculationExecutor Co-authored-by: Michael Hackstein <michael@arangodb.com>
|
This branch has been created by multiple subbranches, which have been reviewed individually. |
…re/AqlSubqueryExecutionBlockImplExecuteImplementation
ObiWahn
added a commit
that referenced
this pull request
Mar 16, 2020
…-fix/validation-fixes-and-improvements * 'devel' of https://github.com/arangodb/arangodb: (25 commits) Do not instantiate snipped if not collection is found on the server. (#11281) Add entries related to search features in 3.7 fix bug (#11279) Docs: Add DocuBlocks for document validation. (#11228) Feature/ngram similarity function (#11276) Fixed production check, removed assertion (#11273) fix compile warning Cluster Metrics (#11234) Feature/satellite graphs (#11015) fix newly created supervision bug with incremental updates (#11269) remove useless std::cout output fix yet more compile warnings Implement memory detection override. (#11268) implement a global deadline when running testcode in the local arangosh (#11123) Encryption key rotation (#11080) fix compile warnings fix compile warnings Feature/aql subquery execution block impl execute implementation (#10606) missing metrics (#10625) Bug fix/supervision server cleanup (#11187) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First implementation to pass through the AqlCall stacks.
This branch has an issue with profiles, as we do overfetch now, if we do precise fetching, other profiler tests fail.
We need to discuss hardLimit propagation as this would avoid the overmatching issues.