8000 Improve use of single shard restriction by dhly-etc · Pull Request #6182 · arangodb/arangodb · GitHub
[go: up one dir, main page]
Skip to content

Improve use of single shard restriction #6182

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 41 commits into from
Sep 14, 2018
Merged

Improve use of single shard restriction #6182

merged 41 commits into from
Sep 14, 2018

Conversation

dhly-etc
Copy link
Contributor

The optimizer should now pick up filters which restrict the search to a single shard, even if they are not part of an index node.

@dhly-etc dhly-etc added this to the devel milestone Aug 16, 2018
@dhly-etc dhly-etc requested a review from jsteemann August 16, 2018 17:38
@ghost ghost assigned dhly-etc Aug 16, 2018
@ghost ghost added the 2 - Working label Aug 16, 2018
@dhly-etc
Copy link
Contributor Author

http://jenkins01.arangodb.biz:8080/job/arangodb-matrix-pr-linux/1547/

jsteemann
jsteemann suggested changes Aug 16, 2018
View reviewed changes
arangod/Aql/ExecutionNode.cpp Outdated

Variable const* inputVariable = nullptr;
std::vector<Variable const*> v = getVariablesUsedHere();
if (v.size() > 1) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a FilterNode will always return exactly one variable from its getVariablesUsedHere function. We can add an assert for this here.

arangod/Aql/OptimizerRules.cpp Outdated
return "";
}

auto set = it->second;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will copy the entire std::unordered_map unnecessarily. Can we use auto const& here instead?

arangod/Aql/OptimizerRules.cpp Outdated
if (shardId.empty() && _shardsUsed[collection].empty()) {
_shardsUsed[collection].emplace("all");
} else if (!shardId.empty()) {
if (1 == _shardsUsed[collection].size() &&
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if this is safe like this.
What happens if we use the same collection twice in a query, e.g.

FOR doc1 IN collection
  FOR doc2 IN collection
    FILTER doc2.shardKey == @whatever
...

If this restricts the query to a single shard, it would be wrong. The reason is that the inner loop can be restricted to one shard, but the outer loop (for the same collection) must not. Is this somehow handled by the state machine?

arangod/Aql/OptimizerRules.cpp Outdated
break;
}

case EN::UPSERT: {
// track usage of the collection
auto collection = ExecutionNode::castTo<ModificationNode const*>(en)->collection();
auto collection =
ExecutionNode::castTo<ModificationNode const*>(en)->collection();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An UPSERT on a collection should previously have turned off the single-shard-optimization for a collection.
I think it has to stay like this, as we do not support single-shard-optimization for UPSERT queries yet AFAIK.

arangod/Aql/OptimizerRules.cpp Outdated
} else if (!shardId.empty()) {
if (1 == _shardsUsed[collection].size() &&
"all" == *_shardsUsed[collection].begin()) {
_shardsUsed[collection].clear();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens by the way, if we find a FILTER on a collection after an INSERT/UPDATE/REMOVE, e.g.

FOR doc IN collection
  UPDATE doc WITH ... IN collection
  FILTER doc.shardKey == @whatever
  RETURN doc

AFAIK this must do the updates on all shards, and then only return the documents from one shard. Is this covered?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a contrived example, however, we need to support all types of queries.

@dhly-etc dhly-etc modified the milestones: devel, 3.5 Aug 27, 2018
@dhly-etc
Copy link
Contributor Author

http://jenkins01.arangodb.biz:8080/job/arangodb-matrix-pr/879/

@dhly-etc
Copy link
Contributor Author
dhly-etc commented Aug 30, 2018 •
edited
Loading

http://jenkins01.arangodb.biz:8080/job/arangodb-matrix-pr/891/

@dhly-etc
Copy link
Contributor Author

http://jenkins01.arangodb.biz:8080/job/arangodb-matrix-pr/1060/

@dhly-etc
Copy link
Contributor Author

http://jenkins01.arangodb.biz:8080/job/arangodb-matrix-pr/1078/

graetzer
graetzer reviewed Sep 12, 2018
View reviewed changes
arangod/Aql/OptimizerRules.cpp
case EN::FILTER: {
auto node = ExecutionNode::castTo<arangodb::aql::FilterNode const*>(en);
auto v = node->getVariablesUsedHere();
TRI_ASSERT(v.size() == 1);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is this enforced ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A FilterNode always has a single input variable, which is normally produced by a CalculationNode.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But a FilterNode never has zero or more than one input variables, so the assertion will never fail.

@dhly-etc
Copy link
Contributor Author

Looks like the remaining failure in Jenkins is actually a Jenkins timeout. Some of the new tests run quite a while. I imagine I need to rewrite with shorter tests, right, @jsteemann?

@jsteemann
Copy link
Contributor

@danielhlarkin : if only I knew about what timeouts are in effect and how one can be sure tests failed due to a timeout...
I suggest we let it run again to see if it aborts reproducibly at that stage.

@dhly-etc
Copy link
Contributor Author

@jsteemann Oskar limits the PR tests to 30 minutes. The test was still running when Jenkins decided it had had enough time so it marked it failed. The tests do take quite a while to run, so I'm going to rewrite them shorter.

@jsteemann
Copy link
Contributor

@danielhlarkin : is there the possibility to reduce the time spent in setUp for the tests, e.g. by using batch inserts? Maybe we can also modify the AQL tests in oskar to use more buckets.

@dhly-etc
Copy link
Contributor Author

@jsteemann The tests were honestly doing much more work than necessary, so I'm just removing some of the iterations essentially. Should cut about 20 tests down from >1min to a few seconds each.

@jsteemann
Copy link
Contributor

Ok, sounds good.

@dhly-etc
Copy link
Contributor Author
dhly-etc commented Sep 13, 2018 •
edited by jsteemann
Loading

http://jenkins01.arangodb.biz:8080/job/arangodb-matrix-pr/1099/
(all blue)

@jsteemann jsteemann removed the 9 WIP label Sep 13, 2018
@jsteemann jsteemann merged commit 52e0fac into devel Sep 14, 2018
@ghost ghost removed the 2 - Working label Sep 14, 2018
ObiWahn added a commit that referenced this pull request Sep 17, 2018
@ObiWahn
Merge remote-tracking branch 'origin/devel' into feature/cpp17
1b9e580 
* origin/devel: (38 commits)
  Maintenance delayed by incomplete hashing maintenance actions (#6448)
  Do not hide indexes (#6507)
  Fix WebUI Login with read-only (#6500)
  Improve use of single shard restriction (#6182)
  make cid values in /_api/replication/logger-follow strings as advertised (#6499)
  less dynamic memory allocations on index operations (#6497)
  Fix several TSan warnings (#6473)
  Copy installation files on startup (#6491)
  nightly frontend build
  fix cluster selectivity estimates (#6488)
  this moves aql::Query* out of the call parameter to any aql::Function (#6485)
  Add advertised endpoints. (#6104)
  nightly frontend build
  fix leak in regex cache (#6480)
  Removing ClusterComm ClientTransactionID (#6294)
  pass variables by const reference
  remove unused variable
  remove unused variable
  fix init-order fiasco with static strings (#6475)
  minor issues (#6471)
  ...
ObiWahn added a commit that referenced this pull request Sep 17, 2018
@ObiWahn
Merge branch 'devel' of https://github.com/arangodb/arangodb into fea…
66d7d0e 
…ture/aql-blocks-by-line

* 'devel' of https://github.com/arangodb/arangodb:
  Maintenance delayed by incomplete hashing maintenance actions (#6448)
  Do not hide indexes (#6507)
  Fix WebUI Login with read-only (#6500)
  Improve use of single shard restriction (#6182)
  make cid values in /_api/replication/logger-follow strings as advertised (#6499)
  less dynamic memory allocations on index operations (#6497)
  Fix several TSan warnings (#6473)
  Copy installation files on startup (#6491)
  nightly frontend build
  fix cluster selectivity estimates (#6488)
  this moves aql::Query* out of the call parameter to any aql::Function (#6485)
  Add advertised endpoints. (#6104)
  nightly frontend build
  fix leak in regex cache (#6480)
  Removing ClusterComm ClientTransactionID (#6294)
  pass variables by const reference
  remove unused variable
  remove unused variable
  fix init-order fiasco with static strings (#6475)
dhly-etc pushed a commit that referenced this pull request Sep 20, 2018
Backport #6182 and #6549.
95af9a3
@fceller fceller deleted the feature/improve-use-of-single-shard-restriction branch October 12, 2018 10:43
ObiWahn added a commit that referenced this pull request Jan 10, 2019
@ObiWahn
Squashed commit of the following:
f847b76 
commit dd4b85fcaa634ed829e79e266ee547c0522e71f6
Merge: 9fc1548 fa7de56
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Thu Jan 10 10:21:26 2019 +0100

    Merge branch 'devel' into feature/cpp17

    * devel:
      upgrade to boost 1.69.0 (#7910)
      Doc - Add a paragraph that JS JS trxs are excluded from intermediate commits. (#7919)
      issue 511.7.3: restore single-document optimization, implement call to update data-store options, add IResearchView upgrade step (#7918)
      improve assertions (#7830)
      fix MSVC warning
      remove unused function
      fix #7900 - null bind values do not change to empty string anymore (#7917)

commit 9fc1548
Merge: f2ecee8 e7810e1
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Jan 9 09:36:19 2019 +0100

    Merge remote-tracking branch 'origin/devel' into feature/cpp17

    * origin/devel:
      Doc - Update remark about per-database replication (#7891)
      Doc - Known Issues - new links (#7908)
      support db._explain with all plans (#7895)
      JWT base64url encoding (#7899)
      cppcheck/Scheduler (#7909)
      multiplex REPLICATION-APPLIER-STATE files for RocksDB engine (#7898)
      fix compile warnings
      upgrade bundled curl library to version 7.63 (#7846)
      Feature/new server infra (#7733)
      Doc - check_function (#7885)
      simran (#7881)
      added AQL function CHECK_DOCUMENT (#7842)
      added arangorestore option `--cleanup-duplicate-attributes` (#7877)
      Doc - arangorestore improvements (#7878)
      added arangorestore options `--number-of-shards` and `--replication-f… (#7869)

commit f2ecee8
Merge: 44adf97 5c53ebc
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Thu Jan 3 10:51:20 2019 +0100

    Merge remote-tracking branch 'origin/devel' into feature/cpp17

    * origin/devel: (40 commits)
      Feature/jwt keyfile (#7863)
      prevent duplicate attributes being generated by AQL queries (#7837)
      fixed issue #7834 (#7843)
      Bug fix/issues 23122018 (#7847)
      Documents that arangorestore/arangodump threads option is from v3.4.0 on (#7701)
      Update ISSUE_TEMPLATE.md (#7103)
      Doc - Forwardports to devel - 2018-12-27 (#7858)
      Doc - Sync-external-repo-2018-12-25 (#7854)
      Doc - re-add switching storage engine section (#7781)
      Doc - Fix build (whitespaces, triple-backticks, comments) (#7857)
      Add http_server test for creating view with links (#7852)
      big reformat
      added new clang style and reformat script for clang-format 6.0
      Doc - Improve MacOS Installation page (#7744)
      Fix issue with geo iterator reset. (#7839)
      Doc - Single Instance vs. Cluster (diff + migration) (#7739)
      Rdb index background (preliminary) (#7644)
      iResearch asan issue (#7832)
      Feature/improve edgeindex covered (#7718)
      issue 511.6: update iresearch to 6e870904eae5eaf891359ed06e7332666c02cb19 (#7819)
      ...

commit 44adf97
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Tue Dec 18 13:39:03 2018 +0100

    Add COMPILER_* variables to VERSIONS file.

    The variables will be used to select a fitting compiler in the CI.
    Developers need to ensure themselves that their development environment
    supports the c++ standard.

commit 8124eb3
Merge: 127bb7d 4423125
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Tue Dec 18 11:09:42 2018 +0100

    Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/cpp17

    * 'devel' of https://github.com/arangodb/arangodb: (381 commits)
      tools are now looking into the build to find the file (#7128)
      issue 511.3: remove IResearchViewDBServer and use the IResearchView directly on db-server (#7748)
      Fix S2 for aarch64. (#7770)
      equalising devel and 3.4 in agency/cluster (#7755)
      wording
      startLocalCluster in rr debugging mode cannot be started with --console on agents / db servers (#7776)
      fix cmake parameters coincidence (#7774)
      Fix a rare deadlock situation in Replication sync phase (#7759)
      add virtual dtor
      wait for procdump to exit too. (#7731)
      fix invalid handling of `_lastValue` in case of multiple coordinators (#7734)
      fixed item 3 of issue #7009 (#7747)
      updated 3rdparty licenses
      remove inclusion of valgrind.h
      remove now-unused file
      Bug fix/fix thread shutdown (#7728)
      Doc - fix build by adding missing SUMMARY entries (#7741)
      issue 511.2.1: use references instead of raw pointers for Index operations to avoid null pointer access (#7725)
      port 3.4 change to stopBackgroundThreads.  Move ClusterCommThread routines to same place in source file. (#7721)
      Ignore invaild geo coordinates when indexing. (#7724)
      ...

commit 127bb7d
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Mon Oct 15 22:06:27 2018 +0200

    fix invoke for c++17

commit 65a514c
Author: Frank Celler <frank@arangodb.com>
Date:   Mon Oct 15 18:58:31 2018 +0200

    trying to use 17

commit 8344d96
Merge: 2822dae c7933bc
Author: Frank Celler <frank@arangodb.com>
Date:   Mon Oct 15 18:08:52 2018 +0200

    Merge branch 'devel' of ssh://github.com/arangodb/ArangoDB into feature/cpp17

commit 2822dae
Merge: 4001d11 af2f18f
Author: Frank Celler <frank@arangodb.com>
Date:   Fri Oct 12 17:44:04 2018 +0200

    Merge branch 'feature/boost-168' of ssh://github.com/arangodb/ArangoDB into feature/cpp17

commit 4001d11
Author: Frank Celler <frank@arangodb.com>
Date:   Fri Oct 12 17:43:46 2018 +0200

    fixed merge conflict

commit 94cb4b7
Merge: cbc2f34 d3c4461
Author: Fran
EED3
k Celler <frank@arangodb.com>
Date:   Fri Oct 12 14:35:42 2018 +0200

    Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/cpp17

commit af2f18f
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Fri Sep 28 14:44:16 2018 +0200

    add bcrypt.lib to make uuid work

commit 11efd12
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 12:31:25 2018 +0200

    fix boost path for iresearch that is not able to use the provided
    information

commit 57ed116
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 10:24:10 2018 +0200

    add missing boost core includes

commit 5d52b32
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 09:22:43 2018 +0200

    upgrade boost to version 1.68.0

commit cbc2f34
Merge: 3de8d48 1b11426
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 12:08:13 2018 +0200

    Merge branch 'feature/cpp17' of https://github.com/arangodb/arangodb into feature/cpp17

    * 'feature/cpp17' of https://github.com/arangodb/arangodb:
      allow MSVC to user autopointer with c++17
      nightly frontend build
      Remove unused variable check from arangodump. (#6529)
      fixed JavaScript error
      added a reserve call
      Bug fix/simplify things (#6516)
      catch 501 (not implemented) in the query editor (ui) (#6523)
      return error codes in case a babies remove operation fails (#6527)
      Use physical collection directly (#6526)
      Feature/aql item block compression (#6514)
      move tests to the right location
      take over selectivity estimates (#6505)
      Properly check syncer erros, catch more exceptions (#6520)
      prevent out-of-bounds array access
      Fix deduplication in IndexBlock (#6508)

commit 3de8d48
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 12:05:56 2018 +0200

    add autopointer and tr1

commit 1b11426
Merge: 4b38f0a 0dd277d
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 10:06:01 2018 +0200

    Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/cpp17

    * 'devel' of https://github.com/arangodb/arangodb:
      nightly frontend build
      Remove unused variable check from arangodump. (#6529)
      fixed JavaScript error
      added a reserve call
      Bug fix/simplify things (#6516)
      catch 501 (not implemented) in the query editor (ui) (#6523)
      return error codes in case a babies remove operation fails (#6527)
      Use physical collection directly (#6526)
      Feature/aql item block compression (#6514)
      move tests to the right location
      take over selectivity estimates (#6505)
      Properly check syncer erros, catch more exceptions (#6520)
      prevent out-of-bounds array access
      Fix deduplication in IndexBlock (#6508)

commit 4b38f0a
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 10:04:46 2018 +0200

    allow MSVC to user autopointer with c++17

commit 9c91023
Merge: 1b9e580 e51359b
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Mon Sep 17 12:53:03 2018 +0200

    Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/cpp17

    * 'devel' of https://github.com/arangodb/arangodb:
      port startup fix
      Bugfix/early out invalid links in view creation (#6502)

commit 1b9e580
Merge: b08bc2e 8bd834b
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Mon Sep 17 06:38:24 2018 +0200

    Merge remote-tracking branch 'origin/devel' into feature/cpp17

    * origin/devel: (38 commits)
      Maintenance delayed by incomplete hashing maintenance actions (#6448)
      Do not hide indexes (#6507)
      Fix WebUI Login with read-only (#6500)
      Improve use of single shard restriction (#6182)
      make cid values in /_api/replication/logger-follow strings as advertised (#6499)
      less dynamic memory allocations on index operations (#6497)
      Fix several TSan warnings (#6473)
      Copy installation files on startup (#6491)
      nightly frontend build
      fix cluster selectivity estimates (#6488)
      this moves aql::Query* out of the call parameter to any aql::Function (#6485)
      Add advertised endpoints. (#6104)
      nightly frontend build
      fix leak in regex cache (#6480)
      Removing ClusterComm ClientTransactionID (#6294)
      pass variables by const reference
      remove unused variable
      remove unused variable
      fix init-order fiasco with static strings (#6475)
      minor issues (#6471)
      ...

commit b08bc2e
Merge: 03fabcc 2eb3c2d
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Tue Sep 11 12:46:51 2018 +0200

    Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/cpp17

    * 'devel' of https://github.com/arangodb/arangodb:
      temporarily increase log verbosity for replication log topic
      Fix skipSome PR (#6455)
      bump version number to 3.5 (#6454)
      Feature/new devel versioning scheme (#6312)
      Bug fix/aql modify cleanup (#6380)
      Trace skipSome() calls (#6449)
      nightly frontend build
      Fix ArangoSearch authentication update task test (#6453)
      Forward port changelog changes and typo fixes from #6303 (#6409)
      fix some issues with sorted variant of COLLECT (#6433)
      added tests, updated docu (#6407)
      now that we are on c++14 and std::make_unique is available everywhere, we can safely remove our shim for std::make_unique (#6429)
      Fixed memory leak due to cyclic references of shared pointer. (#6442)
      do not create empty useless transaction data objects (#6430)
      add some debugging for unstable replication_sync test (#6436)
      Address leak sanitizer warnings (#6275)
      Fix Pregel Graph Loading Logic (#6419)
      remove unused variables
      fix profile output for nodes without calls to getSome (#6397)

commit 03fabcc
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Tue Sep 11 07:19:21 2018 +0200

    switch more libs to c++17

commit 3359420
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Mon Sep 10 07:31:49 2018 +0200

    fix static asserts in iresearch

commit 90eab95
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Mon Sep 10 07:10:48 2018 +0200

    move from c++14 to c++17
KVS85 pushed a commit that referenced this pull request Oct 16, 2019
@ObiWahn @jsteemann @KVS85
C++17 (#10178)
b16a5e6 
* Squashed commit of the following:

commit dd4b85fcaa634ed829e79e266ee547c0522e71f6
Merge: 9fc1548 fa7de56
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Thu Jan 10 10:21:26 2019 +0100

    Merge branch 'devel' into feature/cpp17

    * devel:
      upgrade to boost 1.69.0 (#7910)
      Doc - Add a paragraph that JS JS trxs are excluded from intermediate commits. (#7919)
      issue 511.7.3: restore single-document optimization, implement call to update data-store options, add IResearchView upgrade step (#7918)
      improve assertions (#7830)
      fix MSVC warning
      remove unused function
      fix #7900 - null bind values do not change to empty string anymore (#7917)

commit 9fc1548
Merge: f2ecee8 e7810e1
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Jan 9 09:36:19 2019 +0100

    Merge remote-tracking branch 'origin/devel' into feature/cpp17

    * origin/devel:
      Doc - Update remark about per-database replication (#7891)
      Doc - Known Issues - new links (#7908)
      support db._explain with all plans (#7895)
      JWT base64url encoding (#7899)
      cppcheck/Scheduler (#7909)
      multiplex REPLICATION-APPLIER-STATE files for RocksDB engine (#7898)
      fix compile warnings
      upgrade bundled curl library to version 7.63 (#7846)
      Feature/new server infra (#7733)
      Doc - check_function (#7885)
      simran (#7881)
      added AQL function CHECK_DOCUMENT (#7842)
      added arangorestore option `--cleanup-duplicate-attributes` (#7877)
      Doc - arangorestore improvements (#7878)
      added arangorestore options `--number-of-shards` and `--replication-f… (#7869)

commit f2ecee8
Merge: 44adf97 5c53ebc
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Thu Jan 3 10:51:20 2019 +0100

    Merge remote-tracking branch 'origin/devel' into feature/cpp17

    * origin/devel: (40 commits)
      Feature/jwt keyfile (#7863)
      prevent duplicate attributes being generated by AQL queries (#7837)
      fixed issue #7834 (#7843)
      Bug fix/issues 23122018 (#7847)
      Documents that arangorestore/arangodump threads option is from v3.4.0 on (#7701)
      Update ISSUE_TEMPLATE.md (#7103)
      Doc - Forwardports to devel - 2018-12-27 (#7858)
      Doc - Sync-external-repo-2018-12-25 (#7854)
      Doc - re-add switching storage engine section (#7781)
      Doc - Fix build (whitespaces, triple-backticks, comments) (#7857)
      Add http_server test for creating view with links (#7852)
      big reformat
      added new clang style and reformat script for clang-format 6.0
      Doc - Improve MacOS Installation page (#7744)
      Fix issue with geo iterator reset. (#7839)
      Doc - Single Instance vs. Cluster (diff + migration) (#7739)
      Rdb index background (preliminary) (#7644)
      iResearch asan issue (#7832)
      Feature/improve edgeindex covered (#7718)
      issue 511.6: update iresearch to 6e870904eae5eaf891359ed06e7332666c02cb19 (#7819)
      ...

commit 44adf97
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Tue Dec 18 13:39:03 2018 +0100

    Add COMPILER_* variables to VERSIONS file.

    The variables will be used to select a fitting compiler in the CI.
    Developers need to ensure themselves that their development environment
    supports the c++ standard.

commit 8124eb3
Merge: 127bb7d 4423125
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Tue Dec 18 11:09:42 2018 +0100

    Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/cpp17

    * 'devel' of https://github.com/arangodb/arangodb: (381 commits)
      tools are now looking into the build to find the file (#7128)
      issue 511.3: remove IResearchViewDBServer and use the IResearchView directly on db-server (#7748)
      Fix S2 for aarch64. (#7770)
      equalising devel and 3.4 in agency/cluster (#7755)
      wording
      startLocalCluster in rr debugging mode cannot be started with --console on agents / db servers (#7776)
      fix cmake parameters coincidence (#7774)
      Fix a rare deadlock situation in Replication sync phase (#7759)
      add virtual dtor
      wait for procdump to exit too. (#7731)
      fix invalid handling of `_lastValue` in case of multiple coordinators (#7734)
      fixed item 3 of issue #7009 (#7747)
      updated 3rdparty licenses
      remove inclusion of valgrind.h
      remove now-unused file
      Bug fix/fix thread shutdown (#7728)
      Doc - fix build by adding missing SUMMARY entries (#7741)
      issue 511.2.1: use references instead of raw pointers for Index operations to avoid null pointer access (#7725)
      port 3.4 change to stopBackgroundThreads.  Move ClusterCommThread routines to same place in source file. (#7721)
      Ignore invaild geo coordinates when indexing. (#7724)
      ...

commit 127bb7d
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Mon Oct 15 22:06:27 2018 +0200

    fix invoke for c++17

commit 65a514c
Author: Frank Celler <frank@arangodb.com>
Date:   Mon Oct 15 18:58:31 2018 +0200

    trying to use 17

commit 8344d96
Merge: 2822dae c7933bc
Author: Frank Celler <frank@arangodb.com>
Date:   Mon Oct 15 18:08:52 2018 +0200

    Merge branch 'devel' of ssh://github.com/arangodb/ArangoDB into feature/cpp17

commit 2822dae
Merge: 4001d11 af2f18f
Author: Frank Celler <frank@arangodb.com>
Date:   Fri Oct 12 17:44:04 2018 +0200

    Merge branch 'feature/boost-168' of ssh://github.com/arangodb/ArangoDB into feature/cpp17

commit 4001d11
Author: Frank Celler <frank@arangodb.com>
Date:   Fri Oct 12 17:43:46 2018 +0200

    fixed merge conflict

commit 94cb4b7
Merge: cbc2f34 d3c4461
Author: Frank Celler <frank@arangodb.com>
Date:   Fri Oct 12 14:35:42 2018 +0200

    Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/cpp17

commit af2f18f
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Fri Sep 28 14:44:16 2018 +0200

    add bcrypt.lib to make uuid work

commit 11efd12
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 12:31:25 2018 +0200

    fix boost path for iresearch that is not able to use the provided
    information

commit 57ed116
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 10:24:10 2018 +0200

    add missing boost core includes

commit 5d52b32
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 09:22:43 2018 +0200

    upgrade boost to version 1.68.0

commit cbc2f34
Merge: 3de8d48 1b11426
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 12:08:13 2018 +0200

    Merge branch 'feature/cpp17' of https://github.com/arangodb/arangodb into feature/cpp17

    * 'feature/cpp17' of https://github.com/arangodb/arangodb:
      allow MSVC to user autopointer with c++17
      nightly frontend build
      Remove unused variable check from arangodump. (#6529)
      fixed JavaScript error
      added a reserve call
      Bug fix/simplify things (#6516)
      catch 501 (not implemented) in the query editor (ui) (#6523)
      return error codes in case a babies remove operation fails (#6527)
      Use physical collection directly (#6526)
      Feature/aql item block compression (#6514)
      move tests to the right location
      take over selectivity estimates (#6505)
      Properly check syncer erros, catch more exceptions (#6520)
      prevent out-of-bounds array access
      Fix deduplication in IndexBlock (#6508)

commit 3de8d48
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 12:05:56 2018 +0200

    add autopointer and tr1

commit 1b11426
Merge: 4b38f0a 0dd277d
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 10:06:01 2018 +0200

    Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/cpp17

    * 'devel' of https://github.com/arangodb/arangodb:
      nightly frontend build
      Remove unused variable check from arangodump. (#6529)
      fixed JavaScript error
      added a reserve call
      Bug fix/simplify things (#6516)
      catch 501 (not implemented) in the query editor (ui) (#6523)
      return error codes in case a babies remove operation fails (#6527)
      Use physical collection directly (#6526)
      Feature/aql item block compression (#6514)
      move tests to the right location
      take over selectivity estimates (#6505)
      Properly check syncer erros, catch more exceptions (#6520)
      prevent out-of-bounds array access
      Fix deduplication in IndexBlock (#6508)

commit 4b38f0a
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Wed Sep 19 10:04:46 2018 +0200

    allow MSVC to user autopointer with c++17

commit 9c91023
Merge: 1b9e580 e51359b
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Mon Sep 17 12:53:03 2018 +0200

    Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/cpp17

    * 'devel' of https://github.com/arangodb/arangodb:
      port startup fix
      Bugfix/early out invalid links in view creation (#6502)

commit 1b9e580
Merge: b08bc2e 8bd834b
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Mon Sep 17 06:38:24 2018 +0200

    Merge remote-tracking branch 'origin/devel' into feature/cpp17

    * origin/devel: (38 commits)
      Maintenance delayed by incomplete hashing maintenance actions (#6448)
      Do not hide indexes (#6507)
      Fix WebUI Login with read-only (#6500)
      Improve use of single shard restriction (#6182)
      make cid values in /_api/replication/logger-follow strings as advertised (#6499)
      less dynamic memory allocations on index operations (#6497)
      Fix several TSan warnings (#6473)
      Copy installation files on startup (#6491)
      nightly frontend build
      fix cluster selectivity estimates (#6488)
      this moves aql::Query* out of the call parameter to any aql::Function (#6485)
      Add advertised endpoints. (#6104)
      nightly frontend build
      fix leak in regex cache (#6480)
      Removing ClusterComm ClientTransactionID (#6294)
      pass variables by const reference
      remove unused variable
      remove unused variable
      fix init-order fiasco with static strings (#6475)
      minor issues (#6471)
      ...

commit b08bc2e
Merge: 03fabcc 2eb3c2d
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Tue Sep 11 12:46:51 2018 +0200

    Merge branch 'devel' of https://github.com/arangodb/arangodb into feature/cpp17

    * 'devel' of https://github.com/arangodb/arangodb:
      temporarily increase log verbosity for replication log topic
      Fix skipSome PR (#6455)
      bump version number to 3.5 (#6454)
      Feature/new devel versioning scheme (#6312)
      Bug fix/aql modify cleanup (#6380)
      Trace skipSome() calls (#6449)
      nightly frontend build
      Fix ArangoSearch authentication update task test (#6453)
      Forward port changelog changes and typo fixes from #6303 (#6409)
      fix some issues with sorted variant of COLLECT (#6433)
      added tests, updated docu (#6407)
      now that we are on c++14 and std::make_unique is available everywhere, we can safely remove our shim for std::make_unique (#6429)
      Fixed memory leak due to cyclic references of shared pointer. (#6442)
      do not create empty useless transaction data objects (#6430)
      add some debugging for unstable replication_sync test (#6436)
      Address leak sanitizer warnings (#6275)
      Fix Pregel Graph Loading Logic (#6419)
      remove unused variables
      fix profile output for nodes without calls to getSome (#6397)

commit 03fabcc
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Tue Sep 11 07:19:21 2018 +0200

    switch more libs to c++17

commit 3359420
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Mon Sep 10 07:31:49 2018 +0200

    fix static asserts in iresearch

commit 90eab95
Author: Jan Christoph Uhde <Jan@UhdeJC.com>
Date:   Mon Sep 10 07:10:48 2018 +0200

    move from c++14 to c++17

* specify required clang and gcc versions

* fix openssl 1.3 usage

* change random_shuffle to shuffle as the former has been removed from the c++ standard

https://meetingcpp.com/blog/items/stdrandom_shuffle-is-deprecated.html

* disable uncaught_exceptions in catch2 for darwin

In file included from /Users/jenkins/Git/cpp17/tests/main.cpp:2:
/Users/jenkins/Git/cpp17/3rdParty/catch/catch.hpp:10877:21: error: 'uncaught_exceptions' is unavailable: introduced in macOS 10.12
        return std::uncaught_exceptions() > 0;
                    ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/exception:130:63: note: 'uncaught_exceptions' has been explicitly marked unavailable here
_LIBCPP_FUNC_VIS _LIBCPP_AVAILABILITY_UNCAUGHT_EXCEPTIONS int uncaught_exceptions() _NOEXCEPT;
                                                              ^

* boost locale now depends on boost thread

* add fix for VS2019

* try to fix build for windows

* add some change required by older macos

* add miserable hack to circumvent std::stack implementation problem on older Xcode

* try to fix windows build

* remove unary function as in v8 upstream

* clean up CMakeLists.txt

* Remove emplace hack that was introduced for OSX

* Revert "Remove emplace hack that was introduced for OSX"

This reverts commit 2b0a4cd.

* Remove emplace hack that was introduced on OSX to su
7973
pport C++17 (#10184)

* Revert "Revert "Remove emplace hack that was introduced for OSX""

This reverts commit 7df9edd.

* fixed MACOSX_DEPLOYMENT_TARGET

* Fix quotes usage in conditions

* Added MACOS_MIN

* Adjust variables

* Fix compilation

* remove iresearch patches

* Update CMakeLists.txt

Co-Authored-By: Jan <jsteemann@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@graetzer graetzer graetzer left review comments

@jsteemann jsteemann jsteemann requested changes

Assignees

@dhly-etc dhly-etc

Labels
None yet
Projects
None yet
Milestone
3.5
Development

Successfully merging this pull request may close these issues.
3 participants
@dhly-etc @graetzer @jsteemann
0