8000 Merge branch 'devel' of https://github.com/arangodb/arangodb into 3.4 · mnemosdev/arangodb@d160c98 · GitHub
[go: up one dir, main page]
Skip to content

Commit d160c98

Browse files
fcellerfceller
committed
Merge branch 'devel' of https://github.com/arangodb/arangodb into 3.4
2 parents d7a450e + a669d5a commit d160c98

File tree

677 files changed

+1158
-671
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

677 files changed

+1158
-671
lines changed

‎3rdParty/fuerte/src/HttpConnection.cpp

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,11 @@ using namespace arangodb::fuerte::v1;
3939
using namespace arangodb::fuerte::v1::http;
4040

4141
int on_message_began(http_parser* parser) { return 0; }
42-
int on_status(http_parser* parser, const char* at, size_t len) { return 0; }
42+
int on_status(http_parser* parser, const char* at, size_t len) {
43+
RequestItem* data = static_cast<RequestItem*>(parser->data);
44+
data->_response->header.meta.emplace(std::string("http/") + std::to_string(parser->http_major) + '.' + std::to_string(parser->http_minor), std::string(at, len));
45+
return 0;
46+
}
4347
int on_header_field(http_parser* parser, const char* at, size_t len) {
4448
RequestItem* data = static_cast<RequestItem*>(parser->data);
4549
if (data->last_header_was_a_value) {

‎CHANGELOG

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,12 @@ devel
1515
on path (vertices and edges) have not been applied. In SmartGraphs the checks
1616
have been executed properly.
1717

18+
* added option `--rocksdb.use-file-logging` to enable writing of RocksDB's own
19+
informational LOG files into RocksDB's database directory.
20+
21+
This option is turned off by default, but can be enabled for debugging RocksDB
22+
internals and performance.
23+
1824
* improved error messages when managing Foxx services
1925

2026
Install/replace/upgrade will now provide additional information when an error

‎Documentation/Books/AQL/Functions/Document.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -317,7 +317,7 @@ PARSE_IDENTIFIER()
317317
`PARSE_IDENTIFIER(documentHandle) → parts`
318318

319319
Parse a [document handle](../../Manual/Appendix/Glossary.html#document-handle) and return its
320-
individual parts a separate attributes.
320+
individual parts as separate attributes.
321321

322322
This function can be used to easily determine the
323323
[collection name](../../Manual/Appendix/Glossary.html#collection-name) and key of a given document.

‎Documentation/Books/Manual/Administration/Configuration/RocksDB.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -240,3 +240,20 @@ have been executed without the *waitForSync* attribute.
240240

241241
Note: this option is not supported on Windows platforms. Setting the option to
242242
a value greater 0 will produce a startup warning.
243+
244+
`--rocksdb.use-file-logging`
245+
246+
When set to *true*, enables writing of RocksDB's own informational LOG files into
247+
RocksDB's database directory.
248+
249+
This option is turned off by default, but can be enabled for debugging RocksDB
250+
internals and performance.
251+
252+
`--rocksdb.debug-logging`
253+
254+
When set to *true*, enables verbose logging of RocksDB's actions into the logfile
255+
written by ArangoDB (if option `--rocksdb.use-file-logging` is off) or RocksDB's
256+
own log (if option `--rocksdb.use-file-logging` is on).
257+
258+
This option is turned off by default, but can be enabled for debugging RocksDB
259+
internals and performance.

‎Documentation/Books/Manual/ReleaseNotes/KnownIssues34.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,10 @@ APIs
1818
* the REST API for retrieving indexes at endpoint GET `/_api/index/?collection=<collection>` will
1919
currently also return all links (type `arangosearch`) for views that refer to this collection. The links
2020
will be removed from the results of this API in a later version.
21-
* the REST API for retrieving a single index at endpoint GET `/_api/index/<indexname>` will current
22-
succeed for indexes used internally for links of views. These requests will return a notfound error in
23-
a later version.
21+
22+
* the REST API for retrieving a single index at endpoint GET `/_api/index/<indexname>` will
23+
currently succeed for indexes used internally for links of views. These requests will
24+
return a "not found" error in a later version.
2425

2526
Cluster
2627
-------

‎Documentation/Books/Manual/ReleaseNotes/NewFeatures34.md

Lines changed: 12 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -868,8 +868,8 @@ However, streaming cursors are enabled for the following parts of ArangoDB in 3.
868868
Native implementations
869869
----------------------
870870

871-
The following internal and external functionality has been ported from JavaScript-based
872-
implementations to C++-based implementations in ArangoDB 3.4:
871+
The following internal and user-facing functionality has been ported from
872+
JavaScript-based implementations to C++-based implementations in ArangoDB 3.4:
873873

874874
* the statistics gathering background thread
875875
* the REST APIs for
@@ -879,24 +879,23 @@ implementations to C++-based implementations in ArangoDB 3.4:
879879
- edge management
880880
* the implementations of all built-in AQL functions
881881
* all other parts of AQL except user-defined functions
882+
* database creation and setup
882883
* all the DBserver internal maintenance tasks for shard creation, index
883884
creation and the like in the cluster
884885

885-
By making the listed functionality not use and depend on the V8 JavaScript engine,
886-
the respective functionality can now be invoked more efficiently, without requiring
887-
the conversion of data between ArangoDB's native format and V8's
888-
internal format. For the maintenance operations this will lead to
886+
By making the listed functionality not use and not depend on the V8 JavaScript
887+
engine, the respective functionality can now be invoked more efficiently in the
888+
server, without requiring the conversion of data between ArangoDB's native format
889+
and V8's internal formats. For the maintenance operations this will lead to
889890
improved stability in the cluster.
890891

891-
As less functionality depends on the V8 JavaScript engine, an ArangoDB 3.4 server
892-
will not require as many V8 contexts as previous versions.
893-
This should reduce problems with servers running out of available V8 contexts or
894-
using a lot of memory just for keeping V8 contexts around.
895-
896892
As a consequence, ArangoDB agency and database server nodes in an ArangoDB 3.4
897893
cluster will now turn off the V8 JavaScript engine at startup entirely and automatically.
898-
The V8 engine will still be enabled on cluster coordinators, single server and
899-
active failover instances.
894+
The V8 engine will still be enabled on cluster coordinators, single servers and
895+
active failover instances. But even the latter instance types will not require as
896+
many V8 contexts as previous versions of ArangoDB.
897+
This should reduce problems with servers running out of available V8 contexts or
898+
using a lot of memory just for keeping V8 contexts around.
900899

901900

902901
Foxx

‎Documentation/Books/Manual/ReleaseNotes/UpgradingChanges34.md

Lines changed: 35 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,21 @@ The options `intermediateCommitCount` and `intermediateCommitSize` will have no
175175
anymore on transactions started via `/_api/transaction`, or `db._executeTransaction()`.
176176

177177

178+
RocksDB background sync thread
179+
------------------------------
180+
181+
The RocksDB storage engine in 3.4 has a background WAL syncing thread that by default
182+
syncs RocksDB's WAL to disk every 100 milliseconds. This may cause additional background
183+
I/Os compared to ArangoDB 3.3, but will distribute the sync calls more evenly over time
184+
than the all-or-nothing file syncs that were performed by previous versions of ArangoDB.
185+
186+
The syncing interval can be configured by adjusting the configuration option
187+
`--rocksdb.sync-interval`.
188+
189+
Note: this option is not supported on Windows platforms. Setting the sync interval to
190+
to a value greater than 0 will produce a startup warning on Windows.
191+
192+
178193
Threading and request handling
179194
------------------------------
180195

@@ -475,29 +490,42 @@ instead of error 1582 (`ERROR_QUERY_FUNCTION_NOT_FOUND`) in some situations.
475490
Usage of V8
476491
-----------
477492

478-
The internal usage of the V8 JavaScript for non-user actions has been reduced
479-
in ArangoDB. Several APIs have been rewritten to not depend on V8 and thus do
480-
not require using a V8 context for execution.
493+
The internal usage of the V8 JavaScript engine for non-user actions has been
494+
reduced in ArangoDB 3.4. Several APIs have been rewritten to not depend on V8
495+
and thus do not require using the V8 engine nor a V8 context for execution
496+
anymore.
481497

482498
Compared to ArangoDB 3.3, the following parts of ArangoDB can now be used
483-
without requiring V8 contexts:
499+
without the V8 engine:
484500

501+
- agency nodes in a cluster
502+
- database server nodes in a cluster
503+
- cluster plan application on database server nodes
485504
- all of AQL (with the exception of user-defined functions)
486505
- the graph modification APIs at endpoint `/_api/gharial`
487-
- background server statistics gathering
506+
- background statistics gathering
488507

489-
Reduced usage of V8 by ArangoDB may allow end users to lower the configured
508+
Reduced usage of V8 in ArangoDB may allow end users to lower the configured
490509
numbers of V8 contexts to start. In terms of configuration options, these
491510
are:
492511

493512
- `--javascript.v8-contexts`: the maximum number of V8 contexts to create
513+
(high-water mark)
494514
- `--javascript.v8-contexts-minimum`: the minimum number of V8 contexts to
495-
create at server start and to keep around
515+
create at server start and to keep around permanently (low-water mark)
496516

497517
The default values for these startup options have not been changed in ArangoDB
498518
3.4, but depending on the actual workload, 3.4 ArangoDB instances may need
499519
less V8 contexts than 3.3.
500520

521+
As mentioned above, agency and database server nodes in a cluster does not
522+
require V8 for any operation in 3.4, so the V8 engine is turned off entirely on
523+
such nodes, regardless of the number of configured V8 contexts there.
524+
525+
The V8 engine is still enabled on coordinator servers in a cluster and on single
526+
server instances. Here the numbe of started V8 contexts may actually be reduced
527+
in case a lot of the above features are used.
528+
501529

502530
Startup option changes
503531
----------------------

‎Documentation/Scripts/codeBlockReader.py

Lines changed: 33 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -149,12 +149,16 @@ def example_content(filepath, fh, tag, blockType, placeIntoFilePath):
149149
else:
150150
shortable = True
151151

152-
if blockType == "AQL":
152+
if blockType == "AQL" or blockType == "EXPLAIN":
153153
if line.startswith("@Q"): # query part
154154
blockCount = 0
155155
aqlState = AQL_STATE_QUERY
156-
short += "<strong>Query:</strong>\n<pre>\n"
157-
longText += "<strong>Query:</strong>\n<pre>\n"
156+
if blockType == "EXPLAIN":
157+
short += "<strong>Explain Query:</strong>\n<pre>\n"
158+
longText += "<strong>Explain Query:</strong>\n<pre>\n"
159+
else:
160+
short += "<strong>Query:</strong>\n<pre>\n"
161+
longText += "<strong>Query:</strong>\n<pre>\n"
158162
continue # skip this line - it is only here for this.
159163
elif line.startswith("@B"): # bind values part
160164
short += "</pre>\n<strong>Bind Values:</strong>\n<pre>\n"
@@ -164,7 +168,10 @@ def example_content(filepath, fh, tag, blockType, placeIntoFilePath):
164168
continue # skip this line - it is only here for this.
165169
elif line.startswith("@R"): # result part
166170
shortable = True
167-
longText += "</pre>\n<strong>Results:</strong>\n<pre>\n"
171+
if blockType == "EXPLAIN":
172+
longText += "</pre>\n<strong>Explain output:</strong>\n<pre>\n"
173+
else:
174+
longText += "</pre>\n<strong>Query results:</strong>\n<pre>\n"
168175
blockCount = 0
169176
aqlState = AQL_STATE_RESULT
170177
continue # skip this line - it is only here for this.
@@ -209,20 +216,22 @@ def example_content(filepath, fh, tag, blockType, placeIntoFilePath):
209216
else:
210217
fh.write(unicode("<div id=\"%s\">\n" % longTag))
211218

212-
if blockType != "AQL":
219+
if blockType != "AQL" and blockType != "EXPLAIN":
213220
fh.write(unicode("<pre>\n"))
214221
fh.write(unicode("%s" % longText))
215222
fh.write(unicode("</pre>\n"))
216223
if shortable:
217224
hideText=""
218225
if blockType == "arangosh":
219-
hideText = u"hide execution results"
226+
hideText = u"Hide execution results"
220227
elif blockType == "curl":
221-
hideText = u"hide response body"
228+
hideText = u"Hide response body"
222229
elif blockType == "AQL":
223-
hideText = u"hide query result"
230+
hideText = u"Hide query result"
231+
elif blockType == "EXPLAIN":
232+
hideText = u"Hide explain output"
224233
else:
225-
hideText = u"hide"
234+
hideText = u"Hide"
226235
fh.write(unicode('<div id="%s_collapse" onclick="%s" class="example_show_button">%s</div>' % (
227236
utag,
228237
longToggle,
@@ -232,18 +241,20 @@ def example_content(filepath, fh, tag, blockType, placeIntoFilePath):
232241

233242
if shortable:
234243
fh.write(unicode("<div id=\"%s\" onclick=\"%s\">\n" % (shortTag, shortToggle)))
235-
if blockType != "AQL":
244+
if blockType != "AQL" and blockType != "EXPLAIN":
236245
fh.write(unicode("<pre>\n"))
237246
fh.write(unicode("%s" % short))
238247

239248
if blockType == "arangosh":
240-
fh.write(unicode("</pre><div class=\"example_show_button\">show execution results</div>\n"))
249+
fh.write(unicode("</pre><div class=\"example_show_button\">Show execution results</div>\n"))
241250
elif blockType == "curl":
242-
fh.write(unicode("</pre><div class=\"example_show_button\">show response body</div>\n"))
251+
fh.write(unicode("</pre><div class=\"example_show_button\">Show response body</div>\n"))
243252
elif blockType == "AQL":
244-
fh.write(unicode("</pre><div class=\"example_show_button\">show query result</div>\n"))
253+
fh.write(unicode("</pre><div class=\"example_show_button\">Show query result</div>\n"))
254+
elif blockType == "EXPLAIN":
255+
fh.write(unicode("</pre><div class=\"example_show_button\">Show explain output</div>\n"))
245256
else:
246-
fh.write(unicode("</pre><div class=\"example_show_button\">show</div>\n"))
257+
fh.write(unicode("</pre><div class=\"example_show_button\">Show</div>\n"))
247258

248259
fh.write(unicode("</div>\n"))
249260

@@ -268,6 +279,10 @@ def fetch_comments(dirpath, forceDokuBlockContent):
268279
file_comments = file_content(filepath, forceDokuBlockContent)
269280
for comment in file_comments:
270281
fh.write(unicode("\n<!-- filename: %s -->\n" % filepath))
282+
explain = False
283+
for _com in comment:
284+
if "@EXPLAIN{TRUE}" in _com:
285+
explain = True
271286
for _com in comment:
272287
_text = re.sub(r"//(/)+\s*\n", "<br />\n", _com) # place in temporary brs...
273288
_text = re.sub(r"///+(\s+\s+)([-\*\d])", r" \2", _text)
@@ -287,7 +302,10 @@ def fetch_comments(dirpath, forceDokuBlockContent):
287302
elif "@EXAMPLE_ARANGOSH_RUN" in _text:
288303
blockType = "curl"
289304
elif "@EXAMPLE_AQL" in _text:
290-
blockType = "AQL"
305+
if explain:
306+
blockType = "EXPLAIN"
307+
else:
308+
blockType = "AQL"
291309

292310
shouldIgnoreLine = True
293311
try:

‎UnitTests/OskarTestSuitesBlackList

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,5 @@
1-
1+
ldap
2+
ldaprole
3+
ldaprolesimple
4+
ldapsearch
5+
ldapsearchsimple

‎arangod/Agency/AgencyFeature.cpp

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -239,6 +239,9 @@ void AgencyFeature::validateOptions(std::shared_ptr<ProgramOptions> options) {
239239
{"MMFilesPersistentIndex", "ArangoSearch", "Statistics", "Action", "Script", "FoxxQueues", "Frontend"}
240240
);
241241
if (!result.touched("console") || !*(options->get<BooleanParameter>("console")->ptr)) {
242+
// specifiying --console requires JavaScript, so we can only turn it off
243+
// if not specified
244+
242245
// console mode inactive. so we can turn off V8
243246
disabledFeatures.emplace_back("V8Platform");
244247
disabledFeatures.emplace_back("V8Dealer");

0 commit comments

Comments
 (0)
0