8000 DOC-145: Added documentation for APM-84 (#1060) · arangodb/docs@445c6a2 · GitHub
[go: up one dir, main page]

Skip to content
This repository was archived by the owner on Dec 13, 2023. It is now read-only.

Commit 445c6a2

Browse files
cpjuliaansoboleva
andauthored
DOC-145: Added documentation for APM-84 (#1060)
* Added documentation for APM-84 * Fixed query option name * Removed explanation from release * review * Update invocation-with-arangosh.md * wording, formatting * formatting, wording, added links to release notes * Update release-notes-new-features310.md * Update release-notes-new-features310.md * Update invocation-with-arangosh.md Co-authored-by: ansoboleva <93702078+ansoboleva@users.noreply.github.com>
1 parent 83bf64e commit 445c6a2

File tree

3 files changed

+130
-0
lines changed

3 files changed

+130
-0
lines changed

3.10/aql/invocation-with-arangosh.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,44 @@ In the ArangoDB Enterprise Edition there is an additional parameter:
242242
into sync. The default value is 60.0 (seconds). When the max time has been reached
243243
the query will be stopped.
244244

245+
Additional parameters for spilling data from the query onto disk
246+
-----------------------------------------------------------------
247+
248+
Starting from ArangoDB 3.10, there are two additional parameters that allow spilling
249+
intermediate data from a query onto a disk to descrease the memory usage.
250+
251+
{% hint 'info' %}
252+
The option of spilling data from RAM onto disk is experimental and is turned off
253+
by default. This parameter currently only has effect for sorting -
254+
for a query that uses the SORT keyword, but without LIMIT.
255+
Also, the query results are still built up entirely in RAM on coordinators
256+
and single servers for non-streaming queries. To avoid the buildup of
257+
the entire query result in RAM, a streaming query should be used.
258+
{% endhint %}
259+
260+
- `spillOverThresholdNumRows`: This parameter allows for input data and
261+
intermediate results to be spilled onto disk for a query execution
262+
after the number of rows reaches the specified value. This is
263+
used for decreasing the memory usage during the query execution. In a query
264+
that iterates over a collection that contains documents, each row is a
265+
document and, in a query that iterates over temporary values
266+
(i.e. `FOR i IN 1..100`), each row is one of such temporary values.
267+
This parameter is experimental and is only taken into account if a path for a
268+
directory to store the temporary data is provided with the [`--temp.intermediate-results-path`
269+
server startup option](../programs-arangod-query.html#aql-query-with-spilling-input-data-to-disk).
270+
Default value: 5000000 rows.
271+
272+
273+
- `spillOverThresholdMemoryUsage`: This parameter allows for input data and
274+
intermediate results to be spilled onto disk for a query execution
275+
after the memory usage reaches the specified value (in bytes). This
276+
is used for decreasing the memory usage during the query execution. This
277+
parameter is experimental and is only taken into account if a path for a
278+
directory to store the temporary data is provided with the [`--temp.intermediate-results-path`
279+
server startup option](../programs-arangod-query.html#aql-query-with-spilling-input-data-to-disk).
280+
Default value: 128MB.
281+
282+
245283
With _createStatement (ArangoStatement)
246284
---------------------------------------
247285

3.10/programs-arangod-query.md

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -314,3 +314,63 @@ query results cache, as queries on system collections are internal to ArangoDB a
314314
only use space in the query results cache unnecessarily.
315315

316316
The default value is *false*.
317+
318+
319+
## AQL Query with spilling input data to disk
320+
321+
<small>Introduced in: v3.10.0 </small>
322+
323+
With the parameters mentioned below, queries can execute with storing input
324+
and intermediate results temporarily on disk to decrease the memory usage
325+
a specified threshold is reached.
326+
327+
{% hint 'info' %}
328+
This feature is experimental and is turned off by default.
329+
Also, the query results are still built up entirely in RAM on coordinators
330+
and single servers for non-streaming queries. To avoid the buildup of
331+
the entire query result in RAM, a streaming query should be used.
332+
{% endhint %}
333+
334+
The threshold value to start spilling data onto disk is either
335+
a number of rows in the query input or an amount of memory used in bytes,
336+
which are both set as query options.
337+
338+
The main parameter that must be provided for this feature to be active is
339+
`--temp.intermediate-results-path`. This parameter specifies a path to a directory
340+
used to store temporary data. If such path is not provided, the feature of spilling data
341+
onto the disk will not be activated.
342+
343+
Hence, the following parameters do not have any effect, unless the parameter
344+
mentioned above is provided with a directory path.
345+
The directory specified here must not be located underneath the instance's
346+
database directory.
347+
348+
349+
- `--temp.-intermediate-results-encryption-hardware-acceleration`
350+
351+
Use Intel intrinsics-based encryption, requiring a CPU with the AES-NI
352+
instruction set. If turned off, then OpenSSL is used, which may use hardware-
353+
accelarated encryption too.
354+
Default: `true`.
355+
356+
- `--temp.intermediate-results-capacity`
357+
358+
Maximum capacity, in bytes, to use for ephemeral, intermediate results, meaning
359+
the maximum size allowed for the mentioned temporary storage.
360+
Default: 0 (unlimited).
361+
362+
- `--temp.intermediate-results-encryption`
363+
364+
Encrypt ephemeral, intermediate results on disk.
365+
Default: `false`.
366+
367+
368+
- `--temp.intermediate-results-spillover-threshold-num-rows`
369+
370+
A number of result rows from which on a spillover from RAM to disk will happen.
371+
Default: 5000000.
372+
373+
- `--temp.intermediate-results-spillover-threshold-memory-usage`
374+
375+
Memory usage, in bytes, after which a spillover from RAM to disk will happen.
376+
Default: 128MB.

3.10/release-notes-new-features310.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -517,6 +517,38 @@ arangoexport \
517517
...
518518
```
519519

520+
521+
Query changes for decreasing memory usage
522+
-----------------------------------------
523+
524+
Queries can be executed with storing input and intermediate results temporarily
525+
on disk to descrease memory usage when a specified threshold is reached.
526+
527+
{% hint 'info' %}
528+
This feature is experimental and is turned off by default.
529+
{% endhint %}
530+
531+
The new parameters are listed below:
532+
533+
- `--temp.intermediate-results-path`
534+
- `--temp.-intermediate-results-encryption-hardware-acceleration`
535+
- `--temp.intermediate-results-encryption`
536+
- `--temp.intermediate-results-spillover-threshold-num-rows`
537+
- `--temp.intermediate-results-spillover-threshold-memory-usage`
538+
539+
Example:
540+
541+
```
542+
arangod --temp.intermediate-results-path "tempDir"
543+
--database.directory "myDir"
544+
--temp.-intermediate-results-encryption-hardware-acceleration true
545+
--temp.intermediate-results-encryption true
546+
--temp.intermediate-results-spillover-threshold-num-rows 50000
547+
--temp.intermediate-results-spillover-threshold-memory-usage 134217728
548+
```
549+
550+
For more information, refer to the [Query invocation](aql/invocation-with-arangosh.html#additional-parameters-for-spilling-data-from-the-query-onto-disk) and [Query options](programs-arangod-query.html#aql-query-with-spilling-input-data-to-disk) topics.
551+
520552
Internal changes
521553
----------------
522554

0 commit comments

Comments
 (0)
0