8000 APM-84 | Add `spillOverThreshold*` options to Cursor API DocuBlock (#… · rohan-flutterint/arangodb@65826a5 · GitHub
[go: up one dir, main page]
Skip to content

Commit 65826a5

Browse files
Simran-BKVS85
Simran-B
and
KVS85
authored
APM-84 | Add spillOverThreshold* options to Cursor API DocuBlock (arangodb#17700)
* Cursor API: Add spillOverThreshold* options to DocuBlock * Query spillover startup option: fix typo, add unit * Fix note * Cursor API supports a maxNumberOfPlans option, not maxPlans * Document allowDirtyReads query options of Cursor API Co-authored-by: Vadim Kondratyev <vadim@arangodb.com>
1 parent 880b195 commit 65826a5

File tree

2 files changed

+55
-3
lines changed

2 files changed

+55
-3
lines changed

‎Documentation/DocuBlocks/Rest/Cursors/post_api_cursor.md

Lines changed: 52 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ that read data which are known to be outside of the hot set. By setting the opti
8383
to *false*, data read by the query will not make it into the RocksDB block cache if
8484
not already in there, thus leaving more room for the actual hot set.
8585

86-
@RESTSTRUCT{maxPlans,post_api_cursor_opts,integer,optional,int64}
86+
@RESTSTRUCT{maxNumberOfPlans,post_api_cursor_opts,integer,optional,int64}
8787
Limits the maximum number of plans that are created by the AQL query optimizer.
8888

8989
@RESTSTRUCT{maxNodesPerCallstack,post_api_cursor_opts,int 8000 eger,optional,int64}
@@ -140,6 +140,45 @@ stores the full result in memory (on the contacted Coordinator if in a cluster).
140140
All other resources are freed immediately (locks, RocksDB snapshots). The query
141141
will fail before it returns results in case of a conflict.
142142

143+
@RESTSTRUCT{spillOverThresholdMemoryUsage,post_api_cursor_opts,integer,optional,}
144+
This option allows queries to store intermediate and final results temporarily
145+
on disk if the amount of memory used (in bytes) exceeds the specified value.
146+
This is used for decreasing the memory usage during the query execution.
147+
148+
This option only has an effect on queries that use the `SORT` operation but
149+
without a `LIMIT`, and if you enable the spillover feature by setting a path
150+
for the directory to store the temporary data in with the
151+
`--temp.intermediate-results-path` startup option.
152+
153+
Default value: 128MB.
154+
155+
**Note**:
156+
Spilling data from RAM onto disk is an experimental feature and is turned off
157+
by default. The query results are still built up entirely in RAM on Coordinators
158+
and single servers for non-streaming queries. To avoid the buildup of
159+
the entire query result in RAM, use a streaming query (see the `stream` option).
160+
161+
@RESTSTRUCT{spillOverThresholdNumRows,post_api_cursor_opts,integer,optional,}
162+
This option allows queries to store intermediate and final results temporarily
163+
on disk if the number of rows produced by the query exceeds the specified value.
164+
This is used for decreasing the memory usage during the query execution. In a
165+
query that iterates over a collection that contains documents, each row is a
166+
document, and in a query that iterates over temporary values
167+
(i.e. `FOR i IN 1..100`), each row is one of such temporary values.
168+
169+
This option only has an effect on queries that use the `SORT` operation but
170+
without a `LIMIT`, and if you enable the spillover feature by setting a path
171+
for the directory to store the temporary data in with the
172+
`--temp.intermediate-results-path` startup option.
173+
174+
Default value: `5000000` rows.
175+
176+
**Note**:
177+
Spilling data from RAM onto disk is an experimental feature and is turned off
178+
by default. The query results are still built up entirely in RAM on Coordinators
179+
and single servers for non-streaming queries. To avoid the buildup of
180+
the entire query result in RAM, use a streaming query (see the `stream` option).
181+
143182
@RESTSTRUCT{optimizer,post_api_cursor_opts,object,optional,post_api_cursor_opts_optimizer}
144183
Options related to the query optimizer.
145184

@@ -187,6 +226,18 @@ results by changing the access rights of users on collections.
187226

188227
This feature is only available in the Enterprise Edition.
189228

229+
@RESTSTRUCT{allowDirtyReads,post_api_cursor_opts,boolean,optional,}
230+
If you set this option to `true` and execute the query against a cluster
231+
deployment, then the Coordinator is allowed to read from any shard replica and
232+
not only from the leader.
233+
234+
You may observe data inconsistencies (dirty reads) when reading from followers,
235+
namely obsolete revisions of documents because changes have not yet been
236+
replicated to the follower, as well as changes to documents before they are
237+
officially committed on the leader.
238+
239+
This feature is only available in the Enterprise Edition.
240+
190241
@RESTDESCRIPTION
191242
The query details include the query string plus optional query options and
192243
bind parameters. These values need to be passed in a JSON representation in

‎arangod/RestServer/TemporaryStorageFeature.cpp

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -176,8 +176,9 @@ query result in RAM, use a streaming query.)");
176176
options
177177
->addOption(
178178
"--temp.intermediate-results-spillover-threshold-memory-usage",
179-
"The memory usage threshold after which a spillover form RAM to disk "
180-
"happens for intermediate results (threshold per query executor).",
179+
"The memory usage threshold (in bytes) after which a spillover from "
180+
"RAM to disk happens for intermediate results "
181+
"(threshold per query executor).",
181182
new SizeTParameter(&_spillOverThresholdMemoryUsage),
182183
arangodb::options::makeDefaultFlags(
183184
arangodb::options::Flags::Experimental))

0 commit comments

Comments
 (0)
0