-
Notifications
You must be signed in to change notification settings - Fork 854
Deprecate rocksdb.max-write-buffer-number startup option #9654
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Code change verified to be working, arangod.json (for backport): "rocksdb.max-write-buffer-number" : {
"category" : "option",
"default" : 2,
"deprecatedIn" : [
"v3.4.8",
"v3.5.0"
],
"description" : "maximum number of write buffers that build up in memory. This option is ignored! There is a hardcoded maximum of number of column families + 2 (currently 9) write buffers based on performance research",
"dynamic" : false,
"enterpriseOnly" : false,
"hidden" : false,
"introducedIn" : null,
"obsolete" : false,
"requiresValue" : true,
"section" : "rocksdb",
"type" : "uint64"
}, |
@jsteemann says that we should think about making this option actually work instead of deprecating it. |
"make it work" then means the minimum should be number of column families plus two, and user could increase above that. Going lower creates performance bottleneck on write buffer flush. |
Okay, but I think the value the user passes should be the total max, and ignored if it's below column families plus two (with a log warning). The current default seems to be the offset of plus two? That's kinda confusing, especially because the true max is 9 by default. Will leave this PR pending until further notice. |
"true max" is 2: https://github.com/arangodb/arangodb/blob/devel/3rdParty/rocksdb/6.2/include/rocksdb/advanced_options.h#L167 ... I am the person that raised it to 9 ... 7 column families plus 2. |
Let's just honor the user's setting for this option if it is explicitly set by the user. |
Yes. I object to allowing less than 9. The user is not going to understand the performance impact. They are likely thinking "this will decrease my memory footprint". Then we end up having to debug their environment when server suddenly has periodic stalls which can lead to cluster replication sync issues and other nasty items. |
I think the above PRs of mine obsolete this issue. @Simran-B |
…ture/mimalloc * 'devel' of https://github.com/arangodb/arangodb: (83 commits) Bug fix/internal issue #622 (#9781) mark AQL functions FULLTEXT, NEAR, WITHIN, WITHIN_RECTANGLE as cacheable (#9771) reduce wait timeout, use move Bug fix/implement windows maintenance tests (#9763) show query string length and cacheability in explain output (#9767) fix potential spurious wakeups in scheduler code (#9770) Bug fix/issue #9612 (#9764) downgrade WARN messages to INFO level (#9761) slightly reorder boolean members to reduce struct sizes (#9762) make index selection more deterministic (#9735) Update PULL_REQUEST_TEMPLATE.md (#9758) [devel] Move Shard Bug 4567124 (#9746) Bug fix/fix signed int overflow (#9717) Bug fix/fix invalid cast (#9755) Enforce stricter transaction limits (#9740) Check scheduler queue return value (#9754) dont fill cache on truncate (#9721) fix pasting from the documentation (#9742) tell that procdump is gone - it seems this happenes in reality without coredumps being written (#9748) issue #9654: make `--rocksdb.max-write-buffer-number` work (#9750) ...
Removed the deprecation part, reworded the description to:
|
…x-write-buffer-number
194b6fd
to
e45cf42
Compare
…ture/one-shard-clean-up-2 * 'devel' of https://github.com/arangodb/arangodb: Bug fix/improve stringutils performance (#10208) add option to talk to the SUT using VST (#10217) Doc - Added "log-output" example (#10207) fix it! (#10198) add missing include Bug fix/fix simple example dep proxy skip some regression test (#10213) fixed ui behaviour when replacing a foxx app (#9719) [devel] Fix document search (Ctrl+F/Cmd+F) (#10216) Convert many uses of ClusterComm to Fuerte (#10154) Remove invokeOnAllElements (#10212) AQL Subquery: MultiDependencyRowFetcher (#10101) Bug fix/fix remote executor races (#10206) fix several inefficiencies in Store (#10189) Deprecate rocksdb.max-write-buffer-number startup option (#9654) fix arangosh with vst
Scope & Purpose
Mark the startup option
rocksdb.max-write-buffer-number
as deprecated in v3.4.8 and v3.5.0 because it does nothing. It is hardcoded to number of column families (7) + 2 = 9.matthew:
Related Information
(Please reference tickets / specification etc )
Testing & Verification
This change is a trivial rework / code cleanup without any test coverage - while technically a code change, it should affect nothing but the documentation (and
arangod --help
output)