From 5334006dc8fd7173eb93859bc7d48c5478a0ec28 Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Mon, 20 Sep 2021 12:28:28 +0200 Subject: [PATCH 01/14] add more documentation about how tests work --- README_maintainers.md | 7 +- tests/README.md | 194 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 200 insertions(+), 1 deletion(-) create mode 100644 tests/README.md diff --git a/README_maintainers.md b/README_maintainers.md index 47cbdcd0d001..8c4f2f6ab70f 100644 --- a/README_maintainers.md +++ b/README_maintainers.md @@ -16,7 +16,12 @@ Main sections: - [Linux Core Dumps](#linux-core-dumps) - [Windows Core Dumps](#windows-core-dumps) - [Unittests](#unittests) - + - [invoking driver tests](#driver-tests) via scripts/unittests + - [capturing test communication](#running-tcpdump--windump-for-the-sut) but + [maybe also downgrade to JSON](#forcing-downgrade-from-vpack-to-json) + - [Evaluating previous testruns](#evaluating-json-test-reports-from-previous-testruns) + sorting by setup time etc. +- [what to test where and how](tests/README.md) --- ## Source Code diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 000000000000..933523495867 --- /dev/null +++ b/tests/README.md @@ -0,0 +1,194 @@ +# Tests +When creating tests, please bear in mind, that every test added will add additional time during the PR-Runs. + +So every test added *will* slow down PRs. + +Please also note that tests in the CI share a system with other runs in parallel. +So CI tests are not a place to max out I/O, Memory, or such. + +So please use *BULK* inserts to slimline the creation of your test data. +If you can use a common dataset for several tests, please use `setUpAll` and `tearDownAll` to reduce resource usage. +If splitting a testsuite between read only and writing tests is neccessary for this, please do so. + +### Testsuites +Testsuites live in `js/client/modules/@arangodb/testsuites/`. This folder is spidered during the startup of `testing.js` and loads the files as plugins. +Please think twice, whether you need to create a new testsuite, or your test will fit into one of the existing ones. Suites need to be registered within Oskar. +The Plugin structure works as follows: + +``` +const functionsDocumentation = { + 'shell_server': 'shell server tests', +}; +const optionsDocumentation = [ + ' - `skipAql`: if set to true the AQL tests are skipped', +]; + +const testPaths = { + 'shell_server': [ tu.pathForTesting('common/shell'), tu.pathForTesting('server/shell') ], +}; + +function shellServer (options) { + // spider all testcases, including twin paths in the enterprise directory: + let testCases = tu.scanTestPaths(testPaths.shell_server, options); + // split the testcases by `--testBuckets` parameter + testCases = tu.splitBuckets(options, testCases); + + // spawn the SUT, execute all tests via the `tu.runThere` function + // alternatives are: + // - runThere (use javascript shell exec to run the test *in* the server + // - runInLocalArangosh (run in the very same arangosh as testing.js) + // - runInArangosh (spawn a new arangosh, collect results via result file, more frequent process spawning) + // - runInRSpec (launch tests via ruby rspec) + // - runInDriverTest implemented inside the testsuite - see testsuites/driver.js as example + let rc = tu.performTests(opts, testCases, 'shell_client, tu.runThere, /* startStopHandlers*/ ); + // startStopHandlers can be implemented to add additional code during the startup / shutdown of the SUT: + // - preStart - before starting the SUT, can be used to add additional params etc. + // - postStart - invoked after the primary SUT is online. can be used to start other systems + // - healthCheck - to be invoked after each test to check whether all is well + // - preStop - just before the stop to do things that would hinder the shutdown + // - postStop - after the stop to clean up things. + + // pass up eventual errors that should prohibit flushing the SUT data + options.cleanup = options.cleanup && opts.cleanup; + return rc; +} + +// plugin hook. This is executed during loading of the modules. +exports.setup = function (testFns, defaultFns, opts, fnDocs, optionsDoc, allTestPaths) { + // add all used testpaths, so i.e. `auto` or `find` can find the suite for single `--test` files + Object.assign(allTestPaths, testPaths); + // Register the testsuite itself + testFns['shell_server'] = shellServer; + // enable it to be ran with `./scripts/unittest all` ; if not ommit - adds the [x] in help the list. + defaultFns.push('shell_server'); + // if your testsuite has CLI parameters, specify them including the default param here: + opts['skipAql'] = false; + + // + for (var attrname in functionsDocumentation) { fnDocs[attrname] = functionsDocumentation[attrname]; } + for (var i = 0; i < optionsDocumentation.length; i++) { optionsDoc.push(optionsDocumentation[i]); } +}; +``` + +### existing Javascript tests + +#### [permission] => tests/js/client/permissions / [server_parameters] => tests/js/client/server_parameters / [server_permissions] => client/server_permissions +This set of testsuites is here to revalidate startup parameters of arangodb processes. + +The testfile persists of two parts, and the file is invoked twice: + +Controlling the startup parameters can be done with the `getOptions` part. In `server_permissions` this can also be used to setup data which may not be permitted during the later run: +``` +if (getOptions === true) { + return { + 'javascript.allow-port-testing': false + }; +} +``` +The later part of the test may contain regular jsunity testsuites. + +If you want to revalidate a parameter that is common to arangosh & arangod, you *should* use a test in the `permission` testsuite, +since spawning an arangosh requires way less resources than spawning a full arangodb setup. + +If your testcase requires a setup with more permissions than the testcode itself, you should use `server_permissions`, since it will launch the SUT twice. + +You should however *prefer* `server_parameters` which will launch the SUT only once with the parameters you've provided. + +You should try to use the `-[non]cluster` filename filters to also control whether your test is ran on cluster too to reduce computing power. + +#### [dump*] to test dump and restore procedures +The dump family of testsuites lives in `js/client/modules/@arangodb/testsuites/dump.js` +and orchestrates a sequence: +- launching a SUT +- create test data within it +- dump this test data +- execute some manual sanity checks on the dump files +- restore it to a fresh SUT +- check data + +Since many create & check functions should be common to *all* suites, the default behavior is to enable them, and only blacklist them out if unsupported. + +The individual files to be used are controlled by code in the testsuite like this: + +``` + let tstFiles = { + dumpSetup: 'dump-setup' + c.cluster + '.js', + dumpCheckDumpFiles: 'dump-check-dump-files-encrypted.js', + dumpCleanup: 'cleanup-nothing.js', + dumpAgain: 'dump' + c.cluster + '.js', + dumpTearDown: 'dump-teardown' + c.cluster + '.js', + foxxTest: 'check-foxx.js' + }; +``` + +So `tests/js/server/dump/dump-setup[-cluster].js` will be invoked during setting up the first SUT. +It derives all its functions from `dump-setup-common.inc` and only invokes them in sequence. + +Revalidating will be done using `dump[-cluster].js`, which will obtain all its test functions from `dump-test.inc`. + +Testdata to be used during the first create etc can be put into `tests/js/server/dump`. + + +#### [communication, chaos] testsuite to launch parallel load +These testsuites are intended to launch several arangosh, in order to generate + + + +#### [driver | go | java | js] run driver tests against the local testsuite +These testsuites aim to integrate the respective drivers test with testing.js, +so you can easily invoke them on your workstation. + +# more to come... +#### shell_server +tests/js/server +tests/js/server/aql +tests/js/server/import +tests/js/server/recovery +tests/js/server/replication +tests/js/server/replication/fuzz +tests/js/server/replication/aql +tests/js/server/replication/random +tests/js/server/replication/ongoing +tests/js/server/replication/ongoing/frompresent +tests/js/server/replication/ongoing/global +tests/js/server/replication/ongoing/global/spec +tests/js/server/replication/static +tests/js/server/replication/sync +tests/js/server/backup +tests/js/server/agency-restart +tests/js/server/paths +tests/js/server/upgrade-data +tests/js/server/stress +tests/js/server/resilience +tests/js/server/resilience/move-view +tests/js/server/resilience/failover-view +tests/js/server/resilience/transactions +tests/js/server/resilience/failover +tests/js/server/resilience/move +tests/js/server/resilience/analyzers +tests/js/server/resilience/failover-failure +tests/js/server/resilience/sharddist +tests/js/server/export +tests/js/server/shell +tests/js/common/aql +tests/js/common/replication +tests/js/common/http +tests/js/common/shell +tests/js/client +tests/js/client/authentication +tests/js/client/permissions +tests/js/client/ldap +tests/js/client/server_secrets +tests/js/client/wal_cleanup +tests/js/client/restart +tests/js/client/assets +tests/js/client/assets/queuetest +tests/js/client/active-failover +tests/js/client/load-balancing +tests/js/client/communication +tests/js/client/chaos +tests/js/client/http +tests/js/client/agency +tests/js/client/replication2 +tests/js/client/shell +tests/js/client/server_permissions From 33156d8e45b2cf8867476d8099c0d54d95578c19 Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Mon, 20 Sep 2021 18:27:37 +0200 Subject: [PATCH 02/14] Update README_maintainers.md Co-authored-by: Jan --- README_maintainers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README_maintainers.md b/README_maintainers.md index 8c4f2f6ab70f..242619ad5ef3 100644 --- a/README_maintainers.md +++ b/README_maintainers.md @@ -18,7 +18,7 @@ Main sections: - [Unittests](#unittests) - [invoking driver tests](#driver-tests) via scripts/unittests - [capturing test communication](#running-tcpdump--windump-for-the-sut) but - [maybe also downgrade to JSON](#forcing-downgrade-from-vpack-to-json) + [forcing communication to use plain-text JSON](#forcing-downgrade-from-vpack-to-json) - [Evaluating previous testruns](#evaluating-json-test-reports-from-previous-testruns) sorting by setup time etc. - [what to test where and how](tests/README.md) From a3582c28d3a8f29504c2c7c015e625d0f91c9370 Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Mon, 20 Sep 2021 18:27:56 +0200 Subject: [PATCH 03/14] Update tests/README.md Co-authored-by: Jan --- tests/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/README.md b/tests/README.md index 933523495867..a554d1b540af 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1,7 +1,7 @@ # Tests -When creating tests, please bear in mind, that every test added will add additional time during the PR-Runs. +When creating tests, please bear in mind, that every test added will add additional time during the PR runs. -So every test added *will* slow down PRs. +So every added test *will* slow down PR runs. Please also note that tests in the CI share a system with other runs in parallel. So CI tests are not a place to max out I/O, Memory, or such. From 7f6e3fa015b6974c58e312a77c0fc31cf0e6f88a Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Mon, 20 Sep 2021 18:28:06 +0200 Subject: [PATCH 04/14] Update tests/README.md Co-authored-by: Jan --- tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/README.md b/tests/README.md index a554d1b540af..be98fd379835 100644 --- a/tests/README.md +++ b/tests/README.md @@ -4,7 +4,7 @@ When creating tests, please bear in mind, that every test added will add additio So every added test *will* slow down PR runs. Please also note that tests in the CI share a system with other runs in parallel. -So CI tests are not a place to max out I/O, Memory, or such. +So CI tests are not a place to max out I/O, memory, or such. So please use *BULK* inserts to slimline the creation of your test data. If you can use a common dataset for several tests, please use `setUpAll` and `tearDownAll` to reduce resource usage. From 23b799c59dd92f840b29a8face5fc8bd26ff4035 Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Mon, 20 Sep 2021 18:28:18 +0200 Subject: [PATCH 05/14] Update tests/README.md Co-authored-by: Jan --- tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/README.md b/tests/README.md index be98fd379835..76b8578489c2 100644 --- a/tests/README.md +++ b/tests/README.md @@ -3,7 +3,7 @@ When creating tests, please bear in mind, that every test added will add additio So every added test *will* slow down PR runs. -Please also note that tests in the CI share a system with other runs in parallel. +Please also note that test runs in the CI share the system with other parallel runs. So CI tests are not a place to max out I/O, memory, or such. So please use *BULK* inserts to slimline the creation of your test data. From 3e9ca22cbac283de3310217c48e2a53236c2f0eb Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Mon, 20 Sep 2021 18:28:29 +0200 Subject: [PATCH 06/14] Update tests/README.md Co-authored-by: Jan --- tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/README.md b/tests/README.md index 76b8578489c2..30fd4649fef0 100644 --- a/tests/README.md +++ b/tests/README.md @@ -8,7 +8,7 @@ So CI tests are not a place to max out I/O, memory, or such. So please use *BULK* inserts to slimline the creation of your test data. If you can use a common dataset for several tests, please use `setUpAll` and `tearDownAll` to reduce resource usage. -If splitting a testsuite between read only and writing tests is neccessary for this, please do so. +If splitting a testsuite between read only and writing tests is necessary for this, please do so. ### Testsuites Testsuites live in `js/client/modules/@arangodb/testsuites/`. This folder is spidered during the startup of `testing.js` and loads the files as plugins. From 8be6d47625be2f514fc5acd24b02fac1bdc92d4b Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Mon, 20 Sep 2021 18:28:45 +0200 Subject: [PATCH 07/14] Update tests/README.md Co-authored-by: Jan --- tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/README.md b/tests/README.md index 30fd4649fef0..c792a32dcbda 100644 --- a/tests/README.md +++ b/tests/README.md @@ -11,7 +11,7 @@ If you can use a common dataset for several tests, please use `setUpAll` and `te If splitting a testsuite between read only and writing tests is necessary for this, please do so. ### Testsuites -Testsuites live in `js/client/modules/@arangodb/testsuites/`. This folder is spidered during the startup of `testing.js` and loads the files as plugins. +Testsuites live in `js/client/modules/@arangodb/testsuites/`. This folder is spidered during the startup of `testing.js` and all files in it are loaded automatically. Please think twice, whether you need to create a new testsuite, or your test will fit into one of the existing ones. Suites need to be registered within Oskar. The Plugin structure works as follows: From 34f754499f01ddda798260cec4ef936794435559 Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Mon, 20 Sep 2021 18:28:59 +0200 Subject: [PATCH 08/14] Update tests/README.md Co-authored-by: Jan --- tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/README.md b/tests/README.md index c792a32dcbda..6c2bba84266c 100644 --- a/tests/README.md +++ b/tests/README.md @@ -12,7 +12,7 @@ If splitting a testsuite between read only and writing tests is necessary for th ### Testsuites Testsuites live in `js/client/modules/@arangodb/testsuites/`. This folder is spidered during the startup of `testing.js` and all files in it are loaded automatically. -Please think twice, whether you need to create a new testsuite, or your test will fit into one of the existing ones. Suites need to be registered within Oskar. +Please think twice whether you need to create a new testsuite or your test will fit into one of the existing ones. All test suites need to be registered within Oskar in order to be executed during CI tests. The Plugin structure works as follows: ``` From e51ae812d36e05b0f19ad5cd32cf6a090a4cc6e5 Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Mon, 20 Sep 2021 18:32:58 +0200 Subject: [PATCH 09/14] Update tests/README.md Co-authored-by: Jan --- tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/README.md b/tests/README.md index 6c2bba84266c..f52a39014a24 100644 --- a/tests/README.md +++ b/tests/README.md @@ -13,7 +13,7 @@ If splitting a testsuite between read only and writing tests is necessary for th ### Testsuites Testsuites live in `js/client/modules/@arangodb/testsuites/`. This folder is spidered during the startup of `testing.js` and all files in it are loaded automatically. Please think twice whether you need to create a new testsuite or your test will fit into one of the existing ones. All test suites need to be registered within Oskar in order to be executed during CI tests. -The Plugin structure works as follows: +The structure of each testsuite is as follows: ``` const functionsDocumentation = { From b3f2af0f57e8359335c75e633ea25b3819164d35 Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Mon, 20 Sep 2021 18:33:10 +0200 Subject: [PATCH 10/14] Update tests/README.md Co-authored-by: Jan --- tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/README.md b/tests/README.md index f52a39014a24..0563f321327f 100644 --- a/tests/README.md +++ b/tests/README.md @@ -70,7 +70,7 @@ exports.setup = function (testFns, defaultFns, opts, fnDocs, optionsDoc, allTest }; ``` -### existing Javascript tests +### Existing Javascript tests #### [permission] => tests/js/client/permissions / [server_parameters] => tests/js/client/server_parameters / [server_permissions] => client/server_permissions This set of testsuites is here to revalidate startup parameters of arangodb processes. From a5cd05cd3ae91880ed9cb61d8db78c678e567cbc Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Mon, 20 Sep 2021 18:33:20 +0200 Subject: [PATCH 11/14] Update tests/README.md Co-authored-by: Jan --- tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/README.md b/tests/README.md index 0563f321327f..9d9fc36caf43 100644 --- a/tests/README.md +++ b/tests/README.md @@ -73,7 +73,7 @@ exports.setup = function (testFns, defaultFns, opts, fnDocs, optionsDoc, allTest ### Existing Javascript tests #### [permission] => tests/js/client/permissions / [server_parameters] => tests/js/client/server_parameters / [server_permissions] => client/server_permissions -This set of testsuites is here to revalidate startup parameters of arangodb processes. +This set of testsuites is here to revalidate startup parameters of arangod processes. The testfile persists of two parts, and the file is invoked twice: From 0a0fab507ee2609ca0b47379fc1d8ffe208a4683 Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Mon, 20 Sep 2021 18:34:06 +0200 Subject: [PATCH 12/14] Update tests/README.md Co-authored-by: Jan --- tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/README.md b/tests/README.md index 9d9fc36caf43..d700fb9381dd 100644 --- a/tests/README.md +++ b/tests/README.md @@ -106,7 +106,7 @@ and orchestrates a sequence: - restore it to a fresh SUT - check data -Since many create & check functions should be common to *all* suites, the default behavior is to enable them, and only blacklist them out if unsupported. +Since many create & check functions should be common to *all* suites, the default behavior is to enable them, and only exclude them if unsupported. The individual files to be used are controlled by code in the testsuite like this: From 684bf52322343c9c6718dd4d3e807bfc07a385cf Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Mon, 20 Sep 2021 18:34:21 +0200 Subject: [PATCH 13/14] Update tests/README.md Co-authored-by: Jan --- tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/README.md b/tests/README.md index d700fb9381dd..a7173f3d98b7 100644 --- a/tests/README.md +++ b/tests/README.md @@ -130,7 +130,7 @@ Testdata to be used during the first create etc can be put into `tests/js/server #### [communication, chaos] testsuite to launch parallel load -These testsuites are intended to launch several arangosh, in order to generate +These testsuites are intended to launch several arangosh instances, in order to generate concurrent requests. From 5009ce2c3bafb6cc4dea93c664e06e304e024cc5 Mon Sep 17 00:00:00 2001 From: Wilfried Goesgens Date: Tue, 21 Sep 2021 09:10:18 +0200 Subject: [PATCH 14/14] review --- README_maintainers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README_maintainers.md b/README_maintainers.md index 242619ad5ef3..5eecd6138c61 100644 --- a/README_maintainers.md +++ b/README_maintainers.md @@ -17,7 +17,7 @@ Main sections: - [Windows Core Dumps](#windows-core-dumps) - [Unittests](#unittests) - [invoking driver tests](#driver-tests) via scripts/unittests - - [capturing test communication](#running-tcpdump--windump-for-the-sut) but + - [capturing test HTTP communication](#running-tcpdump--windump-for-the-sut) but [forcing communication to use plain-text JSON](#forcing-downgrade-from-vpack-to-json) - [Evaluating previous testruns](#evaluating-json-test-reports-from-previous-testruns) sorting by setup time etc.