8000 Cleaner test output, enhanced filters. by adityamukho · Pull Request #16021 · arangodb/arangodb · GitHub
[go: up one dir, main page]
Skip to content

Cleaner test output, enhanced filters. #16021

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 14 commits into from
Mar 31, 2022
Merged

Cleaner test output, enhanced filters. #16021

merged 14 commits into from
Mar 31, 2022

Conversation

adityamukho
Copy link
Contributor

Scope & Purpose

This PR solves 2 issues:

  1. The JS test script output is too verbose, containing server startup logs and lifecycle management logs emitted by the process manager that are usually of no use to a tester running normal unit/integration tests on their local machine. They just want to see the test output (pass/fail + stacktrace). A startup command-line option noStartStopLogs is added to suppress these extraneous messages. This option is overridden when extremeVerbosity is set to true.
  2. Test case filters were rudimentary, requiring the user to furnish the complete name of a single test case, and even then the script would complain about all the test suites which did not contain that test case. The filters have now been enhanced in the following ways:
    1. All filters now support extended pattern matching, as explained in https://realguess.net/2014/07/05/extended-pattern-matching/.
    2. A filter that contains a '::' separator uses the prefix to filter the test suites and the suffix to filter the test cases within the matching suites. A filter without a '::' separator uses the entire filter expression to match on the test case, as before.
    3. Any suite that does not match the filter condition is silently ignored.
  • 💩 Bugfix
  • 🍕 New feature
  • 🔥 Performance improvement
  • 🔨 Refactoring/simplification

Checklist

  • Tests
    • Regression tests
    • C++ Unit tests
    • integration tests
    • resilience tests
  • 📖 CHANGELOG entry made
  • 📚 documentation written (release notes, API changes, ...)
  • Backports
    • Backport for 3.9: (Please link PR)
    • Backport for 3.8: (Please link PR)
    • Backport for 3.7: (Please link PR)

Related Information

(Please reference tickets / specification / other PRs etc)

@adityamukho adityamukho added this to the 3.10 milestone Mar 29, 2022
@adityamukho adityamukho marked this pull request as ready for review March 29, 2022 10:52
@dothebart
Copy link
Contributor

in testing.js:

function unitTest (cases, options) {
  if (typeof options !== 'object') {
    options = {};
  }
  loadTestSu
8000
ites(options);
  // testsuites may register more defaults...
  _.defaults(options, optionsDefaults);

appending:

options.noStartStopLogs = !options.extremeVerbosity && options.noStartStopLogs;

seems nicer to me than having these double-if'ed expressions all over the place...

dothebart
dothebart requested changes Mar 29, 2022
View reviewed changes
Copy link
Contributor
@dothebart dothebart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please apply the two suggested simplifications.

@adityamukho adityamukho requested a review from dothebart March 29, 2022 15:05
@adityamukho
Copy link
Contributor Author

in testing.js:

function unitTest (cases, options) {
  if (typeof options !== 'object') {
    options = {};
  }
  loadTestSuites(options);
  // testsuites may register more defaults...
  _.defaults(options, optionsDefaults);

appending:

options.noStartStopLogs = !options.extremeVerbosity && options.noStartStopLogs;

seems nicer to me than having these double-if'ed expressions all over the place...

Done.

@hkernbach
Copy link
Member

in testing.js:

function unitTest (cases, options) {
  if (typeof options !== 'object') {
    options = {};
  }
  loadTestSuites(options);
  // testsuites may register more defaults...
  _.defaults(options, optionsDefaults);

appending:

options.noStartStopLogs = !options.extremeVerbosity && options.noStartStopLogs;

seems nicer to me than having these double-if'ed expressions all over the place...

Done.

Just as an idea: Can this also be set with a configuration file or via environment variables?

dothebart
dothebart approved these changes Mar 29, 2022
View reviewed changes
Copy link
Contributor
@dothebart dothebart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please fix this one condition, else LGTM.

@adityamukho
Copy link
Contributor Author

@hkernbach In reply to
#16021 (comment)

Yes it could be picked up from a config file for sure. We could read overrides from a file in etc/testing. In every case, the priority for any option specified multiple times would be set as follows:

  1. option specified as a command line arg.
  2. option specified as a override in a config file.
  3. options specified as default in testing.js.

@mchacki mchacki merged commit a752455 into devel Mar 31, 2022
@mchacki mchacki deleted the refactor/test-framework branch March 31, 2022 09:35
@mchacki
Copy link
Member
mchacki commented Mar 31, 2022

The reference test has been aborted, and aborted sub-jobs have been restarted.
I ahne validated that Main Job + Restarted sub-Jobs together all have been green, even though the linked PR is declared red here.

dothebart added a commit that referenced this pull request Mar 31, 2022
@dothebart
re-apply changes from #16021
74ea805
dothebart added a commit that referenced this pull request Apr 4, 2022
@dothebart
re-apply changes from #16021
0cbc104
KVS85 pushed a commit that referenced this pull request Apr 5, 2022
@dothebart @jsteemann
Feature/refactor testrunner (#15998)
fdc51c7 
* add separate list of testfiles

* start modularizing the testrunner function into a deriveable object

* start modularizing the testrunner function into a deriveable object

* Update js/client/modules/@arangodb/testutils/testrunner.js

Co-authored-by: Jan <jsteemann@users.noreply.github.com>

* convert more testsuites to the new logic

* properly convert the agency runner

* modify audit test

* convert auth tests

* convert chaos

* adjust communication suite

* convert more tests

* lint

* fix testcase path

* cleanup, convert two missing suites

* endpoints suite

* disable graph checks for backup suite

* lint

* documentation

* implement permissions test

* fix database to be used for checking

* one last convert

* lint

* re-apply changes from #16021

* Update js/client/modules/@arangodb/testutils/testrunner.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/backup.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/backup.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/communication.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/go.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/go.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/rta_makedata.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/rta_makedata.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/endpoints.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/endpoints.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/go.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/drivers.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/go.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/drivers.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testutils/test-utils.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testutils/test-utils.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testutils/test-utils.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/drivers.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/js.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/php.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/rta_makedata.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/go.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/java.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/endpoints.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/export.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/importing.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/permissions_client.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/server_permissions.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testutils/testrunner.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* Update js/client/modules/@arangodb/testsuites/replication.js

Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>

* disable bypass

* add check whether arangods exited early during startup

* move haxxman string into one common place

* fix typo, bucket splitting

* fix authentification setting in restart

Co-authored-by: Jan <jsteemann@users.noreply.github.com>
Co-authored-by: Aditya Mukhopadhyay <aditya@arangodb.com>
@goedderz goedderz mentioned this pull request May 20, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dothebart dothebart dothebart approved these changes

@hkernbach hkernbach Awaiting requested review from hkernbach

Assignees

@mchacki mchacki

Labels
1 Feature 3 - Testing
Projects
None yet
Milestone
3.10
Development

Successfully merging this pull request may close these issues.
4 participants
@adityamukho @dothebart @hkernbach @mchacki
0