10000 [DOCS-10234] Test Health (new page) (#28926) · DataDog/documentation@6de2e85 · GitHub
[go: up one dir, main page]

Skip to content

Commit 6de2e85

Browse files
[DOCS-10234] Test Health (new page) (#28926)
* First draft * More edits, convert to tables, add further reading * Add screenshot * Address some Vale suggestions * Apply suggestions from Jen's review Co-authored-by: Jen Gilbert <jen.gilbert@datadoghq.com> --------- Co-authored-by: Jen Gilbert <jen.gilbert@datadoghq.com>
1 parent 6ecff12 commit 6de2e85

File tree

3 files changed

+132
-21
lines changed

3 files changed

+132
-21
lines changed

config/_default/menus/main.en.yaml

Lines changed: 26 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -4770,111 +4770,116 @@ menu:
47704770
parent: tests
47714771
identifier: tests_monitors
47724772
weight: 6
4773+
- name: Test Health
4774+
url: tests/test_health
4775+
parent: tests
4776+
identifier: tests_test_health
4777+
weight: 7
47734778
- name: Flaky Test Management
47744779
url: tests/flaky_test_management
47754780
parent: tests
47764781
identifier: tests_flaky_test_management
4777-
weight: 7
4782+
weight: 8
47784783
- name: Early Flake Detection
47794784
url: tests/flaky_test_management/early_flake_detection
47804785
parent: tests_flaky_test_management
47814786
identifier: tests_early_flake_detection
4782-
weight: 701
4787+
weight: 801
47834788
- name: Auto Test Retries
47844789
url: tests/flaky_test_management/auto_test_retries
47854790
parent: tests_flaky_test_management
47864791
identifier: tests_auto_test_retries
4787-
weight: 702
4792+
weight: 802
47884793
- name: Test Impact Analysis
47894794
url: tests/test_impact_analysis
47904795
parent: tests
47914796
identifier: test_impact_analysis
4792-
weight: 8
4797+
weight: 9
47934798
- name: Setup
47944799
url: tests/test_impact_analysis/setup/
47954800
parent: test_impact_analysis
47964801
identifier: test_impact_analysis_setup
4797-
weight: 801
4802+
weight: 901
47984803
- name: .NET
47994804
url: tests/test_impact_analysis/setup/dotnet/
48004805
parent: test_impact_analysis_setup
48014806
identifier: test_impact_analysis_setup_dotnet
4802-
weight: 8101
4807+
weight: 9101
48034808
- name: JavaScript and TypeScript
48044809
url: tests/test_impact_analysis/setup/javascript/
48054810
parent: test_impact_analysis_setup
48064811
identifier: test_impact_analysis_setup_javascript
4807-
weight: 8102
4812+
weight: 9102
48084813
- name: Python
48094814
url: tests/test_impact_analysis/setup/python/
48104815
parent: test_impact_analysis_setup
48114816
identifier: test_impact_analysis_setup_python
4812-
weight: 8103
4817+
weight: 9103
48134818
- name: Swift
48144819
url: tests/test_impact_analysis/setup/swift/
48154820
parent: test_impact_analysis_setup
48164821
identifier: test_impact_analysis_setup_swift
4817-
weight: 8104
4822+
weight: 9104
48184823
- name: Java
48194824
url: tests/test_impact_analysis/setup/java/
48204825
parent: test_impact_analysis_setup
48214826
identifier: test_impact_analysis_setup_java
4822-
weight: 8105
4827+
weight: 9105
48234828
- name: Ruby
48244829
url: tests/test_impact_analysis/setup/ruby/
48254830
parent: test_impact_analysis_setup
48264831
identifier: test_impact_analysis_setup_ruby
4827-
weight: 8106
4832+
weight: 9106
48284833
- name: Go
48294834
url: tests/test_impact_analysis/setup/go/
48304835
parent: test_impact_analysis_setup
48314836
identifier: test_impact_analysis_setup_go
4832-
weight: 8107
4837+
weight: 9107
48334838
- name: How It Works
48344839
url: tests/test_impact_analysis/how_it_works/
48354840
parent: test_impact_analysis
48364841
identifier: test_impact_analysis_how_it_works
4837-
weight: 802
4842+
weight: 902
48384843
- name: Troubleshooting
48394844
url: tests/test_impact_analysis/troubleshooting/
48404845
parent: test_impact_analysis
48414846
identifier: test_impact_analysis_troubleshooting
4842-
weight: 803
4847+
weight: 903
48434848
- name: Developer Workflows
48444849
url: tests/developer_workflows
48454850
parent: tests
48464851
identifier: tests_developer_workflows
4847-
weight: 9
4852+
weight: 10
48484853
- name: Code Coverage
48494854
url: tests/code_coverage
48504855
parent: tests
48514856
identifier: tests_code_coverage
4852-
weight: 10
4857+
weight: 11
48534858
- name: Instrument Browser Tests with RUM
48544859
url: tests/browser_tests
48554860
parent: tests
48564861
identifier: tests_browser_tests
4857-
weight: 11
4862+
weight: 12
48584863
- name: Instrument Swift Tests with RUM
48594864
url: tests/swift_tests
48604865
parent: tests
48614866
identifier: tests_swift_tests
4862-
weight: 12
4867+
weight: 13
48634868
- name: Correlate Logs and Tests
48644869
url: tests/correlate_logs_and_tests
48654870
parent: tests
48664871
identifier: tests_correlate_logs_and_tests
4867-
weight: 13
4872+
weight: 14
48684873
- name: Guides
48694874
url: tests/guides/
48704875
parent: tests
48714876
identifier: tests_guides
4872-
weight: 14
4877+
weight: 15
48734878
- name: Troubleshooting
48744879
url: tests/troubleshooting/
48754880
parent: tests
48764881
identifier: tests_troubleshooting
4877-
weight: 15
4882+
weight: 16
48784883
- name: Quality Gates
48794884
url: quality_gates/
48804885
pre: ci

content/en/tests/test_health.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
---
2+
title: Test Health
3+
description: "Measure the impact of flaky tests and how Test Optimization improves CI usage."
4+
further_reading:
5+
- link: "tests/flaky_test_management"
6+
tag: "Documentation"
7+
text: "Learn about managing flaky tests"
8+
- link: "tests/flaky_test_management/auto_test_retries"
9+
tag: "Documentation"
10+
text: "Learn about Auto Test Retries"
11+
- link: "tests/test_impact_analysis"
12+
tag: "Documentation"
13+
text: "Learn about Test Impact Analysis"
14+
- link: "tests/flaky_test_management/early_flake_detection"
15+
tag: "Documentation"
16+
text: "Learn about Early Flake Detection"
17+
- link: "quality_gates"
18+
tag: "Documentation"
19+
text: "Learn about Quality Gates"
20+
---
21+
22+
## Overview
23+
24+
The [Test Health][5] dashboard provides analytics to help teams manage and optimize their testing in CI. This includes sections showing the current impact of test flakiness and how Test Optimization is mitigating these problems.
25+
26+
{{< img src="tests/test-health.png" alt="Test Health dashboard" style="width:100%;" >}}
27+
28+
## Summary metrics
29+
30+
Based on the current time frame and filters applied, the dashboard highlights the following key metrics:
31+
32+
- [**Pipelines Failed**](#pipelines-failed): Sum total of pipelines that failed due to flaky tests
33+
- [**Time Wasted in CI**](#time-wasted-in-ci): Total time spent in CI due to flaky tests
34+
- [**Pipelines Saved**](#pipelines-saved): How many pipelines were prevented from failing by Auto Test Retries
35+
- [**Time Saved in CI**](#time-saved-in-ci): How much time has been saved by Test Impact Analysis and Auto Test Retries
36+
37+
### Pipelines Failed
38+
39+
This table provides details on pipeline executions, failures, and their impact on developer experience.
40+
41+
| Metric | Description |
42+
|--------|-------------|
43+
| **Pipeline Executions with Tests** | Number of pipeline executions with one or more test sessions. |
44+
| **Failures Due to Flaky Tests 10000 ** | Number of pipeline executions that failed solely due to flaky tests. All tests that failed have one or more of the following tags: `@test.is_known_flaky` or `@test.is_new_flaky`. |
45+
| **Failures Due to Non-Flaky Tests** | Number of pipeline executions that failed due to tests without any flakiness. None of the failing tests have any of the following tags: `@test.is_known_flaky`, `@test.is_new_flaky`, and `@test.is_flaky`. |
46+
| **Dev Experience - Test Failure Breakdown** | Ratio of flaky to non-flaky test failures. When pipelines fail due to tests, how often is it a flaky test? A higher ratio of flaky test failures erodes trust in test results. Developers may stop paying attention to failing tests, assume they're flakes, and manually retry. |
47+
48+
### Time Wasted in CI
49+
50+
This table provides details on testing time, time lost due to failures, and the impact on developer experience.
51+
52+
| Metric | Description |
53+
|--------|-------------|
54+
| **Total Testing Time** | Sum of the duration of all test sessions. |
55+
| **Time Lost Due to Flaky Tests** | Total duration of test sessions that failed solely due to flaky tests. All tests that failed have one or more of the following tags: `@test.is_known_flaky`, `@test.is_new_flaky`, or `@test.is_flaky`. |
56+
| **Time Lost Due to Non-Flaky Tests** | Total duration of test sessions that failed due to tests without any flakiness. All tests that failed do not have any of the following tags: `@test.is_known_flaky`, `@test.is_new_flaky`, and `@test.is_flaky`. |
57+
| **Dev Experience - Time Lost Breakdown** | Ratio of time lost due to flaky vs. non-flaky test failures. When you lose time due to tests, how much is due to flaky tests? A higher ratio of time lost to flaky test failures leads to developer frustration. |
58+
59+
### Pipelines Saved
60+
61+
This table shows how many pipelines [Auto Test Retries][1] have prevented from failing.
62+
63+
<div class="alert alert-info">These metrics include test sessions in which a flaky test failed, and then was automatically retried and passed. Newer versions of test libraries provide more accurate results, as they include more precise telemetry regarding test retries.</div>
64+
65+
| Metric | Description |
66+
|--------|-------------|
67+
| **Pipeline Executions with Tests** | Number of pipeline executions with one or more test sessions. |
68+
| **Saved by Auto Test Retries** | Number of CI pipelines with passed test sessions containing tests with `@test.is_retry:true` and `@test.is_new:false`. |
69+
70+
### Time Saved in CI
71+
72+
This table shows how much CI usage time [Test Impact Analysis][4] and [Auto Test Retries][1] have saved.
73+
74+
| Metric | Description |
75+
|--------|-------------|
76+
| **Total Testing Time** | Sum of the duration of all test sessions. |
77+
| **Total Time Saved** | Sum of time saved by Test Impact Analysis and Auto Test Retries. **% of Testing Time** is the percentage of time saved out of total testing time. Total time saved can exceed total testing time if you prevent a lot of unnecessary pipeline and job retries. |
78+
| **Saved by Test Impact Analysis** | Total duration indicated by `@test_session.itr.time_saved`. |
79+
| **Saved by Auto Test Retries** | Total duration of passed test sessions in which some tests initially failed but later passed due to Auto Test Retries. These tests are tagged with `@test.is_retry:true` and `@test.is_new:false`. |
80+
81+
## Use cases
82+
83+
### Enhance developer experience
84+
Use **Dev Experience - Test Failure Breakdown** and **Dev Experience - Time Lost Breakdown** to identify how often flaky tests in particular cause failures and waste CI time.
85+
86+
These Test Optimization features improve developer experience by reducing test failures and wasted time:
87+
- **[Auto Test Retries][1]** reduces the likelihood a flaky test causes a pipeline to fail. This includes your known flaky tests and flaky tests that have yet to be identified. This also provides developers with confidence in test results when a test is actually broken, as it will have failed all retries.
88+
- **[Early Flake Detection][2]**, combined with **[Quality Gates][3]**, prevents new tests that are flaky from entering your default branch.
89+
- **[Test Impact Analysis][4]** minimizes the flaky tests that run, by only running relevant tests based on code coverage. Skipping irrelevant tests also shortens the feedback loop for developers.
90+
91+
### Maximize pipeline efficiency and reduce costs
92+
Lengthy test suites slow down feedback loops to developers, and running irrelevant tests incurs unnecessary costs.
93+
94+
These Test Optimization features help you save CI time and costs:
95+
- **[Auto Test Retries][1]**: If a single flaky test fails during your session, the entire duration of the CI job is lost. Auto Test Retries allow flaky tests to rerun, increasing the likelihood of passing.
96+
- **[Test Impact Analysis][4]**: By running only tests relevant to your code changes, you reduce the overall duration of the test session. This also prevents pipelines from failing due to unrelated flaky tests if you skip them.
97+
98+
## Further reading
99+
100+
{{< partial name="whats-next/whats-next.html" >}}
101+
102+
[1]: /tests/flaky_test_management/auto_test_retries/
103+
[2]: /tests/flaky_test_management/early_flake_detection/
104+
[3]: /quality_gates/
105+
[4]: /tests/test_impact_analysis/
106+
[5]: https://app.datadoghq.com/ci/test/health

static/images/tests/test-health.png

1000 KB
Loading

0 commit comments

Comments
 (0)
0