8000 Fix continuous integration tests across incompatible editor versions … · Unity-Technologies/ml-agents@93221d4 · GitHub
[go: up one dir, main page]
Skip to content

Commit 93221d4

Browse files
alex-mccarthy-unityalex-mccarthy-unity
authored
Fix continuous integration tests across incompatible editor versions (2022 and trunk) (#6079)
* Fix com.unity.ml-agents test 2022.3 on win by upgrading Unity versions Run all CI tests against 2023.2 (required by Sentis), not 2022.3 Sample failing sub-jobs before: https://unity-ci.cds.internal.unity3d.com/job/35022030/dependency-graph Sample passing sub-jobs after: https://unity-ci.cds.internal.unity3d.com/job/35033024/dependency-graph Note that `trunk` jobs are still failing after this fix. Those will be investigated separately since they've been failing since March 8: https://unity-ci.cds.internal.unity3d.com/job/34919178/dependency-graph * Ignore yamato-parser output files * Disable `trunk` tests, which break with Unity 6 Clean run of "Run All Combinations of Editors/Platforms Tests" after this change: https://unity-ci.cds.internal.unity3d.com/job/35037130/dependency-graph * Print full diffs when a sensor mismatch occurs in tests * Refactor tests to use positions instead of hardcoded numbers * Disable `RigidBodySensorTests.TestBodiesWithJoint` which fails in 2023.2 * Fix editor version in test_versions.metafile (use 2023.2)
1 parent c4dc0a2 commit 93221d4

14 files changed

+53
-40
lines changed

‎.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,3 +95,6 @@ coverage.xml
9595
/htmlcov/
9696

9797
**/UserSettings/*
98+
99+
# yamato-parser temporary output files:
100+
.yamato/unfolded/

‎.yamato/com.unity.ml-agents-coverage.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
{% for platform in coverage_test_platforms %}
66
{% capture coverageOptions %} --enable-code-coverage --code-coverage-options 'generateHtmlReport;assemblyFilters:+{{ package.assembly }}'{% endcapture %}
77

8-
test_coverage_{{ package.name }}_{{ platform.name }}_{{ editor.version }}:
9-
name : Coverage {{ package.name }} test {{ editor.version }} on {{ platform.name }}
8+
test_coverage_{{ package.name }}_{{ platform.name }}_{{ editor.version }}_{{ editor.testProject }}:
9+
name : Coverage {{ package.name }} test {{ editor.version }} on {{ platform.name }} in {{ editor.testProject }}
1010
agent:
1111
type: {{ platform.type }}
1212
image: {{ platform.image }}

‎.yamato/com.unity.ml-agents-performance.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
test_editors:
2-
- version: 2022.3
32
- version: 2023.2
43
---
54
{% for editor in test_editors %}

‎.yamato/com.unity.ml-agents-promotion.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
test_editors:
2-
- version: 2022.3
32
- version: 2023.2
43
test_platforms:
54
- name: win

‎.yamato/com.unity.ml-agents-test.yml

Lines changed: 15 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{% metadata_file .yamato/coverage_tests.metafile %}
22
test_editors:
3-
- version: 2022.3
3+
- version: 2023.2
44
# We want some scene tests to run in the DevProject, but packages there only support 2020+
55
testProject: Project
66
enableNoDefaultPackages: !!bool true
@@ -9,9 +9,11 @@ test_editors:
99
enableNoDefaultPackages: !!bool true
1010

1111
trunk_editor:
12-
- version: trunk
13-
# Workaround for MLA-1596 - need to make sure we load the right results.
14-
testProject: DevProject
12+
# TODO: re-enable trunk tests after adding support for Unity 6. As of 12 Mar 2024, tests fail
13+
# with "Error: '6000.0.0b12' is not a valid Unity version"
14+
# - version: trunk
15+
# # Workaround for MLA-1596 - need to make sure we load the right results.
16+
# testProject: DevProject
1517

1618
test_platforms:
1719
- name: win
@@ -42,18 +44,18 @@ all_package_tests:
4244
{% for coverage_editor in coverage_test_editors %}
4345
{% for coverage_platform in coverage_test_platforms %}
4446
{% for coverage_package in coverage_test_packages %}
45-
- .yamato/com.unity.ml-agents-coverage.yml#test_coverage_{{ coverage_package.name }}_{{ coverage_platform.name }}_{{ coverage_editor.version }}
47+
- .yamato/com.unity.ml-agents-coverage.yml#test_coverage_{{ coverage_package.name }}_{{ coverage_platform.name }}_{{ coverage_editor.version }}_{{ coverage_editor.testProject }}
4648
{% endfor %}
4749
{% endfor %}
4850
{% endfor %}
4951
{% for editor in test_editors %}
5052
{% for platform in test_platforms %}
51-
- .yamato/com.unity.ml-agents-test.yml#test_com.unity.ml-agents_{{ platform.name }}_{{ editor.version }}
53+
- .yamato/com.unity.ml-agents-test.yml#test_com.unity.ml-agents_{{ platform.name }}_{{ editor.version }}_{{ editor.testProject }}
5254
{% endfor %}
5355
{% endfor %}
5456
{% for editor in trunk_editor %}
5557
{% for platform in test_platforms %}
56-
- .yamato/com.unity.ml-agents-test.yml#test_com.unity.ml-agents_{{ platform.name }}_{{ editor.version }}
58+
- .yamato/com.unity.ml-agents-test.yml#test_com.unity.ml-agents_{{ platform.name }}_{{ editor.version }}_{{ editor.testProject }}
5759
{% endfor %}
5860
{% endfor %}
5961
triggers:
@@ -72,8 +74,8 @@ all_package_tests:
72< F438 /td>74
{% assign noDefaultPackagesOptions = "" %}
7375
{% endif %}
7476

75-
test_{{ package.name }}_{{ platform.name }}_{{ editor.version }}:
76-
name : {{ package.name }} test {{ editor.version }} on {{ platform.name }}
77+
test_{{ package.name }}_{{ platform.name }}_{{ editor.version }}_{{ editor.testProject }}:
78+
name : {{ package.name }} test {{ editor.version }} on {{ platform.name }} in {{ editor.testProject }}
7779
agent:
7880
type: {{ platform.type }}
7981
image: {{ platform.image }}
@@ -90,7 +92,7 @@ test_{{ package.name }}_{{ platform.name }}_{{ editor.version }}:
9092
{% for coverage_editor in coverage_test_editors %}
9193
{% for coverage_platform in coverage_test_platforms %}
9294
{% for coverage_package in coverage_test_packages %}
93-
- .yamato/com.unity.ml-agents-coverage.yml#test_coverage_{{ coverage_package.name }}_{{ coverage_platform.name }}_{{ coverage_editor.version }}
95+
- .yamato/com.unity.ml-agents-coverage.yml#test_coverage_{{ coverage_package.name }}_{{ coverage_platform.name }}_{{ coverage_editor.version }}_{{ coverage_editor.testProject }}
9496
{% endfor %}
9597
{% endfor %}
9698
{% endfor %}
@@ -117,8 +119,8 @@ test_{{ package.name }}_{{ platform.name }}_{{ editor.version }}:
117119
{% for editor in trunk_editor %}
118120
{% for platform in test_platforms %}
119121

120-
test_{{ package.name }}_{{ platform.name }}_trunk:
121-
name : {{ package.name }} test {{ editor.version }} on {{ platform.name }}
122+
test_{{ package.name }}_{{ platform.name }}_{{ editor.version}}_{{ editor.testProject }}:
123+
name : {{ package.name }} test {{ editor.version }} on {{ platform.name }} in {{ editor.testProject }}
122124
agent:
123125
type: {{ platform.type }}
124126
image: {{ platform.image }}
@@ -141,7 +143,7 @@ test_{{ package.name }}_{{ platform.name }}_trunk:
141143
{% for coverage_editor in coverage_test_editors %}
142144
{% for coverage_platform in coverage_test_platforms %}
143145
{% for coverage_package in coverage_test_packages %}
144-
- .yamato/com.unity.ml-agents-coverage.yml#test_coverage_{{ coverage_package.name }}_{{ coverage_platform.name }}_{{ coverage_editor.version }}
146+
- .yamato/com.unity.ml-agents-coverage.yml#test_coverage_{{ coverage_package.name }}_{{ coverage_platform.name }}_{{ coverage_editor.version }}_{{ coverage_editor.testProject }}
145147
{% endfor %}
146148
{% endfor %}
147149
{% endfor %}

‎.yamato/compressed-sensor-test.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
{% metadata_file .yamato/test_versions.metafile %}
22
---
33
{% for editor in test_editors %}
4-
test_compressed_obs_{{ editor.version }}:
5-
name: Test Compressed Sensor Observation {{ editor.version }}
4+
test_compressed_obs_{{ editor.version }}_{{ editor.extra_test }}:
5+
name: Test Compressed Sensor Observation {{ editor.version }} {{ editor.extra_test }}
66
agent:
77
type: Unity::VM
88
image: ml-agents/ml-agents-ubuntu-18.04:latest
@@ -18,7 +18,7 @@ test_compressed_obs_{{ editor.version }}:
1818
python ml-agents/tests/yamato/scripts/run_compressed_sensor.py --env=artifacts/testPlayer-TestGridCompressed
1919
python ml-agents/tests/yamato/scripts/run_compressed_sensor.py --env=artifacts/testPlayer-TestTextureCompressed
2020
dependencies:
21-
- .yamato/standalone-build-test.yml#test_linux_standalone_{{ editor.version }}
21+
- .yamato/standalone-build-test.yml#test_linux_standalone_{{ editor.version }}_{{ editor.extra_test }}
2222
triggers:
2323
cancel_old_ci: true
2424
{% if editor.extra_test == "sensor" %}

‎.yamato/gym-interface-test.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
{% metadata_file .yamato/test_versions.metafile %}
22
---
33
{% for editor in test_editors %}
4-
test_gym_interface_{{ editor.version }}:
5-
name: Test Linux Gym Interface {{ editor.version }}
4+
test_gym_interface_{{ editor.version }}_{{ editor.extra_test }}:
5+
name: Test Linux Gym Interface {{ editor.version }} {{ editor.extra_test }}
66
agent:
77
type: Unity::VM
88
image: ml-agents/ml-agents-ubuntu-18.04:latest
@@ -18,7 +18,7 @@ test_gym_interface_{{ editor.version }}:
1818
python -u -m ml-agents.tests.yamato.setup_venv
1919
python ml-agents/tests/yamato/scripts/run_gym.py --env=artifacts/testPlayer-Basic
2020
dependencies:
21-
- .yamato/standalone-build-test.yml#test_linux_standalone_{{ editor.version }}
21+
- .yamato/standalone-build-test.yml#test_linux_standalone_{{ editor.version }}_{{ editor.extra_test }}
2222
triggers:
2323
cancel_old_ci: true
2424
{% if editor.extra_test == "gym" %}

‎.yamato/python-ll-api-test.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
{% metadata_file .yamato/test_versions.metafile %}
22
---
33
{% for editor in test_editors %}
4-
test_linux_ll_api_{{ editor.version }}:
5-
name: Test Linux LL-API {{ editor.version }}
4+
test_linux_ll_api_{{ editor.version }}_{{ editor.extra_test }}:
5+
name: Test Linux LL-API {{ editor.version }} {{ editor.extra_test }}
66
agent:
77
type: Unity::VM
88
image: ml-agents/ml-agents-ubuntu-18.04:latest
@@ -20,7 +20,7 @@ test_linux_ll_api_{{ editor.version }}:
2020
python ml-agents/tests/yamato/scripts/run_llapi.py --env=artifacts/testPlayer-WallJump
2121
python ml-agents/tests/yamato/scripts/run_llapi.py --env=artifacts/testPlayer-Match3
2222
dependencies:
23-
- .yamato/standalone-build-test.yml#test_linux_standalone_{{ editor.version }}
23+
- .yamato/standalone-build-test.yml#test_linux_standalone_{{ editor.version }}_{{ editor.extra_test }}
2424
triggers:
2525
cancel_old_ci: true
2626
{% if editor.extra_test == "llapi" %}

‎.yamato/standalone-build-test.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
{% metadata_file .yamato/test_versions.metafile %}
22
---
33
{% for editor in test_editors %}
4-
test_linux_standalone_{{ editor.version }}:
5-
name: Test Linux Standalone {{ editor.version }}
4+
test_linux_standalone_{{ editor.version }}_{{ editor.extra_test }}:
5+
name: Test Linux Standalone {{ editor.version }} {{ editor.extra_test }}
66
agent:
77
type: Unity::VM
88
image: ml-agents/ml-agents-ubuntu-18.04:latest

‎.yamato/test_versions.metafile

Lines changed: 2 additions & 2 deletions
< 1241 td data-grid-cell-id="diff-421fb31658d140ff0fd69640a7ed70c6b32574c997fee1357f9bf1d7553e4912-5-5-0" data-selected="false" role="gridcell" style="background-color:var(--bgColor-default);text-align:center" tabindex="-1" valign="top" class="focusable-grid-cell diff-line-number position-relative diff-line-number-neutral left-side">5
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
# For each "other" test, we only run it against a single version of the
44
# editor to reduce the number of yamato jobs
5
test_editors:
6-
- version: 2022.3
6+
- version: 2023.2
77
extra_test: gym
8-
- version: 2023.3
8+
- version: 2023.2
99
extra_test: sensor
1010
- version: trunk
1111
extra_test: llapi

0 commit comments

Comments
 (0)
0