CI: WebDriver test on try #37498
CI: WebDriver test on try #37498
Conversation
|
Example of
|
There was a problem hiding this comment.
I think it would be better to reuse existing functionality for this, something like I done for vello: 794e568#diff-e5a800a414ff1b0236f85f7e56dbf6a8906198a9f49ef7436026b8188a7935d7
This way you would only need to set wpt_args to ./tests/wpt/tests/webdriver/tests/classic/ --pref=dom_testing_html_input_element_select_files_enabled --product servodriver and reduce number_of_wpt_chunks to ~2
Or am I missing something?
Make sense, I think you are right. Let me think of a proper way. |
|
@sagudev On second thought, we also need |
Ah, normal driver uses |
Thanks, that's interesting. |
If it's okay, do you know the difference between servodriver --headless and servo --headless? I see that webdriver test can run without servodriver. However, I chose them since a lot of interaction test require servodriver, and I think it's more proper to test webdriver command using servodriver. I am a bit confused on understanding how they both differ in the proccess. |
|
I think just servodrived uses webdriver, other runner does not support webdriver stuff AFAIK. I think it should be ok to run webdriver via servodriver using headless too. |
The servo test executor supports webdriver conformance tests ("wdspec") by launching instances of Servo that have webdriver enabled for only those particular tests. Since wdspec tests are a special test type, this is not turned on for other test types which is why testdriver.js only works in the servodriver executor. |
|
I see, thanks for both of you. |
.github/workflows/linux-wpt.yml
Outdated
| --${{ inputs.profile }} \ | ||
| --processes 1 \ | ||
| --timeout-multiplier 2 \ | ||
| --always-succeed \ |
There was a problem hiding this comment.
A bit change on the plan. Here we include the webdriver workflow when merging/pushing. However, it is not required to be passed and only to track the progression/regression of webdriver command caused by the PR. Not all PR attached try result, so I think this makes sense. Is this fine @jdm @sagudev ?
Also because we want to run both wpt and webdriver test, for now I keep the duplication of job for webdriver (I don't know how to run both of them with just changing the wpt argument, feel free to give suggestion).
There was a problem hiding this comment.
My suggestion is to do it as we do it for webgpu, where we do not run webgpu tests unless explicitly requested by the PR (via T-webgpu label).
I don't know how to run both of them with just changing the wpt argument, feel free to give suggestion
I will try to whip out PR tomorrow.
There was a problem hiding this comment.
This is what I was having in my mind: main...sagudev:servo:wd-ci, but as you discovered it does have problem when running both wpt and webdriver tests due to artifact name collision, but I think we should fix this in our CI architecture more generically (as webgpu has same problem). I was sure I opened issue about this somewhere before ...
|
Hmm, right now I have managed to reuse the linux-wpt jobs in the last commit. If you have time you can take a look and tell me what you think @sagudev. Thanks in advance. The reporting is also splitted between normal wpt and webdriver, like https://github.com/PotatoCP/servo/actions/runs/15772380811/job/44460453140. |
Signed-off-by: PotatoCP <kenzieradityatirtarahardja18@gmail.com> Co-authored-by: sagudev <16504129+sagudev@users.noreply.github.com>
|
@sagudev For now I just follow your instruction and code. Later we can improve it again when we have fix the CI architecture more generally. |
|
This is now also available as label: T-webdriver |
Add webdriver test on the workflow with
--product servodriver. This will make tracking progression/regression in webdriver development easier.However, since webdriver test is still unstable, the webdriver test is only enabled on try.
To run try:
./mach try wd,./mach try webdriverTesting: This PR add webdriver test on try