8000 conformance: add Hook in ConformanceTestSuite by zirain · Pull Request #3786 · kubernetes-sigs/gateway-api · GitHub
[go: up one dir, main page]
Skip to content

conformance: add Hook in ConformanceTestSuite #3786

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
May 27, 2025
Merged

conformance: add Hook in ConformanceTestSuite #3786

merged 3 commits into from
May 27, 2025

Conversation

zirain
Copy link
Contributor
@zirain zirain commented May 11, 2025

What type of PR is this?

/kind feature
/kind test
/area conformance-test

What this PR does / why we need it:

Add hook points support in conformance test suite

Which issue(s) this PR fixes:

Fixes #3131

Does this PR introduce a user-facing change?:

Add hook points support in conformance test suite

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/test area/conformance-test Issues or PRs related to Conformance tests. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 11, 2025
@k8s-ci-robot k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label May 11, 2025
@arkodg
Copy link
Contributor
arkodg commented May 12, 2025

/approve thanks !

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: arkodg, zirain

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 12, 2025
@zirain zirain mentioned this pull request May 12, 2025
Closed
@youngnick
Copy link
Contributor

I think this is okay, as it only runs after the test has been completed. It's super important that we don't allow hooks before, as we don't want the behavior of the conformance tests to be able to be modified (or else we can't be sure that the same behavior is executed for every implementation, defeating the point of conformance).

I won't LGTM though, as I'd like @mlavacca to confirm he's okay with this. Holding for his review.

/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 12, 2025
@zirain
Copy link
Contributor Author
zirain commented May 12, 2025

I want to clearify that, we want this to run dump function for debugging a flaky test.
That's why this was only added to post-test.

@robscott
Copy link
Member

Thanks @zirain!

/lgtm

FYI @spencerhance @SinaChavoshi in case this is helpful on inference extension tests

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 14, 2025
@robscott
Copy link
Member

I think this is okay, as it only runs after the test has been completed. It's super important that we don't allow hooks before, as we don't want the behavior of the conformance tests to be able to be modified (or else we can't be sure that the same behavior is executed for every implementation, defeating the point of conformance).

I won't LGTM though, as I'd like @mlavacca to confirm he's okay with this. Holding for his review.

/hold

Missed this, will leave the hold in place for @mlavacca.

@zirain
Copy link
Contributor Author
zirain commented May 17, 2025

@mlavacca can you take a look?

LiorLieberman
LiorLieberman reviewed May 24, 2025
View reviewed changes
Copy link
Member
@LiorLieberman LiorLieberman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @zirain lgtm as well with a tiny nit.

conformance/utils/suite/suite.go
// call the hook function if it was provided,
// this's useful for running custom logic after each test,
// such as collecting current state of the cluster for debugging.
if suite.Hook != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: is it only useful on the suite level, meaning it will run the same hook for each test? or there is value in doing it on a more granular level like tests/profile/supportedFeature etc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it's for suite level.
For specified test, you can use t.Cleanup()

@zirain
update comment
a4456ae 
Signed-off-by: zirain <zirain2009@gmail.com>
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 25, 2025
@LiorLieberman
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 25, 2025
@zirain
Copy link
Contributor Author
zirain commented May 26, 2025

@youngnick can we remove the hold?

@youngnick
Copy link
Contributor

I'd still like @mlavacca to take a look, I'll ping him.

mlavacca
mlavacca reviewed May 26, 2025
View reviewed changes
Copy link
Member
@mlavacca mlavacca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR, @zirain! What's missing here is the value in the Options struct to build a suite with a hook value by using the NewConformanceTestSuite helper.

Apart from that, the PR looks good to me 👍🏻

@zirain
add Hook in ConformanceOptions
80c24e6 
Signed-off-by: zirain <zirain2009@gmail.com>
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 26, 2025
@zirain zirain requested a review from mlavacca May 26, 2025 23:24
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 26, 2025
mlavacca
mlavacca reviewed May 27, 2025
View reviewed changes
Copy link
Member
@mlavacca mlavacca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @zirain!

/lgtm
/unhold

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 27, 2025
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 27, 2025
@k8s-ci-robot k8s-ci-robot merged commit ce0cfeb into kubernetes-sigs:main May 27, 2025
13 checks passed
@zirain zirain deleted the suite-hook branch May 27, 2025 10:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mlavacca mlavacca mlavacca left review comments

@mikemorris mikemorris Awaiting requested review from mikemorris

@sunjayBhatia sunjayBhatia Awaiting requested review from sunjayBhatia

+1 more reviewer

@LiorLieberman LiorLieberman LiorLieberman left review comments

Reviewers whose approvals may not affect merge requirements
Assignees

@robscott robscott

@mlavacca mlavacca

@LiorLieberman LiorLieberman

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/conformance-test Issues or PRs related to Conformance tests. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. kind/test lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add hook points support in conformance test suite
7 participants
@zirain @arkodg @k8s-ci-robot @youngnick @robscott @LiorLieberman @mlavacca
0