8000 Config fuzzer by exclamaforte · Pull Request #139736 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

Config fuzzer #139736

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 11 commits into from
Closed

Config fuzzer #139736

wants to merge 11 commits into from

Conversation

exclamaforte
Copy link
Contributor
@exclamaforte exclamaforte commented Nov 5, 2024 •
edited
Loading

This tool makes it easy to search through config state-space with a minimal reproduction or test. It presents a similar interface to the config bisector by taking a test_function that should either raise on Exception or return False upon failure.

It has two entry points: fuzz_n_tuple, which tries every combination of n configs, and bisect, which randomly flips configs and tries to find the minimal reproduction upon failure. bisect is a much more efficient way to search the space, but fuzz_n_tuple can give you peace of mind that a new config will compose with every other config.

It's been used to find three bugs so far in the inductor config:
#140220 #140219
#143524

This PR also adds a bunch of missing types to the inductor config to get them to play nice with the fuzzer, so it can be a good forcing function for adding types to config.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @aakhundov

@pytorch-bot PyTorch Bot
Copy link
pytorch-bot bot commented Nov 5, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/139736

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 1 Pending

As of commit 6dde38c with merge base fd382f1 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@exclamaforte exclamaforte force-pushed the exclamaforte/config-fuzzer-1 branch 2 times, most recently from b93f94a to 2812500 Compare November 5, 2024 22:11
@exclamaforte exclamaforte mentioned this pull request Nov 6, 2024
Closed
@exclamaforte exclamaforte force-pushed the exclamaforte/config-fuzzer-1 branch 2 times, most recently from 63de493 to 543da65 Compare November 6, 2024 03:09
@c00w c00w requested review from c00w, jansel, Chillee and eellison November 6, 2024 20:28
@exclamaforte exclamaforte force-pushed the exclamaforte/config-fuzzer-1 branch 6 times, most recently from 8a89a60 to 3d45863 Compare November 6, 2024 23:10
@exclamaforte exclamaforte force-pushed the exclamaforte/config-fuzzer-1 branch 2 times, most recently from 7f66a53 to 50619b4 Compare November 7, 2024 00:16
pytorchmergebot pushed a commit that referenced this pull request Nov 7, 2024
@exclamaforte @pytorchmergebot
Add type annotations to Configs (#139833)
2037ea3 
Summary:
Adds types to Configs, and fixes a bug in options that was caused by the lack of types.

fixes: #139822

Configs are used by many modules so not sure which label to put.

Types also allow #139736 to fuzz configs

Pull Request resolved: #139833
Approved by: https://github.com/c00w
@exclamaforte exclamaforte force-pushed the exclamaforte/config-fuzzer-1 branch 5 times, most recently from ef24f3f to 234d251 Compare November 10, 2024 20:47
@exclamaforte exclamaforte force-pushed the exclamaforte/config-fuzzer-1 branch from fa11c3d to f87cea5 Compare December 18, 2024 23:32
@exclamaforte
Copy link
Contributor Author

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 18, 2024
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: 2 mandatory check(s) failed. The first few are:

Dig deeper by viewing the failures on hud

Details for Dev Infra team Raised by workflow job

Failing merge rule: Core Maintainers

@exclamaforte
Copy link 8000
Contributor Author

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

@exclamaforte
Config Fuzzer/Bisector
411c2ea 
This tool makes it easy to search through config state-space with a minimal reproduction or test. It presents a similar interface to the config bisector by taking a test_function that should either raise on Exception or return False upon failure.

It has two entry points: fuzz_n_tuple, which tries every combination of n configs, and fuzz_with_bisect, which randomly flips configs and tries to find the minimal reproduction upon failure. fuzz_with_bisect is a much more efficient way to search the space, but fuzz_n_tuple can give you peace of mind that a new config will compose with every other config.

It's been used to find two bugs so far in the inductor config:

This PR also adds a bunch of missing types to the inductor config to get them to play nice with the fuzzer, so it can be a good forcing function for adding types to config.
@exclamaforte
fix set linter
ca22e43 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@exclamaforte
refactor error msg
a7d57d8 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@exclamaforte
fix bugs add tests
dea4655 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@exclamaforte
improvements
6c72618 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@exclamaforte
improvements
81df0f8 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@exclamaforte
num_ran
770461e 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@exclamaforte
small improvements
57fe8dd 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@exclamaforte
fix a merge race
8000
7894b1a 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@exclamaforte
fix and skip 3.9
658cfd1 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@exclamaforte exclamaforte force-pushed the exclamaforte/config-fuzzer-1 branch from 0bcb9d2 to 658cfd1 Compare January 12, 2025 19:35
@exclamaforte
lint
6dde38c 
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@exclamaforte
Copy link
Contributor Author

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@github-actions github-actions bot deleted the exclamaforte/config-fuzzer-1 branch February 12, 2025 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eellison eellison eellison approved these changes

Assignees
No one assigned
Labels
ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged module: inductor release notes: inductor
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@exclamaforte @eellison @pytorchmergebot
0