[cutlass backend] Reduce log level for cutlass compilation error #153397
Conversation
Differential Revision: [D74596410](https://our.internmc.facebook.com/intern/diff/D74596410/) [ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/153397
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ✅ No FailuresAs of commit bf99f95 with merge base f7798d8 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Differential Revision: [D74596410](https://our.internmc.facebook.com/intern/diff/D74596410/) ghstack-source-id: 283491682 Pull Request resolved: #153397
|
This pull request was exported from Phabricator. Differential Revision: D74596410 |
torch/_inductor/select_algorithm.py
Outdated
| futures[future], CUDATemplateCaller | ||
| ): | ||
| log.debug( | ||
| "Exception %s for benchmark choice %s", e, futures[future] |
There was a problem hiding this comment.
| "Exception %s for benchmark choice %s", e, futures[future] | |
| "Exception %s for benchmark choice %s", e, futures[future], exc_info=True |
Will ensure the same level of detail is printed as log.error
… error" Differential Revision: [D74596410](https://our.internmc.facebook.com/intern/diff/D74596410/) cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov [ghstack-poisoned]
Pull Request resolved: #153397 Differential Revision: [D74596410](https://our.internmc.facebook.com/intern/diff/D74596410/) ghstack-source-id: 283505757
|
This pull request was exported from Phabricator. Differential Revision: D74596410 |
… error" Differential Revision: [D74596410](https://our.internmc.facebook.com/intern/diff/D74596410/) cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov [ghstack-poisoned]
Pull Request resolved: #153397 Differential Revision: [D74596410](https://our.internmc.facebook.com/intern/diff/D74596410/) ghstack-source-id: 283506277
|
This pull request was exported from Phabricator. Differential Revision: D74596410 |
torch/_inductor/select_algorithm.py
Outdated
| if not isinstance(choice, CUDATemplateCaller): | ||
| log.error( | ||
| "CUDA compilation error during autotuning: \n%s. \nIgnoring this choice.", | ||
| str(e), |
There was a problem hiding this comment.
| str(e), | |
| e, |
There was a problem hiding this comment.
@Skylion007 I will make the change, but at this point it might be worth to change all str(e) in the codebase to e (at least for the logging cases)
… error" Differential Revision: [D74596410](https://our.internmc.facebook.com/intern/diff/D74596410/) cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov [ghstack-poisoned]
Pull Request resolved: #153397 ghstack-source-id: 283518476 Differential Revision: [D74596410](https://our.internmc.facebook.com/intern/diff/D74596410/)
|
This pull request was exported from Phabricator. Differential Revision: D74596410 |
… error" Differential Revision: [D74596410](https://our.internmc.facebook.com/intern/diff/D74596410/) This change should only affect cutlass backend. We realize that we are going to have Cuda compilation errors, and we do a really good job handling them and caching them. So reduce the logging levels there. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov [ghstack-poisoned]
|
This pull request was exported from Phabricator. Differential Revision: D74596410 |
| "CUDA compilation error during autotuning: \n%s. \nIgnoring this choice.", | ||
| str(e), | ||
| ) | ||
| from torch._inductor.codegen.cuda.cuda_kernel import CUDATemplateCaller |
There was a problem hiding this comment.
I was told local imports like these can cause unwanted compile times.
There was a problem hiding this comment.
I was told local imports like these can cause unwanted compile times.
I feel like if we try to import that at the top, it would cause circular import
There was a problem hiding this comment.
give it a try and see? if not possible, it's probably fine since this is in exception handling path
There was a problem hiding this comment.
give it a try and see? if not possible, it's probably fine since this is in exception handling path
actually I want to test it in a subsequent PR
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Stack from ghstack (oldest at bottom):
Differential Revision: D74596410
This change should only affect cutlass backend. We realize that we are going to have Cuda compilation errors, and we do a really good job handling them and caching them. So reduce the logging levels there.
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov