[symbolic shapes] Log SymNode id for provenance #146532
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ghstack-poisoned]
This was referenced Feb 5, 2025
cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv [ghstack-poisoned]
This was referenced Feb 10, 2025
cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv [ghstack-poisoned]
bobrenjc93
approved these changes
Feb 10, 2025
We can use the SymNode id to point us back to how previous expressions were created, and construct this nice tree in tlparse: <img width="761" alt="image" src="https://github.com/user-attachments/assets/531b03e8-4398-4d0a-bd11-16078256041c" /> cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv [ghstack-poisoned]
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 2 jobs have failed, first few of them are: inductor / unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu), inductor / unit-test / cuda12.4-py3.13-gcc9-sm86 / test (inductor, 1, 2, linux.g5.4xlarge.nvidia.gpu) Details for Dev Infra teamRaised by workflow job |
We can use the SymNode id to point us back to how previous expressions were created, and construct this nice tree in tlparse: <img width="761" alt="image" src="https://github.com/user-attachments/assets/531b03e8-4398-4d0a-bd11-16078256041c" /> cc ezyang SherlockNoMad EikanWang jgong5 wenzhe-nrv [ghstack-poisoned]
This was referenced Feb 11, 2025
pytorchmergebot
pushed a commit
that referenced
this pull request
Feb 13, 2025
Using a custom logger so that we can store our own buffer to dedup logs that look the same. The schema for deduping is as follows:
```python
if key == "missing_fake_kernel":
return hash((key, data["op"])) # Same ops get deduped
elif key == "mismatched_fake_kernel":
return hash((key, data["op"], data["reason"])) # Same op and reason for errors get deduped
elif key == "propagate_real_tensors":
return hash((key, json.dumps(data["stack"]))) # Guards appearing on the same stacktrace get deduped
elif key == "create_unbacked_symbol":
return hash((key, json.dumps(data["stack"]))) # Unbacked symbols appearing on the same stacktrace get deduped
```
Notably, guards appearing on the same stacktrace get deduped. This is because there are some cases in PT2I models where a piece of code which creates a new unbacked symint + runs into a DDE gets called 800 times, causing 800 new symints to be created, and 800 propagate_real_tensor errors that are all the same expression. This is hard to look at, so we should just deduplicate this.
The con of this is that if there exists multiple DDE on the same stacktrace, we will only show the first issue.
Pull Request resolved: #146533
Approved by: https://github.com/avikchaudhuri
ghstack dependencies: #146532
pytorchmergebot
pushed a commit
that referenced
this pull request
Feb 13, 2025
Added some additional logging so we can also run tlparse on generic export errors Pull Request resolved: #146534 Approved by: https://github.com/pianpwk ghstack dependencies: #146532, #146533
pytorchmergebot
pushed a commit
that referenced
this pull request
Feb 13, 2025
Added a utility function for capturing the user stack and framework stacktrace. Pull Request resolved: #146858 Approved by: https://github.com/bobrenjc93 ghstack dependencies: #146532, #146533, #146534
Raymo111
pushed a commit
that referenced
this pull request
Feb 20, 2025
We can use the SymNode id to point us back to how previous expressions were created, and construct this nice tree in tlparse: <img width="761" alt="image" src="https://github.com/user-attachments/assets/531b03e8-4398-4d0a-bd11-16078256041c" /> Pull Request resolved: #146532 Approved by: https://github.com/bobrenjc93
Raymo111
pushed a commit
that referenced
this pull request
Feb 20, 2025
Using a custom logger so that we can store our own buffer to dedup logs that look the same. The schema for deduping is as follows:
```python
if key == "missing_fake_kernel":
return hash((key, data["op"])) # Same ops get deduped
elif key == "mismatched_fake_kernel":
return hash((key, data["op"], data["reason"])) # Same op and reason for errors get deduped
elif key == "propagate_real_tensors":
return hash((key, json.dumps(data["stack"]))) # Guards appearing on the same stacktrace get deduped
elif key == "create_unbacked_symbol":
return hash((key, json.dumps(data["stack"]))) # Unbacked symbols appearing on the same stacktrace get deduped
```
Notably, guards appearing on the same stacktrace get deduped. This is because there are some cases in PT2I models where a piece of code which creates a new unbacked symint + runs into a DDE gets called 800 times, causing 800 new symints to be created, and 800 propagate_real_tensor errors that are all the same expression. This is hard to look at, so we should just deduplicate this.
The con of this is that if there exists multiple DDE on the same stacktrace, we will only show the first issue.
Pull Request resolved: #146533
Approved by: https://github.com/avikchaudhuri
ghstack dependencies: #146532
Raymo111
pushed a commit
that referenced
this pull request
Feb 20, 2025
Added some additional logging so we can also run tlparse on generic export errors Pull Request resolved: #146534 Approved by: https://github.com/pianpwk ghstack dependencies: #146532, #146533
Raymo111
pushed a commit
that referenced
this pull request
Feb 20, 2025
Added a utility function for capturing the user stack and framework stacktrace. Pull Request resolved: #146858 Approved by: https://github.com/bobrenjc93 ghstack dependencies: #146532, #146533, #146534
Ryo-not-rio
pushed a commit
to Ryo-not-rio/pytorch
that referenced
this pull request
Feb 24, 2025
We can use the SymNode id to point us back to how previous expressions were created, and construct this nice tree in tlparse: <img width="761" alt="image" src="https://github.com/user-attachments/assets/531b03e8-4398-4d0a-bd11-16078256041c" /> Pull Request resolved: pytorch#146532 Approved by: https://github.com/bobrenjc93
Ryo-not-rio
pushed a commit
to Ryo-not-rio/pytorch
that referenced
this pull request
Feb 24, 2025
Using a custom logger so that we can store our own buffer to dedup logs that look the same. The schema for deduping is as follows:
```python
if key == "missing_fake_kernel":
return hash((key, data["op"])) # Same ops get deduped
elif key == "mismatched_fake_kernel":
return hash((key, data["op"], data["reason"])) # Same op and reason for errors get deduped
elif key == "propagate_real_tensors":
return hash((key, json.dumps(data["stack"]))) # Guards appearing on the same stacktrace get deduped
elif key == "create_unbacked_symbol":
return hash((key, json.dumps(data["stack"]))) # Unbacked symbols appearing on the same stacktrace get deduped
```
Notably, guards appearing on the same stacktrace get deduped. This is because there are some cases in PT2I models where a piece of code which creates a new unbacked symint + runs into a DDE gets called 800 times, causing 800 new symints to be created, and 800 propagate_real_tensor errors that are all the same expression. This is hard to look at, so we should just deduplicate this.
The con of this is that if there exists multiple DDE on the same stacktrace, we will only show the first issue.
Pull Request resolved: pytorch#146533
Approved by: https://github.com/avikchaudhuri
ghstack dependencies: pytorch#146532
Ryo-not-rio
pushed a commit
to Ryo-not-rio/pytorch
that referenced
this pull request
Feb 24, 2025
Added some additional logging so we can also run tlparse on generic export errors Pull Request resolved: pytorch#146534 Approved by: https://github.com/pianpwk ghstack dependencies: pytorch#146532, pytorch#146533
Ryo-not-rio
pushed a commit
to Ryo-not-rio/pytorch
that referenced
this pull request
Feb 24, 2025
Added a utility function for capturing the user stack and framework stacktrace. Pull Request resolved: pytorch#146858 Approved by: https://github.com/bobrenjc93 ghstack dependencies: pytorch#146532, pytorch#146533, pytorch#146534
Ryo-not-rio
pushed a commit
to Ryo-not-rio/pytorch
that referenced
this pull request
Feb 24, 2025
Pull Request resolved: pytorch#146859 Approved by: https://github.com/pianpwk ghstack dependencies: pytorch#146532, pytorch#146533, pytorch#146534, pytorch#146858
majing921201
pushed a commit
to majing921201/pytorch
that referenced
this pull request
Mar 4, 2025
We can use the SymNode id to point us back to how previous expressions were created, and construct this nice tree in tlparse: <img width="761" alt="image" src="https://github.com/user-attachments/assets/531b03e8-4398-4d0a-bd11-16078256041c" /> Pull Request resolved: pytorch#146532 Approved by: https://github.com/bobrenjc93
majing921201
pushed a commit
to majing921201/pytorch
that referenced
this pull request
Mar 4, 2025
Using a custom logger so that we can store our own buffer to dedup logs that look the same. The schema for deduping is as follows:
```python
if key == "missing_fake_kernel":
return hash((key, data["op"])) # Same ops get deduped
elif key == "mismatched_fake_kernel":
return hash((key, data["op"], data["reason"])) # Same op and reason for errors get deduped
elif key == "propagate_real_tensors":
return hash((key, json.dumps(data["stack"]))) # Guards appearing on the same stacktrace get deduped
elif key == "create_unbacked_symbol":
return hash((key, json.dumps(data["stack"]))) # Unbacked symbols appearing on the same stacktrace get deduped
```
Notably, guards appearing on the same stacktrace get deduped. This is because there are some cases in PT2I models where a piece of code which creates a new unbacked symint + runs into a DDE gets called 800 times, causing 800 new symints to be created, and 800 propagate_real_tensor errors that are all the same expression. This is hard to look at, so we should just deduplicate this.
The con of this is that if there exists multiple DDE on the same stacktrace, we will only show the first issue.
Pull Request resolved: pytorch#146533
Approved by: https://github.com/avikchaudhuri
ghstack dependencies: pytorch#146532
majing921201
pushed a commit
to majing921201/pytorch
that referenced
this pull request
Mar 4, 2025
Added some additional logging so we can also run tlparse on generic export errors Pull Request resolved: pytorch#146534 Approved by: https://github.com/pianpwk ghstack dependencies: pytorch#146532, pytorch#146533
majing921201
pushed a commit
to majing921201/pytorch
that referenced
this pull request
Mar 4, 2025
Added a utility function for capturing the user stack and framework stacktrace. Pull Request resolved: pytorch#146858 Approved by: https://github.com/bobrenjc93 ghstack dependencies: pytorch#146532, pytorch#146533, pytorch#146534
majing921201
pushed a commit
to majing921201/pytorch
that referenced
this pull request
Mar 4, 2025
Pull Request resolved: pytorch#146859 Approved by: https://github.com/pianpwk ghstack dependencies: pytorch#146532, pytorch#146533, pytorch#146534, pytorch#146858
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
We can use the SymNode id to point us back to how previous expressions were created, and construct this nice tree in tlparse:
cc @ezyang @SherlockNoMad @EikanWang @jgong5 @wenzhe-nrv