Make Adam, AdamW work with nonzero-dim Tensor betas
#149939
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/149939
Note: Links to docs will display an error until the docs builds have been completed. ❌ 5 New Failures, 1 Unrelated FailureAs of commit 2536db1 with merge base 27e9d9b ( NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
janeyx99
reviewed
Apr 28, 2025
janeyx99
reviewed
May 2, 2025
b601fd2 to
2536db1
Compare
zeshengzong
commented
May 14, 2025
| "betas": (torch.tensor([[[0.9]]]), torch.tensor([[0.99]])), | ||
| "amsgrad": False, | ||
| "capturable": True, | ||
| "fused": True, |
There was a problem hiding this comment.
Hi @janeyx99 , looks like fused=True failed with complex type due to backend not supported yet, is there a way to control input type in test and skip complex type? Thanks
Failed in here
pytorch/torch/optim/optimizer.py
Lines 195 to 198 in d51bc27
pytest -s test/test_optim.py -k test_complex_2d_AdamW_cuda_complex64
______________________________________________________________________________________________ TestOptimRenewedCUDA.test_complex_2d_AdamW_cuda_complex64 _______________________________________________________________________________________________
Traceback (most recent call last):
File "/home/zong/miniconda3/envs/torch/lib/python3.12/unittest/case.py", line 58, in testPartExecutor
yield
File "/home/zong/miniconda3/envs/torch/lib/python3.12/unittest/case.py", line 634, in run
self._callTestMethod(testMethod)
File "/home/zong/miniconda3/envs/torch/lib/python3.12/unittest/case.py", line 589, in _callTestMethod
if method() is not None:
^^^^^^^^
File "/home/zong/code/pytorch/torch/testing/_internal/common_utils.py", line 3154, in wrapper
method(*args, **kwargs)
File "/home/zong/code/pytorch/torch/testing/_internal/common_utils.py", line 3154, in wrapper
method(*args, **kwargs)
File "/home/zong/code/pytorch/torch/testing/_internal/common_device_type.py", line 446, in instantiated_test
raise rte
File "/home/zong/code/pytorch/torch/testing/_internal/common_device_type.py", line 426, in instantiated_test
result = test(self, **param_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zong/code/pytorch/torch/testing/_internal/common_utils.py", line 1612, in wrapper
fn(*args, **kwargs)
File "/home/zong/code/pytorch/torch/testing/_internal/common_optimizers.py", line 226, in test_wrapper
return test(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/zong/code/pytorch/torch/testing/_internal/common_device_type.py", line 1215, in dep_fn
return fn(slf, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/zong/code/pytorch/test/test_optim.py", line 664, in test_complex_2d
optim1.step()
File "/home/zong/code/pytorch/torch/optim/optimizer.py", line 507, in wrapper
out = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/zong/code/pytorch/torch/optim/optimizer.py", line 80, in _use_grad
ret = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/zong/code/pytorch/torch/optim/adam.py", line 237, in step
has_complex = self._init_group(
^^^^^^^^^^^^^^^^^
File "/home/zong/code/pytorch/torch/optim/adam.py", line 163, in _init_group
_device_dtype_check_for_fused(p)
File "/home/zong/code/pytorch/torch/optim/optimizer.py", line 197, in _device_dtype_check_for_fused
raise RuntimeError(
RuntimeError: `fused=True` requires all the params to be floating point Tensors of supported devices: ['mps', 'cuda', 'xpu', 'hpu', 'cpu', 'privateuseone'] but torch.complex64 and cuda
To execute this test, run the following from the base repo dir:
python test/test_optim.py TestOptimRenewedCUDA.test_complex_2d_AdamW_cuda_complex64
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
=============================================================================================================== short test summary info ================================================================================================================
FAILED [2.1007s] test/test_optim.py::TestOptimRenewedCUDA::test_complex_2d_AdamW_cuda_complex64 - RuntimeError: `fused=True` requires all the params to be floating point Tensors of supported devices: ['mps', 'cuda', 'xpu', 'hpu', 'cpu', 'privateuseone'] but torch.complex64 and cudaThere was a problem hiding this comment.
Ah sorry, we shouldn't specify "fused" in these OptimInfos after all--that was my bad. (Please delete this config)
What I care about is that the fused optimizer is tested with these betas. Can you confirm such a configuration with Tensor betas and fused is tested in one of the existing fused tests?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #147921
Changes
betasusing_to_scalarbetasparamTest Result