[testing] Port `torch.{repeat, tile}` tests to use OpInfo machinery #50199

kshitij12345 · 2021-01-07T13:04:20Z

Reference: #50013

facebook-github-bot · 2021-01-07T13:14:56Z

💊 CI failures summary and remediations

As of commit ea92bca (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

kshitij12345 · 2021-01-07T13:21:36Z

Test Timings

test_shape_ops.py

torch.repeat

============================================================================= slowest 10 durations =============================================================================
1.15s call     test/test_shape_ops.py::TestShapeFuncsCUDA::test_repeat_tile_vs_numpy_repeat_cuda_complex128
0.14s call     test/test_shape_ops.py::TestShapeFuncsCPU::test_repeat_tile_vs_numpy_repeat_cpu_complex128
0.06s call     test/test_shape_ops.py::TestShapeFuncsCUDA::test_repeat_tile_vs_numpy_repeat_cuda_float64
0.06s call     test/test_shape_ops.py::TestShapeFuncsCUDA::test_repeat_tile_vs_numpy_repeat_cuda_int64
0.06s call     test/test_shape_ops.py::TestShapeFuncsCUDA::test_repeat_tile_vs_numpy_repeat_cuda_uint8
0.01s call     test/test_shape_ops.py::TestShapeFuncsCPU::test_repeat_tile_vs_numpy_repeat_cpu_float64
0.01s call     test/test_shape_ops.py::TestShapeFuncsCPU::test_repeat_tile_vs_numpy_repeat_cpu_int64
0.01s call     test/test_shape_ops.py::TestShapeFuncsCPU::test_repeat_tile_vs_numpy_repeat_cpu_uint8

(2 durations < 0.005s hidden.  Use -vv to show these durations.)
====================================================================== 8 passed, 117 deselected in 3.01s =======================================================================

torch.tile

============================================================================= slowest 10 durations =============================================================================
1.35s call     test/test_shape_ops.py::TestShapeFuncsCUDA::test_repeat_tile_vs_numpy_tile_cuda_complex128
0.24s call     test/test_shape_ops.py::TestShapeFuncsCPU::test_repeat_tile_vs_numpy_tile_cpu_complex128
0.08s call     test/test_shape_ops.py::TestShapeFuncsCUDA::test_repeat_tile_vs_numpy_tile_cuda_uint8
0.07s call     test/test_shape_ops.py::TestShapeFuncsCUDA::test_repeat_tile_vs_numpy_tile_cuda_float64
0.07s call     test/test_shape_ops.py::TestShapeFuncsCUDA::test_repeat_tile_vs_numpy_tile_cuda_int64
0.01s call     test/test_shape_ops.py::TestShapeFuncsCPU::test_repeat_tile_vs_numpy_tile_cpu_float64
0.01s call     test/test_shape_ops.py::TestShapeFuncsCPU::test_repeat_tile_vs_numpy_tile_cpu_uint8
0.01s call     test/test_shape_ops.py::TestShapeFuncsCPU::test_repeat_tile_vs_numpy_tile_cpu_int64

(2 durations < 0.005s hidden.  Use -vv to show these durations.)
====================================================================== 8 passed, 117 deselected in 3.32s =======================================================================

test_ops.py

torch.repeat

============================================================================= slowest 10 durations =============================================================================
2.18s call     test/test_ops.py::TestGradientsCUDA::test_fn_gradgrad_repeat_cuda_complex128
1.49s call     test/test_ops.py::TestOpInfoCUDA::test_supported_dtypes_repeat_cuda_bfloat16
0.72s call     test/test_ops.py::TestGradientsCUDA::test_fn_gradgrad_repeat_cuda_float64
0.65s call     test/test_ops.py::TestGradientsCPU::test_fn_gradgrad_repeat_cpu_complex128
0.45s call     test/test_ops.py::TestGradientsCUDA::test_fn_grad_repeat_cuda_complex128
0.28s call     test/test_ops.py::TestOpInfoCPU::test_supported_dtypes_repeat_cpu_bfloat16
0.27s call     test/test_ops.py::TestGradientsCUDA::test_fn_grad_repeat_cuda_float64
0.18s call     test/test_ops.py::TestGradientsCPU::test_fn_gradgrad_repeat_cpu_float64
0.17s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_repeat_cuda_complex64
0.17s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_repeat_cuda_complex128
=============================================================== 56 passed, 56 skipped, 5570 deselected in 13.90s ===============================================================

torch.tile

============================================================================= slowest 10 durations =============================================================================
2.97s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_jit_tile_cuda_complex64
2.87s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_jit_tile_cuda_complex128
2.68s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_jit_tile_cuda_float32
2.64s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_jit_tile_cuda_float16
2.62s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_jit_tile_cuda_bfloat16
2.61s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_jit_tile_cuda_float64
2.08s call     test/test_ops.py::TestCommonCPU::test_variant_consistency_jit_tile_cpu_complex64
2.07s call     test/test_ops.py::TestGradientsCUDA::test_fn_gradgrad_tile_cuda_complex128
2.02s call     test/test_ops.py::TestCommonCPU::test_variant_consistency_jit_tile_cpu_complex128
1.98s call     test/test_ops.py::TestCommonCPU::test_variant_consistency_jit_tile_cpu_float16
=============================================================== 80 passed, 32 skipped, 5570 deselected in 48.66s ===============================================================

kshitij12345 · 2021-01-07T13:32:32Z

In terms of time required by the test, I see that test_variant_consistency_jit requires a lot of time. Just running the test_variant_consistency_jit for torch.tile takes about 40s.

============================================================================= slowest 10 durations =============================================================================
3.80s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_jit_tile_cuda_bfloat16
2.98s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_jit_tile_cuda_complex64
2.88s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_jit_tile_cuda_complex128
2.63s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_jit_tile_cuda_float64
2.59s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_jit_tile_cuda_float32
2.56s call     test/test_ops.py::TestCommonCUDA::test_variant_consistency_jit_tile_cuda_float16
2.18s call     test/test_ops.py::TestCommonCPU::test_variant_consistency_jit_tile_cpu_complex64
2.17s call     test/test_ops.py::TestCommonCPU::test_variant_consistency_jit_tile_cpu_bfloat16
2.15s call     test/test_ops.py::TestCommonCPU::test_variant_consistency_jit_tile_cpu_complex128
2.03s call     test/test_ops.py::TestCommonCPU::test_variant_consistency_jit_tile_cpu_float16
===================================================================== 24 passed, 5658 deselected in 40.08s =====================================================================

Running all other Op Tests (gradcheck, gradgradcheck, etc) with test_variant_consistency_jit takes about 48-50s.

kshitij12345 · 2021-01-07T13:59:12Z

can torch.repeat be implemented as a call to torch.tile? I understand that torch.tile is actually implemented as a call to repeat currently, but from a UX standpoint, could we alias torch.repeat to torch.tile? It's true that torch.tile can accept more inputs than torch.repeat, but will every valid input to torch.repeat produce the same output when given to torch.tile?

torch.tile is more general than torch.repeat. torch.tile supports cases where passed dims could be less, more or same as the actual dim of the tensor to be repeated. While torch.repeat only accepts cases where passed dims is more or same as the actual dim.

So we can implement torch.repeat in terms of torch.tile by just checking for reps.size() >= self.dim()

Example

Tensor repeat(const Tensor& self, IntArrayRef reps){
  TORCH_CHECK(reps.size() >= self.dim(),
           "Number of dimensions of repeat dims can not be smaller than number of dimensions of tensor");
  return self.tile(reps);
}

codecov · 2021-01-07T17:29:57Z

Codecov Report

Merging #50199 (f7171b7) into master (3f052ba) will decrease coverage by 0.08%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #50199      +/-   ##
==========================================
- Coverage   80.66%   80.57%   -0.09%     
==========================================
  Files        1912     1912              
  Lines      208058   208078      +20     
==========================================
- Hits       167820   167662     -158     
- Misses      40238    40416     +178

mruberry · 2021-01-08T15:28:32Z

test/test_shape_ops.py

@@ -605,7 +606,19 @@ def test_nonzero_non_diff(self, device):
        nz = x.nonzero()
        self.assertFalse(nz.requires_grad)

+class TestShapeFuncs(TestCase):
+    @dtypes(*(torch.uint8, torch.int64, torch.double, torch.complex128))
+    @ops([op for op in shape_funcs if op.name in ['tile', 'repeat']])


This is a good way to filter the OpInfos. In the future we may want to consider allowing filtering directly by name or function. That is,

@ops('tile', 'repeat') @ops(torch.tile, torch.repeat)

Which might be simpler to remember and more readable.

mruberry · 2021-01-08T15:37:02Z

test/test_shape_ops.py

+class TestShapeFuncs(TestCase):
+    @dtypes(*(torch.uint8, torch.int64, torch.double, torch.complex128))
+    @ops([op for op in shape_funcs if op.name in ['tile', 'repeat']])
+    def test_repeat_tile_vs_numpy(self, device, dtype, op):


Since torch.repeat is more restrictive than tile, does this test not test where torch.tile and np.tile are compatible but torch.repeat isn't?

mruberry · 2021-01-08T15:37:48Z

torch/testing/_internal/common_methods_invocations.py

+    rep_dims = ((), (0, ), (1, ), (0, 2), (1, 1), (2, 3), (2, 3, 2), (0, 2, 3), (2, 1, 1, 1),)
+    shapes = ((), (0,), (2,), (3, 0), (3, 2), (3, 0, 1))
+
+    if requires_grad:


This is a good way to filter the samples for gradcheck and gradgradcheck.

mruberry · 2021-01-08T15:38:26Z

torch/testing/_internal/common_methods_invocations.py

+        for t in (tensor, tensor.T):
+            if op_info.name == 'repeat' and len(rep_dim) >= t.dim():
+                samples.append(SampleInput((t, rep_dim),))
+            elif op_info.name == 'tile':


Add a comment here explaining the filtering for tile and repeat. This is a clever way to filter, and this answers my previous question about test coverage.

mruberry · 2021-01-08T16:48:10Z

torch/testing/_internal/common_methods_invocations.py

@@ -500,6 +526,26 @@ def sample_inputs(self, device, dtype, requires_grad=False):
            ]


+class ShapeFuncInfo(OpInfo):


Add a brief comment here explaining what this derived class is intended for (maybe something like, "Early version of a specialized OpInfo for "shape" operations like tile and roll"?)

mruberry · 2021-01-08T16:52:44Z

test/test_shape_ops.py

@@ -605,7 +606,19 @@ def test_nonzero_non_diff(self, device):
        nz = x.nonzero()
        self.assertFalse(nz.requires_grad)

+class TestShapeFuncs(TestCase):


Add a comment here describing what this class is for.

Do we really want to add another test class instead of just putting this function into the existing TestShapeOps?

I just thought, going forward all ShapeFuncsInfo Op tests might as well go under TestShapeFuncs. But we can just move it to the existing class as well.

Do let me know if that is preferred.

Thanks!

Either way is fine, just add a comment explaining why people should put a test here, for example, vs. the other test suite.

mruberry

Hey @kshitij12345, thanks for taking a look at these functions! I appreciate the new tricks you've used to acquire just their OpInfos and filter the sample inputs based on whether they'll be used for {grad}gradchecks or not. Nice work.

I've made a few small comments. Also, repeat has an entry here that we can remove:

pytorch/test/test_torch.py

Line 6818 in 55919a4

    
           ('repeat', '', _small_2d, lambda t, d: [2, 2, 2], 1e-5, 1e-5, 1e-5, _types, _cpu_types, False),

* Add comment for TestShapeFuncs class * Add comment for ShapeFuncInfo class * Add comment for filtering inputs for repeat * Remove redundant test from test_torch.py

kshitij12345 · 2021-01-18T09:24:07Z

@mruberry PTAL :)

mruberry

Awesome! This looks great, thanks @kshitij12345!

facebook-github-bot

@mruberry has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2021-01-19T15:12:58Z

@mruberry merged this pull request in 316f0b8.

port repeat tile tests to use OpInfo machinery

c20dd9f

facebook-github-bot added the cla signed label Jan 7, 2021

pytorchbot added the open source label Jan 7, 2021

use list comprehension instead of filter

126058c

kshitij12345 requested a review from mruberry January 7, 2021 15:02

kshitij12345 marked this pull request as ready for review January 7, 2021 15:02

mruberry reviewed Jan 8, 2021

View reviewed changes

kshitij12345 added 4 commits January 8, 2021 11:31

address comments

af87706

* Add comment for TestShapeFuncs class * Add comment for ShapeFuncInfo class * Add comment for filtering inputs for repeat * Remove redundant test from test_torch.py

Merge branch 'master' into develop/opinfo/repeat-tile

ea92bca

Merge branch 'master' into develop/opinfo/repeat-tile

1973767

Merge branch 'master' into develop/opinfo/repeat-tile

f7171b7

mruberry approved these changes Jan 19, 2021

View reviewed changes

facebook-github-bot reviewed Jan 19, 2021

View reviewed changes

facebook-github-bot closed this in 316f0b8 Jan 19, 2021

kshitij12345 deleted the develop/opinfo/repeat-tile branch January 19, 2021 14:40

facebook-github-bot added the Merged label Jan 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[testing] Port `torch.{repeat, tile}` tests to use OpInfo machinery #50199

[testing] Port `torch.{repeat, tile}` tests to use OpInfo machinery #50199

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

		@@ -500,6 +526,26 @@ def sample_inputs(self, device, dtype, requires_grad=False):
		]


		class ShapeFuncInfo(OpInfo):

[testing] Port torch.{repeat, tile} tests to use OpInfo machinery #50199

[testing] Port torch.{repeat, tile} tests to use OpInfo machinery #50199

Uh oh!

Conversation

Uh oh!

Uh oh!

💊 CI failures summary and remediations

Uh oh!

test_shape_ops.py

test_ops.py

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

[testing] Port `torch.{repeat, tile}` tests to use OpInfo machinery #50199

[testing] Port `torch.{repeat, tile}` tests to use OpInfo machinery #50199