XLA PoC #1275

Scitator · 2021-08-15T08:44:24Z

Before submitting (checklist)

Description

PoC: https://colab.research.google.com/drive/1jG8qcp8plm7YEsFoeIInxtKZoxIRz1CT#scrollTo=r9mDrhuETaPG

Related Issue

Type of Change

Examples / docs / tutorials / contributors update
Bug fix (non-breaking change which fixes an issue)
Improvement (non-breaking change which improves an existing feature)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

FAQ

Please review the FAQ before submitting an issue:

I have read the documentation and FAQ
I have reviewed the minimal examples section
I have checked the changelog for main framework updates
I have read the contribution guide
I have joined Catalyst slack (#__questions channel) for issue discussion

catalyst/engines/xla.py

catalyst/core/engine.py

catalyst/engines/xla.py

github-actions · 2021-08-15T10:40:29Z

catalyst/engines/torch.py

@@ -1,4 +1,4 @@
-from typing import Any, Dict, List, Mapping, Tuple, Union
+from typing import Any, Callable, Dict, List, Mapping, Tuple, Union


[pep8] _{reported by reviewdog 🐶}
F401 'typing.Callable' imported but unused

catalyst/engines/xla.py

pep8speaks · 2021-08-15T13:01:39Z

Hello @Scitator! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file examples/engines/train.py:

Line 34:61: E231 missing whitespace after ','

Comment last updated at 2021-08-20 06:52:53 UTC

catalyst/core/engine.py

github-actions · 2021-08-18T05:39:36Z

catalyst/engines/deepspeed.py

+    def spawn(self, fn: Callable, *args: Any, **kwargs: Any) -> None:
+        """Spawns abstraction for``nprocs`` creation with specified ``fn`` and ``args``/``kwargs``.
+
+        Args:


[pep8] _{reported by reviewdog 🐶}
DAR101 Missing parameter(s) in Docstring: - **kwargs

github-actions · 2021-08-18T05:39:37Z

catalyst/engines/deepspeed.py

+    def spawn(self, fn: Callable, *args: Any, **kwargs: Any) -> None:
+        """Spawns abstraction for``nprocs`` creation with specified ``fn`` and ``args``/``kwargs``.
+
+        Args:


[pep8] _{reported by reviewdog 🐶}
DAR101 Missing parameter(s) in Docstring: - *args

github-actions · 2021-08-18T05:39:37Z

catalyst/engines/xla.py

+    def spawn(self, fn: Callable, *args: Any, **kwargs: Any) -> None:
+        """Spawns abstraction for``nprocs`` creation with specified ``fn`` and ``args``/``kwargs``.
+
+        Args:


[pep8] _{reported by reviewdog 🐶}
DAR101 Missing parameter(s) in Docstring: - **kwargs

github-actions · 2021-08-18T05:39:37Z

catalyst/engines/xla.py

+    def spawn(self, fn: Callable, *args: Any, **kwargs: Any) -> None:
+        """Spawns abstraction for``nprocs`` creation with specified ``fn`` and ``args``/``kwargs``.
+
+        Args:


[pep8] _{reported by reviewdog 🐶}
DAR101 Missing parameter(s) in Docstring: - *args

catalyst/engines/torch.py

github-actions · 2021-08-18T06:47:49Z

catalyst/utils/distributed.py

@@ -64,20 +72,57 @@ def get_nn_from_ddp_module(model: nn.Module) -> nn.Module:
    return model


+def get_backend() -> Optional[str]:


[pep8] _{reported by reviewdog 🐶}
D103 Missing docstring in public function

github-actions · 2021-08-18T06:47:50Z

catalyst/utils/distributed.py

+#     pass
+
+
+def get_world_size() -> int:


[pep8] _{reported by reviewdog 🐶}
D103 Missing docstring in public function

mergify · 2021-08-19T05:33:13Z

This pull request is now in conflicts. @Scitator, could you fix it? 🙏

mergify · 2021-08-19T20:45:23Z

This pull request is now in conflicts. @Scitator, could you fix it? 🙏

review-notebook-app · 2021-08-20T06:24:41Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

github-actions · 2021-08-20T06:34:17Z

catalyst/engines/amp.py

@@ -238,12 +304,15 @@ def get_engine(self):
        stages:
            ...

+    .. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#


[pep8] _{reported by reviewdog 🐶}
E501 line too long (102 > 99 characters)

github-actions · 2021-08-20T06:34:17Z

catalyst/engines/amp.py

@@ -238,12 +304,15 @@ def get_engine(self):
        stages:
            ...

+    .. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#


[pep8] _{reported by reviewdog 🐶}
W505 doc line too long (102 > 99 characters)

github-actions · 2021-08-20T06:34:17Z

catalyst/engines/fairscale.py

@@ -203,103 +199,18 @@ def get_engine(self):

        stages:
            ...
-
+    .. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#


[pep8] _{reported by reviewdog 🐶}
E501 line too long (102 > 99 characters)

github-actions · 2021-08-20T06:34:17Z

catalyst/engines/fairscale.py

@@ -203,103 +199,18 @@ def get_engine(self):

        stages:
            ...
-
+    .. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#


[pep8] _{reported by reviewdog 🐶}
W505 doc line too long (102 > 99 characters)

github-actions · 2021-08-20T06:34:17Z

catalyst/engines/fairscale.py

@@ -424,6 +326,8 @@ def get_engine(self):
        stages:
            ...

+    .. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#


[pep8] _{reported by reviewdog 🐶}
E501 line too long (102 > 99 characters)

github-actions · 2021-08-20T06:34:18Z

catalyst/engines/fairscale.py

@@ -526,6 +433,8 @@ def get_engine(self):
        stages:
            ...

+    .. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#


[pep8] _{reported by reviewdog 🐶}
W505 doc line too long (102 > 99 characters)

github-actions · 2021-08-20T06:34:18Z

catalyst/engines/torch.py

@@ -357,19 +369,24 @@ def get_engine(self):
        stages:
            ...

+    .. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#


[pep8] _{reported by reviewdog 🐶}
E501 line too long (102 > 99 characters)

github-actions · 2021-08-20T06:34:18Z

catalyst/engines/torch.py

@@ -357,19 +369,24 @@ def get_engine(self):
        stages:
            ...

+    .. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#


[pep8] _{reported by reviewdog 🐶}
W505 doc line too long (102 > 99 characters)

github-actions · 2021-08-20T06:34:18Z

catalyst/engines/apex.py

@@ -439,19 +442,27 @@ def get_engine(self):

        stages:
            ...
+
+    .. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#


[pep8] _{reported by reviewdog 🐶}
E501 line too long (102 > 99 characters)

github-actions · 2021-08-20T06:34:18Z

catalyst/engines/apex.py

@@ -439,19 +442,27 @@ def get_engine(self):

        stages:
            ...
+
+    .. _convert_sync_batchnorm: https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#


[pep8] _{reported by reviewdog 🐶}
W505 doc line too long (102 > 99 characters)

xla poc

f409f6b

Scitator requested review from bagxi and ditwoo as code owners August 15, 2021 08:44

github-actions bot reviewed Aug 15, 2021

View reviewed changes

catalyst/engines/xla.py Show resolved Hide resolved

catalyst/engines/xla.py Show resolved Hide resolved

Scitator force-pushed the engines/tpu branch 3 times, most recently from b03b044 to 8cb5eab Compare August 15, 2021 10:32

github-actions bot reviewed Aug 15, 2021

View reviewed changes

xla requirements hotfix

84641e8

Scitator force-pushed the engines/tpu branch 3 times, most recently from fe1c5b9 to dfd6ae6 Compare August 15, 2021 11:47

github-actions bot reviewed Aug 15, 2021

View reviewed changes

catalyst/engines/xla.py Show resolved Hide resolved

Scitator force-pushed the engines/tpu branch 6 times, most recently from 57f5adb to 53d9ddd Compare August 15, 2021 13:01

Scitator force-pushed the engines/tpu branch 10 times, most recently from 2538074 to 758ec82 Compare August 15, 2021 14:03

github-actions bot reviewed Aug 18, 2021

View reviewed changes

Scitator force-pushed the engines/tpu branch 8 times, most recently from b865672 to 3c88377 Compare August 18, 2021 19:03

cleanup 2

740d6c4

Scitator force-pushed the engines/tpu branch from 3c88377 to 740d6c4 Compare August 18, 2021 19:13

Scitator added 2 commits August 19, 2021 07:48

cleanup 3

197474f

refactoring

ee931b9

Scitator force-pushed the engines/tpu branch from d94cfcb to 68b4ff5 Compare August 19, 2021 06:10

refactoring

811e2ac

Scitator force-pushed the engines/tpu branch from 68b4ff5 to 811e2ac Compare August 19, 2021 06:13

Scitator added 2 commits August 19, 2021 09:18

merged

dfe8565

codestyle

5977fe0

Scitator added 4 commits August 20, 2021 08:34

fix

1e8ad7c

merged

8fc9622

sync-bn in one commit

5bbec8e

xla notebooks

3ff2b0b

github-actions bot reviewed Aug 20, 2021

View reviewed changes

docs

d472d44

Scitator merged commit 6a2b929 into master Aug 20, 2021

mergify bot deleted the engines/tpu branch August 20, 2021 07:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

XLA PoC #1275

XLA PoC #1275

		@@ -1,4 +1,4 @@
		from typing import Any, Dict, List, Mapping, Tuple, Union
		from typing import Any, Callable, Dict, List, Mapping, Tuple, Union

		@@ -64,20 +72,57 @@ def get_nn_from_ddp_module(model: nn.Module) -> nn.Module:
		return model


		def get_backend() -> Optional[str]:

XLA PoC #1275

XLA PoC #1275

Conversation

Before submitting (checklist)

Description

Related Issue

Type of Change

PR review

FAQ

Choose a reason for hiding this comment

Comment last updated at 2021-08-20 06:52:53 UTC

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment