Python-API created tensors to go through `Tensor.init` #26566

yaroslavvb · 2019-09-20T21:13:37Z

It would be easier to customize PyTorch behavior if Tensors constructed through Python API called Tensor.__init__ method because user could monkey-patch that method for new behavior

Examples:

CPU vs GPU placement. To run someone's CPU code on my data I had to mechanically append .to(device) to every instance torch.ones, torch.randn and torch.eye. This could instead by with myutils.MoveToGpu(). This could be a temporary workaround [feature request] Global GPU Flag #7535 (also known as permanent workaround ;)
Debugging. TorchSnooper allows tracking tensor shapes automatically. However it 1) requires modifying user code with decorators 2) is complicated -- it's based on dynamically rewriting function representation. A monkey-patching solution would be much simpler and work without code modification, ie python -m find_strange_tensors mycode.py
Supporting custom types. I'm looking at a representation where certain operations can be made more efficient by incorporating tensor structure. Monkey-patching Tensor would allow me to try it out on existing PyTorch training code without modifying that code.

In TensorFlow shape debugging was possible through monkey-patching because all Python tensor construction went through Python create_op function.

The text was updated successfully, but these errors were encountered:

VitalyFedyunin · 2019-09-24T13:53:44Z

Need to discuss if it is even possible to implement.

yaroslavvb · 2019-09-25T00:30:29Z

I'm wondering what the experience of adding non-Python client to PyTorch is. If this use-case is well-supported, then one could conceivably add a more Pythonic client to PyTorch using the same API as other languages, rather being a first-class citizen as it is now.

This would also make Python type-inference more consistent, current solution relies on .pyi files which get out of sync with code (example #25736)

ezyang · 2019-09-30T17:25:30Z

cc @rgommers @prasunanand

rgommers · 2019-09-30T21:57:24Z

Hmm, making changes to how Tensor initialization works to "support monkeypatching" without nailing down what changes are allowed/supported is probably not a healthy approach. These three issues seem like they can be treated separately, it's not clear to me they require a single solution.

An explicit hook to register some code that needs to be run at the end of object creation could be put in, but even if it's do-nothing by default that is likely to affect performance a little. If we can find a way to do this without extra overhead, then it would perhaps be a handy thing to have. Something like

import torch

def move_to_device(x, device='cuda:0'):
    """Gets called at the end of every Tensor init, once registered."""
    x.to(device):

torch.hide_this_somewhere.register_init_hook(move_to_device)

No idea if it's feasible, but it requires some global state and likely some tens of nanoseconds of overhead.

CPU vs GPU placement. To run someone's CPU code on my data I had to mechanically append .to(device) to every instance torch.ones, torch.randn and torch.eye. This could instead by with myutils.MoveToGpu(). This could be a temporary workaround [feature request] Global GPU Flag #7535 (also known as permanent workaround ;)

Rather than data.to(device) as a separate call (as suggested in gh-7535), tensor creation functions can be used like torch.ones(..., device=device) directly. Or use zeros_like. That should be documented as best practice anyway I'd think? Doesn't help you with code that you receive from someone else of course, but introducing a magic MoveToGpu to work around code that's not written in a general enough way doesn't sound quite right.

2. Debugging. TorchSnooper allows tracking tensor shapes automatically. However it 1) requires modifying user code with decorators 2) is complicated -- it's based on dynamically rewriting function representation. A monkey-patching solution would be much simpler and work without code modification

Shapes are not just determined at tensor creation time, but also when creating views (e.g. reshape). So just monkeypatching __init__ may not be reliable. This does start to smell like the __array_finalize__-like thing as discussed in gh-22402.

3. Supporting custom types. I'm looking at a representation where certain operations can be made more efficient by incorporating tensor structure. Monkey-patching Tensor would allow me to try it out on existing PyTorch training code without modifying that code.

Since comprehensive support for custom tensor types and subclasses is still a work in progress (see gh-22402), I wouldn't worry about this one. Anyway it seems like the more interesting custom types (like NestedTensor) would need a lot more than just a tweak at the end of __init__?

yaroslavvb · 2019-09-30T23:25:38Z

@rgommers thanks for the in-depth response. The hook you suggest would obviate the need for having Tensor.__init__

MoveToGpu is indeed too hacky to incorporate into PyTorch, I was thinking of it as a user-level workaround to to deal with code that doesn't have device annotations.
Related use-case: I'm porting calculations from a language (Mathematica) which supports scalar/matrix matmul. Equivalent Python operator @ operator does not support it due to PEP-0465 . A tensor creation hook would make the porting process a bit easier -- first make it work on MyTensor type that has desired semantics. Use the hook to wrap every Tensor into MyTensor
Calls like reshape should not be a problem for shape tracking -- the hook could would wrap Tensor into ShapeTrackingTensor which intercepts relevant methods reshape and view. This should cover all the shape-use cases that TorchSnooper currently supports (ie, relevant calls have to go through Python API)
Custom type use-case I had in mind has to do with structured Tensors -- certain quantities coming up in neural networks can be Kronecker factored. A smart tensor object would recognize it and keep things in factored form, by implementing its own version of matmul and related ops. In other words, an object that behaves identically to Tensor but uses more efficient implementation. An install_smart_tensor hook would make this easier to test on third-party code.

rgommers · 2019-10-01T17:18:17Z

A tensor creation hook would make the porting process a bit easier -- first make it work on MyTensor type that has desired semantics. Use the hook to wrap every Tensor into MyTensor

Could you define "wrap into" a little more? It sounds like you want a subclass like

class MyTensor(torch.Tensor):
    def __init__(...):
        super().__init__()
        do_something_arbitrary()

    def reshape(*shape):
        do_something_else_arbitrary()
        super().reshape(*shape)

but that then has class torch.Tensor rather than MyTensor, and literally replaces Tensor. Just the init hook I thought of won't be enough it sounds like.

yaroslavvb · 2019-10-01T18:35:30Z

Yes, that's the wrapping I had in mind. To think of it, this seems hard with init hook. It may require manipulating CPython internal structures, if possible at all.

rgommers · 2019-10-01T20:28:29Z

It may require manipulating CPython internal structures, if possible at all.

This may be what you want: #22402 (comment). It's probably fragile, but it seems to work today.

gchanan · 2020-02-06T23:05:43Z

I'm going to close this because it seems like #22402 subsumes this. If you disagree, please reopen.

yaroslavvb mentioned this issue Sep 21, 2019

Support matmul for scalar tensors #26552

Closed

VitalyFedyunin added needs research We need to decide whether or not this merits inclusion, based on research world triage review triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Sep 24, 2019

ezyang added module: pybind Related to our Python bindings / interactions with other Python libraries and removed triage review labels Sep 30, 2019

rgommers mentioned this issue Oct 1, 2019

add __torch_function__ API override mechanism #27064

Closed

gchanan closed this as completed Feb 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python-API created tensors to go through `Tensor.init` #26566

Python-API created tensors to go through `Tensor.init` #26566

Python-API created tensors to go through Tensor.__init__ #26566

Python-API created tensors to go through Tensor.__init__ #26566

Comments

Python-API created tensors to go through `Tensor.init` #26566

Python-API created tensors to go through `Tensor.init` #26566