PyTorch Glossary

Operation and Kernel
JIT Compilation
- JIT
- TorchScript
- Tracing
- Scripting

Operation and Kernel

ATen

Short for "A Tensor Library". The foundational tensor and mathematical operation library on which all else is built.

Operation

A unit of work. For example, the work of matrix multiplication is an operation called aten::matmul.

Native Operation

An operation that comes natively with PyTorch ATen, for example aten::matmul.

Custom Operation

An Operation that is defined by users and is usually a Compound Operation. For example, this tutorial details how to create Custom Operations.

Kernel

Implementation of a PyTorch operation, specifying what should be done when an operation executes.

Compound Operation

A Compound Operation is composed of other operations. Its kernel is usually device-agnostic. Normally it doesn't have its own derivative functions defined. Instead, AutoGrad automatically computes its derivative based on operations it uses.

Composite Operation

Same as Compound Operation.

Non-Leaf Operation

Same as Compound Operation.

Leaf Operation

An operation that's considered a basic operation, as opposed to a Compound Operation. Leaf Operation always has dispatch functions defined, usually has a derivative function defined as well.

Device Kernel

Device-specific kernel of a leaf operation.

Compound Kernel

Opposed to Device Kernels, Compound kernels are usually device-agnostic and belong to Compound Operations.

JIT Compilation

JIT

Just-In-Time Compilation.

TorchScript

An interface to the TorchScript JIT compiler and interpreter.

Tracing

Using torch.jit.trace on a function to get an executable that can be optimized using just-in-time compilation.

Scripting

Using torch.jit.script on a function to inspect source code and compile it as TorchScript code.