Introduction to PyTorch

What Is PyTorch?

PyTorch is an open-source deep learning framework developed by Meta AI. It’s the most widely used framework in machine learning research and increasingly the standard for production systems. PyTorch’s key insight is treating neural network computations as standard Python code — tensors are just arrays, operations are just function calls, and debugging works with standard Python tools.

The examples in this lesson were run with PyTorch 2.14 on a CPU-only machine.

Installation

# Linux wheels on PyPI already include CUDA support
pip install torch torchvision

# Smaller CPU-only build
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

The PyPI install above reports its build in the version string — 2.14.0+cu130 on the machine these examples ran on.

Tensors — The Core Data Structure

A tensor carries three pieces of metadata alongside its values: a dtype, a shape, and the device it lives on.

import torch

x = torch.tensor([1.0, 2.0, 3.0, 4.0])
print(x.dtype, x.shape, x.device)

ints = torch.tensor([1, 2, 3])
print(ints.dtype)

matrix = torch.tensor([[1., 2., 3.], [4., 5., 6.]])
print(matrix.shape, matrix.ndim)

print(torch.zeros(2, 3))
print(torch.arange(0, 10, 2))
print(torch.linspace(0, 1, 5))

$ python tensors.py
torch.float32 torch.Size([4]) cpu
torch.int64
torch.Size([2, 3]) 2
tensor([[0., 0., 0.],
        [0., 0., 0.]])
tensor([0, 2, 4, 6, 8])
tensor([0.0000, 0.2500, 0.5000, 0.7500, 1.0000])

A tensor built from Python floats is float32 — not NumPy’s float64 — while one built from Python ints is int64. torch.arange excludes its end point (no 10), whereas torch.linspace includes both ends and chooses the step for you.

Tensors and NumPy Can Share Memory

torch.from_numpy and torch.tensor both turn an array into a tensor, but only one of them copies.

import numpy as np
import torch

arr = np.array([1.0, 2.0, 3.0])
shared = torch.from_numpy(arr)   # same memory as arr
copied = torch.tensor(arr)       # new memory

arr[0] = 100.0
print("numpy: ", arr)
print("shared:", shared)
print("copied:", copied)
print(shared.dtype)

$ python numpy_bridge.py
numpy:  [100.   2.   3.]
shared: tensor([100.,   2.,   3.], dtype=torch.float64)
copied: tensor([1., 2., 3.], dtype=torch.float64)
torch.float64

Writing to the NumPy array changed shared too, because from_numpy wraps the same buffer. It also kept NumPy’s float64 dtype, which is why a model expecting float32 input can complain about tensors that came from NumPy. torch.tensor copied, so copied still holds the original values.

Tensor Operations

import torch

a = torch.tensor([[1., 2.], [3., 4.]])
b = torch.tensor([[5., 6.], [7., 8.]])

print(a * b)        # element-wise
print(a @ b)        # matrix multiply
print(a.sum())
print(a.sum(dim=0)) # collapse the rows -> one value per column
print(a.mean(dim=1))
print(a.argmax())   # index into the flattened tensor
print(a[a > 2])     # boolean mask -> 1-D result

$ python ops.py
tensor([[ 5., 12.],
        [21., 32.]])
tensor([[19., 22.],
        [43., 50.]])
tensor(10.)
tensor([4., 6.])
tensor([1.5000, 3.5000])
tensor(3)
tensor([3., 4.])

* multiplies matching positions; @ is matrix multiplication — [1, 2] · [5, 7] = 19 for the top-left cell. sum(dim=0) removes dimension 0 (the rows), leaving one total per column: [1+3, 2+4] = [4, 6]. argmax() without a dim returns 3, an index into the flattened tensor, not a (row, column) pair. A boolean mask always produces a 1-D result, whatever shape it started from.

Views, Strides, and Contiguity

view reinterprets existing memory with a new shape; it never copies.

import torch

x = torch.arange(6.)
v = x.view(2, 3)
v[0, 0] = 99.
print(x)            # the view shares storage with x

t = v.t()           # transpose: same storage, different strides
print(t.is_contiguous())
print(t.reshape(6)) # reshape copies when it has to

try:
    t.view(6)
except RuntimeError as e:
    print("view failed:", e)

$ python views.py
tensor([99.,  1.,  2.,  3.,  4.,  5.])
False
tensor([99.,  3.,  1.,  4.,  2.,  5.])
view failed: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

Writing 99 through the view changed x — they are the same storage. Transposing only swaps strides, so t reads memory out of order and is no longer contiguous. reshape handles that by copying (notice the reordered values 99, 3, 1, 4, 2, 5), while view refuses and tells you to use reshape.

Dtype Promotion

Arithmetic between different dtypes picks a result dtype for you, and the rules are not always the ones you would guess.

import torch

ints = torch.tensor([1, 2, 3])
print(ints / 2)              # true division promotes to float
print(ints // 2)             # floor division stays int64
print(ints + 0.5)            # int tensor + Python float -> float32

halves = torch.tensor([0.1, 0.2], dtype=torch.float16)
print(halves + torch.tensor([1, 1]))   # float16 wins over int64
print(ints.float().dtype, ints.to(torch.float64).dtype)

$ python dtypes.py
tensor([0.5000, 1.0000, 1.5000])
tensor([0, 1, 1])
tensor([1.5000, 2.5000, 3.5000])
tensor([1.0996, 1.2002], dtype=torch.float16)
torch.float32 torch.float64

/ on integers always produces floats, while // stays integer. Adding a Python float to an int64 tensor gives float32, the default float dtype, not float64. When two tensors mix a float and an integer dtype, the float dtype wins even if it is the smaller float16, which is why 0.1 + 1 shows up as 1.0996 — the nearest value float16 can represent. Cast explicitly with .float() or .to(dtype) when you need a particular type.

Moving to GPU

import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")

x = torch.randn(100, 10).to(device)
w = torch.randn(10, 5, device=device)   # or create it on the device directly
result = x @ w
print(result.shape, result.device)

back = result.cpu().numpy()   # NumPy only understands CPU memory
print(type(back), back.shape)

$ python device.py
Using device: cpu
torch.Size([100, 5]) cpu
<class 'numpy.ndarray'> (100, 5)

On a machine without CUDA this prints cpu, and the same code runs unchanged on a GPU machine — which is the point of writing device once and using .to(device) everywhere. Every tensor in one operation must live on the same device. .numpy() needs CPU memory, so on a GPU you call .cpu() first; on the CPU it is a no-op.

Autograd — Automatic Differentiation

Autograd records operations on tensors with requires_grad=True, then backward() walks that record in reverse to compute gradients.

import torch

x = torch.tensor(3.0, requires_grad=True)

y = x ** 2 + 2 * x + 1       # dy/dx = 2x + 2
y.backward()
print(x.grad)

y = x ** 2 + 2 * x + 1       # same computation, second backward
y.backward()
print(x.grad)                # gradients accumulate

x.grad.zero_()
(x ** 2 + 2 * x + 1).backward()
print(x.grad)

$ python grad.py
tensor(8.)
tensor(16.)
tensor(8.)

The derivative of x² + 2x + 1 at x = 3 is 2·3 + 2 = 8, and the first backward() gets it right. The second call prints 16, not 8: backward() adds into .grad rather than replacing it. That is why every training loop clears gradients each step, and why zero_() brings the answer back to 8.

Turning Tracking Off

import torch

x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = (x ** 2).sum()
y.backward()
print(x.grad)                 # d(sum x_i^2)/dx_i = 2 * x_i

print(y.requires_grad, type(y.grad_fn).__name__)

with torch.no_grad():
    z = x ** 2
print(z.requires_grad, z.grad_fn)

d = x.detach()
print(d.requires_grad, d.data_ptr() == x.data_ptr())

$ python no_grad.py
tensor([2., 4., 6.])
True SumBackward0
False None
False True

For y = x₁² + x₂² + x₃² the gradient is 2xᵢ, so [2, 4, 6]. y remembers how it was made (SumBackward0); the same arithmetic inside torch.no_grad() produces a tensor with no history, which is what you want for inference — no graph is built, so no memory is spent on it. detach() also drops tracking, but the final True shows it shares storage with x: modifying one modifies the other.

A Trap: Scaling a Tensor After requires_grad=True

A common way to write “small random weights” silently produces a tensor that never receives a gradient.

import warnings
import torch

W1 = torch.randn(2, 4, requires_grad=True) * 0.3   # scaled after creation
print(W1.is_leaf, W1.requires_grad, type(W1.grad_fn).__name__)

(W1 ** 2).sum().backward()

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    print(W1.grad)
print(str(caught[0].message).split(". ")[0])

$ python xor_broken.py
False True MulBackward0
None
The .grad attribute of a Tensor that is not a leaf Tensor is being accessed

torch.randn(..., requires_grad=True) is the leaf; * 0.3 creates a new tensor computed from it (MulBackward0). Autograd only fills .grad on leaves, so W1.grad stays None and a later W1 -= lr * W1.grad would crash. The fix is to finish building the value first and turn on tracking last, with .requires_grad_().

Your First Neural Network with Autograd

A two-layer network, trained on XOR with nothing but tensors, autograd, and a manual update step:

import torch
import torch.nn.functional as F

torch.manual_seed(42)
X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
y = torch.tensor([[0.], [1.], [1.], [0.]])

W1 = (torch.randn(2, 8) * 0.5).requires_grad_()   # scale first, then track
b1 = torch.zeros(8, requires_grad=True)
W2 = (torch.randn(8, 1) * 0.5).requires_grad_()
b2 = torch.zeros(1, requires_grad=True)
params = [W1, b1, W2, b2]

for epoch in range(1, 2001):
    out = torch.sigmoid(F.relu(X @ W1 + b1) @ W2 + b2)
    loss = F.binary_cross_entropy(out, y)
    loss.backward()
    with torch.no_grad():
        for p in params:
            p -= 0.5 * p.grad
            p.grad.zero_()
    if epoch % 400 == 0:
        print(f"epoch {epoch:4d}  loss {loss.item():.4f}")

print("predictions:", (out > 0.5).int().flatten().tolist())

$ python xor.py
epoch  400  loss 0.0056
epoch  800  loss 0.0021
epoch 1200  loss 0.0012
epoch 1600  loss 0.0008
epoch 2000  loss 0.0006
predictions: [0, 1, 1, 0]

The update happens under torch.no_grad() because the weight change itself must not become part of the next step’s graph, and each gradient is zeroed right after it is used. The network reaches the XOR truth table [0, 1, 1, 0] — something no single linear layer can represent, as the third practice task shows. Everything nn.Module and torch.optim add in the next lesson is a tidier way to write these same four parameters and this same loop.

Practice

import numpy as np
import torch

t = torch.from_numpy(np.zeros(3))
arr = t.numpy()          # also shares memory
t.add_(5)                # in-place: writes into the shared buffer
print(arr)

t = t + 1                # out-of-place: t now points at new memory
print(arr, t)

$ python p1_reverse.py
[5. 5. 5.]
[5. 5. 5.] tensor([6., 6., 6.], dtype=torch.float64)

import torch

x = torch.tensor(2.0, requires_grad=True)
for _ in range(3):
    x.grad = None                 # reset instead of accumulating
    (x ** 3).backward()           # d/dx x^3 = 3x^2 = 12 at x=2
    print(x.grad)

$ python p2_cube.py
tensor(12.)
tensor(12.)
tensor(12.)

import torch
import torch.nn.functional as F

torch.manual_seed(42)
X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
y = torch.tensor([[0.], [1.], [1.], [0.]])

W1 = (torch.randn(2, 8) * 0.5).requires_grad_()
b1 = torch.zeros(8, requires_grad=True)
W2 = (torch.randn(8, 1) * 0.5).requires_grad_()
b2 = torch.zeros(1, requires_grad=True)
params = [W1, b1, W2, b2]

for epoch in range(1, 2001):
    out = torch.sigmoid((X @ W1 + b1) @ W2 + b2)     # no ReLU
    loss = F.binary_cross_entropy(out, y)
    loss.backward()
    with torch.no_grad():
        for p in params:
            p -= 0.5 * p.grad
            p.grad.zero_()

print(f"final loss {loss.item():.4f}")
print("probabilities:", [round(v, 3) for v in out.flatten().tolist()])

$ python p3_linear_xor.py
final loss 0.6931
probabilities: [0.5, 0.5, 0.5, 0.5]

import torch

t = torch.arange(6.).view(2, 3).t()
fixed = t.contiguous()            # copies into row-major order
print(fixed.is_contiguous())
print(fixed.view(6))
print(t.data_ptr() == fixed.data_ptr())

$ python p4_contiguous.py
True
tensor([0., 3., 1., 4., 2., 5.])
False