NumPy Matrix Operations

Matrix Multiplication with @

import numpy as np

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

print(A @ B)
print(A * B)
print(np.dot([1, 2, 3], [4, 5, 6]))

$ python matmul.py
[[19 22]
 [43 50]]
[[ 5 12]
 [21 32]]
32

A @ B is the matrix product: the top-left 19 is row 0 of A dotted with column 0 of B (1·5 + 2·7). A * B multiplies matching positions only. Both are valid for square matrices, so mixing them up gives wrong numbers rather than an error.

Inner Dimensions Must Match

import numpy as np

X = np.ones((100, 20))
W = np.ones((20, 10))
print((X @ W).shape)

try:
    X @ np.ones((10, 20))
except ValueError as e:
    print("ValueError:", e)

$ python matmul_shapes.py
(100, 10)
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 10 is different from 20)

(100, 20) @ (20, 10) works because the inner 20s match, and the result takes the outer dimensions, (100, 10) — the shape of a dense neural-network layer. Swap in a (10, 20) weight matrix and the error message spells out which core dimensions disagree.

Transpose

import numpy as np

A = np.array([[1, 2, 3], [4, 5, 6]])
AT = A.T
print(AT.shape)
print(AT)

AT[0, 0] = 99
print(A[0, 0])

batch = np.ones((8, 28, 28, 3))
print(np.transpose(batch, (0, 3, 1, 2)).shape)

$ python transpose.py
(3, 2)
[[1 4]
 [2 5]
 [3 6]]
99
(8, 3, 28, 28)

.T is a view, so writing through it changed A as well. For more than two dimensions, np.transpose takes the new axis order explicitly — here converting images from channels-last (N, H, W, C) to channels-first (N, C, H, W).

Reshape and -1

import numpy as np

arr = np.arange(24)
print(arr.reshape(4, 6).shape, arr.reshape(2, 3, -1).shape)

images = np.ones((32, 28, 28))
print(images.reshape(32, -1).shape)

try:
    arr.reshape(5, -1)
except ValueError as e:
    print("ValueError:", e)

$ python reshape.py
(4, 6) (2, 3, 4)
(32, 784)
ValueError: cannot reshape array of size 24 into shape (5,newaxis)

-1 asks NumPy to compute one dimension from the others: 24 / (2·3) = 4, and 28·28 = 784 for flattening each image. The total size must still work out, and 24 doesn’t divide by 5, so that reshape fails.

Stacking

import numpy as np

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
print(np.stack([a, b]).shape, np.stack([a, b], axis=1).shape)
print(np.concatenate([a, b]))

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
print(np.vstack([A, B]).shape, np.hstack([A, B]).shape)

$ python stacking.py
(2, 3) (3, 2)
[1 2 3 4 5 6]
(4, 2) (2, 4)

np.stack creates a new axis, so two length-3 vectors become (2, 3) or (3, 2). concatenate, vstack and hstack join along an existing axis, so the dimension count stays the same and only one length grows.

Splitting

import numpy as np

arr = np.arange(12).reshape(3, 4)

left, right = np.hsplit(arr, [2])
print(left.shape, right.shape)
print([p.shape for p in np.array_split(arr, 3, axis=1)])

try:
    np.hsplit(arr, 3)
except ValueError as e:
    print("ValueError:", e)

$ python splitting.py
(3, 2) (3, 2)
[(3, 2), (3, 1), (3, 1)]
ValueError: array split does not result in an equal division

A list gives split positions: [2] cuts before column 2. An integer asks for that many equal pieces, and 4 columns can’t be split evenly into 3, so hsplit refuses. array_split allows unequal pieces, giving widths 2, 1 and 1.

A Neural-Network Forward Pass

import numpy as np

rng = np.random.default_rng(42)
sizes = [784, 256, 10]
Ws = [rng.standard_normal((i, o)) * np.sqrt(2 / i) for i, o in zip(sizes, sizes[1:])]
bs = [np.zeros(o) for o in sizes[1:]]

X = rng.standard_normal((32, 784))
h = np.maximum(0, X @ Ws[0] + bs[0])
logits = h @ Ws[1] + bs[1]

exp = np.exp(logits - logits.max(axis=1, keepdims=True))
probs = exp / exp.sum(axis=1, keepdims=True)

print(h.shape, logits.shape)
print(probs.sum(axis=1)[:3])
print(probs[0].round(3))

$ python forward_pass.py
(32, 256) (32, 10)
[1. 1. 1.]
[0.062 0.015 0.047 0.013 0.116 0.283 0.05  0.301 0.026 0.086]

Each layer is one @ plus a bias that broadcasts across the batch of 32. Softmax subtracts each row’s maximum before exp so large logits can’t overflow, and every row of probabilities sums to

1. With random weights the predictions are meaningless, but the shapes are exactly those of a real model.

Practice

1. Compute A.T @ A for A = np.arange(6).reshape(3, 2). What shape do you expect?
2. Stack three length-4 vectors into a (4, 3) matrix, one vector per column.
3. Reshape np.arange(12) into shape (3, 4) and then back to 1-D. Does the round trip share memory with the original?

import numpy as np

A = np.arange(6).reshape(3, 2)
print(A.T @ A)

$ python practice_gram.py
[[20 26]
 [26 35]]

import numpy as np

vs = [np.arange(4), np.arange(4) * 10, np.arange(4) * 100]
M = np.stack(vs, axis=1)
print(M.shape)
print(M)

$ python practice_columns.py
(4, 3)
[[  0   0   0]
 [  1  10 100]
 [  2  20 200]
 [  3  30 300]]

import numpy as np

a = np.arange(12)
back = a.reshape(3, 4).reshape(-1)
print(back.shape, np.shares_memory(a, back))

$ python practice_roundtrip.py
(12,) True