Matrix Transformations: Visualizing Linear Algebra for Machine Learning

There’s a particular wall that people hit when studying deep learning — somewhere around the point where the tutorials run out and the papers begin. The PyTorch calls still work, the loss still goes down, but nothing quite makes sense anymore. Gradients vanish. Models don’t converge. Architectures that “should” work don’t.

Nine times out of ten, the root cause is that matrices were never understood as anything other than data containers — glorified spreadsheets. That framing is useful for storage, but it actively obscures what a neural network is doing to your data.

In linear algebra, a matrix is an active transformation of space. When a feed-forward layer processes an input, it isn’t querying a table. It’s warping the geometric coordinate system the data lives in — rotating it, scaling it, shearing it — until the structure of the data becomes legible. Everything else in deep learning is built on top of that operation.

The Coordinate System Is Not Fixed

Before we can talk about transforming space, we need to be precise about what defines it.

Any point in a 2D plane is specified by two numbers: how far it travels horizontally and how far it travels vertically. But that description depends on what “horizontal” and “vertical” mean — and those directions are defined by two foundational vectors we call basis vectors:

$\hat{i}$ : one unit to the right, $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$
$\hat{j}$ : one unit upward, $\begin{bmatrix} 0 \\ 1 \end{bmatrix}$

Every vector in 2D space is a linear combination of these two. When you write $\vec{v} = \begin{bmatrix} 3 \\ 2 \end{bmatrix}$ , you’re saying: take $\hat{i}$ , scale it by 3, take $\hat{j}$ , scale it by 2, add them:

$\vec{v} = 3\begin{bmatrix} 1 \\ 0 \end{bmatrix} + 2\begin{bmatrix} 0 \\ 1 \end{bmatrix} = \begin{bmatrix} 3 \\ 2 \end{bmatrix}$

The key shift: the grid itself is downstream of $\hat{i}$ and $\hat{j}$ . Move the basis vectors, and every point in the space moves with them — because every point was defined in terms of those vectors. A matrix is precisely a description of where the basis vectors land after a transformation.

Matrices as Spatial Instructions

A $2 \times 2$ matrix encodes exactly two pieces of information: the new position of $\hat{i}$ (first column) and the new position of $\hat{j}$ (second column):

$W = \begin{bmatrix} a & b \\ c & d \end{bmatrix}$

The first column $\begin{bmatrix} a \\ c \end{bmatrix}$ is where $\hat{i}$ lands. The second column $\begin{bmatrix} b \\ d \end{bmatrix}$ is where $\hat{j}$ lands. Matrix-vector multiplication then follows from that definition directly: to find where input $\vec{x}$ ends up, scale the new $\hat{i}$ by $x$ and the new $\hat{j}$ by $y$ , then add:

$W\vec{x} = \begin{bmatrix} a & b \\ c & d \end{bmatrix}\begin{bmatrix} x \\ y \end{bmatrix} = x\begin{bmatrix} a \\ c \end{bmatrix} + y\begin{bmatrix} b \\ d \end{bmatrix}$

The Shear Transformation

A shear transformation pushes one axis while leaving the other fixed. To shear horizontally — sliding the top of the space right while keeping the bottom anchored — we leave $\hat{i}$ in place and push $\hat{j}$ one unit to the right:

$W_{shear} = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}$

Apply this to an arbitrary point:

$W_{shear}\begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} x + y \\ y \end{bmatrix}$

A square input becomes a parallelogram. This is, geometrically, what happens to your data in the first dense layer of a network.

Determinant: How Much Does Space Scale?

The determinant of a $2 \times 2$ matrix measures how much the transformation scales area. If the original input occupies a region of area 1, after applying $W$ , that region has area $|\det(W)|$ :

$\det\begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc$

For $W_{shear}$ above: $\det = (1)(1) - (1)(0) = 1$ . Area is preserved — the parallelogram has the same area as the original square.

A negative determinant means the transformation flips orientation (a reflection is involved). And a determinant of exactly zero means the transformation is singular: the entire 2D plane collapses into a line (or a point).

Dimensional Collapse

This is the degenerate case worth understanding carefully. Consider:

$W_{collapse} = \begin{bmatrix} 2 & -1 \\ 4 & -2 \end{bmatrix}$

The new $\hat{i}$ is $\begin{bmatrix} 2 \\ 4 \end{bmatrix}$ and the new $\hat{j}$ is $\begin{bmatrix} -1 \\ -2 \end{bmatrix}$ . Notice that $\hat{j} = -\frac{1}{2}\hat{i}$ — they’re collinear. When the two basis vectors land on the same line, the entire 2D plane gets crushed onto that line. Every point that was anywhere in the plane is mapped to some point on that one diagonal.

Verify: $\det(W_{collapse}) = (2)(-2) - (-1)(4) = -4 + 4 = 0$ . Zero determinant, collapsed space.

This is irreversible. Once you’ve crushed a 2D plane to a 1D line, you can’t reconstruct where any particular point came from — infinitely many input points map to the same output. There’s no inverse matrix because information has been permanently destroyed.

In ML, this collapse is used intentionally. Embedding layers and autoencoders use non-square weight matrices to project high-dimensional data into lower-dimensional spaces, discarding redundant dimensions and keeping the principal structure. The non-invertibility isn’t a bug; it’s the compression step.

Linear Layers in Neural Networks

A dense (fully connected) layer implements:

$\vec{y} = W\vec{x} + \vec{b}$

Each term has a geometric role:

$W\vec{x}$ applies the transformation — rotating, scaling, shearing the space. This part is a linear map: lines stay lines, the origin stays fixed.

$+\,\vec{b}$ is a translation, shifting the entire transformed space by a constant offset. This breaks the origin-fixed constraint of pure linear maps and produces an affine transformation (linear transformation + translation). Without the bias, a neuron with zero input would always output zero regardless of weights, which severely limits representational capacity.

The goal during training is to find a $W$ and $\vec{b}$ that reshapes the input space so that whatever you’re trying to predict — spam vs. not spam, cat vs. dog — becomes geometrically separable. Backpropagation is the algorithm that calculates how to adjust the entries of $W$ to move the loss in the right direction.

Why Depth Alone Doesn’t Help: The Linearity Collapse Problem

Here’s the architectural trap. Suppose we stack three dense layers with no activation functions between them:

$\vec{y_1} = W_1\vec{x} + \vec{b_1}$ $\vec{y_2} = W_2\vec{y_1} + \vec{b_2} = W_2(W_1\vec{x} + \vec{b_1}) + \vec{b_2} = W_2W_1\vec{x} + (W_2\vec{b_1} + \vec{b_2})$ $\vec{y_3} = W_3\vec{y_2} + \vec{b_3} = W_3W_2W_1\vec{x} + (W_3W_2\vec{b_1} + W_3\vec{b_2} + \vec{b_3})$

The final expression has the form $W_{eff}\vec{x} + \vec{b}_{eff}$ , where both $W_{eff} = W_3W_2W_1$ and $\vec{b}_{eff}$ are just constants. Three layers of linear transformations collapse into one single affine map. A network with 1,000 such layers is mathematically equivalent to one with exactly one.

The consequence: a purely linear network can only learn linear decision boundaries. No matter how many layers you add, it can only draw hyperplanes through the data. Spirals, concentric rings, XOR — none of these are linearly separable, and a linear model cannot learn them at any depth.

Nonlinearity: Breaking the Collapse

The solution is to introduce a nonlinear function between layers — an activation function — that breaks the composability of linear maps.

The most common choice is ReLU (Rectified Linear Unit):

$f(x) = \max(0, x)$

ReLU sets all negative values to zero and leaves positives unchanged. Applied element-wise after each linear layer, it folds the transformed space: any coordinates pushed into negative territory get snapped to the axis. This folding is genuinely nonlinear — it cannot be represented by any matrix — so the composability argument no longer applies. Three ReLU layers are not equivalent to one.

Why ReLU specifically, rather than Sigmoid or tanh? The short answer is that Sigmoid and tanh both saturate: for large inputs (positive or negative), their gradients approach zero. During backpropagation through deep networks, this causes gradients to vanish — updates to early layers become negligible, and the network stops learning. ReLU doesn’t saturate on the positive side, so gradients flow through cleanly. The cost is “dead neurons” (units stuck outputting zero when their input is persistently negative), which is why variants like Leaky ReLU or GELU are used in practice, but the core motivation is gradient preservation.

Implementation: Tracing a Vector Through a Layer

The following demonstrates a single dense layer with ReLU, tracking what happens to a specific input vector at each step. One thing to notice: no explicit loops over data points. The @ operator performs matrix multiplication on the entire batch simultaneously, which on modern hardware maps directly to SIMD vector instructions.

import numpy as np

def relu(x: np.ndarray) -> np.ndarray:
    return np.maximum(0, x)

# Weight matrix and bias for a single dense layer
W = np.array([
    [ 1.5, -0.5],
    [ 0.8,  2.0]
])

b = np.array([[-1.0],
              [-0.5]])

# A single input vector in the bottom-right quadrant
x = np.array([[1.0],
              [-1.0]])

print("Input vector:")
print(x)

linear_out = (W @ x) + b
print("\nAfter linear transform (W @ x + b):")
print(linear_out)

final_out = relu(linear_out)
print("\nAfter ReLU:")
print(final_out)

Input vector:
[[ 1.]
 [-1.]]

After linear transform (W @ x + b):
[[ 1. ]
 [-1.7]]

After ReLU:
[[1.]
 [0.]]

The input started at $(1, -1)$ . The weight matrix and bias moved it to $(1, -1.7)$ . ReLU then clamped the negative $y$ -component to zero, landing it at $(1, 0)$ on the horizontal axis.

That sequence — linear map, then nonlinear clamp — is the atomic operation that every layer in a feed-forward network repeats. Stack enough of them, and the network can carve out arbitrarily complex decision regions in the input space. The training process (gradient descent + backpropagation) is entirely about finding the $W$ and $\vec{b}$ values that make those regions align with the labels in your data.

Understanding that the weights are spatial instructions, not lookup values, is what makes the rest of the math fall into place: why initialization matters (you’re setting the initial geometry), why batch normalization helps (it reconditions the transformed space between layers), and why very deep networks without residual connections struggle (the effective transformation accumulates numerical instability across many matrix products).