How a CNN actually sees.

00 · The big picture

Two ideas that make vision tractable.

Start with the naive approach and watch it explode. An image is just numbers — flatten 32 × 32 × 1 into a vector of 1,024 values and feed it to a dense layer of 4,096 neurons. Cost: 4.2M parameters — for one layer. Do that to a 224 × 224 photo and you're at 616M before the network has done anything. Worse, a cat shifted two pixels right becomes a completely different input. The dense layer has to re-learn "cat" at every position separately.

Convolution fixes both problems with two commitments:

• Locality. A neuron looks only at a small patch — say 3×3 — not the whole image. Nearby pixels are related; far ones mostly aren't. Bake that in.
• Weight sharing. The same tiny filter slides over every position. An edge detector useful in the top-left is useful everywhere. One set of weights, reused thousands of times.

Together they buy you translation equivariance — shift the input, the feature map shifts with it, for free — and a parameter count that depends on the filter, not the image size. That's the entire trick. Everything else is stacking.

The lineage, briefly

• 1980Neocognitron. Fukushima sketches the local-receptive-field hierarchy, inspired by Hubel & Wiesel's cat-cortex experiments — simple cells detect edges, complex cells tolerate position.
• 1989–98LeNet. LeCun adds backprop and ships it: reading handwritten digits on real US checks. Our reference model. Proof the idea works, decades before the hype.
• 2012AlexNet. Two GPUs, ReLU, dropout, 60M params — and the ImageNet top-5 error drops by ~10 points overnight. The moment deep learning stopped being a niche.
• 2014VGG. The depth thesis, stated cleanly: nothing but 3×3 convs, stacked 16 layers deep. Our second reference model.
• 2015ResNet. Skip connections — x + F(x) — and suddenly 100+ layers train. The same residual idea that later carries transformers (§ LLM guide).
• nowStill everywhere. ViTs lead some leaderboards, but CNNs own the edge: phones, medical imaging, detection, segmentation — and healthcare document pipelines like mine.

01 · The convolution, worked by hand

One output pixel = one dot product.

Here's the whole operation. Lay a k×k kernel on a patch of the input, multiply element-wise, sum, add a bias. That single number is one pixel of the output. Slide, repeat:

Watch it happen, one step at a time — a 6×6 input with a bright vertical bar, convolved with a 3×3 vertical-edge kernel. The two edges of the bar come out with opposite signs:

The three knobs

• Kernel size k — how much context each output pixel sees directly. 3×3 won; bigger kernels lost to stacking (§05).
• Stride s — how far the window jumps. Stride 2 halves the resolution: cheap downsampling built into the conv.
• Padding p — zeros around the border. "Valid" (p=0) shrinks the map by k−1; "same" (p=⌊k/2⌋) preserves the size so you can stack layers without eroding the image away.

The part diagrams hide: channels

Real inputs have depth — RGB is H × W × 3. A "3×3" filter is actually a 3×3×C_in volume: it dots across all input channels at once and still yields one 2D map. Want C_out different feature maps? Learn that many filters. Which gives the parameter formula:

02 · Channels & feature maps

A deep pixel is a description, not a color.

After the first layer, forget colors. The tensor flowing through the network is H × W × C, and the right way to read it: at each spatial position sits a C-dimensional vector describing the patch around that position. Channel 12 might respond to vertical edges, channel 40 to a texture. By the last conv layer of LeNet-5 that vector is 120 numbers summarizing an entire region.

If you've read my LLM guide, this should feel familiar — it's the same shape of idea as a token's residual-stream vector. Transformers carry n × d; CNNs carry H × W × C. Position on a grid instead of position in a sequence, but in both cases: a vector per location, refined layer by layer.

The 1×1 convolution — weirder and more useful than it sounds

A 1×1 conv looks at a single pixel. Pointless? No — it mixes channels: a tiny dense layer applied at every position, C_in → C_out, with k²=1 so it's dirt cheap. Networks use it to compress channels before expensive 3×3s (bottlenecks), to re-project after concatenations, and as half of the depthwise-separable trick below. Spatial reasoning is 3×3's job; channel reasoning is 1×1's.

03 · ReLU & pooling

Cheap nonlinearity, deliberate forgetting.

Between convolutions, two humble operations. First, ReLU: max(0, x), applied elementwise. Without a nonlinearity, stacked convolutions collapse into one big linear filter — the depth would be fake. ReLU keeps positive evidence, zeroes negative, costs one comparison, and (unlike the sigmoids it replaced) doesn't squash gradients for positive activations. That last property is a real part of why deep nets became trainable. Its one pathology: a neuron stuck outputting 0 stops learning ("dead ReLU") — cousins like LeakyReLU and GELU exist partly for that.

Second, max-pooling — typically 2×2, stride 2: keep the strongest response in each little window, discard the rest. H × W × C → H/2 × W/2 × C, with zero parameters. You're throwing away exact positions on purpose: "there's an edge around here" matters; "at pixel (14, 7) precisely" usually doesn't. The reward is tolerance to small shifts and a quadratically shrinking compute bill.

04 · The full shape walk

Follow one image through LeNet-5.

The table below is generated live from the architecture config (input 32×32×1) — every layer, its output tensor, and its exact parameter count. This is the CNN analog of tracing [n × d] through a transformer:

Two distributions worth staring at until they feel obvious:

• Activations concentrate early. The biggest tensor in memory is 28 × 28 × 6 = 4,704 values, right after the first conv — early layers have huge maps with few channels. This is what fills GPU memory during training (you keep them all for backprop).
• Compute concentrates early too, parameters late. The heaviest layer by multiply-accumulates is C3: 240k MACs — big maps × growing channels — while the parameter mass sits in the late, fat layers. Params ≠ FLOPs ≠ memory; experts budget all three separately.

05 · Receptive fields

How 3×3 filters end up seeing the whole image.

A single 3×3 conv sees 3 pixels of context. Useless for recognizing a face — unless you stack. Each layer looks at the previous layer's outputs, which already summarize patches, so the region of the original image influencing one deep neuron — its receptive field — grows with every layer: 3 → 5 → 7 → 9 for stacked 3×3s (RF = 2L+1), and it grows in leaps every time you pool or stride. Slide the layers and watch:

Why stacking beat big kernels

Two stacked 3×3s see 5×5 of context. So why not one 5×5? Count, for C channels in and out: two 3×3s cost 2 · 9C² = 18C²; one 5×5 costs 25C². At C=64 that's 73,728 vs 102,400 — the stack is ~28% cheaper, and it fits an extra ReLU between the layers, making the composed filter more expressive. That inequality, applied ruthlessly, is the entire VGG design philosophy. (Dilated convs are the other lever: spread the 9 taps apart to inflate the RF without new params — standard in segmentation.)

06 · What filters learn

Nobody programs the edge detectors.

In my Conv Lab you set kernels by hand — Sobel, Laplacian, emboss. Here's the remarkable part: train a CNN from random weights and layer 1 rediscovers those filters on its own, every single run. Oriented edges, color-opponent blobs, little Gabor-like gratings — they emerge because they're simply the most useful first questions to ask about natural images. Gradient descent keeps converging on what took vision scientists decades to formalize.

Go deeper and the questions compose. The classic feature-visualization work (Zeiler & Fergus 2014, and later the circuits line of research) shows the hierarchy directly: layer 1 edges → layer 2 corners, textures → middle layers repeated motifs, fur, mesh, text-like patterns → late layers object parts: wheels, eyes, keyboards. Exactly the simple-to-complex cascade the neocognitron guessed at in 1980.

The practical payoff: transfer learning

Because early layers learn generic visual vocabulary and late layers learn task-specific combinations, you almost never train from scratch. Take a network pretrained on ImageNet, chop the head, fine-tune the late layers on your few thousand medical scans or claim forms. The early edge/texture machinery transfers untouched. This asymmetry — general early, specific late — is one of the most load-bearing facts in applied deep learning; it's how small-data domains (like healthcare documents) get big-data performance.

07 · Training a CNN

Same machinery, plus two vision-specific tricks.

Nothing exotic at the core: the final layer emits one logit per class, softmax turns them into probabilities, cross-entropy (−log p of the true class) scores them, and backprop pushes gradients down the stack — the same loop you can watch live in my MLP Lab. One elegant detail: the gradient of a convolution is itself a convolution (with a flipped kernel), so the backward pass reuses the same efficient machinery as the forward pass. Weight sharing means each filter's gradient is summed from every position it visited — one filter, thousands of teachers per image.

Two additions did heavy lifting in the CNN era:

• Batch normalization. After a conv, normalize each channel's activations (per batch) to zero-mean/unit-variance, then let the network re-scale with two learned numbers per channel (γ, β — just 2C parameters). Effect: activations stop drifting between layers, large learning rates become safe, training goes from fragile to boring. The standard sandwich is conv → BN → ReLU. It's the CNN cousin of the LayerNorm you met in transformers — normalizing over different axes for the same reason: stability.
• Data augmentation. Flip, crop, jitter colors, occlude patches (cutout), blend images (mixup). Every transform your label should survive is free training data — and a way of whispering invariances to the network: "a mirrored cat is still a cat." For my document models the equivalents are rotations, scan-noise, and contrast shifts — augmentation is usually the cheapest accuracy you'll ever buy.

Optimizer lore, condensed: classic CNNs trained with SGD + momentum and step-decay schedules (and pure SGD often generalizes slightly better on vision); modern practice frequently uses AdamW + cosine decay for convenience. Weight decay throughout. Watch train/val curves diverge for overfitting; augment or decay harder before reaching for a smaller model.

08 · The architecture zoo

Five ideas, forty years.

Every famous CNN is remembered for one idea. The table is the tour; the ideas below it are the ones worth internalizing:

The ResNet idea deserves its own paragraph. Before 2015, deeper networks often trained worse — not overfitting, just failing to optimize. The fix: make each block compute a correction to its input rather than a replacement, x + F(x). Gradients get an identity highway; 152 layers become routine. If you've read the LLM guide you've seen this exact equation as the residual stream — it is the single most transferable idea in deep learning, and it was born here, in vision.

U-Net matters when the output is an image too — segmentation masks, document layout, denoising (it's the backbone inside diffusion models). Downsample to understand context, upsample to decide per pixel, and pass skip connections across so the decoder recovers the spatial detail that pooling destroyed. It's the architecture I reach for when a claim form needs per-region understanding, not just a label.

09 · Where the parameters live

Auditing all 61.7k, layer by layer.

Same exercise as the LLM guide, same rule: if you can rebuild this table from the formula k²·C_in·C_out + C_out, you understand the architecture. Generated live from the config — audit it (input 32×32×1):

The headline pattern: the convolutional stack — all the actual seeing — is 50.7k, while the dense classifier head is 11.0k (18% of the model). Now flip to VGG-16 and look again: 89% of its 138M parameters sit in three dense layers — the 7×7×512 → 4096 flatten alone costs 102.8M. That absurdity is why the field switched to global average pooling (ResNet's head: 512 × 1000 ≈ 0.5M). Whole eras of architecture design are visible in this one table.

10 · CNNs vs Transformers

Baked-in bias vs learned bias.

A CNN assumes locality and translation-sameness — it's in the wiring. A vision transformer (ViT) chops the image into 16×16 patches, treats them as tokens, and lets attention learn any spatial relationship from scratch. That's the whole disagreement: how much prior knowledge do you hard-code?

• Small data: CNN priors are a gift — they're correct about images, so you don't spend data learning them. CNNs (or heavy augmentation) win.
• Huge data: priors become a ceiling. Given hundreds of millions of images, ViTs learn subtler-than-local structure and pull ahead at the frontier.
• Deployment: convs are compute-dense, cache-friendly, quantize well — phones, embedded, real-time detection stay largely convolutional. Attention's O(n²) over patches bites at high resolution.
• Dense prediction (segmentation, layout, diffusion backbones): conv encoders/decoders and U-Nets remain everywhere, often with attention only in the low-resolution middle.

And the convergence is real: ConvNeXt showed a pure convnet trained with modern recipes matches ViTs; hybrid models put convs early and attention late. The two families share residuals, normalization, the same losses, the same optimizers. Different bets on geometry, same everything else.

11 · Beginner → expert

The revision checklist.

Fourteen sentences again. Check them honestly — each one you can explain with numbers is yours for good.

• Convolution = one small filter slid everywhere; params = k²·C_in·C_out + C_out, independent of image size.
• Output size = ⌊(n + 2p − k)/s⌋ + 1 — memorized cold, including what valid vs same padding does.
• A filter is a k×k×C_in volume producing one 2D map; C_out filters stack into the output tensor.
• After layer 1, channels aren't colors — each spatial position carries a C-dim descriptor of its patch (the CNN's "residual stream").
• 1×1 convs mix channels (per-pixel dense layer); 3×3s handle space; depthwise-separable = those two factored apart, ~9× cheaper.
• ReLU makes stacked convs genuinely nonlinear; pooling/striding buys shift-tolerance and compute at zero parameters.
• Receptive field grows +2 per 3×3 layer and leaps at every downsample; two 3×3s beat one 5×5 (18C² vs 25C², plus an extra ReLU).
• Activations and FLOPs concentrate in early layers, parameters in late ones — memory, compute, and size are three separate budgets.
• BatchNorm normalizes per channel then re-scales with learned γ, β (2C params) — the stability trick that made big learning rates safe.
• Augmentation encodes invariances as free data — the cheapest accuracy available.
• Training is the universal recipe: softmax + cross-entropy + SGD/AdamW; conv gradients are themselves convolutions.
• Layer-1 filters (edges, Gabors) are rediscovered from random init every run; the hierarchy goes edges → textures → parts → objects.
• Transfer learning works because early layers are universal — freeze them, fine-tune the top on your small dataset.
• x + F(x) — ResNet's skip — is the same residual idea that later made 100-layer transformers trainable.

Where to go from here

Draw on the Conv Lab and watch the exact math from §01 run live on your own strokes. Then the rite of passage: implement LeNet-5 yourself and train it on MNIST until the §09 table matches your framework's parameter count digit for digit — 61,706 or you have a bug. After that, read the ResNet paper and carry x + F(x) with you into the LLM guide. The two halves of this site are one story.