A CNN with No Framework: Pothole Detection with LeNet-5 and NumPy

Classifying potholed road versus intact road with a Convolutional Neural Network whose core is written by hand in NumPy. No PyTorch, no TensorFlow, no autograd. Convolution, pooling, ReLU, softmax, cross-entropy and the whole of backpropagation were derived on paper first, then coded.

Third-party libraries only appear at the edges: Pillow for image I/O, scikit-learn for the split, matplotlib for figures. Everything that learns lives in src/cnn/.

Why do it this way

With a framework, loss.backward() hides exactly the part I wanted to understand. Writing the backward pass myself forced answers to questions a framework never raises: what is MaxPool’s derivative with respect to the inputs it didn’t select, and where does a Conv2D gradient flow when it’s strided.

The architecture is LeNet-5, with each layer’s dimensions computed from output = (W − N + 2P)/S + 1:

Input 3×48×48
  → Conv1 6@5×5 + ReLU   → 6×44×44
  → MaxPool 2×2          → 6×22×22
  → Conv2 16@5×5 + ReLU  → 16×18×18
  → MaxPool 2×2          → 16×9×9
  → Flatten              → 1296
  → FC-120 + ReLU
  → FC-84  + ReLU
  → FC-2   + Softmax     → {normal, pothole}

The convolution isn’t four nested loops. tensor_utils.py holds im2col/col2im, which turns convolution into a single matrix multiply — and that’s also what keeps the backward pass from becoming an indexing nightmare, since the gradient is just col2im applied to the result of the same matrix multiply.

Proving the backprop is correct

This is the part people usually skip, and the exact part you cannot skip when there is no autograd guaranteeing anything.

src/cnn/gradcheck.py computes numerical gradients by finite differences and compares them against the analytic gradients from my backward pass. If the difference exceeds 1e-5, some derivative is wrong. It’s its own Makefile target, ahead of training:

make gradcheck   # verify backprop correctness, error < 1e-5
make train

Training without passing gradcheck first means burning hours working out why the loss won’t drop, when the answer is one minus sign.

75.7% → 94.4%

The baseline was bad and overfitting: grayscale 32×32, single model, SGD+momentum, 15 epochs — 75.70% test accuracy against a training accuracy near 100%. The fixes came in stages, and their order of impact wasn’t what I expected:

1. The input signal — by far the biggest lever. Moving from grayscale to RGB and from 32×32 to 48×48. A pothole is distinguished by shadow and texture, and grayscale throws away half the evidence. This one change took a single model from ~79% to ~92.5%.
2. Regularisation. Dropout 0.3 (inverted, with separate train/eval modes), L2 weight decay 1e-4 on the W weights only, and random online augmentation per batch — flip, ±3px shift, contrast, brightness, noise. The static offline augmentation was dropped because the model memorised the augmented images too.
3. Optimisation. Adam at lr 1e-3 with a cosine schedule, 40 epochs, best-validation weights saved and restored at the end.
4. Inference. A three-seed ensemble [42, 7, 123] averaging softmax outputs, combined with test-time augmentation (averaging the original image with its flip). That’s what lifted 92.5% to 94.39%.

Final results over 107 test images: 94.39% accuracy, 91.53% precision, 98.18% recall, 94.74% F1. Confusion matrix: TP 54 / TN 47 / FP 5 / FN 1.

The number I care about most isn’t the accuracy but 98.18% recall with a single false negative. For this problem, missing a pothole costs far more than flagging a smooth stretch of road.

The documentation had to catch up

There’s a non-technical lesson here that was fairly embarrassing. The commit that raised accuracy only changed code and figures. The paper, docs/, and the site still said 75.7%, still described a grayscale 32×32 architecture, and never mentioned the ensemble at all. For several days the repo held two contradictory truths.

docs/ now runs to nine pages — architecture, forward pass, backpropagation, training loop, code structure, formulas, glossary, development notes — with Mermaid diagrams, plus an Indonesian-language paper and an appendix dissecting 17 code modules line by line. The glossary exists because this has to be readable by people who don’t look at ∂L/∂W every day.

Running it

pip install -r requirements.txt
make data        # download the dataset
make label       # build classes → labels.csv
make preprocess  # dataset.npz
make gradcheck
make train
make eval        # test metrics + confusion matrix
make visualize   # feature maps across layers

The web demo (web/server.py, with its own Dockerfile) is deployed to the homelab and can be tried at potholes.arisjirat.com.

Notation follows R. Munir, “21, Convolutional Neural Network”, IF4073 Digital Image Processing, Informatics Engineering, ITB, 2024.