Neural Network Playground

A two-input feed-forward classifier trained from scratch with full-batch gradient descent. Configure architecture, learning rate, and dataset; watch each hidden unit develop its own response surface and the global decision boundary form.
Network architecture each unit shows its learned response surface over $[-1, 1]^2$
positive weight negative weight line thickness ∝ |weight|
Decision boundary input space $[-1, 1]^2$
Class 0 Class 1 Test point
Training loss binary cross-entropy

What is happening

Each hidden unit computes $a_j^{(l)} = \sigma\!\left(\sum_k W_{jk}^{(l)} a_k^{(l-1)} + b_j^{(l)}\right)$, with $\sigma$ a non-linearity. The output applies a logistic squash $\hat{y} = (1+e^{-z})^{-1}$, giving binary cross-entropy

$$\mathcal{L}(\theta) = -\frac{1}{N}\sum_{i=1}^{N}\Big[y_i \log \hat{y}_i + (1-y_i)\log(1-\hat{y}_i)\Big].$$

Gradients are obtained by backpropagation (chain rule) and applied via gradient descent $\theta \leftarrow \theta - \eta\, \nabla_\theta \mathcal{L}$. Weights use Glorot initialisation $W \sim \mathcal{U}(-\sqrt{6/(n_\text{in}+n_\text{out})}, +\sqrt{6/(n_\text{in}+n_\text{out})})$. Data are split 80/20 train/test; the loss curve plots training and test loss per epoch. Each unit's heatmap visualises $a_j^{(l)}(x_1, x_2)$ — its activation as a function of inputs, holding learned weights fixed.

Things to try. XOR with a single hidden unit (cannot solve — non-linearly separable; the lone unit's heatmap is just a half-plane). Spiral with $\eta = 1.5$ (diverges). Spiral with $\eta = 0.1$, hidden $[8,\, 8,\, 4]$ — late-layer units develop curved, dataset-specific selectivities. Switch to ReLU on Spiral and look for "dead" units (uniform light heatmaps, zero gradient flow). Increase noise to 0.3 on Checkerboard — test loss eventually rises while training loss continues to fall, a clean overfitting trace.

Exercises

Each exercise asks you to commit to a prediction before running the simulation. The "Apply settings" button configures the playground for you, but the prediction step is non-negotiable — write it down before you click anything.

Exercise 1 — The XOR threshold Architecture & capacity
Setup XOR, noise = 0.05, tanh, $\eta = 0.1$. Train networks with hidden = 1, then 2, then 3.
Predict What is the minimum number of hidden units required to solve XOR?
Run Train each configuration to convergence (around 300 epochs).
Observe
  • Final test accuracy in each case.
  • With hidden = 2 (once it converges): examine the two hidden-unit response surfaces. What does each unit appear to specialise in?
Reflect XOR can be written as $(x_1 \vee x_2) \wedge \neg(x_1 \wedge x_2)$. Do the two hidden units appear to compute something resembling these two clauses? Why does one unit fundamentally fail, regardless of how long you train?
Exercise 2 — Initialisation matters Optimisation dynamics
Setup XOR, noise = 0.05, hidden = 2, tanh, $\eta = 0.1$.
Predict Click Reset 10 times in succession, training each fresh network for ~300 epochs. Will all 10 runs converge to the same solution?
Run For each Reset, record the final test accuracy in a small table.
Observe
  • Distribution of final accuracies across the 10 runs.
  • How often does the network get stuck below 80%?
  • For runs that fail: what do the hidden-unit response surfaces look like? Are both units pointing in similar directions?
Reflect The loss landscape is non-convex — there are local minima and saddle points. What does this exercise reveal about the sensitivity of training to initialisation? Why is it standard practice in deep learning research to report results across multiple random seeds rather than a single run?
then click Reset 10 times
Exercise 3 — Overfitting in real time Bias-variance trade-off
Setup Checkerboard, noise = 0.30, hidden = 16, 16, tanh, $\eta = 0.05$. Train for 500+ epochs.
Predict Sketch what you expect the train and test loss curves to look like. Will they track each other? Diverge? When?
Run Set steps-per-frame to 10 and let it train. Watch the loss panel.
Observe
  • The epoch at which test loss begins to rise while training loss continues falling.
  • Pause around that point. Examine the decision boundary — does it look "wiggly", curving around individual training points?
  • Continue training. How wide does the train/test gap eventually become?
Reflect Mark on your loss plot the epoch at which early stopping would have been optimal. Why doesn't the model "know" to stop on its own — what information would it need that it doesn't have during training?
Exercise 4 — What does each layer compute? Feature learning
Setup Spiral, noise = 0.05, hidden = 8, 8, 4, tanh, $\eta = 0.1$. Train to convergence (500+ epochs).
Predict Will deeper layers have simpler or more complex response surfaces? Why?
Run Train slowly first (steps-per-frame = 1) for 50 epochs to watch features emerge, then increase to 10 to reach convergence.
Observe Categorise the response surfaces in each hidden layer:
  • Layer 1 (h1): Are these oriented half-planes — linear features rotated and shifted across the input space?
  • Layer 2 (h2): Do you see curved or compound features that combine pairs of layer-1 units?
  • Layer 3 (h3): Do these resemble spiral-segment selectivities — features tailored to the dataset structure?
Reflect This is hierarchical feature learning made visible. The universal approximation theorem tells us a single hidden layer suffices in principle — but does that match what you're seeing? Why might depth produce qualitatively different features rather than just "more" features? How does this connect to the way convolutional networks learn edges → textures → object parts?