Why use tanh instead of ReLU here?

**ReLU** is the default in deep networks for speed and sparsity, but **tanh** is smooth and bounded, which makes the loss landscape easier for this tiny teaching model and matches many textbook derivations. You can still see vanishing gradients if weights saturate.

Is full-batch the same as stochastic gradient descent?

**SGD** uses small random subsets (or one point) per step; **full-batch** averages the gradient over **all** points each update. Here batch = entire dataset for clarity; noise-free curves but slower movement on large N.

Why might spirals stay partly wrong even after many epochs?

This is a **very small** network with **axis-free** but still limited capacity; spirals are hard. Try **more hidden units**, a **smaller learning rate** for stability, or many more epochs—sometimes the boundary needs fine “wiggles” that require width and patience.

Toy 2-Layer MLP + Backprop (XOR / spiral)

Click-labeled 2D data; tanh hidden layer + logistic output trained by full-batch gradient descent on binary cross-entropy. Heatmap shows P(class 1) evolving across epoch blocks — watch the 0.5 decision contour wrap XOR or untangle spirals.

About this model

This page implements the smallest nonlinear classifier that still matches the “neural network” story taught in intro ML: a single hidden layer with tanh activations and a logistic (sigmoid) output for binary labels. Given labeled points (x, y), the network computes z₁ = W₁ x + b₁, a₁ = tanh(z₁), z₂ = w₂ᵀ a₁ + b₂, and ŷ = σ(z₂). Training minimizes mean binary cross-entropy over the full dataset using full-batch gradient descent—the same gradient-averaging idea as large minibatches, but deterministic and easy to read on a chalkboard. Backpropagation is just the chain rule applied in reverse: the output error ŷ − y propagates to W₂ and b₂, then through tanh′(z₁) = 1 − tanh²(z₁) to W₁ and b₁. A dense heatmap of σ(z₂(x, y)) shows how the decision boundary (roughly the 0.5 level set) reshapes after each epoch block; misclassified training points get a white ring so overfitting is visible instantly. Presets include XOR (linearly inseparable) and two spirals (highly non-convex).

Who it's for: Students learning MLP notation, BCE loss, and manual backprop before frameworks; pairs naturally with the axis-aligned decision-tree lab on this site.

Key terms

Multilayer perceptron
Tanh activation
Sigmoid output
Binary cross-entropy
Full-batch gradient descent
Backpropagation
Decision boundary
XOR problem

How it works

A two-layer perceptron maps (x, y) through a hidden vector with tanh, then a logistic output σ(z) trained by full-batch gradient descent on binary cross-entropy. Backpropagation applies the chain rule: error at the sigmoid is ŷ − y, flows to W₂ and biases, then through tanh′ = 1 − tanh² to W₁. The heatmap shows P(class 1) over the plane so you can watch the decision boundary (≈0.5 contour) move after each epoch block. Try XOR (needs curvature) vs intertwined spirals (needs many cuts); increase hidden width or learning rate to see speed vs stability trade-offs.

Frequently asked questions

Why use tanh instead of ReLU here?: ReLU is the default in deep networks for speed and sparsity, but tanh is smooth and bounded, which makes the loss landscape easier for this tiny teaching model and matches many textbook derivations. You can still see vanishing gradients if weights saturate.
Is full-batch the same as stochastic gradient descent?: SGD uses small random subsets (or one point) per step; full-batch averages the gradient over all points each update. Here batch = entire dataset for clarity; noise-free curves but slower movement on large N.
Why might spirals stay partly wrong even after many epochs?: This is a very small network with axis-free but still limited capacity; spirals are hard. Try more hidden units, a smaller learning rate for stability, or many more epochs—sometimes the boundary needs fine “wiggles” that require width and patience.

Continue with similar topics in this category — or all 85 in Math Visualization.

View category →

NewSchool

Decision Tree Classifier (2D toy)

Launch Simulator

Greedy axis-aligned splits on a click-labeled scatter: compare **Gini** vs **entropy** impurity, max depth, and min-samples-per-leaf; shaded rectangles show leaf decisions, dashed lines show recursive partitions.

NewSchool

Gradient Descent (2D)

Launch Simulator

Level sets of f(x,y) and path (x,y) ← (x,y) − η∇f; bowl or elliptic well.

NewSchool

K-Means Clustering (Lloyd)

Launch Simulator

Click to add points, choose k, randomize centroids, then step Lloyd iterations (assign to nearest centroid, update means). Optional Gaussian-mixture demo; watch within-cluster SSE decrease.

NewSchool

Convex Hull (Graham & QuickHull)

Launch Simulator

Click to add points, drag to move; Graham scan with step playback or QuickHull divide-by-farthest; compare vertex sets.

NewSchool

Logistic Growth

Launch Simulator

dN/dt = rN(1−N/K); exact S-curve vs carrying capacity K.

NewUniversity / research

Conjugate Gradient Solver

Launch Simulator

SPD system Ax=b as quadratic minimization: contour geometry, CG vs steepest descent path, residual norm, and condition number.

Toy 2-Layer MLP + Backprop (XOR / spiral)

About this model

How it works

Frequently asked questions

Related simulators

Decision Tree Classifier (2D toy)

Gradient Descent (2D)

K-Means Clustering (Lloyd)

Convex Hull (Graham & QuickHull)

Logistic Growth

Conjugate Gradient Solver

Toy 2-Layer MLP + Backprop (XOR / spiral)

About this model

How it works

Frequently asked questions