The proof

For seven chapters you wrote parts and tested each one green on its own. Now comes the test that counts: a task no straight line can solve, computed from start to finish with your own framework.

The task is two interlocking half moons. It is the standard example for "not linearly separable": no straight line cuts the two classes apart, whereas a network with a ReLU layer in the middle manages it easily. If your framework can do that, it can do what frameworks are built for.

The goal

It runs, and it runs without torch. That sentence is the whole point of the course, and it is checkable: this chapter's milestone test searches your own files for import torch and looks whether the module was loaded anywhere in the process. Both have to come up empty.

The dataset does not come from a library for the same reason. The obvious move would be sklearn.datasets.make_moons - one line, done. But then this course would end on a pip install scikit-learn, and the half moons would be the only piece of the way that somebody else made for you. Two shifted semicircles plus noise are ten lines of NumPy. Zero magic applies to the data as well.

Zero magic in this step

NumPy is allowed, torch, jax, tinygrad and autograd are forbidden - and in this chapter scikit-learn on top, along with any other library that would hand you datasets, splits or metrics. Dataset, split and accuracy you write yourself; together they are not twenty lines.

The exercise

Two files. First the dataset, meintorch/datasets.py:

import numpy as np


def make_moons(n=400, noise=0.15, seed=0):
    """Two interlocking half moons. Ten lines of NumPy, no library."""
    rng = np.random.default_rng(seed)
    n_upper = n // 2
    n_lower = n - n_upper
    # TODO(du): the upper moon is the unit semicircle (cos, sin) for angles
    # from 0 to pi. The lower one is the same semicircle, point-mirrored and
    # shifted down to the right so the two interlock. Stack both classes,
    # append labels 0 and 1 as a column, and finally add noise on top with
    # `rng.normal`.
    raise NotImplementedError

Then the proof script, beweis.py, in the root directory next to meintorch/:

import numpy as np

from meintorch.data import DataLoader
from meintorch.datasets import make_moons
from meintorch.nn import Linear, ReLU, Sequential
from meintorch.tensor import Tensor
from meintorch.train import fit


def accuracy(model, x, y):
    """Share of correctly guessed classes. Threshold 0.5."""
    # TODO(du): predict, threshold at 0.5, compare with y, take the mean.
    raise NotImplementedError


def main():
    x, y = make_moons(n=600, noise=0.12, seed=0)
    # TODO(du): split 80/20 into training and test - permute once, then cut.
    # Build a network 2 -> 16 -> 16 -> 1 with ReLU in between, create a data
    # loader, train with `fit` and print both accuracies at the end.
    raise NotImplementedError


if __name__ == "__main__":
    main()

Hint 1 - the direction

For the moons: a semicircle is (cos t, sin t) for t from 0 to pi. The second moon is the same arc, the other way round and offset - try (1 - cos t, 0.5 - sin t). If you are unsure whether it fits: the two clouds should embrace each other like two brackets, not sit side by side. A scatter plot tells you in five seconds whether the shape is right.

The noise comes last, on both moons together. Without noise all points would lie exactly on two curves, and the task would be boring instead of easy.

For the split: np.random.default_rng(1).permutation(len(x)) gives you a shuffled list of indices. The first 80 percent are training, the rest is test. The test part must not appear in training anywhere - otherwise you measure memorised instead of understood.

For the accuracy: pred > 0.5 and y > 0.5 are both boolean arrays. Their elementwise comparison, averaged, is the hit rate.

Hint 2 - more concrete

• The moons:

      angle_upper = np.linspace(0.0, np.pi, n_upper)
      angle_lower = np.linspace(0.0, np.pi, n_lower)
      upper = np.stack([np.cos(angle_upper), np.sin(angle_upper)], axis=1)
      lower = np.stack(
          [1.0 - np.cos(angle_lower), 0.5 - np.sin(angle_lower)], axis=1
      )
      x = np.concatenate([upper, lower])
      y = np.concatenate([np.zeros(n_upper), np.ones(n_lower)]).reshape(-1, 1)
      x += rng.normal(scale=noise, size=x.shape)
      return x, y
  

• The network: Sequential(Linear(2, 16, seed=0), ReLU(), Linear(16, 16, seed=1), ReLU(), Linear(16, 1, seed=2)). One layer fewer works too, two ReLU layers leave more room.

• Training: fit(net, DataLoader(x_train, y_train, batch_size=32, seed=2), epochs=60, lr=0.1). With those values test accuracy typically lands beyond 97 percent - the required 90 are generous on purpose.

• Accuracy: float(((model(Tensor(x)).data > 0.5) == (y > 0.5)).mean()).

# in datasets.py
def make_moons(n=400, noise=0.15, seed=0):
    rng = np.random.default_rng(seed)
    n_upper = n // 2
    n_lower = n - n_upper
    angle_upper = np.linspace(0.0, np.pi, n_upper)
    angle_lower = np.linspace(0.0, np.pi, n_lower)
    upper = np.stack([np.cos(angle_upper), np.sin(angle_upper)], axis=1)
    lower = np.stack(
        [1.0 - np.cos(angle_lower), 0.5 - np.sin(angle_lower)], axis=1
    )
    x = np.concatenate([upper, lower])
    y = np.concatenate([np.zeros(n_upper), np.ones(n_lower)]).reshape(-1, 1)
    x += rng.normal(scale=noise, size=x.shape)
    return x, y


# in beweis.py
def accuracy(model, x, y):
    pred = model(Tensor(x)).data
    return float(((pred > 0.5) == (y > 0.5)).mean())


def main():
    x, y = make_moons(n=600, noise=0.12, seed=0)
    idx = np.random.default_rng(1).permutation(len(x))
    cut = int(0.8 * len(x))
    train_idx, test_idx = idx[:cut], idx[cut:]

    net = Sequential(
        Linear(2, 16, seed=0),
        ReLU(),
        Linear(16, 16, seed=1),
        ReLU(),
        Linear(16, 1, seed=2),
    )
    loader = DataLoader(x[train_idx], y[train_idx], batch_size=32, seed=2)
    history = fit(net, loader, epochs=60, lr=0.1)

    print("loss first %.4f, last %.4f" % (history[0], history[-1]))
    print("training %.3f" % accuracy(net, x[train_idx], y[train_idx]))
    print("test     %.3f" % accuracy(net, x[test_idx], y[test_idx]))

In goal mode

The last autonomous brief is the only one that checks the whole stretch:

Create `meintorch/datasets.py` with `make_moons(n=400, noise=0.15, seed=0)`: two
interlocking half moons from pure NumPy, returning (x of shape (n, 2), y of
shape (n, 1) with labels 0/1), classes of equal size. Write `beweis.py` that
builds an 80/20 split from it, trains a network 2-16-16-1 with ReLU via `fit`
and a data loader, and prints training and test accuracy.
Mandatory: NumPy allowed, torch/jax/tinygrad/autograd forbidden, plus no
scikit-learn - write dataset, split and metric yourself.
Write `tests/test_08_beweis.py`: the moons have to give two equally sized
classes in the right shape, no file under `meintorch/` may contain
`import torch` and `torch` must not appear in `sys.modules`, and the trained
network has to reach over 90 percent accuracy on the test part.
Run `python -m pytest` until everything is green.

The milestone test

tests/test_08_beweis.py, run from the root directory. The second test is the title of the course as an assertion, the third is the milestone: over 90 percent on data the network has never seen.

import pathlib
import sys

import numpy as np

from meintorch.data import DataLoader
from meintorch.datasets import make_moons
from meintorch.nn import Linear, ReLU, Sequential
from meintorch.tensor import Tensor
from meintorch.train import fit


def test_the_moons_are_two_equally_sized_classes():
    x, y = make_moons(n=600, noise=0.12, seed=0)
    assert x.shape == (600, 2)
    assert y.shape == (600, 1)
    assert int(y.sum()) == 300


def test_no_torch_involved():
    assert "torch" not in sys.modules
    for file in pathlib.Path("meintorch").glob("*.py"):
        assert "import torch" not in file.read_text()


def test_the_network_separates_the_moons_above_90_percent():
    x, y = make_moons(n=600, noise=0.12, seed=0)
    idx = np.random.default_rng(1).permutation(len(x))
    cut = int(0.8 * len(x))
    train_idx, test_idx = idx[:cut], idx[cut:]

    net = Sequential(
        Linear(2, 16, seed=0),
        ReLU(),
        Linear(16, 16, seed=1),
        ReLU(),
        Linear(16, 1, seed=2),
    )
    loader = DataLoader(x[train_idx], y[train_idx], batch_size=32, seed=2)
    history = fit(net, loader, epochs=60, lr=0.1)
    assert history[-1] < history[0] / 5

    pred = net(Tensor(x[test_idx])).data
    accuracy = float(((pred > 0.5) == (y[test_idx] > 0.5)).mean())
    assert accuracy > 0.9

What you have now

A directory with seven files and no dependency other than NumPy. Inside it: a tensor that computes and remembers what it came from; an autograd that handles the chain rule as bookkeeping; layers, a loss, an optimizer; a loop that turns all of it into training, and a data loader that feeds it.

The difference to before is not that you can train networks now - you could do that with three lines of PyTorch before. The difference is that loss.backward() is no longer a black box for you. When a training run does not converge, you now have candidates instead of bafflement: a forgotten zero_grad(), too large a step size, a gradient that falls out of the graph somewhere because somebody computed on a raw array.

And the way there was the same as with the agents next door: build it once yourself. The five lines from chapter 06 are the same figure as the loop in the tools and loop building block - repeat first, and then it gets clever.