This post aims to give a quick and simple explanation of the paper “Self-Play Pretraining with Zero Data”

The goal of the method is to be able to train an in-context next token predictor, without externally supplied training data. The predictor can predict the next token given examples in context.

Input: two randomly initialized networks, a generator and a predictor, plus a fixed program interpreter (The interpreter is ordinary, fixed software that executes programs.)

Output: a trained predictor that predicts the token (here, a byte, number from 0 to 255). This predictor can predict the next sequence item, given a few in-context (few-shot) examples.

Consider this toy example, written in Python-like pseudocode for readability. (The paper uses a tiny Brainfuck-like language instead).

1. The generator produces a program, for example:

repeat 3 times:
    emit(0)
    emit(1)

Initially, its programs are random. Nobody supplies this example we show here. The program code can also expect input (optional).

2. The interpreter executes the program.

The interpreter runs the program, supplying the program input with random bytes whenever the program requests them.

The program output becomes training data, e.g.

0, 1, 0, 1, 0, 1

3. The predictor (transformer) practices next-byte prediction. It sees the sequence, without seeing the program.

Standard cross-entropy loss training increases the probabilities of the correct answers. Compute the gradients of this loss, then use them to update the predictor’s weights with an AdamW step.

4. The generator receives a reward.

The paper’s authors define a fixed reward function; the user does not choose it, and it is not learned.

Its inputs are the predictor gradients (from step 3), the predictor’s current and earlier (a checkpoint from roughly halfway through training so far) step weights, and its optimizer statistics

Let $g_j$ be the derivative of this loss with respect to predictor weight $j$, evaluated at the current weights before the update.

\[r = \left|\sum_j g_j P_j\left(w_{\text{earlier},j} - w_{\text{current},j}\right)\right|. \tag{1}\]

The index $j$ runs over every scalar entry in all the predictor’s weight tensors.

$P_j$ is the AdamW scaling factor for that weight, computed from the learning rate and optimizer statistics.

The earlier weights come from a checkpoint roughly halfway back through training. The reward is a single nonnegative number.

Motivation for the reward function: accumulated weight changes indicate what the predictor has been learning.

The reward favors sequences whose gradients strongly overlap that history, as a proxy for related, learnable structure. Already mastered sequences usually give small gradients; unrelated noise is expected to have weak alignment. Because of the absolute value, strong opposition also earns reward, so this is not a guarantee of improvement. Reinforcement learning encourages the generator to produce higher-reward programs.

The reward is higher when the sequence’s gradients have a larger optimizer-weighted overlap, in either direction, with the predictor’s past weight changes, which serves as a proxy for useful training data.

5. Repeat: generate programs, execute them, train the predictor, update the generator.

The implementation also reuses and mutates previous programs.

“Mutates” means creating variations of promising programs by inserting, deleting, or replacing one instruction, helping explore new patterns while preserving earlier discoveries.

“Reuses” means running previously generated programs again to produce training data.

6. After we ran the loop many times, now we discard the generator and keep the predictor, which predicts the next byte given previous bytes.

They put examples of the task into the predictor’s context, followed by a new input, and ask it to predict the answer, without changing its weights.

Another example: An illustrative input would show “2 and 3 gives 5; 4 and 6 gives 10,” then ask “7 and 8 gives what?”, expecting 15.

The paper tested this with byte values, with sums wrapping around at 256, and found accuracy improved as more examples were supplied.

the concrete demonstrations are fairly basic: reversing strings, stack operations, retrieving associated values, addition, minimum, and maximum.

Also, the generator discovered a program producing Fibonacci numbers because its reward encouraged sequences estimated to be useful for the predictor.