neural-chess-engine

How it is built

Architecture overview

A neural chess engine picks a move by running a tree search whose every leaf is judged by one neural network. The network reads a position and returns two things: a policy (how promising each legal move looks) and a value (who is winning). The search spends its visits where the policy points and backs the values up the tree; the most visited root move is played. The same engine, run against itself with exploration noise, produces the training data for the next network, so playing strength comes from a closed loop of search and learning, not from hand-written chess knowledge. Leela Chess Zero (Lc0) is the open reference; it follows the AlphaZero recipe and has moved from convolutional to transformer networks.

Parts:

• Board and move generator — legal moves, make/unmake, position hash — Bitboards, Zobrist hashing
• Search — PUCT tree search over the legal moves, one network call per new leaf — Monte Carlo tree search, Multi-armed bandits, Transposition table
• Evaluator — the policy-value network — Transformer block, Attention variants, Embeddings, Rotary position embedding
• Inference runtime — batches leaves from many searches onto the GPU — Batched MCTS inference, Dynamic batching, Flash attention, Weight quantization
• Self-play workers — play engine against engine, write positions with search targets — AlphaZero self-play
• Data pipeline — turns millions of game files into shuffled training batches — Data loading pipelines, Object storage
• Trainer — fits the network to the search targets — AdamW, Learning rate warmup + cosine, Mixed precision training, ZeRO / FSDP sharding, Ring all-reduce

            ┌──────────────────────── play (UCI) ────────────────────────┐
 position ─►│ move gen ─► PUCT search ──leaves──► batch queue ─► network │─► best move
            │      ▲          ▲   ◄──policy, value──┘   (GPU)           │
            │      └─ TT ─────┘                                          │
            └────────────────────────────────────────────────────────────┘
                                   ▲ new weights
 self-play workers ──games + visit counts──► object store ──► data loader ──► trainer
 (same search + net, noise on)                                (shuffle, augment)  (GPU cluster)

There are two products of one design: the play engine (one process, latency per move matters) and the training loop (many processes, throughput of games and gradient steps matters). Both share the move generator, the search and the network code.

How the parts fit together

Board, move generator and hashing

The search needs legal moves and a position key millions of times per second, so the board is a set of 64-bit masks (Bitboards) and the key is updated incrementally with XOR (Zobrist hashing). Unlike an alpha-beta engine, move generation is not the bottleneck: a neural engine visits far fewer nodes, because each node costs a network evaluation. What matters is that the generator is exact (perft-tested) and that the encoder below can read the board cheaply.

The encoder is the contract between the board and the network. Lc0 feeds 112 input planes of 8×8: 13 planes per position (pieces of each type and colour, plus repetition) for the current and the last 7 positions, plus 8 auxiliary planes (castling rights, side to move, the fifty-move counter and constants) (source: lc0 src/neural/encoder.h, network.h). History is part of the input because repetitions and the fifty-move rule change the result of a position. The board is flipped so the side to move is always "us"; the network never needs to learn both colours separately.

Search: PUCT over a tree of positions

The search is Monte Carlo tree search with the PUCT selection rule from AlphaZero: at each node pick the child that maximises its mean value Q plus an exploration bonus proportional to the network's prior P for that move and to √(parent visits)/(1 + child visits). This is the bandit idea of Multi-armed bandits (UCB) with the network's policy as a prior, so good moves get searched early and bad moves are almost never expanded. There are no random rollouts: the value head replaces them.

Contracts with its neighbours:

• To the network: a leaf is expanded once; its policy is stored as priors on its children and its value is backed up along the path with the sign flipped at each ply.
• Unvisited children: need a first-play urgency (FPU) value; Lc0 exposes it as FpuValue (source: lc0 src/search/classic/params.cc). Too optimistic and the search wastes visits on bad moves, too pessimistic and it never tries a good move with a low prior.
• Terminal positions (mate, stalemate, draw by rule) are scored exactly by the move generator, not by the net.
• Transpositions: chess reaches the same position by different move orders. A cache keyed by the Zobrist hash (Transposition table) avoids evaluating the same position twice; Lc0 keeps an NN cache of network outputs for this.

Evaluator: the policy-value network

The network is a stack of Transformer blocks over 64 tokens, one per square. Each square's input planes are projected into a vector (Embeddings); attention lets every square look at every other square, which matches chess well: a bishop on a1 and a king on h8 interact in one layer, where a 3×3 convolution needs many layers to connect them. Older Lc0 and AlphaZero networks use residual Convolutional networks; they are still a valid, cheaper choice for small nets.

Heads on top of the shared body:

• Policy head: one logit per possible move. Lc0's transformer nets compute it with attention between the from-square and to-square tokens, which suits the from-to structure of chess moves. Illegal moves are masked before the softmax.
• Value head: win/draw/loss probabilities (WDL) rather than one scalar, so the engine knows a 0.0 that is "certain draw" from a 0.0 that is "sharp, either side can win".
• Moves-left head (Lc0): predicts how long the game will last, used to prefer faster wins and slower losses.

Square positions enter through learned position embeddings or relative attention biases; the 64 squares are a fixed grid, so the rotary trick of Rotary position embedding is optional here. See Attention variants for the head layouts.

Inference runtime: keeping the GPU busy

A single search asks for one leaf at a time, but a GPU is efficient only on batches of hundreds. This is the main engineering problem of the play engine, solved by Batched MCTS inference:

• Virtual loss: while a leaf waits for its evaluation, its path is temporarily scored as a loss, so the next selection descends a different path; one search collects a minibatch of distinct leaves (Lc0: MinibatchSize).
• Many searches per GPU in self-play: hundreds of games run at once and their leaves share one batch queue (Dynamic batching): dispatch when the batch is full or the oldest request has waited too long.
• Kernels: attention via Flash attention; fp16 or int8 weights (Weight quantization) cut memory traffic. Lc0 ships CUDA/cuDNN, ROCm, Metal, ONNX runtime and BLAS backends behind one interface (source: lc0 README), so the search code never knows the hardware.

The trade-off is explicit: bigger batches mean more throughput but more virtual-loss distortion inside one tree, so the play engine uses small batches per search and the self-play workers use many parallel games instead.

Self-play and training data

AlphaZero self-play closes the loop. A worker plays the current best network against itself with the same search, but with exploration switched on:

• Dirichlet noise added to the root priors (AlphaZero used α = 0.3 for chess, mixed in at 25 %; Silver et al. 2018), so the engine keeps trying moves the net currently dislikes.
• Moves sampled in proportion to visit counts early in the game, greedy later, so the games are varied but the endings are played well.
• A fixed, small number of visits per move (AlphaZero: 800 simulations per move), much less than in a real game: the goal is many decent games, not a few perfect ones.

Each position is written with its search policy (the root visit distribution) and, once the game ends, the result. These are the training targets: the policy head learns to predict what search found, the value head learns the outcome. Because search is stronger than the raw network, the network improves toward search, and the better network makes the next search stronger.

Data pipeline and trainer

Self-play produces a stream of small game files. The trainer reads a sliding window of the most recent games, because old games were played by weaker networks. Data loading pipelines handle the rest: sharded files in Object storage, shuffled across many games so consecutive samples are not from the same game, decoded on CPU workers, prefetched to the GPU. The board's symmetries can multiply the data (symmetry-data-augmentation, no page yet), but chess has only the left-right mirror, and only when castling rights are gone; AlphaZero did not augment chess data (Silver et al. 2018).

The trainer minimises policy cross-entropy plus value loss plus weight decay, with AdamW or SGD with momentum and a warmup + cosine schedule, in mixed precision. Large nets train data-parallel across GPUs with gradients summed by Ring all-reduce and, when the model plus optimizer state no longer fits one GPU, ZeRO / FSDP sharding. New weights are published to the self-play workers on a fixed cadence; Lc0 runs this as a distributed project where volunteers' clients download the latest network and upload games.

Data and control flow

One move in a game (play engine):

1. The GUI sends the position and the clock over UCI; the engine sets the move history.
2. Time management decides the budget for this move from the clock and the increment.
3. The search loop runs: select a path by PUCT with virtual loss → reach a leaf → check the NN cache → queue the leaf. When a minibatch is full, the backend runs the network once for all queued leaves.
4. Each result expands its leaf (priors on children) and backs up its value; virtual losses are removed.
5. Repeat until the budget is spent or the best move can no longer be overtaken by the remaining visits (Lc0 calls this smart pruning). Most of the time goes to network evaluation on the GPU; the CPU side must keep the next batch ready.
6. Play the most visited root move; keep the subtree under it for the next move (tree reuse).

One training cycle (training loop):

1. Workers download network N and play games with noise and a fixed visit count.
2. Each game is uploaded as positions + search policy + result.
3. The data loader samples from the window of the latest games, shuffles and batches.
4. The trainer runs a number of gradient steps and exports network N+1.
5. Optionally a gating match: N+1 must beat N to be published (AlphaGo Zero did this; AlphaZero dropped it and always used the latest network; Silver et al. 2018).

Scaling

Limits, in the order they bite, and the lever for each:

1. Play strength per move is bound by network evaluations per second. Levers: a smaller or quantized net, fp16 kernels, bigger GPU, better batching; then a better net (more search does not fix a weak evaluator).
2. Self-play throughput is bound by GPU time per game. Levers: many games per GPU to fill batches, fewer visits per move, a smaller net for self-play, more workers (it is embarrassingly parallel).
3. Training throughput is bound by data loading as often as by compute: keep the GPU fed (prefetch, many decode workers), then add GPUs data-parallel, then shard optimizer state.
4. Learning quality is bound by data freshness and diversity: window size, noise level, and how fast new nets reach the workers. The original AlphaZero run used about 5,000 first-generation TPUs for self-play and 64 second-generation TPUs for training (Silver et al. 2018): self-play, not gradient steps, dominates the compute.

Failure modes

Real implementations

The alternative: an alpha-beta engine with an efficiently updatable CPU network (NNUE, Alpha-beta search, Move ordering and pruning) searches millions of nodes per second with a much weaker evaluator per node, and needs no GPU. Pick the neural MCTS design when a GPU is available and positional judgement matters most; pick alpha-beta + NNUE for CPU-only targets and very short time controls.

Sources

• D. Silver et al., A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play, Science 2018 (preprint: Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm). paper
• A. Ruoss et al., Amortized Planning with Large-Scale Transformers: A Case Study on Chess, 2024: a transformer trained on engine-labelled positions, playing without search. paper
• Leela Chess Zero: lczero.org, lc0 source (encoder src/neural/encoder.h, search parameters src/search/classic/params.cc, backends in README).