← Back
Home › Systems › Frontier AI research

reasoning-rl

RL with verifiable rewards for long reasoning

o-series, DeepSeek-R1, Claude thinking

SystemTechniques · 12Primitives · 52 reasoning-rl Benchmark contamination detectionmachine-learning · medium: Detect test-set leakage into training data via n-gram overlap, perplexity and canary checksLLM-as-judgemachine-learning · medium: Grade open-ended outputs with a language model under a rubric, then correct its known biases by order swapping, voting and calibration against human labels.›Policy gradient and PPOmachine-learning · medium: Raise the log-probability of actions in proportion to their advantage; PPO clips the probability ratio so several epochs on one rollout stay safe.›Distributed checkpointingml-training · medium: Sharded, asynchronous checkpoint save/load of model and optimizer state, with resharding to new world sizesGrpoml-training · medium: Group relative policy optimization: advantages from rewards of several samples per prompt, no value networkKnowledge distillationml-training · medium: Train a small student to match a large teacher's softened outputs (plus the hard labels) so it keeps most of the teacher's accuracy at a fraction of the cost.›Prefill decode disaggregationml-training · medium: Running prompt prefill and token decode on separate GPU pools with KV-cache transferToken samplingml-training · low: Turn a vector of scores into one chosen item with temperature, top-k, top-p, min-p, repetition penalties and a seed.›ZeRO and FSDP shardingml-training · low: Data parallelism that shards optimizer state, gradients and weights across ranks and gathers weights just in time; per-GPU model memory drops from 16Ψ to 16Ψ/N.›Continuous batchingml-training · low: Schedule LLM inference per iteration, not per request; finished sequences leave and new ones join the running batch every step, which keeps the GPU full.›Paged KV cacheml-training · low: Store each sequence's attention keys/values in fixed-size blocks mapped through a block table, like virtual memory pages; near-zero fragmentation and copy-on-write sharing.›Pass at kmachine-learning · low: Unbiased pass@k estimator from n samples with c correct, for code and agent benchmarksContext parallelismml-training · low: Splitting the sequence dimension across GPUs (ring attention, Ulysses all-to-all) for long-context trainingMixed-precision trainingml-training · low: Run matmuls in bf16/fp16/fp8 on tensor cores while keeping master weights, optimizer state and reductions in fp32; about 2x speed, half the activation memory.›Rejection samplingml-training · low: Best-of-n and filtered sampling: generate many candidates, keep those passing a verifier or reward thresholdSelf consistency votingml-training · low: Sample several answers and pick by majority or execution agreement to raise reliabilitySequence packingml-training · low: Packing variable-length sequences into fixed-size batches with attention masks to avoid padding wasteTensor parallelismml-training · low: Split each layer's weight matrices across GPUs (Megatron-style column then row splits) so one layer's math runs on t devices with one all-reduce per block.›Weight synchronizationml-training · low: Pushing updated policy weights from trainers to inference/actor workers (NCCL broadcast, sharded resharding)Importance samplingnumerics · low: Sample from a proposal distribution and reweight to cut Monte Carlo varianceBloom filterdata-structures · low: Bit array answering "definitely not present" or "probably present" in about 10 bits per key for 1 % false positives; inserts are cheap, deletes impossible.›Rolling hashdata-structures · low: Rabin-Karp polynomial rolling hashes and winnowing for substring fingerprints and document matchingLoglikelihood scoringmachine-learning · low: Scoring choices by summed token log-probabilities, normalization by length, perplexity computation+ 5 more techniques+ 36 more primitives

Similar systems

Share the most building blocks with this one.

rl-post-training 39 %ai-for-math 22 %synthetic-data 21 %