← Back
Home › Systems › AI and ML infrastructure

rl-post-training

RLHF / RL post-training pipeline

OpenRLHF

SystemTechniques · 10Primitives · 46 rl-post-training Policy gradient and PPOmachine-learning · medium: Raise the log-probability of actions in proportion to their advantage; PPO clips the probability ratio so several epochs on one rollout stay safe.›Direct preference optimizationml-training · medium: DPO and variants: optimize a policy directly on preference pairs with a reference-model log-ratio lossGrpoml-training · medium: Group relative policy optimization: advantages from rewards of several samples per prompt, no value networkLLM fine-tuningml-training · medium: Adapting a pretrained LLM with supervised fine-tuning, LoRA/QLoRA adapters and preference optimization (DPO), and evaluating that it improved without forgetting.›Reward modelingml-training · medium: Training a scalar reward model on preference pairs with a Bradley-Terry loss; overoptimization pitfallsVerifiable rewardsml-training · medium: Rewards from programmatic checkers (unit tests, math answer matching) for RL and data filteringEgress proxy filteringsecurity · medium: Forcing sandbox network traffic through a proxy with domain allowlists, TLS SNI checks and request loggingLora adaptersml-training · low: Low-rank adapter matrices for fine-tuning; QLoRA; serving many adapters on one base modelBradley terry modelstatistics · low: Pairwise comparison model P(i beats j)=s_i/(s_i+s_j); MLE fit, link to Elo and reward modelsToken samplingml-training · low: Turn a vector of scores into one chosen item with temperature, top-k, top-p, min-p, repetition penalties and a seed.›Agent sandboxingagentic-ai · unknown: Continuous batchingml-training · low: Schedule LLM inference per iteration, not per request; finished sequences leave and new ones join the running batch every step, which keeps the GPU full.›Mixed-precision trainingml-training · low: Run matmuls in bf16/fp16/fp8 on tensor cores while keeping master weights, optimizer state and reductions in fp32; about 2x speed, half the activation memory.›Paged KV cacheml-training · low: Store each sequence's attention keys/values in fixed-size blocks mapped through a block table, like virtual memory pages; near-zero fragmentation and copy-on-write sharing.›Rejection samplingml-training · low: Best-of-n and filtered sampling: generate many candidates, keep those passing a verifier or reward thresholdWeight synchronizationml-training · low: Pushing updated policy weights from trainers to inference/actor workers (NCCL broadcast, sharded resharding)ZeRO and FSDP shardingml-training · low: Data parallelism that shards optimizer state, gradients and weights across ranks and gathers weights just in time; per-GPU model memory drops from 16Ψ to 16Ψ/N.›AdamWml-training · low: Adam's per-parameter adaptive steps with weight decay applied directly to the weights; the default optimizer for transformers and LLM pretraining.›Gradient clippingml-training · low: Clipping gradients by global norm (or value) to prevent loss spikes; interaction with mixed precisionReverse-mode automatic differentiat…ml-training · low: Record the forward computation on a tape, then sweep it backwards once to get the gradient of a scalar loss for all inputs.›Loss functionsmachine-learning · low: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.›Gradient checkpointingml-training · low: Save only some activations in the forward pass and recompute the rest during backward, trading about 33% extra compute for O(√n) activation memory.›Learning-rate warmup and cosine dec…ml-training · low: Ramp the learning rate up linearly for the first ~1% of steps, then decay it along a half cosine to ~10% of peak.›+ 3 more techniques+ 30 more primitives

Similar systems

Share the most building blocks with this one.

reasoning-rl 39 %synthetic-data 22 %ai-for-math 16 %