← Back
Home › Systems › Agentic AI

agent-evals

agent evaluation harness

SWE-bench, τ-bench

SystemTechniques · 18Primitives · 65 agent-evals User simulationagentic-ai · medium: LLM-driven simulated users following scripted goals to test conversational agentsDurable job queuedistributed · medium: Persisted tasks that workers lease for a visibility timeout, ack on success and retry with backoff; at-least-once execution that survives worker and queue crashes.›Benchmark contamination detectionmachine-learning · medium: Detect test-set leakage into training data via n-gram overlap, perplexity and canary checksLLM-as-judgemachine-learning · medium: Grade open-ended outputs with a language model under a rubric, then correct its known biases by order swapping, voting and calibration against human labels.›Near-duplicate detection with MinHa…machine-learning · medium: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.›Verifiable rewardsml-training · medium: Rewards from programmatic checkers (unit tests, math answer matching) for RL and data filteringDialogue state trackingmachine-learning · medium: Keep a structured belief of the user's goal (slot → value, plus requests) and update it every turn from NLU output, the old state and the last system action.›Pass at kmachine-learning · low: Unbiased pass@k estimator from n samples with c correct, for code and agent benchmarksAgent evaluationagentic-ai · unknown: Agent sandboxingagentic-ai · unknown: Agent tracingagentic-ai · unknown: Record replaydistributed · low: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuildEvaluation metricsmachine-learning · low: Precision, recall, ROC AUC, average precision, log loss, NDCG and pass@k computed correctly, plus paired confidence intervals for comparing models.›Bootstrap confidence intervalsstatistics · low: Resampling-based confidence intervals for arbitrary statisticsCopy on write snapshotsstorage · low: Point-in-time snapshots by sharing pages/extents and copying only on writeGit object modelstorage · low: Content-addressed blobs, trees, commits and refs; packfiles with delta compression; worktreesTemplate-based text generationmachine-learning · low: Realise facts as sentences by filling slots in alternative templates, with agreement, referring expressions and seeded variation; every word traces to data or a template.›Token samplingml-training · low: Turn a vector of scores into one chosen item with temperature, top-k, top-p, min-p, repetition penalties and a seed.›Dead-letter queuedistributed · low: A side queue for messages that keep failing: after N delivery attempts or a permanent error, park them with the reason so the main queue keeps moving.›Leasesdistributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Idempotency keysfinance · low: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Exponential backoff with jittersystems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›+ 11 more techniques+ 49 more primitives

Similar systems

Share the most building blocks with this one.

eval-harness 33 %dangerous-capability-evals 24 %synthetic-data 17 %ai-red-teaming 16 %rl-environments 16 %