mhtirogla
Home
Systems
API
Pricing
About
Log in
← Back
Home
›
Systems
›
AI and ML infrastructure
eval-harness
model evaluation harness
lm-eval-harness, HELM
System
Techniques · 11
Primitives · 46
eval-harness
LLM-as-judge
machine-learning · medium: Grade open-ended outputs with a language model under a rubric, then correct its known biases by order swapping, voting and calibration against human labels.
›
Near-duplicate detection with MinHa…
machine-learning · medium: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.
›
User simulation
agentic-ai · medium: LLM-driven simulated users following scripted goals to test conversational agents
Egress proxy filtering
security · medium: Forcing sandbox network traffic through a proxy with domain allowlists, TLS SNI checks and request logging
Microvm isolation
security · medium: Minimal KVM-based VMs (Firecracker-style) with tiny device model, fast boot and snapshot restore
Process sandboxing
security · medium: Run risky work in a separate, deliberately powerless process and let a small privileged broker do the few things it needs, over a narrow pipe.
›
Hermetic execution
systems · medium: Run actions with only declared inputs in a sandbox and fixed environment
Bradley terry model
statistics · low: Pairwise comparison model P(i beats j)=s_i/(s_i+s_j); MLE fit, link to Elo and reward models
Cryptographic hashes (SHA-2, BLAKE3)
security · low: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.
›
Calibration
machine-learning · low: Make predicted probabilities match observed frequencies: measure with ECE, fix post hoc with temperature, Platt or isotonic scaling on held-out data.
›
Agent evaluation
agentic-ai · unknown:
Agent sandboxing
agentic-ai · unknown:
Constrained decoding
agentic-ai · low: Force LLM output to match a regex, JSON schema or grammar by masking, at every step, the tokens that would leave the language.
›
Evaluation metrics
machine-learning · low: Precision, recall, ROC AUC, average precision, log loss, NDCG and pass@k computed correctly, plus paired confidence intervals for comparing models.
›
Loglikelihood scoring
machine-learning · low: Scoring choices by summed token log-probabilities, normalization by length, perplexity computation
Pass at k
machine-learning · low: Unbiased pass@k estimator from n samples with c correct, for code and agent benchmarks
Continuous batching
ml-training · low: Schedule LLM inference per iteration, not per request; finished sequences leave and new ones join the running batch every step, which keeps the GPU full.
›
Bootstrap confidence intervals
statistics · low: Resampling-based confidence intervals for arbitrary statistics
Token sampling
ml-training · low: Turn a vector of scores into one chosen item with temperature, top-k, top-p, min-p, repetition penalties and a seed.
›
Annotator agreement
statistics · low: Inter-annotator agreement measures: Cohen and Fleiss kappa, Krippendorff alpha, gold-question accuracy
Fast hash functions
data-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.
›
Rolling hash
data-structures · low: Rabin-Karp polynomial rolling hashes and winnowing for substring fingerprints and document matching
Swiss table
data-structures · low: Open-addressing hash table that filters 8-16 slots at once with 1-byte hash tags and SIMD, giving fast lookups at 87.5 % load.
›
+ 4 more techniques
+ 30 more primitives
System
Techniques · 11
Primitives · 46
eval-harness
LLM-as-judge
machine-learning · medium: Grade open-ended outputs with a language model under a rubric, then correct its known biases by order swapping, voting and calibration against human labels.
›
Near-duplicate detection with MinHa…
machine-learning · medium: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.
›
User simulation
agentic-ai · medium: LLM-driven simulated users following scripted goals to test conversational agents
Egress proxy filtering
security · medium: Forcing sandbox network traffic through a proxy with domain allowlists, TLS SNI checks and request logging
Microvm isolation
security · medium: Minimal KVM-based VMs (Firecracker-style) with tiny device model, fast boot and snapshot restore
Process sandboxing
security · medium: Run risky work in a separate, deliberately powerless process and let a small privileged broker do the few things it needs, over a narrow pipe.
›
Hermetic execution
systems · medium: Run actions with only declared inputs in a sandbox and fixed environment
Dialogue state tracking
machine-learning · medium: Keep a structured belief of the user's goal (slot → value, plus requests) and update it every turn from NLU output, the old state and the last system action.
›
Intent classification and slot fill…
machine-learning · medium: Turn one utterance into an intent label and typed slot values with a joint model: a sentence-level classifier plus a token-level BIO tagger on a shared encoder.
›
Named entity recognition
machine-learning · medium: Find the character spans of entities in text and give each a type, by labelling every token and decoding a valid label sequence.
›
Text classification
machine-learning · medium: Assign one or more labels to a text; climb a ladder from regexes to linear models on n-grams, embeddings, fine-tuned encoders and LLM prompts, only as far as the data and budget demand.
›
Bradley terry model
statistics · low: Pairwise comparison model P(i beats j)=s_i/(s_i+s_j); MLE fit, link to Elo and reward models
Cryptographic hashes (SHA-2, BLAKE3)
security · low: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.
›
Calibration
machine-learning · low: Make predicted probabilities match observed frequencies: measure with ECE, fix post hoc with temperature, Platt or isotonic scaling on held-out data.
›
Agent evaluation
agentic-ai · unknown:
Agent sandboxing
agentic-ai · unknown:
Constrained decoding
agentic-ai · low: Force LLM output to match a regex, JSON schema or grammar by masking, at every step, the tokens that would leave the language.
›
Evaluation metrics
machine-learning · low: Precision, recall, ROC AUC, average precision, log loss, NDCG and pass@k computed correctly, plus paired confidence intervals for comparing models.
›
Loglikelihood scoring
machine-learning · low: Scoring choices by summed token log-probabilities, normalization by length, perplexity computation
Pass at k
machine-learning · low: Unbiased pass@k estimator from n samples with c correct, for code and agent benchmarks
Continuous batching
ml-training · low: Schedule LLM inference per iteration, not per request; finished sequences leave and new ones join the running batch every step, which keeps the GPU full.
›
Bootstrap confidence intervals
statistics · low: Resampling-based confidence intervals for arbitrary statistics
Token sampling
ml-training · low: Turn a vector of scores into one chosen item with temperature, top-k, top-p, min-p, repetition penalties and a seed.
›
Annotator agreement
statistics · low: Inter-annotator agreement measures: Cohen and Fleiss kappa, Krippendorff alpha, gold-question accuracy
Fast hash functions
data-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.
›
Rolling hash
data-structures · low: Rabin-Karp polynomial rolling hashes and winnowing for substring fingerprints and document matching
Swiss table
data-structures · low: Open-addressing hash table that filters 8-16 slots at once with 1-byte hash tags and SIMD, giving fast lookups at 87.5 % load.
›
Template-based text generation
machine-learning · low: Realise facts as sentences by filling slots in alternative templates, with agreement, referring expressions and seeded variation; every word traces to data or a template.
›
DNS message format and client looku…
networking · low: Binary query/response format of DNS (RFC 1035) and how a stub client uses it: record types, name compression, UDP/TCP/EDNS sizes, TTL caching.
›
Http connect tunneling
networking · low: HTTP CONNECT tunnels for proxying TLS and arbitrary TCP
Public Suffix List
networking · low: Find the registrable domain (eTLD+1) of a host name by matching the longest rule in a community-maintained list of shared suffixes.
›
Glob matching
search-sort · low: Glob/gitignore pattern matching without backtracking blowup
Cgroups and namespaces
hardware-os · low: Linux namespaces limit what a process can see; cgroup v2 limits and accounts what it may use. Containers, sandboxes and job schedulers are built from the two.
›
Hardware virtualization kvm
hardware-os · low: VT-x/AMD-V via KVM: vCPU run loop, VM exits, MMIO emulation
Nested paging ept
hardware-os · low: EPT/NPT two-level address translation for guests
Virtio devices
hardware-os · low: Virtqueue rings for paravirtual block, net and vsock devices
Seccomp bpf
security · low: Syscall filtering with seccomp BPF programs
Overlay filesystem
hardware-os · low: Union mounts (overlayfs) stacking read-only layers under a writable layer
Least privilege and isolation
security · low: Assume every component will be compromised and limit what it can then reach: narrow identities, separate accounts, sandboxed processes.
›
Content-addressed storage
storage · low: Store each blob under the hash of its bytes: immutable, deduplicated by construction, self-verifying on every read, and composable into Merkle trees.
›
Statecharts
graphics · low: State machines with nesting, parallel regions, history and entry/exit actions, so UI and animation logic stays linear in size instead of exploding.
›
Byte-pair encoding (BPE) tokenizati…
machine-learning · low: Learn a subword vocabulary by repeatedly merging the most frequent adjacent pair; byte-level BPE encodes any input losslessly with 32k-200k tokens.
›
Transformer block
ml-training · low: The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.
›
Convolutional codes and the Viterbi…
signal-processing · low: Streaming error-correcting code: each output bit is a parity of the last K input bits; the Viterbi decoder finds the most likely path through a 2^(K-1)-state trellis.
›
Linear and logistic regression
machine-learning · low: Fit a weighted sum of features to a number (least squares) or a probability (logistic); the baseline every model must beat.
›
Reference counting
compilers · low: Keep a count of owners in each object, free it when the count hits zero; deterministic and local, but pays on every copy and leaks cycles.
›
Merkle dag
data-structures · low: Content-addressed DAG of hashed nodes (git trees/commits) for dedupe and verifiable history
Record replay
distributed · low: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuild
Atomic file replacement
hardware-os · low: Write temp file, fsync, rename over target, fsync directory: crash-safe whole-file updates
Loss functions
machine-learning · low: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.
›
Text normalization
machine-learning · low: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.
›
Paged KV cache
ml-training · low: Store each sequence's attention keys/values in fixed-size blocks mapped through a block table, like virtual memory pages; near-zero fragmentation and copy-on-write sharing.
›
L-BFGS
numerics · low: Quasi-Newton minimizer that approximates the inverse Hessian from the last m gradient differences via the two-loop recursion; O(m·n) memory and work per step.
›
LU and Cholesky factorization
numerics · low: Factor A = PLU (any square matrix, partial pivoting) or A = LLᵀ (SPD) once in O(n³), then solve each right-hand side in O(n²).
›
Finite state transducer
search-sort · low: Minimal acyclic FST mapping sorted keys to values; compact term dictionaries with prefix/fuzzy search
JSON Schema validation
security · low: Compile a JSON Schema once, then check every untrusted document against it, within hard size and depth limits, and report failures as JSON Pointers.
›
Hierarchical state machine
systems · low: Nest states so a substate inherits its parent's reactions and only overrides what differs, with ordered entry and exit actions along the path.
›
Copy for AI
Similar systems
Share the most building blocks with this one.
agent-evals
33 %
synthetic-data
25 %
structured-extraction
18 %
dangerous-capability-evals
17 %
machine-translation
16 %