← Back
Home › Systems › AI and ML infrastructure

llm-api-platform

a model provider's public API at global scale: capacity scheduling, priority/batch tiers, prompt caching, server-side tools, metering, safety hooks, model lifecycle

Claude API, OpenAI API, Gemini API

SystemTechniques · 32Primitives · 103 llm-api-platform Model routing and cascadesagentic-ai · medium: Send each request to the cheapest model that will answer it well enough, by rules, a learned router, or a cascade that escalates from small to large on low confidence.›Model context protocolagentic-ai · medium: JSON-RPC protocol exposing tools, resources and prompts from servers to LLM clients, with capability negotiationPrompt injection defenseagentic-ai · medium: Treat every untrusted string as data, track where values came from, and make tool calls and rendered output depend on provenance, not on what the model says.›Deterministic simulation testingdistributed · medium: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Primary-backup replicationdistributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Usage meteringfinance · medium: Counting tokens/requests per user and key, aggregating for quotas and billing with idempotent eventsPrefill decode disaggregationml-training · medium: Running prompt prefill and token decode on separate GPU pools with KV-cache transferCircuit breakersystems · low: Stop calling a failing dependency for a while, probe it with trial calls, and resume on recovery: fail fast instead of piling up timeouts.›Server-sent eventsnetworking · low: One long-lived HTTP response carries text events one way; each event may carry an id, so a reconnecting client names its last one and the server resumes.›Idempotency keysfinance · low: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.›Continuous batchingml-training · low: Schedule LLM inference per iteration, not per request; finished sequences leave and new ones join the running batch every step, which keeps the GPU full.›Paged KV cacheml-training · low: Store each sequence's attention keys/values in fixed-size blocks mapped through a block table, like virtual memory pages; near-zero fragmentation and copy-on-write sharing.›Hdr histogramdata-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentilesGraceful drainingsystems · low: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›Constrained decodingagentic-ai · low: Force LLM output to match a regex, JSON schema or grammar by masking, at every step, the tokens that would leave the language.›Prompt cachingagentic-ai · low: Reuse the attention keys/values of a repeated prompt prefix, so an agent's growing context is prefilled once and later turns pay only for the new tokens.›Tool callingagentic-ai · low: The model emits a structured call (id, name, JSON arguments) instead of prose; your code validates, runs it and returns a result tagged with the same id.›Process supervisionhardware-os · low: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›Speculative decodingml-training · low: A cheap draft proposes γ tokens, the big model verifies them in one pass, and rejection sampling keeps the target's exact distribution at 2-3x speed.›Tensor parallelismml-training · low: Split each layer's weight matrices across GPUs (Megatron-style column then row splits) so one layer's math runs on t devices with one all-reduce per block.›Mutual TLSsecurity · low: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›Calibrationmachine-learning · low: Make predicted probabilities match observed frequencies: measure with ECE, fix post hoc with temperature, Platt or isotonic scaling on held-out data.›Embeddingsmachine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.›+ 25 more techniques+ 87 more primitives

Similar systems

Share the most building blocks with this one.

chat-assistant 44 %llm-gateway 34 %agent-runtime 30 %agent-gateway 30 %mcp-server 29 %