← Back
Home › Systems › AI applications

chat-assistant

consumer chat assistant

Claude.ai, ChatGPT

SystemTechniques · 28Primitives · 103 chat-assistant Model routing and cascadesagentic-ai · medium: Send each request to the cheapest model that will answer it well enough, by rules, a learned router, or a cascade that escalates from small to large on low confidence.›Prompt injection defenseagentic-ai · medium: Treat every untrusted string as data, track where values came from, and make tool calls and rendered output depend on provenance, not on what the model says.›Deterministic simulation testingdistributed · medium: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Primary-backup replicationdistributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Usage meteringfinance · medium: Counting tokens/requests per user and key, aggregating for quotas and billing with idempotent eventsText classificationmachine-learning · medium: Assign one or more labels to a text; climb a ladder from regexes to linear models on n-grams, embeddings, fine-tuned encoders and LLM prompts, only as far as the data and budget demand.›Cross-encoder rerankingsearch-sort · medium: Re-score the top-k candidates of a cheap first stage with a model that reads query and candidate together; more accurate than separate embeddings, but one model call per pair.›Idempotency keysfinance · low: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.›Agent memoryagentic-ai · unknown: Agent sandboxingagentic-ai · unknown: Context window managementagentic-ai · low: Keep an agent's growing history inside the model's window by truncating, clearing and summarizing old turns in a cheap-to-lossy ladder, at turn boundaries.›Prompt cachingagentic-ai · low: Reuse the attention keys/values of a repeated prompt prefix, so an agent's growing context is prefilled once and later turns pay only for the new tokens.›Tool callingagentic-ai · low: The model emits a structured call (id, name, JSON arguments) instead of prose; your code validates, runs it and returns a result tagged with the same id.›Hdr histogramdata-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentilesProcess supervisionhardware-os · low: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›Text chunkingmachine-learning · low: Cut documents into token-bounded pieces along structure boundaries, keeping offsets and section paths, so each piece embeds well and can be cited.›Continuous batchingml-training · low: Schedule LLM inference per iteration, not per request; finished sequences leave and new ones join the running batch every step, which keeps the GPU full.›Paged KV cacheml-training · low: Store each sequence's attention keys/values in fixed-size blocks mapped through a block table, like virtual memory pages; near-zero fragmentation and copy-on-write sharing.›Token samplingml-training · low: Turn a vector of scores into one chosen item with temperature, top-k, top-p, min-p, repetition penalties and a seed.›Speculative decodingml-training · low: A cheap draft proposes γ tokens, the big model verifies them in one pass, and rejection sampling keeps the target's exact distribution at 2-3x speed.›Server-sent eventsnetworking · low: One long-lived HTTP response carries text events one way; each event may carry an id, so a reconnecting client names its last one and the server resumes.›Mutual TLSsecurity · low: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›OAuth 2 and OpenID Connectsecurity · low: Delegated authorization (OAuth 2) and login on top of it (OIDC); use authorization code + PKCE for every user-facing client.›+ 21 more techniques+ 87 more primitives

Similar systems

Share the most building blocks with this one.

llm-api-platform 44 %mcp-server 38 %llm-gateway 37 %mcp-host 35 %agent-runtime 35 %

In-house systems

Named systems of this kind at large companies, with their paper or write-up.

EricaBank of AmericaGigaChatSberbank