Tool calling

Deep dive

How it works

In: tool definitions (name, description, JSON Schema) and the model's call blocks (id, name, JSON arguments) · Out: one result block per call id · State: none (the conversation holds the history) · Cost: O(argument bytes) per call; schemas are input tokens on every request

1. Request: tools = [{name, description, input_schema}]. The provider renders them into the prompt (Anthropic documents a constructed system prompt; the format is the provider's).
2. The model either answers in text or emits call blocks and stops (Anthropic stop_reason: tool_use; OpenAI tool_calls).
3. Per call: look up the name, parse the arguments, validate them against the schema, run, capture output or error.
4. Reply with one result per call, matched by id, then call the model again. This is the agent loop.

Anthropic  assistant: {"type":"tool_use","id":"toolu_01","name":"get_weather","input":{...}}   (parsed JSON)
           user:      {"type":"tool_result","tool_use_id":"toolu_01","content":"15 C","is_error":false}
OpenAI     assistant.tool_calls[i] = {id, function:{name, arguments:"{...}"}}  (arguments = JSON string)
           {"role":"tool","tool_call_id":id,"content":"..."}   (Responses API: call_id / function_call_output)

Anthropic rules: results directly follow the assistant turn, tool_result blocks precede any text, parallel results share one message. A skipped call still gets an is_error: true result.

Code

Registry, JSON parser, strict-style validation and parallel dispatcher, std only. Every failure (bad JSON, unknown tool, wrong type, tool error, panic) becomes an is_error result so the model can retry.

use std::thread;

#[derive(Debug, Clone, PartialEq)]
pub enum Json { Null, Bool(bool), Num(f64), Str(String), Arr(Vec<Json>), Obj(Vec<(String, Json)>) }

pub fn parse(s: &str) -> Result<Json, String> {
    let (v, rest) = value(s.trim_start())?;
    if rest.trim().is_empty() { Ok(v) } else { Err("trailing characters after JSON".into()) }
}

fn value(s: &str) -> Result<(Json, &str), String> {
    let s = s.trim_start();
    match s.chars().next().ok_or("unexpected end of JSON")? {
        '{' => {
            let (mut kv, mut r) = (vec![], s[1..].trim_start());
            if let Some(r2) = r.strip_prefix('}') { return Ok((Json::Obj(kv), r2)); }
            loop {
                let (k, r1) = string(r.trim_start())?;
                let r1 = r1.trim_start().strip_prefix(':').ok_or("expected ':'")?;
                let (v, r2) = value(r1)?;
                kv.push((k, v));
                let r2 = r2.trim_start();
                if let Some(x) = r2.strip_prefix(',') { r = x } else { return Ok((Json::Obj(kv), r2.strip_prefix('}').ok_or("expected '}'")?)); }
            }
        }
        '[' => {
            let (mut xs, mut r) = (vec![], s[1..].trim_start());
            if let Some(r2) = r.strip_prefix(']') { return Ok((Json::Arr(xs), r2)); }
            loop {
                let (v, r1) = value(r)?;
                xs.push(v);
                let r1 = r1.trim_start();
                if let Some(x) = r1.strip_prefix(',') { r = x } else { return Ok((Json::Arr(xs), r1.strip_prefix(']').ok_or("expected ']'")?)); }
            }
        }
        '"' => string(s).map(|(v, r)| (Json::Str(v), r)),
        _ => {
            for (lit, v) in [("true", Json::Bool(true)), ("false", Json::Bool(false)), ("null", Json::Null)] {
                if let Some(r) = s.strip_prefix(lit) { return Ok((v, r)); }
            }
            let end = s.find(|c: char| !"+-.eE0123456789".contains(c)).unwrap_or(s.len());
            let n = s[..end].parse::<f64>().map_err(|_| format!("bad JSON near `{}`", &s[..s.len().min(12)]))?;
            Ok((Json::Num(n), &s[end..]))
        }
    }
}

fn string(s: &str) -> Result<(String, &str), String> {
    let mut it = s.strip_prefix('"').ok_or("expected string")?.char_indices();
    let mut out = String::new();
    while let Some((i, c)) = it.next() {
        match c {
            '"' => return Ok((out, &s[i + 2..])),
            '\\' => match it.next().ok_or("bad escape")?.1 {
                'n' => out.push('\n'), 't' => out.push('\t'), '"' => out.push('"'), '\\' => out.push('\\'), '/' => out.push('/'),
                e => return Err(format!("unsupported escape \\{e}")),
            },
            c => out.push(c),
        }
    }
    Err("unterminated string".into())
}

#[derive(Clone, Copy)]
pub enum Ty { Str, Int, Bool, Enum(&'static [&'static str]) }
pub struct Param { pub name: &'static str, pub ty: Ty, pub required: bool }
pub struct Tool { pub name: &'static str, pub params: &'static [Param], pub run: fn(&Json) -> Result<String, String> }

/// Strict-style check: object, no unknown keys, required present, types exact.
fn validate(t: &Tool, args: &Json) -> Result<(), String> {
    let Json::Obj(kv) = args else { return Err("arguments must be a JSON object".into()) };
    if let Some((k, _)) = kv.iter().find(|(k, _)| !t.params.iter().any(|p| p.name == k)) {
        return Err(format!("unknown parameter `{k}`; allowed: {}", t.params.iter().map(|p| p.name).collect::<Vec<_>>().join(", ")));
    }
    for p in t.params {
        match (kv.iter().find(|(k, _)| k == p.name), p.ty) {
            (None, _) if p.required => return Err(format!("missing required parameter `{}`", p.name)),
            (None, _) | (Some((_, Json::Str(_))), Ty::Str) | (Some((_, Json::Bool(_))), Ty::Bool) => {}
            (Some((_, Json::Num(n))), Ty::Int) if n.fract() == 0.0 => {}
            (Some((_, Json::Str(s))), Ty::Enum(vs)) if vs.contains(&s.as_str()) => {}
            (Some(_), Ty::Enum(vs)) => return Err(format!("`{}` must be one of {vs:?}", p.name)),
            (Some(_), _) => return Err(format!("`{}` has the wrong type", p.name)),
        }
    }
    Ok(())
}

pub struct Call { pub id: String, pub name: String, pub arguments: String } // arguments = raw JSON text from the model
#[derive(Debug)]
pub struct ToolResult { pub id: String, pub content: String, pub is_error: bool }

fn execute(tools: &[Tool], c: &Call) -> Result<String, String> {
    let Some(t) = tools.iter().find(|t| t.name == c.name) else {
        return Err(format!("unknown tool `{}`; available: {}", c.name, tools.iter().map(|t| t.name).collect::<Vec<_>>().join(", ")));
    };
    let args = parse(&c.arguments).map_err(|e| format!("arguments are not valid JSON: {e}"))?;
    validate(t, &args)?;
    (t.run)(&args)
}

/// One result per call, same order and ids. Independent calls run in parallel; failures never abort the batch.
pub fn dispatch(tools: &[Tool], calls: &[Call], max_bytes: usize) -> Vec<ToolResult> {
    thread::scope(|s| {
        let hs: Vec<_> = calls.iter().map(|c| s.spawn(move || execute(tools, c))).collect();
        calls.iter().zip(hs).map(|(c, h)| {
            let r = h.join().unwrap_or_else(|_| Err("tool panicked".into()));
            let (mut content, is_error) = match r { Ok(o) => (o, false), Err(e) => (e, true) };
            if content.len() > max_bytes {
                let mut cut = max_bytes;
                while !content.is_char_boundary(cut) { cut -= 1; }
                content.truncate(cut);
                content.push_str("\n[truncated; narrow the request]");
            }
            ToolResult { id: c.id.clone(), content, is_error }
        }).collect()
    })
}

Compiled and run on ten calls (valid, missing or unknown parameter, wrong type, bad enum, unknown tool, truncated JSON, tool error, panic) plus an oversized result: ids and order preserved, only the two valid calls succeed. The parser is a subset (no \uXXXX); use serde_json in real code.

Complexity and constants

• Parse and validate: O(argument bytes), microseconds; negligible next to a model turn (order of seconds).
• Token cost is the real cost: schemas are input tokens on every request, plus a fixed tool-use system prompt (Anthropic's docs list on the order of 300 tokens). Descriptions dominate; optional input_examples add roughly 20-50 tokens each for simple inputs and 100-200 for nested ones (Anthropic docs).
• Each round trip re-reads the whole context; results stay in it for the rest of the run.

Efficiency

• Cost: parse and validate are O(argument bytes), microseconds, against a model turn of order of seconds. Schemas are input tokens on every request plus a fixed tool-use system prompt (a few hundred tokens, Anthropic docs); input_examples add roughly 20-200 tokens each. Bound: model round trips and input tokens.
• Levers:
  • parallel calls, executed concurrently — N reads take the time of the slowest (LLMCompiler: up to 3.7x latency, 6.7x cost vs ReAct) — independent calls
  • cache the tool prefix (prompt caching) — a stable tool order is cached; editing a description invalidates it (Anthropic docs) — long sessions
  • small results: truncate, paginate — every result byte is re-read on later turns — large outputs
  • fewer, broader tools (one action enum) — fewer schemas, fewer wrong picks — near-duplicate tools
  • defer rarely used tools (defer_loading, tool retrieval) — schemas loaded on demand — large tool sets
• Naive → standard → state of the art:
  • one call per model turn, every schema inline, full results kept · cost grows with tools x turns
  • parallel calls, cached tool prefix, truncated results · one turn per batch
  • plan-then-execute graph (LLMCompiler) or code as the tool, deferred schemas · up to 3.7x faster, 6.7x cheaper than ReAct on its benchmarks

Implementation notes

• Join results to calls by id only.
• Run read-only calls concurrently; serialize writers to shared state.
• Set a per-call timeout and size cap; return timed out after 30 s, try a narrower query instead of hanging. Make side-effecting tools idempotent so a retried turn does not double-apply (idempotency keys).
• Error text is prompt text: say what went wrong and what to try.
• Streamed arguments arrive as partial JSON: parse after the block ends, or use a streaming parser.
• Schemas: additionalProperties: false, required, enums over free strings; descriptions say when not to use the tool.
• Give each tool the least privilege it needs and run risky ones in a sandbox behind approval.

Pitfalls

• Arguments as a string: OpenAI returns arguments as JSON text that may be malformed or contain extra or missing keys; JSON.parse can throw. Without strict mode, nothing guarantees the schema.
• Orphaned calls: an assistant turn with a call and no matching result is rejected by the API (Anthropic: "tool_use ids were found without tool_result blocks immediately after"). Always answer every id, including skipped ones.
• Text before results: a text block ahead of tool_result blocks is a 400 on Anthropic; results split over several messages discourage parallel calls.
• Untrusted results: web pages and emails can carry instructions. Keep them in result blocks, treat as data (injection defense).
• Forced calls: tool_choice any or a named tool skips text and (per Anthropic) thinking; some newer models reject it. Prefer auto plus strict schemas.
• Strict-mode limits: only a JSON Schema subset is supported; schemas are compiled into grammars, so keep them small.

Variants and alternatives

In production

Every major model API ships it: Anthropic Messages, OpenAI (Chat Completions, Responses) and Gemini (function calling). Local servers (llama.cpp, Ollama, vLLM) expose OpenAI-compatible endpoints. Robotics: Code as Policies has an LLM write programs that call perception and control-primitive APIs on robots. Chemistry: ChemCrow gives an LLM 18 expert-designed tools for synthesis planning, drug discovery and materials design.

Implementations

Sources

• Anthropic, Define tools, Handle tool calls, Parallel tool use, Strict tool use, Claude API docs. define · handle · parallel · strict
• OpenAI, Function calling guide. docs
• Timo Schick et al., Toolformer: Language Models Can Teach Themselves to Use Tools, 2023. paper
• Shunyu Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, 2022. paper · code
• Shishir G. Patil et al., Gorilla: Large Language Model Connected with Massive APIs, 2023. paper · code
• Sehoon Kim et al., An LLM Compiler for Parallel Function Calling, 2023. paper · code
• Jacky Liang et al., Code as Policies: Language Model Programs for Embodied Control, 2022. paper
• Andres M. Bran et al., Augmenting large language models with chemistry tools (ChemCrow), Nature Machine Intelligence 2024. paper · arXiv