A customer-service agent that looks up orders and issues refunds through typed, validated tools, but pauses for explicit human approval before any money moves. Every step is logged under a shared run_id for ordered replay. On a 4-task fixture it gated 100% of high-risk actions: the one policy-ineligible refund was stopped at the approval boundary, at $0.0006 per task.
A customer asks for a refund. An agent can look up the order, read the refund policy, check eligibility, and
issue the payment, four steps a returns desk runs hundreds of times a day. The lookups are safe to automate.
The payout is not: it moves real money out of the business and cannot be taken back.
That makes a returns desk both the canonical case for agents (multi-step, tool-dependent, high-volume) and
the canonical case against them. An agent that asks permission for everything is a slow form; one that asks
for nothing is an unbounded liability.
The question was which steps can run unsupervised, what the approval boundary costs in throughput, and where
to place that boundary so a wrong refund is stopped before the money moves rather than explained afterwards.
Technical Approach
What an agent is
Stripped of framework vocabulary, an agent is four things: tools, plain functions the model
may request, each with a typed schema; a loop that asks the model, runs what it asked for,
and feeds the result back; state, an ordered history so a run can be inspected, resumed, or
replayed; and a stopping rule, task complete or step budget exhausted. Everything else, the
plan/act/reflect vocabulary included, is engineering to make those four reliable.
The loop, the gate, and the traces
Every tool call is validated before it runs; ordinary tools execute directly, and high-risk tools stop at a
human approval gate. Three choices make that safe.
Schema validation and a human approval gate stand between the agent and any irreversible action. Measured: 100% of high-risk attempts gated.
Typed tools: Pydantic models already emit JSON Schema, so the API tool definition and the
runtime validator are the same declaration, with no second definition to drift. Validation failures
return as data: a malformed call comes back as an observation the model reads and corrects on the
next turn, so an error recovers the run instead of ending it. Ordered traces: a shared
run_id and monotonic step make replay possible, because timestamps are not reliable
ordering when steps land in the same millisecond, and replay is the only real way to debug an agent.
Design choice: the gate sits after validation, before execution
Approving an unvalidated call means the human is approving something that may not even be well-formed.
Validate first, surface the concrete action, then execute. A refusal is recorded as a normal outcome the
agent can see and respond to, not an exception. The provider's automatic function-calling is deliberately
disabled so the loop executes tools itself and the gate stays inside it, verified by driving a refund on a
policy-ineligible order and confirming the gate blocked it and no money moved.
Results
On a 4-task fixture the agent reached 100% task success and tool-call accuracy, gated 100% of high-risk
actions (one attempt, one gate fired), and cost about $0.0006 per task at 23.8 s mean latency. Per-task cost
is measured from token usage and the full tau-bench sweep is projected from it, roughly 115 tasks at
$0.00063, about $0.07, which is why development runs against a small subset and the full suite runs once. To
be explicit, this is a 4-task fixture, not tau-bench: the numbers prove the machinery works,
not a benchmark result.
Main weakness: the gate relocates judgement, it does not remove it
An approver who says yes to everything lets a bad refund through; one who says no to everything blocks the
legitimate one too. There is no setting that is simply correct. The same shape recurs in the semantic
cache's threshold, the text-to-SQL confidence cutoff, and the retrieval judge's bias, naming that
recurrence across four projects is the through-line of the whole set.
Two things visible in the per-task data are worth stating. Task success and tool-call accuracy can
diverge even when both read 1.0 here, a run can reach the correct outcome while calling a tool wrong and
silently recovering. And latency varied 3.3 s to 68 s across the four tasks (the thinking model plus
free-tier rate-limit backoff), so a single mean hides a wide spread.
Next iteration priorities
Run the full tau-bench retail suite and publish success, tool accuracy, cost, and latency at scale.
Make the approver policy-aware enough to auto-refuse clear cases and escalate only real judgement.
Decide where the amount <= price rule belongs, schema, tool, or approver, and defend it.
Reconstruct a failed run from its trace alone, and see whether the trace is sufficient.
