Production agents need gates, freeze-before-send, and replay—not a better prompt
A freight desk once called us because the agent “sent the quote.” The customer never got it. The model had drafted text in the transcript and stated it was done. No SMTP call. No freeze record. The ops lead had approved from a chat summary, not from the artifact that would have gone out.
That incident is more representative than any benchmark score. We run agents in freight forwarding, drawing-to-MES flows, and shop-floor vision—roughly 600,000 hours of business runtime in aggregate across deployments. The recurring argument in postmortems is never “we needed a smarter model.” It is: who authorized the side effect, on what revision, with what evidence tied to the order ID.
Chat demos lie gently
A chat UI trains everyone—builders included—to optimize fluency. Production optimizes whether a side effect is acceptable: cargo released, EBOM promoted, a gray-zone vision call written to QMS. The gap is structural. In chat, the model’s natural-language answer is the output. In production, the answer is a proposal until a tool result—or a human release—says otherwise.
Teams that port a chat loop to production without re-homing accountability usually discover this on a Tuesday night, when nobody can reconstruct what happened for order 88412.
Harness
We use harness in the engineering sense: the layer between your host system (permissions, master data, UI) and the model loop. It decides which tool calls are eligible, where execution stops for review, and what gets attached to each business order. The model classifies and plans. The harness owns whether anything actually happens.
This is not interchangeable with “we added LangChain.” Orchestration libraries wire steps. A harness answers: if this mail leaves, where is the frozen blob, who released it, and which config revision was active.
Gates
Register every tool with schema, idempotency rules, and a risk class. The runtime enforces the allow-list; the prompt does not get a bypass lane.
| Class | Examples | Default |
|---|---|---|
| Read | Fetch order, parse attachment | Run, log |
| Draft | Quote text, BOM suggestion | Run, version output |
| Write | CRM, MES writeback | Schema + idempotency key |
| Outbound | Mail, EDI, portal post | Freeze first |
| Destructive | Delete, line stop, payment | Human gate; second approver if configured |
User mail and pasted PDFs live in a data zone. They are not system instructions. If natural language can trigger a write, you do not have an agent—you have a remote-control vulnerability with good copy.
Freeze-before-send
Outbound and other irreversible writes produce a frozen artifact first: payload, hash, recipients, fields, bound to config_version. Nothing leaves until something explicit releases it—a person, or a named policy you can audit later.
Freight uses this on customer-facing mail. Drawing flows freeze before EBOM promotion. Vision holds gray-zone QMS writebacks the same way. Same mechanism; different gate tables per plant or desk.
draft = registry.call("draft_reply", order_id, context)
freeze_id = freeze.store(
order_id=order_id,
body=draft.body,
to=draft.to,
config_version=run.config_version,
)
# no SMTP / EDI until release
hitl.queue(freeze_id, role="manager")
# after approve:
registry.call("send_frozen", freeze_id=freeze_id, idempotency_key=...)
The latency cost is real. Desks that used to “just send” now wait on a queue. That is the point. If you cannot show a freeze row for a sent message, you do not have production outbound—you have trust in a transcript.
Auto-release policies are fine when written down: which lanes, which amount bands, which customers. “The model looked confident” is not a policy.
Replay
Replay is not “open the chat and ask again.” Given order_id and run_id, reconstruct inputs, tool calls, freeze/release events, model route, and config revision—without mutating state.
Postmortems download one evidence pack. Regression diffs packs before and after a prompt change. Auditors get rule versions and primitives, not screenshots of a thread. If your incident workflow starts with “what did you type in the playground,” you are still in demo mode.
Run binding
Each run carries config_version: prompt hash, tool registry snapshot, gate table, model route. Canary and rollback are revision operations. “What changed before the bad release?” should be answered with a diff, not a Slack scrollback.
Prompts belong in that revision graph—system block for red lines, task block for order facts, tool results as observations. A better prompt cuts down bad drafts. It does not stop an unapproved send. Red lines in config; outbound enforcement in the harness.
What we sign before more code
We write acceptance with the customer before the second sprint. Typical rows:
| Area | Example pass |
|---|---|
| Loop | Scene closes; deny-listed tools blocked; outbound held until release |
| Abuse | Injection cases run; soak test shows zero unapproved outbound |
| Evidence | Pack downloads by order ID; replay matches config_version |
| Ops | Freeze queue depth visible; idempotent write failures alert; rollback drill done once |
Throughput and savings numbers come from pilot measurement in a signed scope—not from a blog post.
Brownfield mistakes
High temperature on write-path planning gives unstable tool arguments—keep it low and schema-bound. Retries without error class turn transient blips into duplicate shipments. One global gate table breaks the moment Plant B needs a stricter outbound rule than Plant A. “The model said it updated CRM” means you treated language as state; only tool results count.
Demo environments that skip freeze are worse than no demo: they train approvers to sign summaries instead of artifacts. Production then feels broken when the real gate appears.
Three scenes, same spine
Freight: multi-inbox inquiry, extract lanes and charges, draft reply, freeze, manager release, send with idempotency. (freight.crusbro.com if you want to click through a desk-shaped sandbox.)
Drawings: parse, suggest BOM/route, check rev conflicts, freeze before MES promotion; low-confidence fields to a human with primitives and rule version in the pack.
Vision: model proposes; gray zone and QMS writes through gates; dual-sign where the customer requires it. Leak-before-false-call is a configured tradeoff on the line—not a slogan.
Where we still argue internally
Freeze adds latency; replay adds storage; tight gates annoy power users who want the agent to “just do it.” We have not found a way around that triad. Workflow engines are clearer for fixed paths but brittle when the inbox is messy. Agents are messier but fit unstructured intake—if the harness is strict at the boundary.
If you have solved replay without bloating every step, or you draw the autonomy line differently, we are interested. That is what field notes are for.