TL;DR
Catching a problem is half the job; enforcing is the other half. When a guardrail fires you can block, redact, or warn — safest to weakest. And every LLM-judged guardrail buys its judgement with a per-turn model call (latency + money), so you spend that budget where the risk justifies it and lean on cheap deterministic checks everywhere else.
Block, redact, or warn?
When a guardrail catches something, it has three enforcement options, and the right one depends on what you're protecting:
| Action | What happens | When to use it |
|---|---|---|
| Block | Refuse the turn; tell the user to remove the data | Default for sensitive data — the value never reaches the model or logs |
| Redact | Strip the value, replace with a placeholder, let the clean message through | When the rest of the message is legitimate and useful |
| Warn | Let it through but flag it | Weakest — the data still flows; use only when you can't block |
That's the checkpoint's first half. A card number pasted into an otherwise-fine question is the textbook case for redact if the rest of the message needs answering — but your agent's PII filter blocks and records only a redacted copy for the trace, because for support there's rarely a reason the model needs to see a card number to help. Conservative by default; loosen deliberately.
The cost of judging every reply
The second tradeoff is which kind of guardrail to run. Deterministic checks (PII, injection, length, schema) are nearly free — a pattern match adds a millisecond. LLM-judged checks (topic scope, toxicity) are different: every turn they guard makes an extra model call.
| Deterministic (regex/schema) | LLM-judged (topic, toxicity) | |
|---|---|---|
| Cost per turn | ~nothing | A full model call |
| Latency | ~1 ms | The user waits for a second model round-trip |
| Catches | Things with a shape | Judgement calls a regex can't |
So the benefit (catching failures a regex never could) is bought with per-turn latency and spend — on every turn, not just the bad ones. That's the checkpoint's second half: an LLM-judged guardrail is worth its cost when the risk justifies the tax — a public-facing agent, a compliance requirement, a high-stakes domain — and overkill on a low-risk internal tool where the deterministic checks already cover the real danger.
Analogy
Safety is a budget, not a switch. Deterministic checks are the cheap, always-on smoke detectors; LLM-judged checks are hiring a security guard. You put guards where the valuables are — not one at every closet door. Spend your safety budget where the danger actually is.
Common mistake
Turning on every LLM-judged guardrail "to be safe" and doubling the latency and cost of every single turn — including the 99% that were never going to be a problem. Safety that makes the agent twice as slow and expensive for everyone is its own kind of failure (users leave, bills balloon). Layer the cheap deterministic checks broadly; add the expensive judged ones precisely, where the risk is real.
Key takeaways
Key takeaways
- Enforcement has three settings: block (safest) → redact → warn (weakest).
- Block sensitive data by default; redact when the rest of the message is legitimate.
- LLM-judged guardrails cost a model call every turn — latency and money.
- Spend the safety budget where risk is real: deterministic checks broadly, judged checks precisely.
Go deeper: failing safe, and observing your guardrails
Two operational realities make guardrails trustworthy in production:
- Fail safe, not open. What happens if the guardrail itself errors — the toxicity classifier times out, say? A guardrail that "fails open" (lets the turn through on error) silently disables your safety net exactly when something's already wrong. For safety-critical checks, prefer fail closed: on guardrail error, block and return the fallback. Slower, but safe.
- Observe what your guardrails do. Guardrail decisions belong in your traces (Module 6): which guardrail fired, on what, how often. A guardrail blocking 40% of traffic is either under attack or too aggressive — you can't tell which without the data. And a guardrail that never fires might be broken, not unneeded.
- Tune with the red-team suite. The lab's red-team set is also your eval for guardrails (Module 7): does turning a guardrail on block more attacks without blocking legitimate traffic? That's a measurable tradeoff — false-negatives (attacks that slip through) vs. false-positives (real users wrongly blocked) — and you tune it the same way you tune any metric.
Guardrails aren't set-and-forget. They're a safety system you instrument, observe, and tune — with the same trace-and-eval tools you already built.