Part 4: Safety and governance for LLM systems: guardrails, PII, audit, and memory

Summary

When an LLM system makes decisions that matter, Level 4 safety and governance requires four disciplines: layered guardrails, PII handled at boundaries, an immutable audit trail, and scoped memory. Guardrails are independent layers arranged in a pipeline—input, grounding, output scrub, verification, judge, and confidence routing—each with a simple contract, and they fail closed so an error blocks the request. PII is handled at every boundary by classifying fields—unknown defaults to PII, secret never persisted—and redacting or hashing before crossing, storing only an HMAC hash in the audit ledger. The audit ledger is append-only, tamper-evident, and written as the final step of each decision, making past decisions traceable with a one-row query. Memory is split into seed (shipped config) and runtime (earned state); clear_state purges only runtime, returning behavior to baseline, and learning flows through a reviewed merge into seed.

Takeaways

1. 1

   Guardrails must be independent layers with fail-closed behavior: each layer catches a different failure mode (injection, off-schema output, logic errors, plausible-wrongness), and an erroring guardrail blocks instead of silently passing, preventing false assurance.

2. 2

   PII should be handled at every boundary using a declared sensitivity classification: unknown fields default to PII, secret fields are never persisted or logged, and the audit ledger stores only an HMAC hash plus a redacted summary for verifiability without storing raw sensitive data.

3. 3

   Memory should be typed into seed (shipped read-only config) and runtime (earned state); clearing runtime returns behavior to baseline, and new behavior flows through a reviewed eval-gated merge into seed, preventing unversioned, irreproducible behavior changes.

FAQ