“Classic governance was built for systems of record and known analytics pipelines,” he said. “That world is gone. Now you have systems creating systems — new data, new outputs, and much is done on the fly.” In that environment, point-in-time audits create false confidence. Output-focused controls miss where the real risk lives.
“No breach is required for harm to occur — secure systems can still hallucinate, discriminate, or drift,” Butt said, emphasizing that inputs, not outputs, are now the most neglected risk surface. This includes prompts, retrieval sources, context, and any tools AI agents can dynamically access.
What to do: Before writing policy, establish guardrails. Define no-go use cases. Constrain high-risk inputs. Limit tool access for agents. And observe how systems behave in practice. Policy should come after experimentation, not before. Otherwise, organizations hard-code assumptions that are already wrong.
