With agents, the same input doesn't guarantee the same output. Building them at Lexacon, I had to account for that every time. It's how agents work, and it changes how you have to manage them compared to apps.
Most agents going in right now are replacing traditional automation, and they're being managed the way that automation was. That kind of automation is deterministic, so you test it, sign it off in UAT, and it behaves the same way until someone changes it. Agents don't act that way. Passing UAT tells you how an agent behaved on the days you tested it, not how it will behave a month later.
We already manage something that behaves differently from one day to the next. We do it with people. Nobody signs off a new hire after their first week and stops looking. They get performance reviews, regular check-ins and a performance improvement plan for when things slip.
Agents need the same treatment. Not just a one-time test, but a review cycle with an accountable owner, outputs sampled on a schedule by an independent mechanism, and a threshold for when the agent goes back for 'retraining', whether that means reworking the prompts or tools, or swapping the model.
If your agent passed UAT, you've reviewed its first week. Who's running its performance review, and when was the last one?
#AgenticAI #AIGovernance #Automation