Agents at work - Trusting AI-enabled systems

At the end of my previous article, I discussed an elaborate evaluation layer as the last step of the build pipeline, ensuring that what was built is what we intended to build. That it works the way we intended for it to work.

So if the evals tell you that, yes, the system does work as intended, we should be able to deploy it with the same confidence as we did with systems we built "ourselves".

Don't get me wrong; I am not advocating for going from "how we've always done things" to this new reality over night. The whole point of that post, and this one, is to talk about how the new way of doing things can look. And you get there by experimenting, testing, learning. And failing. So we are building out our guardrails using low risk/low consequence systems as our first contenders.

AI adoption vs AI enablement

I separate these two terms into "building with/using AI tools" and "incorporating AI components into your existing systems". Adopting AI is the natural first step: learning the tools, experiencing its quirks and hallucinations, witnessing the non-determinism.

Familiarising ourselves with these things is important, because we need that understanding when we decide to extend our system's functionality with a component that behaves quite differently than we are used to with code.

How non-determinism affects your system

The obvious one is the LLM's probabilistic nature. The same input twice does not produce the same output twice. Sure, it will usually be quite similar, but you don't know. So the system must be built in a way where this variance is tolerated. It also means that we cannot use only our normal test layer to ensure things work as expected.

But even if we account for the inherent unpredictability of the models, that's not enough. In addition to the model unpredictability, there's a variety of other reasons an AI-enabled system can change the way it behaves over time. We call this drift. Here are a few examples:

• A model starts behaving slightly differently after model upgrade (model drift)

• Behavioral drift from changes to the system prompt

• The context-window changes between model versions

• A change in the tokenization behaviour

• Changes to temperature/top-p

• Model routing changes behind a provider endpoint

• Capability regression in a newer model

• Inputs from the system gradually becoming larger than during the system's testing and early deployment

• Unexpected capability increase in newer models creating new risk

• Changes in model behavior near context limits

There is almost an infinite number of such examples, in various categories like data and context, reasoning, agent/tool-use failures, integration failures, autonomous-system risks, evaluation-specific failures, security (even without a malicious attacker), operational causes… I'm sure you can think of some categories specific to your system or your company as well.

Intimidated?

That's not my goal, but respect for unpredictability is healthy. And the encouraging part is that you already know the solution to it, because it's the same remedy we applied to the build pipeline.

When adopting AI into our build pipeline, we asked: how do I know that the system that was built is the system I intended to build?

When AI-enabling our systems, the same problem continues after deployment, because a deployed AI-enabled system is not frozen. The next question to ask, then, becomes:

How do we know the system we deployed is still doing what we intended?

And the answer to that is the same as for the build pipeline: an evaluation layer. Or several. Except these evals run continuously on the system.

From one-time evals to continuous evaluation

These evals will depend greatly on the system being evaluated, but we can imagine some example layers:

Live behavior evals - Do the outputs satisfy the expected format? Did it use the right data? Did it call the right tool? Did it follow policy? Did it complete the task?

Regression evals - Keep a database of known good cases from development and rerun them regularly, and especially when something changes: model version, system prompt, retrieval pipeline etc. This builds your confidence that the system still behaves at least as well as the version that was approved.

Drift detection evals - For this you need to measure several metrics over time. It is not a binary measure, but measurements over the distribution of metrics. Examples of metrics could be answer length, refusal rate, tool selection frequency and token use.

Incident-driven evals - Possibly the most valuable category. Every non-trivial production failure becomes a new eval case, giving you a growing regression suite based on the system's actual history, rather than imagined edge cases. I find this category very interesting, because this is really the system learning from itself. For agentic systems in particular, these could serve as a way to achieve recursive self-improvement. Don't get me started, because this is a super-interesting topic!

Wrapping up

Continuous evals enable you to detect drift early, allowing your systems to handle the situation based on what risk tolerance the system in question has.

For your critical systems, you've done risk assessment long before considering AI-enabling them. So you already have a playbook for how to handle deviations or outages.

Adding AI to the system doesn't meaningfully change that regime, except for the detection/evaluation layer. If your continuous evals start to detect drift, there will be a threshold after which the system will apply the safety measures you already have, like fallbacks, containment, graceful degradation, human escalation etc.

The primary focus of AI-enabling your systems, therefore, is designing the guardrails and policies around them. With those in place, adding the AI component no longer looks so risky.