The Missing Metric in Agentic AI

Why Organisations Should Measure Decision Authority, Not Just Accuracy

For several years, much of the conversation around artificial intelligence has centered on whether a system gives the right answer. That was a reasonable place to start. When a system summarises a document or suggests a response, accuracy matters because you can catch a mistake before anything happens. The operating environment changes when the system is no longer limited to producing an answer.

An agent may receive a request, interpret it, search internal information, decide what should happen, update a record, send a message, and trigger another process without requiring approval at each step. At that point, asking whether the model is accurate tells us only part of what we need to know.The more important question becomes: what is this system allowed to do when it is wrong?

Accuracy vs. Actions

Intelligence and authority are not the same thing. A capable system with limited authority can be straightforward to supervise. The same capability, paired with permission to modify records, communicate externally, or spend money, creates a different risk. The model may not have changed. Its authority has. Accuracy still matters, but it largely describes output quality. Agent-based systems increasingly need to be assessed through the consequences of actions.

A customer-support assistant that produces an incorrect draft creates inconvenience; one that sends it automatically creates an external commitment. A finance assistant that misclassifies an invoice creates a review problem; one permitted to approve payment creates financial exposure. The underlying mistake may be similar, but the failure is not. The missing variable is decision authority.

Decision Authority

I use decision authority to describe the practical freedom a system has to make or execute a decision without fresh human approval. It is not simply a question of whether a system is autonomous. Autonomy is rarely binary. A system may perform one step independently but require approval for another. It may act below a financial threshold or contact an internal team but not an external customer. The meaningful question is not, “Is this autonomous?” It is, “Where does its authority begin, where does it end, and what happens inside those boundaries?”

A useful way to examine that is through a Decision Authority Profile built around five measures.

1. Autonomous action
2. Consequence exposure
3. Human intervention
4. Reversibility
5. Audit latency Let us have a look at each of these measures in detail below.

Measure #1: Autonomous Action

To understand the concept and principles behind this autonomous action, we need to ask ourselves a question...'how much of a process can the system complete without a person approving its decision?'. Consider two systems handling the same request. One identifies the issue, prepares a response, and recommends the next action. The other chooses the resolution, updates the record, issues the refund, and sends the confirmation.

Both may use similar reasoning. Their operational authority is completely different.The useful measurement is not simply how many tasks have been automated, but how many meaningful decisions can move from interpretation to execution without another person becoming involved. Removing repetitive copying from a workflow is not the same as delegating a decision.

Measure #2: Consequence Exposure

Not every autonomous action deserves the same level of control. An incorrect internal tag may have almost no lasting consequence. An incorrect change to someone’s access rights could be far more serious.The question is simple: 'if the system makes the wrong decision and nobody intervenes, what can actually happen?' The consequence may be financial, operational, reputational, or human. What matters is the action, not the technology’s sophistication. A simple automation may deserve stricter controls than a more advanced system if it has access to something more consequential.

Measure #3: Human Intervention

Organisations often talk about reducing human involvement as though the lowest possible level is automatically desirable. I do not think that is always the right objective. Human intervention provides information. If people repeatedly correct the same kind of decision, the problem may not be that they are slowing the system down. Their intervention may be showing that the boundary of automation has been drawn in the wrong place.

The following scenarios offer insight into different problems:

• Where are people stepping in, and why?
• Are they correcting factual errors, handling ambiguity, applying judgment, managing exceptions, or simply checking work because they do not trust it?

Removing the person before understanding why the intervention exists can speed up the process while making the underlying system worse. The better objective is not minimum human involvement, but deliberate human involvement. People should not spend their time approving routine decisions that are already predictable, but they also should not be removed where context, consequence, or ambiguity makes judgement valuable.

Measure #4: Reversibility

Some mistakes are cheap to reverse, i.e., a draft can be deleted, a recommendation ignored, a temporary classification corrected. Other actions become harder to undo. Examples include a message sent to a customer cannot be meaningfully unsent and/or an incorrect payment that may require recovery.

This suggests a practical principle...the easier an action is to reverse, the more comfortable an organization can be granting authority. The harder it is to reverse, the stronger the case for additional control before execution. Instead of asking whether to automate an entire workflow, divide it by reversibility. Let the system perform low-consequence actions that are easy to undo. Introduce approval where the action becomes difficult, costly, or socially awkward to reverse.

Measure #5: Audit Latency

Mistakes will occur in any sufficiently complex operational environment, whether the decision comes from a person, a rule-based system, or an agent. The question is how quickly the organisation can understand what happened. Have a look at the following sequence:

• Request received
• Information used
• Decision made
• Records accessed
• Actions that followed

Can somebody reconstruct the above sequence? If reconstruction takes hours of searching across logs, messages, and applications, the organisation has an operational problem even if the system performs well most of the time. Auditability is therefore not just a compliance concern. It affects how quickly people can diagnose failures, correct them and improve the process. The longer it takes to understand an automated decision, the more expensive that failure becomes to manage.

Together, these measures show how often the system can act independently, how serious the consequences are, where people intervene, how reversible mistakes are, and how quickly decisions can be reconstructed. I would resist combining them into a single score. A single number can hide the differences that matter. Two systems could score similarly while carrying very different operational exposure. The purpose of a Decision Authority Profile is not to label a system safe or unsafe, but to make the distribution of authority visible.Once that authority becomes visible, design decisions become easier.

Imagine an organisation introducing an agent into a customer-support process.The system may retrieve account information, identify the likely issue, and prepare a response automatically. For familiar, low-consequence cases, it may also be permitted to send the response. For refunds below a defined threshold, it might proceed while recording the reasoning and transaction. Above that threshold, a person becomes part of the decision. For unusual or sensitive cases, human review remains mandatory. This is not simply “human in the loop”.

The human-in-the-loop Conundrum

A person who clicks approve hundreds of times a day is technically in the loop. Whether that person is exercising meaningful judgment is another matter. A better design identifies where human authority adds value and places it there. That distinction becomes more important as systems connect to more tools.

An isolated assistant has limited reach. A connected agent may read documents, modify records, send communications, and trigger other processes. Each integration increases usefulness, but can also increase the distance between the original instruction and the final consequence. The main design challenge is therefore not just connecting systems together. It is controlling how authority travels across those connections. This is where permission design becomes part of workflow design.

A system should not receive broad access simply because broad access makes implementation easier. Permissions should reflect the decisions it genuinely needs to make. In practice, convenience often wins. Broad data access and direct update permissions make workflows easier to build, but those shortcuts can quietly turn technical access into decision authority. That is why the authority discussion needs to happen during implementation, not after deployment.

Permissions are Key in Approvals

For every automated decision, the following four questions are worth asking:

• What human problem is being solved?
• What part of the process is creating unnecessary effort or delay?
• What system access is actually required?
• What happens when the automated decision is wrong?

If the last question is difficult to answer, the workflow is probably not ready for greater autonomy.Another reason decision authority deserves attention: organisations can gradually increase autonomy without noticing. A system begins as an assistant. Later it creates drafts, sends them automatically in simple cases, then gains permission to update records. Another integration is added. No single change appears dramatic. Yet after several months, the organisation may have created a system with much greater operational authority than the one originally reviewed. The risk does not always arrive through one large deployment. It accumulates through permissions.Governance therefore cannot be a one-off approval exercise.

Decision authority

Decision authority profiles should change as workflows change. New integrations, permissions, thresholds, and categories of autonomous action should alter the profile. A system that was acceptable when it could only recommend an action may need different controls once it can execute that action.This also changes how we should measure success.A mature deployment is not one where humans have disappeared, nor one where the largest possible number of tasks has been automated.

Success is closer to this: routine decisions move with less friction, people stay involved where judgement matters, mistakes are contained, consequential actions have clear boundaries, and somebody can understand what happened when something goes wrong. There will still be difficult questions. Consequence is not always easy to quantify. Human review can also fail, approvals become habitual, and oversight can turn into ceremony. Reversibility can mislead too...a database field may be easy to restore while the communication it triggered has already had an external effect. These limitations are why the authority profile should be treated as a way of thinking, not a compliance score. Its value lies in forcing a conversation that technical performance measures often miss. We should continue measuring whether systems produce correct outputs. But as those systems begin to act inside organisations, accuracy needs a companion measure. We need to understand the authority attached to those outputs.

Conclusion

The question is no longer simply whether a system can make a good decision. It is whether the organisation has designed sensible boundaries around what that decision is allowed to change. The systems that earn greater autonomy will not necessarily be the ones that appear most intelligent. They will be the ones whose authority has been designed with enough care that an imperfect decision does not automatically become an uncontrolled consequence.