[ITmedia Executive] Where to leave people behind - Governance design

The introduction of AI agents is often thought of as a question of ``to what extent should human work be replaced?'' However, the point of practical work is rather the opposite, and is to design ``where to leave people.'' In the fourth session, we will discuss governance for safely using probabilistic mechanisms as an organization.

The essence of governance is “where should we leave people?”

Last time, I stated that there are theoretical limits to technological defense. The design of human involvement assumes this residual risk. Therefore, the ``human involvement'' mentioned in this series is not just a precautionary measure. It is a component that makes up the security architecture. No matter how robust the guardrails are, the final answer to the question, ``Is it okay to leave this decision to an agent?'' is an organizational decision, not a technology one.

In fact, the design concept of ``humans monitoring machines'' was not first created by AI agents. There is knowledge that has been accumulated over several decades in fields that require high reliability, such as air traffic control, nuclear power, and medical care. One of the most suggestive is the ``irony of automation'' that cognitive psychologists have pointed out. As automation becomes more sophisticated, human attentiveness and skill decline, and it is precisely during abnormal situations, when human intervention is most needed, that the ability to intervene is diminished. Herein lies the pitfall of thinking that full automation is the ideal.

What this lesson shows is that ``retaining human resources'' does not simply mean placing one person in the role of watchdog. In order to retain someone who can make meaningful decisions in times of abnormality, that person must understand the context of the work on a regular basis and be able to take action when the situation calls for it. Even if you have a token supervisor sitting in front of a screen, if they are not given any judgment or authority, nothing will actually be stopped. Human involvement does not end with the arrangement, but the scope of design extends to the point of keeping things in a functional state.

Control proportional to risk

The most common mistake to make is to impose the same level of control on all agents. If you impose the same approval process on agents who respond to internal FAQs and agents who place orders with business partners, the former will be over-managed and useless, and the latter will be passed through with only token approval. Over-management and under-management occur at the same time - this is the structural problem of uniform management.

The answer is clear. Evaluate the risk for each agent and set the management strength proportional to the risk. There are mainly three axes for evaluation. These are the reversibility of the operation (can it be undone later), the scope of influence (whether it extends to an individual, a department, or outside the organization), and the detectability of errors (whether it is noticed immediately or whether it goes unnoticed for a long time).

If we look at these three axes, we can see that even the same function, ``email sending,'' should not be grouped together with the same management strength. Standard communications for internal use are highly reversible, have limited impact, and require simple approval. On the other hand, sending an email to a business partner cannot be canceled and the scope of influence extends outside the company, so it is an operation that requires prior approval. Therefore, management needs to be designed with consideration to the reversibility and scope of influence of individual operations, rather than all at once on an agent-by-agent basis. Coarse-grained risk assessments create a breeding ground for both over-management and under-management.

Three intervention patterns that leave humanity behind

There are three main types of human involvement. These are not exclusive and may be combined depending on the nature of the operation.

Pre-approval A human reviews and approves the content before the agent can operate it. Place it immediately before the most stringent, irreversible operations - money transfers, contracts, customer calls, data deletions. The agent creates a "plan" and humans decide on "execution." This is not a waste that reduces efficiency, but a necessary expense for using the probabilistic mechanism in production.

Post-Verification Operations are executed immediately and results are checked by humans after the fact. Suitable for tasks that require reversible operations and immediacy. However, in order to avoid "formal confirmation," it is necessary to present the basis for judgments in a form that can be grasped in a short time, and to have a system in place to feed back the corrections for improvement.

Periodic audit Rather than looking at each individual operation, it statistically evaluates behavior over a certain period of time. Although management costs are low, irreversible operations should not be managed solely using this method, as it delays the discovery of problems. In practice, these three are distributed according to the weight of the operation. Irreversible and high-impact operations should be approved in advance, reversible and frequently occurring operations should be verified after the fact, and low-risk, routine operations should be audited regularly - by grading things in this way, you can concentrate your limited manpower where it's really needed.

Approval fatigue, kill switch, responsible design

Too many pre-approvals can lead to ``approval fatigue,'' in which approvers automatically approve without checking the content. This is the same problem as the "alarm flood" that contributed to the nuclear accident. Countermeasures include automatic screening of low-risk operations and batch approvals, as well as auditing the quality of approvals themselves (detecting signs such as extremely short approval times and approval rates approaching 100%).

At the same time, prepare a kill switch to immediately stop unexpected behavior. It is designed in three layers: per session, per agent, and as a whole for the entire system, to prepare for ``ideally having it but not using it, and worst case having it but not being able to use it''.

The most important thing that decision makers must decide on is responsibility design. When an agent makes a mistake, the responsibility lies with the department head who approved the introduction, the site where the agent is operated, or the vendor who developed it. If we proceed without this being ambiguous, the organization will be unable to move at the moment of an accident. We would like to apply the ``4eyes principle'' (at least two people are involved in one operation) and the separation of duties among creators, approvers, operators, and auditors that have been established in regulated industries such as finance to agent operations. The reason why an increasing number of companies are appointing a CAIO (Chief AI Officer) is to clarify the location of this responsibility and policy. In addition, it would be too late to set up a design for this responsibility after an accident occurs. At the same time as the decision to implement, document who will approve it, who will stop it, and who will be accountable after the fact. Having decided on this during peacetime, the organization can move without hesitation in an emergency.

Trust grows progressively based on performance.

Human involvement is not fixed. Strict controls at the initial stage of implementation are gradually loosened based on operational performance - this process of ``gradual expansion of trust'' itself is the object of the design. At the initial stage of implementation, all cases are approved in advance, during a stable period, operations for which quality has been confirmed are subject to post-verification, and at a mature stage, routine operations are subject to periodic audits only.

The key is to judge this transition based on quantitative criteria, rather than a sense that "it's about to be okay." Trends in the intervention rate (percentage of human intervention), the number of serious problems discovered during post-tests, and the passing rate of quality evaluations - when these indicators meet the standards, the process moves on to the next stage. And expanding trust is not a one-way street. If quality deterioration or model updates occur, a "rollback" function is designed in advance to return to the previous stage. Rollback is not a failure, but the normal operation of a safety valve.

Governance sounds like a brake that restricts utilization. But actually the opposite is true. If we clarify where humans should remain, we can boldly advance autonomy in other vast areas. Deciding where to leave people is inextricably linked to deciding where to entrust them.

AI agent design theory for decision makers

The essence of introducing an AI agent lies in "controlling uncertainty" rather than technology selection. Unraveling the essentials of design theory for taming and safely implementing autonomous systems that defy conventional wisdom as an organization.

See more articles in this series

Author of this article

Related articles