Human-in-the-Loop Policy Design

1. Why Judgment Remains Central

Public policy is not an optimization problem with one objectively correct answer.

Governments regularly choose between legitimate but competing objectives:

• economic growth and environmental protection;
• innovation and precaution;
• efficiency and accessibility;
• individual flexibility and collective protection;
• short-term relief and long-term fiscal sustainability.

Evidence informs these choices.

It does not make the choices disappear.

An AI system may estimate impacts, identify patterns or generate possible interventions. It cannot legitimately determine which social value should prevail when values conflict.

That requires judgment.

It also requires accountability.

When a policy creates material consequences, institutions must be able to answer:

• Who defined the objective?
• Who reviewed the evidence?
• Who challenged the recommendation?
• Who decided?
• Who is responsible if the outcome is harmful?

This is why human oversight is not simply an ethical preference. It is an institutional requirement.

UNESCO's global Recommendation on the Ethics of AI states that ethical and legal responsibility must remain attributable to people or existing legal entities and that AI systems cannot replace ultimate human responsibility and accountability.

2. Human-in-the-Loop Is More Than Human Approval

Weak human oversight can look like this:

AI generates a recommendation. A busy official receives it. The official clicks approve.

Technically, a human was in the loop.

Substantively, meaningful judgment may never have occurred.

The Institute therefore treats human oversight as a design question:

Those points begin before AI analysis starts.

3. Where Humans Sit in the Policy Intelligence Cycle

The Institute's Policy Intelligence Cycle provides a practical way to allocate responsibility.

Human oversight therefore occurs throughout the cycle, not at its end.

4. Three Models of Human Oversight

Not every AI-supported activity requires identical governance.

It is helpful to distinguish three broad models.

The key question is not whether every action needs manual approval.

It is whether the level of oversight matches the consequence and uncertainty involved.

5. A Principle of Proportionality

Human oversight should increase as four characteristics increase:

This creates a practical spectrum.

Lower-risk AI assistance

Examples include:

• document summarisation
• transcription
• formatting
• translation with review
• classification
• initial literature discovery

These may require ordinary quality assurance.

Medium-risk analytical assistance

Examples include:

• evidence synthesis
• scenario generation
• international policy comparison
• forecast support
• identification of regulatory options

These require stronger verification, source traceability and expert review.

Higher-consequence policy assistance

Examples include:

• prioritising interventions affecting populations
• assessing distributional impacts
• making recommendations involving rights or significant public resources
• generating options for consequential regulatory action

Here, human ownership should be explicit and review significantly stronger.

A trusted system therefore does not ask: Can AI perform this task?

It asks:

6. Designing for Accountability

Five design principles are particularly important.

1. Traceability

A significant factual claim should be traceable to evidence.

If a model produces a conclusion but the team cannot establish where the underlying information came from, that conclusion should not quietly enter a policy recommendation.

UNESCO and NIST both emphasize traceability and accountability as important characteristics of trustworthy AI governance.

2. Contestability

AI-supported analysis should be challengeable.

Analysts should be able to question assumptions, test alternatives and disagree with machine-generated recommendations without treating the system as an authority.

3. Explicit delegation

Institutions should define what AI may: assist with, draft, recommend or execute.

Those are different levels of delegation.

Ambiguity creates accountability gaps.

4. Named ownership

Consequential outputs should have identifiable human owners.

The phrase “the model recommended it” is not a substitute for institutional responsibility.

5. Documented uncertainty

Policy intelligence should not hide uncertainty behind polished output.

Where evidence is contested, data incomplete or forecasts highly sensitive to assumptions, that uncertainty should be visible to the decision-maker.

7. Guarding Against Automation Bias

One of the less visible risks of AI-supported policymaking is not that the system makes a spectacular mistake.

It is that a plausible-looking output receives too little scrutiny because it is fast, coherent and confidently written.

This is automation bias.

Human oversight must therefore be active rather than ceremonial.

Useful practices include:

• asking analysts to independently form an initial view before seeing an AI recommendation
• requiring evidence links for material claims
• deliberately generating counterarguments
• conducting red-team review for consequential proposals
• comparing alternative models or methods where justified
• documenting disagreements
• requiring explicit human rationale for final recommendations

The human in the loop must be capable of saying no.

Otherwise there is no meaningful loop.

8. AI Drafts, People Decide — With an Important Qualification

The Institute uses a simple principle:

The phrase captures the direction of responsibility, but mature governance requires greater precision.

AI may do more than drafting. In defined contexts it can automate analytical or administrative work, detect anomalies, monitor indicators or execute pre-approved functions.

The important boundary is not the word draft.

It is authority.

Machines can perform increasingly sophisticated tasks.

But when a policy choice requires interpretation of public values, acceptance of significant trade-offs or exercise of consequential state authority, responsibility must remain clearly human and institutionally accountable.

Key Takeaway

Selected References

1. UNESCO, Recommendation on the Ethics of Artificial Intelligence (2021), particularly its principles on human oversight, accountability, auditability and traceability.
2. NIST, Artificial Intelligence Risk Management Framework 1.0, which frames AI risk management around governance, mapping, measurement and management and emphasizes organizational accountability.