Scalable oversight is the challenge of providing reliable training signal and supervision to an AI system on tasks where evaluating its output is difficult, expensive, or beyond the overseer’s own ability. The phrase was introduced as one of the concrete problems in the 2016 paper “Concrete Problems in AI Safety,” framed there as the question of how to train an agent when we can only afford to evaluate a small fraction of its actions.
The problem sharpens as models become more capable. If a system can write code, summarize a long legal document, or reason about a scientific claim better than the human supervising it, then the human cannot simply check whether each answer is correct - the very expertise needed to judge the output is what the system was meant to supply. Naively relying on human approval in that regime risks training the model to produce answers that look good to a non-expert rather than answers that are actually right.
Several research directions are aimed squarely at scalable oversight. AI safety via debate has two systems argue so a weaker judge can decide; weak-to-strong generalization studies whether a weak supervisor can still elicit a strong model’s capabilities; and “Measuring Progress on Scalable Oversight” proposed the sandwiching experiment to study the problem empirically with today’s models. RLHF and its AI-fed variants are also, in part, attempts to scale human judgment.
For a business reader, scalable oversight is the crux of trusting AI with work you cannot easily verify yourself. Solving it is what would let organizations safely hand increasingly capable systems increasingly consequential tasks.