Checking work when effort is no longer the signal

Reviewing other people’s work has always leaned on a quiet clue: how much effort it took. A thorough document usually came from somebody who had done the reading. A confident answer usually came from somebody entitled to be confident. Neither holds for a machine, which produces the fluent version and the wrong version at the same speed and in the same tone. Review habits built on the old clue do not transfer, and most organisations have not noticed that yet.

The asymmetry that decides everything

Before automating a task, ask one question: how much easier is it to check the result than to do the work yourself?

Where checking is cheap and the doing is expensive, the gain is real: an extraction you can confirm against the source in seconds, a draft whose errors are obvious to anyone who knows the subject. Where checking costs about what doing costs, you have moved the work rather than removed it, and possibly made it worse: reading somebody else’s plausible output closely is more tiring than writing your own.

That single question sorts most candidate tasks correctly, and it is not the question people tend to start from.

Make the output checkable

Whether work can be checked quickly is a design choice, not a property of the task:

• Have it cite where each fact came from, and link to it, so verifying is a click rather than a search
• Have it show the intermediate steps, not only the conclusion
• Have it say what it could not find, instead of filling the gap
• Keep the source material beside the output, in the same view

A tool that answers with a paragraph and no provenance is asking to be trusted. One that answers with a paragraph and four references is asking to be checked, which is the behaviour you want from it.

Review where the consequence is

Reviewing everything with equal care is how review becomes a formality. Sort the work by what happens if it is wrong: goes to a client, goes to a regulator, moves money, enters a system of record. Those get read properly, every time.

First passes, internal drafts and anything a person will rework anyway can be sampled. Being explicit about which is which is what keeps the first category from being skimmed along with the rest.

Design against the rubber stamp

The failure is predictable. Output that is right nine times out of ten trains the reviewer to skim, and the tenth goes through. Nobody was careless; they were responding rationally to a long run of good results.

What helps is structural. Sample rather than sign off in bulk, so each check is a real read of a small number. Rotate who checks, because a fresh reader notices what a familiar one has stopped seeing. Occasionally have somebody do the work without seeing the generated answer first, which is the only check that does not start from what the machine already decided. And record the corrections. A log of what was wrong is the only honest measure of whether the thing is working, and it is what tells you when the sampling rate can come down, or has to go up.

Some work does not qualify

If the only person who could spot the error is the client, the task is not a candidate: there is no review step that would catch it. That is rarely a permanent verdict. Usually it means the task has to be narrowed until somebody inside the business can check the part being automated.

The boundary is worth stating plainly, because it survives whatever the tools do next: automate the part you can verify, and keep a person answerable for the part you cannot.

The point of checking

None of this comes from distrusting the tool. It is about keeping the ability to tell. An organisation that can say how often its automated work is right, because it looked, can extend it with some confidence. One that cannot is making a bet every time it presses send, and will find out how the bet went at the least convenient moment.