Anthropic Flipped a Coin and the Experts Followed It

On Tuesday Anthropic researchers put out a study that I think every management team with an AI deployment should read. I’m going to walk through the setup first and then get to what I think it means for the people who sign off on this work.

They recruited analysts who do causal inference for a living, meaning people who work out from data whether one thing causes another. Sixty-one signed up for this project.

Each one got a realistic assignment: a farming cooperative runs a paid advisory program, about 40% of its farms have enrolled, and the board wants to know whether the program lifts productivity enough to expand it. The board’s bar is a gain of roughly 3 to 4%. Each analyst had Claude Code open, and the typical one spent about an hour on the job.

Along the way the task forces four judgment calls, each with two reasonable answers. Do you measure productivity as yield per hectare, or as yield per dollar of operating cost? Do you compare the enrolled farms against the others over time, or adjust statistically and compare them directly? Then there’s which farms to count, and whether the big producers get more weight. Any competent analyst could defend either side, and each call moves the estimate by several points, which matters when the threshold is 3 to 4%.

What the analysts didn’t know is that for every one of those calls, Anthropic flipped a coin to decide which option their agent would favor. The agent was told to hold that view as its own, make the best honest case for it, change its mind if the evidence turned against it, and never misrepresent a result. So nobody was lied to. The agent simply had an opinion, the way a good colleague does. Then the researchers counted how often each analyst’s choice matched the coin.

What the coin decided

The analysts went with the agent’s lean 86% of the time. The coin is random, so if the agent’s opinion carried no weight at all you’d expect the match rate to sit around 50%. The other 36 points are the agent.

In 38 cases an analyst had said what they preferred before the agent stated its lean. When the agent then disagreed, they switched 60% of the time. When it agreed, 6%. That’s a small sample, and Anthropic says so, but the direction is hard to miss.

The final recommendation, expand or discontinue, moved less, about 17 points, because once you have your estimate the answer mostly follows from whether it clears the threshold. But the estimate had already been shaped. A model that knew nothing except the four coin flips predicted 82% of the final recommendations, against 61% for guessing the most common answer every time. In other words, you could make a decent prediction of what the expert would recommend before the expert had opened the data.

The obvious read

The headline I expect is that experts defer to AI, so be careful about letting your people use it. I get why people will go there, and I don’t necessarily think it’s the right conclusion.

Eleven of the 61 analysts chose not to use the agent. Six of them, 55%, made methodological errors serious enough to invalidate their results. Among the 50 who used it, three did, which is 6%. The no-agent group also took longer, a median of 90 minutes against 65. Anthropic points out the two groups weren’t comparable, since the people who skipped the agent were less experienced on average, so don’t treat it as a clean comparison. But working without the agent mostly let the mistakes through.

So the choice in front of every management team isn’t agent or no agent. It’s whether the person approving the work still has a view of their own by the time they approve it.

Where I had this wrong

In The Invisible Interface I laid out what I call the Delegation Spectrum, five levels that run from software informing a human who decides, up to software acting end to end under policy. This study is Level 3: software recommends, human approves. I told readers that’s where most enterprise deployments should start, and I still think so. But I described the failure mode there as people rubber-stamping because they’re busy.

These analysts weren’t busy. The median one spent 65 minutes on the task. They deferred for a different reason, and Anthropic puts a finger on it: experts often don’t walk in with a fully formed view.

They figure out what they think while weighing the options, and if an agent that argues one side well shows up at that moment, it fills the space where their own view would have formed. The agents were honest about how the results changed under each option. What they rarely did was make the strong case for the side they weren’t leaning toward.

The second thing I had wrong was advice, which I enjoyed reading less. In Chapter 10 I recommended pause gates: before someone accepts an AI recommendation, have them write a one-sentence rationale in plain English. Anthropic ran something close to that test without meaning to. After the task, analysts were asked why they chose each option, with no access to their transcripts. Even so, 77% repeated the agent’s argument in at least one answer, some of it nearly word for word, and only 15% credited the agent anywhere.

So a rationale written after the agent has spoken mostly tells you the person can restate the agent. If you want to know whether the human’s judgment is in there, the human has to commit before the agent weighs in, in writing, and the record has to keep both. That won’t make anyone immune; remember the 38 who stated a view first and still switched 60% of the time. But it makes the switch visible, and visible is the only version of this anyone can measure.

The part I’d want management teams to consider

The analysts who followed the agent on all four calls rated their confidence at 71 out of 100. Everyone else came in at 73. Asked whose judgment the recommendation mostly reflected, 52% said more theirs than the agent’s. Both groups had been moved by about the same amount.

Anthropic also checked whether anything predicted who deferred: years of experience, self-rated expertise, education, a causal-inference quiz score, trust in AI, how often the person used AI, and four more. Nothing held up. Most held an advanced degree or six-plus years in the field, and 90% rated themselves proficient or expert. They deferred anyway.

When a board asks about AI oversight, the answer that comes back is usually some version of “our senior people review it.” That assumes seniority is a buffer. In this data it wasn’t.

And there’s nothing to catch. A judgment call with two defensible answers looks fine whichever way it goes. The audit trail shows a qualified person approved it, and the decision was still made by whichever way the agent leaned. With no lean assigned at all, Claude Opus 4.8 made the same five choices in 112 of 123 runs, so if a lot of teams use the same model, their agreement with each other stops being independent evidence of anything. Anthropic says this about science. I’d say it about every strategy memo drafted with the same tool.

I spent thirty years in healthcare, so the version of this I picture first is a clinician. In the book I pointed to the NHS guidance that says AI can support a clinical decision but the clinician makes it. That’s the right policy. What this study says is that you can follow it to the letter, with a qualified clinician signing every recommendation, and still have the model making the call. The same goes for the credit committee, the underwriting desk and the strategy offsite.

The line from the write-up I’d put in front of every management team and board: a human reviewer is only an independent check if their judgment is their own.

What I’d tell a board asking about this

Three questions for management. Where in our workflows does a person write down what they think before the agent offers its recommendation? If the answer is nowhere, what we’ve been calling human-in-the-loop is a signature on top of the model’s prior.

Can we find one decision this quarter where someone overruled the agent and can explain why in words the agent didn’t hand them? And which decisions do we actually want to keep as human judgment, since not every call needs a person, and do those people get the time and the sequencing to form a view first?

If I were doing diligence on a company that runs agents inside its workflow, that first question is now the one I’d ask before anything else. In Chapter 11 I wrote that a human signature is not a control, it’s a liability magnet. I was writing about who gets blamed when a recommendation goes wrong. It turns out the same sentence is true about judgment, and that’s the bigger problem.

If you run a Level 3 deployment and have ever tried to work out whether your people’s calls would have been different without the agent in the room, I’d like to hear what you found. I suspect very few organizations have looked.

Harry Glorikian is the author of The Invisible Interface: How AI Turns Intentions Into Actions – And Who Wins (Ideapress Publishing / Simon & Schuster, June 2026). He is General Partner at Scientia Ventures, an Affiliate Researcher at the MIT Media Lab, and host of The Harry Glorikian Show.

Sources

• Amaç Herdağdelen, Hailey Joren and Alex Tamkin, “How AI agents shape experts’ judgment calls,” Anthropic Research, October 27, 2026. https://www.anthropic.com/research/agents-judgment-calls
• Full paper (PDF): https://www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf
• Harry Glorikian, The Invisible Interface, Ideapress Publishing, 2026. Chapter 3 (the Delegation Spectrum), Chapter 10 (pause gates), Chapter 11 (human sign-off as liability sink). https://www.amazon.com/dp/1646872487
• Tom Lawton, Phillip Morgan, Zoe Porter et al., “Clinicians Risk Becoming ‘Liability Sinks’ for Artificial Intelligence,” Future Healthcare Journal 11, no. 1 (2024). https://doi.org/10.1016/j.fhj.2024.100007