Every company selling AI for plan review says the same thing: there's a human in the loop. We say it too. We believe AI should help plan reviewers, not replace them.
But as I dug into this question for our AI product roadmap, I found something worth sharing with the cities we work with. "Human in the loop" is not one thing. The phrase covers very different designs, and those differences change how much control the reviewer actually has.
I'd had a hunch about this for a while. From what reviewers told us, it felt right to have AI suggest a possible code issue rather than declare one. It turns out researchers in other fields have been studying exactly this question, and their findings back up that hunch in ways I didn't expect.
In this conversation, ePlanSoft CEO Debra Senra and Paul Stansik discuss the different kinds of "human in the loop" that are out there and why the difference matters for plan review.
Want to see what this looks like in practice? See how AI works inside e-PlanSoft →
When a vendor says a human stays in the loop, I'd ask two questions.
First, what does the AI hand the reviewer? Does it make a determination ("this is compliant, this is not") and ask the reviewer to agree or disagree? Or does it make a recommendation, pointing the reviewer to a spot on the plan with some context, and leave it to them to make the call?
Second, who sees the AI's output first? Does it go to the reviewer, or does the applicant see it at the same time, or even before the reviewer does?
Those sound like small design choices. But research from other fields indicates that they're more important than you might think.
Radiology got AI tools years before plan review did, so it has had time to study what happens when experts use them. One study stands out.
In 2023, a team led by Dr. Thomas Dratsch published a study in the journal Radiology. They asked 27 radiologists to read mammograms with help from an AI system. The AI gave each image a rating, essentially a verdict on how suspicious it looked. What the radiologists didn't know was that the researchers had deliberately set some of those AI verdicts to be wrong.
When the AI was right, the radiologists were right. When the AI was wrong, the radiologists were wrong with it. The most experienced doctors in the study, averaging more than 15 years on the job, went from about 82% accuracy to about 45% when the AI handed them a bad answer. Experience didn't protect them.
The lesson wasn't that the doctors were careless. It's that when a confident system makes the call for you, your job quietly changes. You're no longer deciding. You're approving a decision that's already been made, and people are bad at overriding that. It doesn't feel good to disagree with the machine.
Researchers call this automation bias. Anyone who has watched ChatGPT state something wrong with total confidence has felt a version of it.
Now compare that to a different design. Another 2023 study in Radiology looked at AI that helped doctors read chest X-rays by highlighting areas worth a closer look, rather than issuing a verdict. In that setup, doctors at every experience level got better. They caught more problems, and they read faster. A 2024 study in Nature Medicine covering 140 radiologists found the same pattern: what mattered most was not how experienced the doctor was, but whether the AI's mistakes could pull the doctor off course.
It could be the same underlying AI in both cases. The difference is whether it decides for you or helps you decide.
The second question, who sees the output first, works the same way, but the pressure comes from a different direction.
Say the AI tells an applicant, "Move your water heater," and the reviewer sees that note at the same time. The reviewer is technically in the loop. But if they think the AI is wrong, they now have to say so in front of the applicant, who has already read the AI's finding and may have started redrawing. Overriding the tool stops being a quiet judgment call and becomes a public correction, in front of the people the department serves.
That's a real cost, and it pushes reviewers toward going along with the AI even when their judgment says otherwise. We wrote more about this in Who Should See AI Plan Review Findings First?
Plan review has parts that are tedious and predictable, and parts that depend on professional judgment. The closer an AI feature gets to the judgment-dependent parts, the more we lean toward the recommendation model: flag it, highlight it, bring the reviewer's attention to it, and let the reviewer make the call.
That's our view of AI in plan review. Take the repetitive work off the reviewer's plate, make them faster at the work that's left, and leave their judgment where it has always been and always will be: at the center of plan review.
So the next time someone tells you their AI keeps a human in the loop, ask them which kind of loop they mean.
Curious what kind of AI we're building?
We're building AI that tackles the tedious, repetitive work involved in plan review - while still leaving the tough judgment calls up to the reviewers.
Want to see what we're building? Book a demo at the link below.