HLM: I want to come back to the idea that in the view of these participating surgeons, this was their best work they submitted, right?
JB: The advantage of having surgeons select videos of their best work makes this even more reproducible. If you didn't control for case difficulty, you'd introduce bias and measurement noise.
HLM: Were you surprised that fellowship completion or teaching hospital practice weren't linked to better outcomes?
JB: No. Keep in mind that we were focusing on surgeons who had been practicing for a mean of a decade. They had done thousands of bariatric operations.
HLM: How can you protect against bias among the peer reviewers? Perhaps some might favor a technique or sequence similar to their own.
JB. We found very little evidence of that. I worried we'd have hard graders and easy graders that would contaminate findings. We re-rated all these videos with a second expert panel, with no attachment to the Michigan collaborative, and they were slightly harsher. And all the best surgeons in the original rating were still the best, and the worst surgeons still the worst.
And we had the best and worst surgeons submit a second video, just to make sure there wasn't something weird about what they submitted on the first pass. And again, when we re-rated those, the best and worst got put in exactly the same place.
And when I showed these videos to non-surgeons, including lay people, with no labels or prompts, every single lay person said, "Oh my God, I definitely would want the first guy," and they were exactly right, of course. Everything wasn't quite as good.