Self-Reported Quality Measures Don't Add Up, Study Says

Data from The Leapfrog Group's doesn't match Medicare data and suggests a lack of reliability in self-reported data, researchers say. Leapfrog says it "goes to extreme lengths" to verify survey data.

In the search for hospital quality measures that matter, all kinds of rating systems are coming under scrutiny, including those used by the Leapfrog Group, a nonprofit advocate of quality and safety in healthcare.

A study from the University of Michigan concludes that the group's Safe Practice Score (SPS) produces different results than those used by Medicare's Hospital Compare to track common complications and readmissions.

The Leapfrog findings "skew toward positive self-report[ing]," according to the study, which appeared in the journal Medical Care and was entitled "Dissecting Leapfrog."

Leapfrog Group Names 'Top Hospitals'

The researchers suggest that some of the differences could stem from "selection effects." Hospitals may be more likely to share information if they generally earn high scores, they write.

"Alternatively, given that hospitals have a clear incentive to score themselves highly, participating hospitals may inflate their SPS reports, resulting in the skewed distributions and undermining the measures' predictive value," they write.

And they note that Leapfrog makes efforts to validate data. It requires "a letter of affirmation" and flags erroneous or misleading reports.

Leapfrog Responds
In a written response to the paper, Leapfrog notes that, while it is possible to "game any ratings system, we find that hospitals attempting this with the Leapfrog Hospital Safety Grade are generally frustrated by the result."

The organization's safety grade is based on 30 measures, "meaning that there are thousands of potential statistical outcomes that cannot be predicted with certainty."

In terms of the reliability of voluntarily reported data, the Leapfrog statement notes that the organization "goes to extreme lengths" to verify survey data. This includes on-site verification at randomly selected hospitals, and independent review of each survey response.

Leapfrog also suggests that the disparity between its own data and that of Medicare could be linked to a difference in data reporting periods or to a difference between Medicare data and Leapfrog reporting at "individual bricks-and-mortar facility level."

Leapfrog notes that there are different ways to measure safe care and the organization doesn't expect its findings to "correlate 100%" with other scores.