Say it Ain't So, Joe by Paul Levy

I heard a great presentation this morning by Joe Newhouse, from the Department of Health Policy and Management at Harvard Medical School. There was one point that he made that really caught my attention. It was a cite to a 2004 article in the Journal of the American Medical Association (Dimick, et al, JAMA 2004; 292: 849) that presented the issue of how many cases you would need to collect of a certain clinical procedure to be able to make a determination that a given hospital's mortality for that procedure was twice the national average. It turns out that only for CABGs (coronary artery bypass grafts) are there enough cases performed to have statistical confidence that a hospital has that poor a record compared to the national average. For other procedures (hip replacements, abdominal aortic aneurysm repairs, pediatric heart surgery, and the like) there are just not enough cases done to make this assessment. (By the way, if you just want to know if a hospital is say, 20%, worse on relative mortality, you need even a bigger sample size.)

I have copied the basic chart above. Sorry, but I couldn't nab the whole slide. The vertical axis is "Observed 3 year hospital case loads", or the number of cases performed over three years. The horizontal access is "Operative mortality rates". The line curving down through the graph shows the frontier at which statistical significance can be determined. As you see, only CABGs are above the line.

And, as Joe pointed out, this chart is based on three years of data for each hospital. With only a year's worth from each hospital, you surely don't have enough cases to draw statistically interesting conclusions about relative mortality. And remember, too, that this is hospital-wide data. No one doctor does enough cases to cross the statistical threshold.

So, this would suggest that publication of hospital mortality rates for many procedures would not be helpful to consumers or to referring physicians.

Meanwhile, though, you might recall a post I wrote on surgical results as calculated by the American College of Surgeons in their NSQIP project. This program produces an accurate calculation of a hospital's actual versus expected outcomes for a variety of surgical procedures. Unfortunately, the ACS does not permit these data to be made public.

Where does this leave us?  Well, as I noted in a Business Week article, the main value of transparency is not necessarily to enable easier consumer choice or to give a hospital a competitive edge. It is to provide creative tension within hospitals so that they hold themselves accountable. This accountability is what will drive doctors, nurses, and administrators to seek constant improvements in the quality and safety of patient care. So, even if we can't compare hospital to hospital on several types of surgical procedures, we can still commend hospitals that publish their results as a sign that they are serious about self-improvement.