Determining Clinical Significance Posted on January 15, 2008 Howard Abikoff (bio) describes a method for ascertaining whether a statistically significant difference is also clinically meaningful.

The challenge, of course, is how do we know if it’s clinically significant? How do we determine that? Well, one way we do it, of course, is by using measures that have been normed on a large sample of individuals, presumably many of whom are typical individuals, so that we can get some sense of what the normative mean is on a particular scale and what the spread of scores is around that mean.

It’s very interesting because when you use ecologically valid measures or attempt to develop them and then use them, it’s often very difficult to develop a normative database for those measures because typically for a normative database, you’re going to need a very large sample. And that’s not easy to acquire when you have, for example, observational methods or analog tasks that you’re developing in a laboratory.

What it does mean is that hopefully you can still obtain information from a normative sample, or from normative or from typical individuals that can be used to help you judge whether or not the data you’ve collected from your clinical sample on these, quote, “ecologically valid measures" are representative of how typical individuals perform on that measure and in that setting. In fact, we used that approach with our classroom observation methodology.

Briefly, what we did was when we went into the classrooms and observed our study participants, we would of course collect baseline data on them and then go in and collect data on them during and at the end of treatment. And if you do that for your experimental group and your control group, the comparisons will yield information about statically significant differences. That’s great, but we wouldn’t know whether or not in fact the change is clinically significant.

Well we also collected data in each classroom on other children in that classroom who were identified by the teacher as typical youngsters. So each time that we went in and observed the study patient, we also observed another child in the classroom, the comparison, and collected data on that child on the same measures at the same time, at base line and over the course of treatment.

That gave us a standard against which to compare change in the study population and to see whether or not over time how well they changed and how much they changed to begin to approximate the behavior of typical children in their own classroom.

So we’ve controlled for setting effects, for context, and also gave us a comparison for each child, and I think that’s a very powerful way of being able to talk about clinical significance in the context of developing and using ecologically valid measures for which there’s not a large normative database.

« Back to Article