Item Paneling and Piloting Posted on February 26, 2006 Peter Mangione (bio) describes the pilot work that should take place before an instrument is ready.

I don't care how experienced you are as a researcher. You have to build in and plan for a developmental process when you're creating an instrument. You need to build in that piloting. You get great ideas from colleagues; if you have enough funding, you can bring together a group of expert advisors and get that developmental framework. You can get it from the literature. You have your concept, but then you have to go try it out, and you ought to try it out in a small way. I think with funding you have to build in that pilot period, and I would build in at least two iterations. Go through it twice because the first time you're going to be fumbling around.

Even before you get to the pilot, one thing that we do in our work, which is a very important piece, is what we call item paneling where we are bringing together people who will be using the instrument and getting their input about our first draft of the instrument before we ever go out and pilot it. That changes the instrument tremendously, just that step alone, and it saves you a lot of trouble down the line. Then you go pilot it, and the advantage of a pilot is you're getting data. You actually see what the measures are yielding for you, and then you can analyze that small amount of data and start to see. Are you getting the trends you want? Are you getting the kind of information? Is it coming out as you expected? Or are there problems with the data? Is it hard to do this instrument reliably? Is it hard to distinguish one developmental level defined by a progress variable from another? Is there ambiguity between the levels? Are your observers or your assessors not able to make those fine distinctions that you thought you had all figured out?

And you find out no I'm not getting reliability with this small data set and be thankful you didn't do it with the big data set because you would have wasted a lot of money. Likewise, are the associations between the different items as you would expect, or is it all over the place? Is there no coherence in a statistical sense with what you're getting from your measure? Again, a pilot sample isn't going to be able to tell you all of that because you need a bigger sample to look at coherence, but it can tell you some things. It can point to some problems you may have when you go to that bigger sample, so I would build in those smaller, less expensive iterations as part of your project and then do your big project, and you have a better chance of avoiding those big problems later.

« Back to Article