Training and Monitoring Clinical Raters Posted on December 3, 2007 Noreen Reilly-Harrington (bio) considers some ways of addressing assessment reliability.

One other area that I’ve worked extensively in is the idea of reliability in assessments. And this is an extremely important area for anyone who is designing a clinical trial. You really need to think through carefully not only what assessments you’re choosing, but how are you going to train your clinical raters? Your clinical raters either make or break a study. And one of the areas that I worked with STEP-BD, and now the bipolar trials network, I’m directing the training and the assessments for that project.

We’ve put an enormous amount of time into developing training to teach raters specific conventions for each clinical rating scale. And to carefully monitor their performance over the course of the trial. It’s very important to not only train your raters at the beginning of a large scale study, but to provide ongoing adherence and monitoring of those raters, because certainly rater drift occurs over the course of a study. This is a very important area to be cautious about.

For the Bipolar Trials Network I have developed a set of training DVDs that each rater is required to watch. So we provide an overview of the rating scale, an overview of the conventions, and agreement is really our goal. Raters who have been trained on these scales for prior studies might say I wasn’t taught to rate it this way in a previous study, but agreement is always the goal. So that’s very important when you’re designing a study to look at agreement and consensus for your trial.

So each rater is required to watch these overview training presentations, and then each rater is required to view a videotape of a patient interview and reach a certain acceptable standard of reliability with that tape. If raters do not succeed on the first try, they do have a second try to certify. And then if they do not certify on the second set of tapes, we really need to call into question whether or not this rater is adequate for the study.

« Back to Article