Lessons Learned from Conducting Randomized Trials in a Criminal Court Setting
Posted on May 31, 2011
Michael Rempel (bio) of the Center for Court Innovation discusses lessons learned while assessing domestic violence treatment programs in criminal court settings.
(1) Make sure you include all stakeholders at an early stage. For our Rochester trial, we ultimately did this by forming a Community Advisory Board (CAB). For our Bronx trial, we did not need a formal "board," because we had preexisting informal relationships with all of the relevant stakeholders, but we still needed to engage in a sizable amount of outreach. In short, stakeholder buy-in is as necessary to successful implementation as funding.
(2) The planning process can be lengthy. Time must be allotted both for the actual planning of the trial and scheduling meetings with stakeholders. In our experience, this will almost always take far longer than you expect. Part of the reason is purely logistic. Sometimes, not much "work" is actually taking place, but it still takes time just to schedule and hold the next planning meeting.
(3) Make sure a clear delineation exists between the intervention and the control group condition. We found that one particular aspect of the Rochester system created problems. Specifically, Rochester operated an "integrated" court that dealt with both domestic violence cases as well as other family-related cases. Therefore, study offenders who had been assigned to a no-monitoring condition might still appear in the same court for unrelated issues. Even though judges were directed to make no mention of the criminal domestic violence case when dealing with defendants' other family cases, it was difficult to say whether, on some level, simply having to appear in the same court and before the same judge might compromise trial fidelity.
(4) When working in a court setting, your trials must address legal due process.
(5) You may need to revise your study eligibility criteria to address stakeholder concerns. In our case, judges in the Bronx setting expressed concern that defendants pleading guilty to a misdemeanor charge rather than the less serious violation charge could receive a less onerous sentence. We therefore agreed to limit inclusion in the trial to only those defendants pleading guilty to a violation charge. Ostensibly, this would ensure that more violent offenders would receive the sentence -- a batter program and monitoring sentence -- that the district attorney deemed appropriate.
(6) Research on violent populations may require additional victim safeguards. At both study sites, victim advocacy groups conducted checks on victims in the months immediately following sentencing. In addition, the local victim advocacy groups conducted the research interviews with victims. Unfortunately, these efforts were largely ineffective because of the lack of reliable contact information for victims (at both sites, we underestimated the degree of transiency in the victim community) and the reluctance of victims to be interviewed.
(7) Approach caseload estimates with a healthy dose of skepticism. Whenever reliable case volume data is unavailable, assume that the volume will fall well below the estimates provided by court staff or others.
(8) Problems with study fidelity can occur at any time throughout implementation. Therefore, in addition to careful planning, ongoing monitoring of study fidelity is crucial.
(9) Accurate documentation of information in a reliable and accessible content management system is key, both for final data analysis and to catch problems. The latter became very apparent for us when our documentation of court appearance data in the Bronx enabled us to uncover the fact that the judicial hearing officer had not adhered to the study protocols. Consequently, we were able to adjust our analysis accordingly.
(10) Engage in early reviews of research results. Randomization, by its nature, results in the withholding of treatment from one group of people. Ethical considerations demand that researchers review data at an early stage to identify any significant treatment effects. This is especially important in studies where victim safety is potentially an issue.
In addition, although the challenges to conducting randomized trials are unique, it doesn't mean that they are fewer than in other types of research. A well-executed randomized trial involves statistically identical groups of individuals at baseline, as a result of which any differences in their outcomes can be safely attributed to the intervention in question. In quasi-experiments, by contrast, where the samples may differ at baseline, a host of statistical adjustment strategies must be undertaken, and these strategies are not always easy to implement or, if the samples are severely biased at baseline, even accurate. So other types of studies have challenges of their own.
Adapted from the 2010 article "Lessons Learned from the Implementation of Two Randomized Trials in a Criminal Court Setting," by Melissa Labriola, Michael Rempel, and Amanda Cissner.
Please note that the feedback is viewed only by 4researchers staff and is not intended for communication with individual contributors.
Use the form below to submit feedback about this article. If you would like a response, please be sure to include your e-mail address.