Home / Topics / Collaboration / Working with Sites / Justice System / Lessons Learned from Conducting Randomized Trials in a Criminal Court Setting
Michael Rempel

Lessons Learned from Conducting Randomized Trials in a Criminal Court Setting

Posted on May 31, 2011

Michael Rempel (bio) of the Center for Court Innovation discusses lessons learned while assessing domestic violence treatment programs in criminal court settings.


Q: Can you describe the court-based trials you conducted?
A: We conducted trials in two locations, the Bronx, New York, and Rochester, New York, with the goal of assessing the effectiveness of court-mandated participation in batterer programs as well as the effectiveness of judicial monitoring in cases of domestic violence. We wanted to determine whether court-mandated attendance at a batterer program or judicial monitoring in conjunction with some type of program -- batterer, substance abuse, mental health -- could elicit real change in a defendant population. We hoped to address some of the shortcomings from which previous studies on the subject had suffered.
Q: What kinds of challenges did you face in conducting these trials?
A: The challenges were manifold, from upholding participants' due process rights without compromising our goals to overestimations of caseload and balancing the need to secure funding while simultaneously soliciting stakeholder buy-in.
Q: Can you provide a summary of the lessons you learned from conducting court-based research?
A: We came away with 10 distinct "lessons learned" from our experiences.

(1) Make sure you include all stakeholders at an early stage. For our Rochester trial, we ultimately did this by forming a Community Advisory Board (CAB). For our Bronx trial, we did not need a formal "board," because we had preexisting informal relationships with all of the relevant stakeholders, but we still needed to engage in a sizable amount of outreach. In short, stakeholder buy-in is as necessary to successful implementation as funding.

(2) The planning process can be lengthy. Time must be allotted both for the actual planning of the trial and scheduling meetings with stakeholders. In our experience, this will almost always take far longer than you expect. Part of the reason is purely logistic. Sometimes, not much "work" is actually taking place, but it still takes time just to schedule and hold the next planning meeting.

(3) Make sure a clear delineation exists between the intervention and the control group condition. We found that one particular aspect of the Rochester system created problems. Specifically, Rochester operated an "integrated" court that dealt with both domestic violence cases as well as other family-related cases. Therefore, study offenders who had been assigned to a no-monitoring condition might still appear in the same court for unrelated issues. Even though judges were directed to make no mention of the criminal domestic violence case when dealing with defendants' other family cases, it was difficult to say whether, on some level, simply having to appear in the same court and before the same judge might compromise trial fidelity.

(4) When working in a court setting, your trials must address legal due process.

(5) You may need to revise your study eligibility criteria to address stakeholder concerns. In our case, judges in the Bronx setting expressed concern that defendants pleading guilty to a misdemeanor charge rather than the less serious violation charge could receive a less onerous sentence. We therefore agreed to limit inclusion in the trial to only those defendants pleading guilty to a violation charge. Ostensibly, this would ensure that more violent offenders would receive the sentence -- a batter program and monitoring sentence -- that the district attorney deemed appropriate.

(6) Research on violent populations may require additional victim safeguards. At both study sites, victim advocacy groups conducted checks on victims in the months immediately following sentencing. In addition, the local victim advocacy groups conducted the research interviews with victims. Unfortunately, these efforts were largely ineffective because of the lack of reliable contact information for victims (at both sites, we underestimated the degree of transiency in the victim community) and the reluctance of victims to be interviewed.

(7) Approach caseload estimates with a healthy dose of skepticism. Whenever reliable case volume data is unavailable, assume that the volume will fall well below the estimates provided by court staff or others.

(8) Problems with study fidelity can occur at any time throughout implementation. Therefore, in addition to careful planning, ongoing monitoring of study fidelity is crucial.

(9) Accurate documentation of information in a reliable and accessible content management system is key, both for final data analysis and to catch problems. The latter became very apparent for us when our documentation of court appearance data in the Bronx enabled us to uncover the fact that the judicial hearing officer had not adhered to the study protocols. Consequently, we were able to adjust our analysis accordingly.

(10) Engage in early reviews of research results. Randomization, by its nature, results in the withholding of treatment from one group of people. Ethical considerations demand that researchers review data at an early stage to identify any significant treatment effects. This is especially important in studies where victim safety is potentially an issue.
Q: With so many challenges to completion and fidelity, did you ever wonder if the scope of the project was simply too large/overwhelming? What, ultimately, made it worth attempting, even with the multiple challenges?
A: Ultimately, the projects were worth attempting, because the research questions need to be answered. In our examples, batterer programs had morphed into the primary sanction of choice nationwide for domestic violence offenders whose cases were deemed legally inappropriate for jail. Yet, the research literature through the 1990s on the effectiveness of batterer programs was of remarkably poor quality, and the few strong studies that had been conducted yielded mixed findings. As for judicial monitoring, studies with other types of offenders (notably substance abusing offenders) suggested that monitoring might be effective, but a rigorous test of any kind had yet to be conducted with a domestic violence population. My colleagues and I are currently conducting three new trials on different topics in the justice system, all of which, to us, merit serious research attention. So obviously, we're willing to "do it again" after our initial experiences in the Bronx and Rochester, and we simply hope each time that we do better at following the above lessons -- and get lucky.

In addition, although the challenges to conducting randomized trials are unique, it doesn't mean that they are fewer than in other types of research. A well-executed randomized trial involves statistically identical groups of individuals at baseline, as a result of which any differences in their outcomes can be safely attributed to the intervention in question. In quasi-experiments, by contrast, where the samples may differ at baseline, a host of statistical adjustment strategies must be undertaken, and these strategies are not always easy to implement or, if the samples are severely biased at baseline, even accurate. So other types of studies have challenges of their own.



Adapted from the 2010 article "Lessons Learned from the Implementation of Two Randomized Trials in a Criminal Court Setting," by Melissa Labriola, Michael Rempel, and Amanda Cissner.


Please note that the feedback is viewed only by 4researchers staff and is not intended for communication with individual contributors.


Use the form below to submit feedback about this article. If you would like a response, please be sure to include your e-mail address.

More About "Justice System"

Challenges of and Tips for Conducting Randomized Trials in a Criminal Court Setting


Considering the Upstream Factors


More From Michael Rempel (bio)

Challenges of and Tips for Conducting Randomized Trials in a Criminal Court Setting