Home / Topics / Study Management / Organization and Planning / Mid-Study Changes / Designing School-based Interventions

Designing School-based Interventions

Posted on May 17, 2007

Read practical advice about designing and implementing a violence prevention program in schools by Lisa Jaycox (bio), Daniel F. McCaffrey (bio), Beverly Weidmer Ocampo (bio), Gene A. Shelley (bio), Susan M. Blake (bio), Joan Kub (bio), and Donna J. Peterson (bio).

 

Q: What are some of the difficulties that you encountered while conducting school-based evaluations of interventions?
A: We evaluated three intervention programs that were developed to prevent intimate partner violence (IPV) among minority populations. While the three school-based interventions varied in design and implementation, they all faced challenges associated with:
  1. constraints related to research design
  2. recruitment of schools and individuals
  3. implementation issues
Q: What research design difficulties did you encounter and how did you respond to them?
A: Randomization was the biggest challenge. For one of the programs, Respect Me (Johns Hopkins University), randomization at the individual student level was not possible because the program was based on a school-wide intervention. In addition, stakeholders felt that randomizing at the school level would not be accepted by the community. The project therefore decided to match schools on particular demographics, choosing two intervention and two comparison schools. The comparison schools were given the intervention at a later time.

In another project called Break the Cycle (a non-profit organization in Los Angeles), the original plan to randomize at the school level had to be changed. At these year-round schools, children were on one of three "tracks," with very little overlap of courses and sports, so that children on different tracks had very little contact with one another. It was decided then to randomize at the track level, across schools.

For the other project, Promoting Healthy Relationships (University of Arizona), neither randomization nor matching was feasible. The program was implemented in a rural area, with only one middle school and one high school. The total number of children in the schools was small as well, which can allow transfer of knowledge between intervention and control groups. The evaluation was therefore designed as a pre-post study.
Q: What issues arose with participation and recruitment?
A: The following issues arose:

Consent procedures
Consent procedures should be considered carefully when conducting school-based research on sensitive topics. Passive consent (a consent letter that is signed only if the parent does NOT want her child in the study) is usually not recommended when studying topics such as IPV or drug use with teens. On the other hand, active consent can result in fewer participants and a biased sample. One way to approach this problem is to increase time and money spent on recruitment efforts, such as having the research team be responsible for distributing and collecting consents.

Also, the consent form itself can be the cause of recruitment problems. IRB's have requirements relating to language used on consents that can be off-putting. For example, the term "Principal Investigator"; had negative connotations for members of the rural Arizona tribal community, so the term was changed to "Principal Researcher." In addition, consent forms should be translated as needed to ensure a more representative sample.

Participant burden
There is so much pressure put on schools to meet testing standards these days, that any project which takes them "off task" may not be welcomed. The schools that we were working in were faced with budget cuts, teacher layoffs, and increased workloads. Therefore, it was essential to the success of the projects that our research staff take on some of the burden. For example, in the Break the Cycle project, the school staff were supposed to prepare and deliver the call slips that allowed students to leave their classrooms for assessments. This was very time-consuming, and the school staff was not completely reliable in delivering the slips, so the research staff took on the task.

Originally, all three programs asked teachers to be responsible for talking to their students about the project, sending home consent packets, and following up. However, this turned out to be too much of a burden on teachers who were already overloaded, so the research teams decided to take on those tasks. The research staff also provided incentives such as ice cream certificates to classrooms who had at least an 80% return rate (allowing or denying permission).

Relationship between principal and teachers
We also needed to take into account the administrator-teacher relationship, which can vary among schools. Some administrators talk with teachers before agreeing to participate, while others may agree without consulting their staff, leaving teachers feeling forced to participate. These teachers may then pass a resentful attitude towards the project on to their students, which can negatively affect recruitment and retention. It's important, therefore, to gain support from teachers and staff as well as from the principal before beginning the project.
Q: How did you handle confidentiality issues when talking to teens about drug use and sexual behavior?
A: There is a delicate balance between what's best from a research standpoint, and concerns about privacy. Anonymous surveys may yield more truthful responses (although not necessarily), but evaluation studies often need to link individual responses over time. Keeping students' answers private can be difficult, so researchers need to take precautions such as asking teachers to leave the room before administering the surveys.

Another issue is reporting of child abuse. State laws differ, but mandatory reporting of suspected abuse can come up against the assurance of confidentiality. Each of the programs handled these challenges differently. Because Arizona's state law requires reporting of all instances of dating violence, the Break the Cycle team felt that teens would be more forthcoming if this portion of the survey was anonymous. The team provided a list of resources about dating violence, and the school counselor was present on survey days in case students had any concerns. On the other hand, the Respect Me project used individual identifiers for their surveys, but stressed that all discussion would remain confidential. They also told their students that the research team would need to tell the authorities if a student reported abuse.
Q: What difficulties came up once the programs began implementation?
A: Scheduling a time and place to conduct the program can be extremely frustrating, particularly in schools with 4 or 5 tracks. The research team needs to be able to accommodate shortened days, earthquake drills, special school activities, state testing, etc. The Arizona project had to postpone the program until after basketball season, for example. A change in course offering in the same project required a quick reshuffling of the program setting. The violence prevention program had originally been designed to take place during 9th grade physical education, but because of poor academic performance, the school decided to cut the 9th grade PE classes. Research staff then had to find an alternate setting and time for the program.

The school-based research team should understand that, just as each school has a different culture, there are many cultures that exist within each school. Different groups (teachers, students in general, ethnic student groups) may require distinct methods of forming a partnership.

A critical element for any research team working in schools is flexibility; research staff need to be able to adapt to changes in a stressful environment while maintaining the integrity of the research. The team should expect modifications to the implementation, and be able to make immediate changes to protocol as needed. Some of these changes may be based on discussions over several meetings, while other may need to be made on the spot. When faced with a need to change protocol, the research team should consider the following questions:
  • Will it result in any ethical or confidentiality violations?
  • Is it feasible?
  • Will it affect the ability to discern differences between intervention and comparison groups?
  • Will there be spillover from intervention to comparison group?
  • Will the quality of data be affected?
It also helps to have good background knowledge of how statistics can be used to deal with issues such as missing data, attrition, and power.
Q: Any final advice for a researcher going into partnerships with schools?
A: Here are a few lessons that we learned that can help you to navigate the bumpy road of research in schools:
  • Understand and be sensitive to the school's cultures
  • Get everyone on board, including the district administrator, principal, teachers, and other staff who will be involved at any point during the study
  • Keep in contact with school staff throughout the project
  • Minimize burden on staff
  • Provide incentives for returning consents, as well as for participating
  • Plan to spend a lot of time on consent forms (including translation if needed, distribution, and reminders)
  • Allow flexibility in implementation and allow time to deal with last minute changes

 

 


Feedback

Please note that the feedback is viewed only by 4researchers staff and is not intended for communication with individual contributors.

 

Use the form below to submit feedback about this article. If you would like a response, please be sure to include your e-mail address.


More About "Mid-Study Changes"

3:04

A Statistician's Role in Interim Analysis

Changing Consent Procedures in Mid-Project

 

More From Lisa Jaycox (bio)

0:00

Keeping Things Confidential