I also think that there is increasingly embedded within the kind of causal, rigorous evaluation that Kris mentioned other approaches simultaneously. Examples of that are implementation research done at the same time as the causal evaluation research where the question is different. It's not 'Did children differ because of an intervention?' or 'Did adolescents differ in their developmental outcomes?' but 'Did we do the program in the way we intended? Did we hit barriers to implementation or facilitators? Were there important facilitators? Did we see important local variations?'

I think we've heard wonderful examples about how that component of research can help from this meeting. An example is the early Head Start impact evaluation, which is this kind of rigorous random assignment evaluation, where they also made judgments about how thoroughly the program model was implemented. They actually found differences in the impacts on children according to whether they were in fully implemented programs or not, so this is very important to do side-by-side.

Another example from this meeting of a different strategy embedded within the causal evaluation kind of research is ethnography. I think we're in a time period where there has been an enormous progress made in the inter-relationships between qualitative and quantitative research. Where these used to be opposed, and the people doing this kind of research sometimes didn't even talk to each other or read each other's work. All of a sudden, now we're having a set of studies that are random assignment experimental studies with embedded qualitative research. Even more than that, unbelievable linkages and bridges between the two kinds of work.

Examples are in the New Hope evaluation where the qualitative research was done based on quantitative sampling approaches, so they randomly selected the families they did the qualitative research on. That's Tom Wisener and the New Hope team. That's really path breaking. You can speak to the larger sample. You're not sampling your ethnography based on rare cases. You can generalize to some extent, not statistically, but you can say, 'This is a reasonable set of examples of what we're seeing out there.'

Other examples include people who had no idea what the meaningful subgroups should be for their quantitative analysis, and they went to the ethnography to inform them. 'Ah, this is the set of subgroups that should be taken.' Of course, ethnography is also very important for just putting a face on what you're seeing in the quantitative data.

« Back to Article