Home / Topics / Research Design / Observational / Analyzing Observational Data

Analyzing Observational Data

Posted on August 13, 2010

Bradley Stein (bio) talks about the 'difference in difference' technique to measure treatment effects in non-randomized trials.


Times in a real world you can't randomize, and if you can't randomize, by definition you're living in a world of observational data. Right? You can influence it somehow, but in the end it's not a random sample. You're using observational data. And so the question then that comes up is using observational data, how do we know that something is having an impact? And I think it's just important for people to realize that there are statistical techniques, many of which have actually been developed in the world of economics, because in the world of economics no one is going to randomize — this country is going to do this with interest rates and another country is going to do this with interest rates. It doesn't work like that. You can't really randomize in the world of economics.

So economists are sort of faced with this problem, is what is the impact of changing a policy when we can't randomize?

So let me explain difference in difference, which is actually a pretty easy one.
Let's say we have two states or two places that are essentially the same, and you have data from what was going on, and then at one point one of the places implements a new policy and the other one doesn't, and then you have data from what happens after that. What you're able to do is look and see did the place that implemented the policy, was there a change from before to after. Now if we were doing just that by itself, maybe you would see a change from before to after, but maybe it's caused by something that has nothing to do with the intervention of the policy.

Well, the advantage of having another state where the policy wasn't implemented is if there's that type of secular change, which is changing everything, then you should be seeing those differences in both of those places. If you're seeing it in the one place and not the other place, then you can walk away with a good deal more confidence that the change that you're seeing may very well be related to the policy. And again, I think most people would think that the gold standard is, and probably will continue to be, randomized controlled trials, but this is an example of how you can use observational data and still walk away with a pretty high degree of confidence that that intervention, that policy change is what really accomplished this difference.

These are statistical analyses that have significance. There are p-values. They're oftentimes slightly different modifications of the types of regression analyses that people are very commonly used to seeing. It's the same type of regression analyses, just there are some different variables that enter the model and they use slightly different approaches, but, in essence, if you look at the computer output it's going to look very, very similar to what you'd see coming out on the regression.

Viewing Preferences




Excerpted from interview with researcher in May 2010.


Please note that the feedback is viewed only by 4researchers staff and is not intended for communication with individual contributors.


Use the form below to submit feedback about this article. If you would like a response, please be sure to include your e-mail address.


More From Bradley Stein (bio)


Developing a Robust School Intervention for Traumatized Children


Early Revisions Improve Interventions

Show All...