Running a difference in differences test

[UnderOver]

I am trying to run a difference in differences test and I think my coding should be working but to the eye the results don't appear to be correct. I am talking about a difference in means of 1,000 compared to a difference in means of 100, which, again, seems it should be different. (It may be totally correct and the data is just got some crazy stuff going on, but I am a bit of a novice with this code and was hoping some more experienced people could give it a once over!)

So I have data with 8 different conditions and am running with some covariates. I am testing here that condition 7-8 > 3-4. So I run a basic ANCOVA with contrast coding:

UNIANOVA DV1 BY Condtion WITH CV1 CV2
  /METHOD = SSTYPE(3)
/LMATRIX 'DifDif' Condtion 0 0 1 -1 0 0 1 -1.

Any help is appreciated. Thanks in advance!

[Bruce Weaver]

I think you've got your signs the wrong way around on one pair of coefficients.  It helps if you include explicitly the contrasts for 7-8 and 3-4, and then compute the difference between them.  

UNIANOVA DV1 BY Condtion WITH CV1 CV2
  /METHOD = SSTYPE(3)
  /LMATRIX '7-8'    Condtion 0 0  0  0 0 0 1 -1
  /LMATRIX '3-4'    Condtion 0 0  1 -1 0 0 0  0
  /LMATRIX 'DifDif' Condtion 0 0 -1  1 0 0 1 -1.

HTH.

UnderOver wrote

I am trying to run a difference in differences test and I think my coding should be working but to the eye the results don't appear to be correct. I am talking about a difference in means of 1,000 compared to a difference in means of 100, which, again, seems it should be different. (It may be totally correct and the data is just got some crazy stuff going on, but I am a bit of a novice with this code and was hoping some more experienced people could give it a once over!)

So I have data with 8 different conditions and am running with some covariates. I am testing here that condition 7-8 > 3-4. So I run a basic ANCOVA with contrast coding:

UNIANOVA DV1 BY Condtion WITH CV1 CV2
  /METHOD = SSTYPE(3)
/LMATRIX 'DifDif' Condtion 0 0 1 -1 0 0 1 -1.

Any help is appreciated. Thanks in advance!

[Art Kendall]

Differences of differences often means crossed IVs.

What are your 8 conditions?  Are they really 8 values of one construct or are they, for example, 3 dichotomous variables? Are all of the values between cases or are some of the values repeats?




The substantive meaning of your variables and their values could well be informative to people who might help you.

What is DV1? CV1? CV2?

[UnderOver]

Thanks for the response

As for what these variables are the 8 conditions stem from 3 sets of dichotomous variables.  

As for the DV it is a continuous variable that ranges from 0 to about 10,000 (dollars for investment). One CV is a categorical variable(wealth) and the other is continuous (Age).

[UnderOver]

Thank you for the response.

Yes you were right I had those backwards, but they are still giving me an odd response. Partly what prompted me to ask about this was that my first contrast (7-8) is shows a p-value >.05 while the second contrast is <.7 yet the difference in difference test is <.2, which just doesn't seem correct, although I realize is totally possible.  

[Art Kendall]

Then it appears that you want a 3 way ANCOVA.
Whichever difference of differences you are interested in would be one of the interactions.

It seems odd that wealth would be categorical.  How many values does it have?  Is it fully crossed with the three dichotomies? That would make a 4way ancova.

How many cases do you have?  Are the cell sizes balanced?

[UnderOver]

Sorry I meant that it is ordinal, was going fast and mistakes abound. It has 4 values to match some regulatory items.

I have run the 3-way ANCOVA and there are significant interactions with the IVs, so when digging into what is actual significant I have looked at the individual contrasts some are some aren't. Instead of having some no result hypotheses and tests I wanted to use this difference in difference test was a way to utilize the no result contrasts as part of the story showing where the contrast is significant vs. where it is not. Hopefully making for a more compelling study then a no results test would.  

[Art Kendall]

An interaction term is a difference of differences.  Did you hypothesize a particular difference of differences?

When you do your visualization do you see crossed lines?

[UnderOver]

Yeah I do, and that is what I thought so when the interaction came back significant I would have thought that the individual test should as well.

[Art Kendall]

The test for the interaction effect is the test stat that you want, i.e., the "lines" are significantly different from parallel.  The visualization should show  that the lines are non-parallel. Perhaps they even cross.