Re: Stats course geared towards Program Evaluation

Posted by Art Kendall on Mar 02, 2015; 6:27pm
URL: http://spssx-discussion.165.s1.nabble.com/Stats-course-geared-towards-Program-Evaluation-tp5728871p5728876.html

It is true that there is no single design for program evaluation.  In earlier years it was called "applied social science".  In fact if you look at the Handbook of Evaluation Research, it was sponsored by SPSSI (Society for Psychological Study of Social Issues).

The whole panoply of social science methods can be used in evaluating a vast array of programs.
The series of books authored by Cook and Campbell and later Shadish can be used as a framework to conceptualize the designs that can be used in program evaluation.  

In an ideal world we would be able to randomly assign individuals or larger aggregation of individuals to treatments.  When we cannot (which is very often) we need to supplement the rhetoric used in our statistical arguments with further information to rule out plausible rival hypotheses for observed differences.

In one sense basic research we are looking for difference in summary observations relevant to theory and in program evaluation we are looking for difference in summary observations relevant to practice and or policy/

Even in true experiments, our reasoning can be strengthened by some form of repeated measures.

Wrt teaching

One point that some miss is that the main hypotheses are interaction hypotheses. The simplest example of this kind of of interaction is the interaction of a variables related to time (repeats) with a variable representing (hopefully randomly assigned) treatment.  A difference in change is a simple example.  We hope that a treated groups shows a larger improvement than a control (or at least comparison group).  In my experience - YMMV - it is helpful for clients to sketch a graph of what they would like to see.  The idealized sketch with the DV on the Y axis and an X axis with 2 values pre and post, and two line segments connecting pre and   post separately would show a definite non-parallelism between the line segments.

In the example syntax below, the same SD is assumed for all 4 means, ideally these would be informed guesses from prior research.  For the treatment group, the "effect size that would make a difference" is 5 points on the DV scale.

Using such a demonstration a few times can help clients understand what they hope for is an interaction, that they need to consider prior research, and that they need to consider what size difference in a DV would be meaningful for theory/practice/policy.

set seed 20101802.
input program.
   loop #i = 1 to 25.
      compute group =1.
      compute DV_pre = rv.normal(22,5).
      compute DV_post = rv.normal(27,5).
      end case.
   end loop.
   loop #i = 1 to 25.
      compute group =2.
      compute DV_pre = rv.normal(22,5).
      compute DV_post= rv.normal(22,5).
      end case.
   end loop.
   end file.
end input program.
value labels group 1 'treatment' 2 'control'.
execute.
GLM DV_pre DV_post BY group
  /WSFACTOR=time 2 Polynomial
  /METHOD=SSTYPE(3)
  /PLOT=PROFILE(time*group)
  /CRITERIA=ALPHA(.05)
  /WSDESIGN=time
  /DESIGN=group.