Mike makes excellent points to which I’ll add that program evaluation looks frequently at program process (activities, outputs) which are not outcomes to the
persons, organizations, and/or communities involved. Statistical analysis of process usually differs substantially from the statistical analysis of outcomes.
B
Brian Dates, M.A.
Director of Evaluation and Research | Evaluation & Research | Southwest Counseling Solutions
Southwest
Solutions
1700 Waterman, Detroit, MI 48209
313-841-8900 (x7442) office | 313-849-2702 fax
[hidden email] |
www.swsol.org
From: Mike Palij [mailto:[hidden email]]
Sent: Monday, March 02, 2015 11:12 AM
To: Dates, Brian; [hidden email]
Cc: Michael Palij
Subject: Re: Stats course geared towards Program Evaluation
Just to add a little to what Brian Dates provides below:
there is no single design for program evaluation,
consequently, as implied below, a large variety of different
statistical analyses can be conducted depending upon
(a) the design of program or the logic model assumed,
(b) how program participants were assigned to different
programs components (e.g., random assignment, is there
an appropriate "control" group that was not in the program,
etc.), and what data was collected at different times of a
participant's involvement in the program (NOTE: I am implicitly
assuming the unit of analysis as being a person/dyad/family
but the unit can larger, such as schools, geographical regions,
social groups, etc.). An analysis can be as simple an independent
groups t-test on some outcome measure or a process model with
empirical indicators (a) prior to program involvement,
(b) at specific times in the program, and (c) at the immediate exit
from the program as well as longer term outcome measures.
The simplest way to think of the analysis of a program is
that it conforms to an experimental/quasi-experimental design
but, as examination of a textbook on program evaluation
(e.g., Rossi, Lipsey, & Freeman "Evaluation") will show this an
oversimplification which may lead one to conduct analyses
that are essentially meaningless. Cost-benefit analysis, degree
of program participant and program administers "investment
in the program" (i.e., degree to which they believe in the
program and will work to achieve program goals), and other
issues need to be taken into account. Positive outcomes
may not be due to the program nor may negative results
indicate program failure (especially if the program was not
properly implemented or was sabotaged by some "stakeholder").
It is also useful to keep in mind that some programs start out
with a well-define design/logic model which would suggest how
to analyze it while other programs developed piecemeal over
time, with different logic models at different times. Which
time/model does one want to analyze?
There is a lot to chew on in program evaluation.
-Mike Palij
New York University