Program evaluation question

[Salbod]

This is a program evaluation question.

I have dataset made up incidents at a school between 2006 and 2013. There are just two variables of interest: date of incident e.g., 12/01/2011) and type of incident (e.g., minor injury, physical aggression to staff). Students stay at the school one to three years. The number of incidents during their stay at the school can vary from 1 to 173. In 2010 a program was instituted at the school to reduce aggressiveness. I’ve recoded type of incident into a new variable named  aggress (0 = No, 1 = Yes). The problem I see is that no students have the same number of dates (incidents).  How can I evaluate the effectiveness of the program?

Any ideas are more than welcomed.

TIA

Stephen Salbod, Pace University, NYC

[Maguin, Eugene]

There could be a sample problem and there is a statistics problem. The likely sample problem is that some kids' enrollment spanned the program start date, e.g., they were there the year before and the year or two after program start. A clean sample design would drop those kids since they have pre and post data while the rest of the kids have either pre or post data. That's the sample.

Crude methods: Divide the number of incidents by the length of enrollment to get incidents per time period and compare the two groups. That will give you a very rough picture of what the story might be.
Better: To start, I'd guess that the data might conform to a poisson distribution, which I think can be checked in the one of the spss test collections that I seldom need to use. If the data, both pre and post do conform to a poisson distribution, I think but I'm not sure that genlin can be used to analyze the data. And, if not genlin, then genlinmixed. However, you might find that there is an excess of zeros--too many kids not doing anything! There's a number of different models that might be relevant although the one I first think of is a zero-inflated poisson. I don't know whether or not that is available in genlinmixed. I know there are others on the list that have a much broader range and depth of experience with these sorts of data than I do and I hope they will give correct advice where I have failed.

Gene Maguin


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Salbod
Sent: Tuesday, April 09, 2013 1:13 PM
To: [hidden email]
Subject: Program evaluation question

This is a program evaluation question.

I have dataset made up incidents at a school between 2006 and 2013. There are just two variables of interest: date of incident e.g., 12/01/2011) and type of incident (e.g., minor injury, physical aggression to staff).
Students stay at the school one to three years. The number of incidents during their stay at the school can vary from 1 to 173. In 2010 a program was instituted at the school to reduce aggressiveness. I’ve recoded type of incident into a new variable named  aggress (0 = No, 1 = Yes). The problem I see is that no students have the same number of dates (incidents).  How can I evaluate the effectiveness of the program?

Any ideas are more than welcomed.

TIA

Stephen Salbod, Pace University, NYC




--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Program-evaluation-question-tp5719367.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[ViAnn Beadle]

How about a time series approach? The hypothesis could be that a program reduced the rate of incidents or the rate of aggressive incidents. The intervention point would be the introduction of the program (ala Campbell).

-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Maguin, Eugene
Sent: Tuesday, April 09, 2013 3:39 PM
To: [hidden email]
Subject: Re: Program evaluation question

There could be a sample problem and there is a statistics problem. The likely sample problem is that some kids' enrollment spanned the program start date, e.g., they were there the year before and the year or two after program start. A clean sample design would drop those kids since they have pre and post data while the rest of the kids have either pre or post data. That's the sample.

Crude methods: Divide the number of incidents by the length of enrollment to get incidents per time period and compare the two groups. That will give you a very rough picture of what the story might be.
Better: To start, I'd guess that the data might conform to a poisson distribution, which I think can be checked in the one of the spss test collections that I seldom need to use. If the data, both pre and post do conform to a poisson distribution, I think but I'm not sure that genlin can be used to analyze the data. And, if not genlin, then genlinmixed. However, you might find that there is an excess of zeros--too many kids not doing anything! There's a number of different models that might be relevant although the one I first think of is a zero-inflated poisson. I don't know whether or not that is available in genlinmixed. I know there are others on the list that have a much broader range and depth of experience with these sorts of data than I do and I hope they will give correct advice where I have failed.

Gene Maguin


-----Original Message-----
From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Salbod
Sent: Tuesday, April 09, 2013 1:13 PM
To: [hidden email]
Subject: Program evaluation question

This is a program evaluation question.

I have dataset made up incidents at a school between 2006 and 2013. There are just two variables of interest: date of incident e.g., 12/01/2011) and type of incident (e.g., minor injury, physical aggression to staff).
Students stay at the school one to three years. The number of incidents during their stay at the school can vary from 1 to 173. In 2010 a program was instituted at the school to reduce aggressiveness. I’ve recoded type of incident into a new variable named  aggress (0 = No, 1 = Yes). The problem I see is that no students have the same number of dates (incidents).  How can I evaluate the effectiveness of the program?

Any ideas are more than welcomed.

TIA

Stephen Salbod, Pace University, NYC




--
View this message in context: http://spssx-discussion.1045642.n5.nabble.com/Program-evaluation-question-tp5719367.html
Sent from the SPSSX Discussion mailing list archive at Nabble.com.

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Salbod]

Thank you Rich Ulrick, ViAnn Beadle, & Eugene Maguin, your questions were priceless. The data was collapsed into one year before the intervention (pretest) and one year after the intervention (posttest). I'm now busy converting a long file (id, incident date, type of incident) into a long file in order to get a time perspective on the data (number of incidents, range in terms of days, etc) so I can eliminate those students who were only there for the pretest or posttest period.  

[Rich Ulrich]

It is a narrower question (and answer) if you look only at  those
people who were there both pre and post.

If it was "normal" before intervention that students had more offenses
when younger, the limited file might give the appearance of an effective
treatment, no matter what.  If it was "normal" that the older students
were cited for more offenses, then the effect of treatment could be
concealed by the other change.

It should be useful to look at effect on particular students, but this
approach seems more problematic than ignoring individuals altogether.

--
Rich Ulrich

> Date: Thu, 11 Apr 2013 09:32:01 -0700

> From: [hidden email]
> Subject: Re: Program evaluation question
> To: [hidden email]
>
> Thank you Rich Ulrick, ViAnn Beadle, & Eugene Maguin, your questions were
> priceless. The data was collapsed into one year before the intervention
> (pretest) and one year after the intervention (posttest). I'm now busy
> converting a long file (id, incident date, type of incident) into a long
> file in order to get a time perspective on the data (number of incidents,
> range in terms of days, etc) so I can eliminate those students who were only
> there for the pretest or posttest period.
>
> ...

[Dittfurth, Monica]

I will be out of the office until Saturday, April 20th with almost no access to email. If you have an urgent need or request before I return, please email [hidden email].

 

Thank you,

Monica

 

[Rich Ulrich]

[trying again, to re-post from April 9, Reply not showing in Nabble]

Let's consider for a minute what is missing from the dataset.

How many students were there who did not have incidents? 
Was the total school population similar from year to year? 
 - If you don't have base population data, you are pretty much
sunk, for estimating "rates" of anything, if it is supposed to
mean something in regard to the total population.

The crudest information might be:  What is the count total
incidents by month?  What is the count of aggressive incidents
by month?  Does that ratio vary?  Does it matter if that ratio
improves? -- That might provide a start to a narrative.

You *might*  look at the first and last incidents for each subject,
taken separately.  That is going to be irregular for startup/ending
effects, but your intervention was in the middle.  Is there a
decrease after the intervention?

A person with 173 incidents might be overly-influential in the
analysis.  But the presence of "bad apples" suggests that you
might look for "bad apples" and see how many of them there are
within various periods.  Do 10% of the people account for 90% of
the incidents?   or 20% for 80%?  What is the concentration?
Does that change by year?  - Your intervention might have cooled
off the worst performers.  Or it might leave them unaffected while
reducing the number of subjects cited.

For looking at bad apples, you could define "high in incidents"
separately for the total incidents and for aggressive incidents.

All of that counting leaves aside the questions that might arise
about whether the same people were using the same standards
to create those incident reports for the whole multi-year period.

--
Rich Ulrich

 

> Date: Tue, 9 Apr 2013 10:13:21 -0700
> From: [hidden email]
> Subject: Program evaluation question
> To: [hidden email]
>
> This is a program evaluation question.
>
> I have dataset made up incidents at a school between 2006 and 2013. There
> are just two variables of interest: date of incident e.g., 12/01/2011) and
> type of incident (e.g., minor injury, physical aggression to staff).
> Students stay at the school one to three years. The number of incidents
> during their stay at the school can vary from 1 to 173. In 2010 a program
> was instituted at the school to reduce aggressiveness. I’ve recoded type of
> incident into a new variable named aggress (0 = No, 1 = Yes). The problem I
> see is that no students have the same number of dates (incidents). How can
> I evaluate the effectiveness of the program?
>
> Any ideas are more than welcomed.
> ...