Can This Research Project Be Saved?

[Annette D. Hawkins]

I was asked to help evaluate an experimental design at my community
college.  When presented with the data, I told the instructor involved
that the design was flawed because no random sampling occurred.  This is
the research:

Three math instructors selected two classes each for the experimental
design.

One taught two classes of Math 070 Introductory Algebra
One taught two classes of Math 080 Intermediate Algebra
One taught two classes of Math 060 Basic Arithmetic

One class would be the treatment class and use MyMathLab, computer
software by Prentice-Hall.  I don’t know how often they used the
software.  I was told at least once a week.  The other class did not use
the software.

The students were matched by one of two methods: grade in a previous
course or college placement score.  For example, if a student in the
treatment group tested into the course, the college placement score was
used.  The instructor then tried to find a student in the control group
who tested in, too.  If the student entered the class based on a grade in
a previous course, then the control group match used the same method.  In
some cases, she matched students more than once in the control group.  For
the Basic Arithmetic class, all of those students used placement scores
because there is no course below this one.  For the Math 080 class, she
included a night class with a day class for the control group. At the end
of the semester, final grades in both groups were compared.

We have a grant and time has come to evaluate whether the treatment group
performed better because of MyMathLab.  I hope I have given enough
details.  I would appreciate any help with this project.  Thanks.

Annette

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

[Gary Oliver]

Hi Annette

I don't think the design as such is flawed. Textbooks acknowledge that random selection often does not occur. Attention would need to be paid to the data analysis. There are clearly nested values - students in classes. This is comprehensively discussed in the HML literature eg
Goldstein: Multilevel statistical models
Raudenbush and Bryk: Hierarchical linear models

Happy New Year

Dr Gary R Oliver, CPA
Discipline of Accounting
School of Business
The University of Sydney
------------------------
      ,-_|\       Building H69, Office 437
     /        \      Corner Codrington Street
     \_,-._*      & Rose Street
                  Darlington 2006
           @      Australia
--------------------------------------
E-mail: [hidden email]
------------------------
Location details:
Travelling from Broadway, turn south off City Road
Navigate toward the Aquatic Centre
------------------------
University Map: http://db.auth.usyd.edu.au/directories/map/index.stm
University Website:
www.usyd.edu.au
Faculty Website
www.econ.usyd.edu.au
------------------------
Faculty Student Information Office
(Timetables, Special Consideration)
Merewether Building
Enter from City Road side
e-mail: [hidden email]
Phone: [phone]
----------------------------------
Executive Officer for the
Discipline of Accounting:
Kate Doyle
Room 302, Building H69
E-mail: : [hidden email]
Phone: [phone]
Fax: [phone]
---------------------------------

>>-----Original Message-----
>>From: SPSSX(r) Discussion [mailto:[hidden email]]
>>On Behalf Of Annette D. Hawkins
>>Sent: Monday, January 07, 2008 2:15 AM
>>To: [hidden email]
>>Subject: Can This Research Project Be Saved?
>>
>>I was asked to help evaluate an experimental design at my
>>community college.  When presented with the data, I told the
>>instructor involved that the design was flawed because no
>>random sampling occurred.  This is the research:
>>
>>Three math instructors selected two classes each for the
>>experimental design.
>>
>>One taught two classes of Math 070 Introductory Algebra One
>>taught two classes of Math 080 Intermediate Algebra One
>>taught two classes of Math 060 Basic Arithmetic
>>
>>One class would be the treatment class and use MyMathLab,
>>computer software by Prentice-Hall.  I donâEUR(tm)t know how often
>>they used the software.  I was told at least once a week.
>>The other class did not use the software.
>>
>>The students were matched by one of two methods: grade in a
>>previous course or college placement score.  For example, if
>>a student in the treatment group tested into the course, the
>>college placement score was used.  The instructor then tried
>>to find a student in the control group who tested in, too.
>>If the student entered the class based on a grade in a
>>previous course, then the control group match used the same
>>method.  In some cases, she matched students more than once
>>in the control group.  For the Basic Arithmetic class, all of
>>those students used placement scores because there is no
>>course below this one.  For the Math 080 class, she included
>>a night class with a day class for the control group. At the
>>end of the semester, final grades in both groups were compared.
>>
>>We have a grant and time has come to evaluate whether the
>>treatment group performed better because of MyMathLab.  I
>>hope I have given enough details.  I would appreciate any
>>help with this project.  Thanks.
>>
>>Annette
>>
>>=====================
>>To manage your subscription to SPSSX-L, send a message to
>>[hidden email] (not to SPSSX-L), with no body text
>>except the command. To leave the list, send the command
>>SIGNOFF SPSSX-L For a list of commands to manage
>>subscriptions, send the command INFO REFCARD
>>

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD