Calculation of power - which N matters

[Robinson Aschoff]

Hello,

we have some doubts about the calculation of power in our design. Maybe
someone can help to disentangle this.

Our design can be described as follows: We would like to know if a certain
internet source (e.g. wikipedia) has higher quality articles than a
printed encyclopedia. So we intend to draw a random sample of articels
from both sources and let them rate for quality by a number of persons
(likert scale). We plan to draw about 20 articles from each source and let
them rate by 5 persons. Since it is not possible that one person rates all
40 articles, we intend to give each person 8 articles (four from wikipedia
and four from encyclopedia).
We intend to run a paired t-test to compare the ratings of the wikipedia
articles with the encyclopedia articles.

Now we are not sure about the power of this test. We would assume that it
depends on the number of articles we draw from the respective source but
the number of raters we choose should also have an impact. Since we want
to compare articles and not people we are not sure which N we have to
consider in which way.

The actual design is a bit more complicated, but this should illustrate
our problem pretty well. Thanks a lot for any suggestions.


        Wikipedia    encyclopedia

Rater1        1                1
             2                2
                     3                3
                     4                4

Rater2        5                5
                      6                6
                     7                7
                    8                8

Rater3        9                9
                    10                10
                   11                11
                   12                12

Rater4        13                13
                      14                14
                      15                15
                      16                16

Rater5  17                17
               18                18
               19                19
               20                20
----------------------------------------------------------------
Felix-Robinson Aschoff
Information Management Research Group
Department of Informatics
University of Zurich
Binzmuehlestrasse 14
CH-8050 Zurich, Switzerland

E-Mail: [hidden email]
Phone: [phone]
Fax: [phone]
Room: 2.D.11
http://www.ifi.unizh.ch/im

[Maguin, Eugene]

Robinson,

I haven't seen any on-list reply to your posting but perhaps you have
received off-list replies. Anyway. It would seem to me that the rating an
article receives depends on the source (Wiki or encyclopedia), the rater,
and the interaction between source and rater. I would also wonder if the
order that the articles were presented would matter. I think your basic
design is a between-within design with rater being the between factor and
source being the within factor.

Calculating power for this sort of design most easily requires a custom
written power anlaysis program (e.g., PASS). However, you can use the Manova
procedure in spss to compute power by inputting through the spss matrix data
routine the means, ns, variances and covariances. If you want to pursue this
method, reply to me off-list and I will send you some information. This is
the referenced article.

D'Amico, Elizabeth J; Neilands, Torsten B; Zambarano, Robert. Power analysis
for multivariate and repeated measures designs: A flexible approach using
the SPSS MANOVA procedure. Behavior Research Methods, Instruments &
Computers. Vol 33(4) Nov 2001, 479-484.

An update to this article has been published.

Osborne, Jason W. Power analysis for multivariate and repeated measurements
designs via SPSS: Correction and extension of D'Amico, Neilands, and
Zambarano (2001). Behavior Research Methods. Vol 38(2) May 2006, 353-354.

Gene Maguin