In Neil Salkind (Ed.), Encyclopedia of Research Design.
Thousand Oaks, CA: Sage. 2010
Contrast Analysis
Herv´e Abdi ⋅ Lynne J. Williams
A standard analysis of variance (a.k.a. anova) provides an F -test, which is called an om-
nibus test because it reflects all possible differences between the means of the groups analyzed
by the anova. However, most experimenters want to draw conclusions more precise than
“the experimental manipulation has an effect on participants’ behavior.” Precise conclusions
can be obtained from contrast analysis because a contrast expresses a specific question about
the pattern of results of an anova. Specifically, a contrast corresponds to a prediction precise
enough to be translated into a set of numbers called contrast coefficients which reflect the
prediction. The correlation between the contrast coefficients and the observed group means
directly evaluates the similarity between the prediction and the results.
When performing a contrast analysis we need to distinguish whether the contrasts are
planned or post hoc. Planned or a priori contrasts are selected before running the experiment.
In general, they reflect the hypotheses the experimenter wanted to test and there are usually
few of them. Post hoc or a posteriori (after the fact) contrasts are decided after the experiment
has been run. The goal of a posteriori contrasts is to ensure that unexp ected results are
reliable.
When performing a planned analysis involving several contrasts, we need to evaluate
if these contrasts are mutually orthogonal or not. Two contrasts are orthogonal when their
contrast coefficients are uncorrelated (i.e., their coefficient of correlation is zero). The number
of possible orthogonal contrasts is one less than the number of levels of the independent
variable.
Herv´e Abdi
The University of Texas at Dallas
Lynne J. Williams
The University of Toronto Scarborough
Address correspondence to:
Herv´e Abdi
Program in Cognition and Neurosciences, MS: Gr.4.1,
The University of Texas at Dallas,
Richardson, TX 75083–0688, USA
E-mail: [email protected] http://www.utd.edu/∼herve
2 Contrast Analysis
All contrasts are evaluated using the same general procedure. First, the contrast is for-
malized as a set of contrast coefficients (also called contrast weights). Second, a specific F
ratio (denoted F
ψ
) is computed. Finally, the probability associated with F
ψ
is evaluated.
This last step changes with the type of analysis performed.
1 How to express a research hypothesis as a contrast
When a research hypothesis is precise, it is possible to express it as a contrast. A research
hypothesis, in general, can be expressed as a shape, a configuration, or a rank ordering of
the experimental means. In all of these cases, we can assign numb ers which will reflect the
predicted values of the experimental means. These numbers are called contrast coefficients
when their mean is zero. To convert a set of numbers into a contrast, it suffices to subtract
their mean from each of them. Often, for convenience we will express contrast coefficients
with integers.
For example, assume that for a 4-group design, a theory predicts that the first and second
groups should be equivalent, the third group should perform better than these two groups
and the fourth group should do better than the third with an advantage of twice the gain of
the third over the first and the second. When translated into a set of ranks this prediction
gives:
C
1
C
2
C
3
C
4
Mean
1 1 2 4 2
After subtracting the mean, we get the following contrast:
C
1
C
2
C
3
C
4
Mean
−1 − 1 0 2 0
In case of doubt, a good heuristic is to draw the predicted configuration of results, and
then to represent the position of the means by ranks.
ABDI & WILLIAMS 3
2 A priori (planned) orthogonal contrasts
2.1 How to correct for multiple tests
When several contrast are evaluated, several statistical tests are performed on the same data
set and this increases the probability of a Type I error ( i.e., rejecting the null hypothesis
when it is true). In order to control the Type I error at the level of the set (a.k.a. the family)
of contrasts one needs to correct the α level used to evaluate each contrast. This correction
for multiple contrasts can be done using the
ˇ
Sid`ak equation, the Bonferroni (a.k.a. Boole,
or Dunn) inequality or the Monte-Carlo technique.
2.1.1
ˇ
Sid`ak and Bonferroni
The probability of making as least one Type I error for a family of orthogonal (i.e., statisti-
cally independent) C contrasts is
α[P F ] = 1 − (1 − α[P C])
C
. (1)
with α[P F ] being the Type I error for the family of contrasts; and α[P C] being the Type I
error per contrast. This equation can be rewritten as
α[P C] = 1 − (1 − α[P F ])
1/C
. (2)
This formula, called the
ˇ
Sid`ak equation, shows how to correct the α[P C] values used for
each contrast.
Because the
ˇ
Sid`ak equation involves a fractional power, ones can use an approximation
known as the Bonferroni inequality, which relates α[P C] to α[P F ] by
α[P C] ≈
α[P F ]
C
. (3)
ˇ
Sid`ak and Bonferroni are related by the inequality
α[P C] = 1 − (1 − α[P F ])
1/C
≥
α[P F ]
C
. (4)
They are, in general, very close to each other. As can be seen, the Bonferroni inequality is
a pessimistic estimation. Consequently
ˇ
Sid`ak should be preferred. However, the Bonferroni
inequality is more well known, and hence, is used and cited more often.
2.1.2 Monte-Carlo
The Monte-Carlo technique can also be used to correct for multiple contrasts. The Monte
Carlo technique consists of running a simulated experiment many times using random data,
4 Contrast Analysis
Table 1: Results of a Monte-Carlo simulation. Numbers of Type I errors when performing C = 5 contrasts for 10, 000 analyses of
variance performed on a 6 group design when the H
0
is true. How to read the table? For example, 192 families over 10, 000 have 2
Type I errors, this gives 2 × 192 = 384 Type I errors.
Number of families X: Number of Type 1 Number of
with X Type I errors errors per family Type I errors
7, 868 0 0
1, 907 1 1, 907
192 2 384
20 3 60
13 4 52
0 5 0
10, 000 2, 403
with the aim of obtaining a pattern of results showing what would happen just on the basis
of chance. This approach can be used to quantify α[P F ], the inflation of Type I error due
to multiple testing. Equation 2 can then be used to set α[P C] in order to control the overall
value of the Type I error.
As an illustration, suppose that 6 groups with 100 observations per group are created with
data randomly sampled from a normal population. By construction, the H
0
is true (i.e., all
population means are equal). Now, construct 5 independent contrasts from these 6 groups.
For each contrast, compute an F -test. If the probability associated with the statistical index
is smaller than α = .05, the contrast is said to reach significance ( i.e., α[P C] is used). Then
have a computer redo the experiment 10, 000 times. In sum, there are 10, 000 experiments,
10, 000 families of contrasts and 5 × 10, 000 = 50, 000 contrasts. The results of this simulation
are given in Table 1.
Table 1 shows that the H
0
is rejected for 2, 403 contrasts over the 50, 000 contrasts actually
performed (5 contrasts times 10, 000 experiments). From these data, an estimation of α[P C]
is computed as:
α[P C] =
number of contrasts having reached significance
total number of contrasts
=
2, 403
50, 000
= .0479 . (5)
This value falls close to the theoretical value of α = .05.
It can be seen also that for 7, 868 experiments no contrast reached significance. Equiva-
lently for 2, 132 experiments (10, 000−7, 868) at least one Type I error was made. From these
data, α[P F ] can be estimated as:
ABDI & WILLIAMS 5
α[P F ] =
number of families with at least 1 Type I error
total number of families
=
2, 132
10, 000
= .2132 . (6)
This value falls close to the theoretical value given by Equation 1:
α[P F ] = 1 − (1 − α[P C])
C
= 1 − (1 − .05)
5
= .226 .
2.2 Checking the orthogonality of two contrasts
Two contrasts are orthogonal (or independent) if their contrast coefficients are uncorrelated.
Recall that contrast coefficients have zero sum (and therefore a zero mean). Therefore, two
contrasts whose A contrast coefficients are denoted C
a,1
and C
a,2
, will be orthogonal if and
only if :
A
a=1
C
a,i
C
a,j
= 0 . (7)
2.3 Computing sum of squares, mean square, and F
The sum of squares for a contrast can be computed using the C
a
coefficients. Specifically,
the sum of squares for a contrast is denoted SS
ψ
, and is computed as
SS
ψ
=
S(
∑
C
a
M
a.
)
2
∑
C
2
a
(8)
where S is the number of subjects in a group.
Also, because the sum of squares for a contrast has one degree of freedom it is equal to
the mean square of effect for this contrast:
MS
ψ
=
SS
ψ
df
ψ
=
SS
ψ
1
= SS
ψ
. (9)
The F
ψ
ratio for a contrast is now computed as
F
ψ
=
MS
ψ
MS
error
. (10)
6 Contrast Analysis
2.4 Evaluating F for orthogonal contrasts
Planned orthogonal contrasts are equivalent to indep endent questions asked to the data.
Because of that independence, the current procedure is to act as if each contrast were the
only contrast tested. This amounts to not using a correction for multiple tests. This procedure
gives maximum power to the test. Practically, the null hypothesis for a contrast is tested
by computing an F ratio as indicated in Equation 10 and evaluating its p value using a
Fisher sampling distribution with ν
1
= 1 and ν
2
being the number of degrees of freedom of
MS
error
[e.g., in independent measurement designs with A groups and S observations per
group ν
2
= A(S − 1)].
2.5 An example
This example is inspired by an experiment by Smith (1979). The main purp ose in this experi-
ment was to show that being in the same mental context for learning and for test gives better
performance than b eing in different contexts. During the learning phase, subjects learned a
list of 80 words in a room painted with an orange color, decorated with posters, paintings
and a decent amount of paraphernalia. A first memory test was performed to give subjects
the impression that the experiment was over. One day later, subjects were unexpectedly
re-tested for their memory. An experimenter asked them to write down all the words of the
list they could remember. The test took place in 5 different experimental conditions. Fifty
subjects (ten per group) were randomly assigned to one of the five experimental groups. The
five experimental conditions were:
1. Same context. Subjects are tested in the same room in which they learned the list.
2. Different context. Subjects are tested in a room very different from the one in which they
learned the list. The new room is located in a different part of the campus, is painted
grey and lo oks very austere.
3. Imaginary context. Subjects are tested in the same room as subjects from Group 2. In
addition, they are told to try to remember the room in which they learned the list. In
order to help them, the experimenter asks them several questions about the room and
the objects in it.
4. Photographed context. Subjects are placed in the same condition as Group 3, and, in
addition, they are shown photos of the orange room in which they learned the list.
5. Placebo context. Subjects are in the same condition as subjects in Group 2. In addition,
before starting to try to recall the words, they are asked first to perform a warm-up task,
namely, to try to remember their living room.
The data and anova results of the replication of Smith’s experiment are given in the Tables 2
and 3.
2.5.1 Research hypotheses for contrast analysis
Several research hypotheses can be tested with Smith’s experiment. Suppose that the exper-
iment was designed to test these hypotheses:
ABDI & WILLIAMS 7
Table 2: Data from a replication of an experiment by Smith (1979). The dependent variable is the number of words recalled.
Experimental Context
Group 1 Group 2 Group 3 Group 4 Group 5
Same Different Imagery Photo Placebo
25 11 14 25 8
26 21 15 15 20
17 9 29 23 10
15 6 10 21 7
14 7 12 18 15
17 14 22 24 7
14 12 14 14 1
20 4 20 27 17
11 7 22 12 11
21 19 12 11 4
Y
a.
180 110 170 190 100
M
a.
18 11 17 19 10
M
a.
− M
..
3 −4 2 4 −5
∑
(Y
as
− M
a.
)
2
218 284 324 300 314
Table 3: anova table for a replication of Smith’s experiment (1979).
Source df SS MS F Pr(F )
Experimental 4 700.00 175.00 5.469
∗∗
.00119
Error 45 1, 440.00 32.00
Total 49 2, 140.00
– Research Hypothesis 1. Groups for which the context at test matches the context during
learning (i.e., is the same or is simulated by imaging or photography) will perform better
than groups with a different or placeb o contexts.
– Research Hypothesis 2. The group with the same context will differ from the group with
imaginary or photographed contexts.
– Research Hypothesis 3. The imaginary context group differs from the photographed con-
text group.
– Research Hypothesis 4. The different context group differs from the placebo group.
2.5.2 Contrasts
The four research hypotheses are easily transformed into statistical hypotheses. For example,
the first research hypothesis is equivalent to stating the following null hypothesis:
The means of the population for groups 1., 3., and 4. have the same value as the
means of the population for groups 2., and 5..
8 Contrast Analysis
Table 4: Orthogonal contrasts for the replication of Smith (1979).
contrast Gr. 1 Gr. 2 Gr. 3 Gr. 4 Gr. 5
∑
C
a
ψ
1
+2 −3 +2 +2 −3 0
ψ
2
+2 0 −1 −1 0 0
ψ
3
0 0 +1 −1 0 0
ψ
4
0 +1 0 0 −1 0
This is equivalent to contrasting groups 1., 3., 4. and groups 2., 5.. This first contrast is
denoted ψ
1
:
ψ
1
= 2µ
1
− 3µ
2
+ 2µ
3
+ 2µ
4
− 3µ
5
.
The null hypothesis to be tested is
H
0,1
∶ ψ
1
= 0
The first contrast is equivalent to defining the following set of coefficients C
a
:
Gr.1 Gr.2 Gr.3 Gr.4 Gr.5
a
C
a
+ 2 − 3 + 2 + 2 − 3 0
Note that the sum of the coefficients C
a
is zero, as it should be for a contrast. Table 4
shows all 4 contrasts.
2.5.3 Are the contrast orthogonal?
Now the problem is to decide if the contrasts constitute an orthogonal family. We check that
every pair of contrasts is orthogonal by using Equation 7. For example, Contrasts 1 and 2
are orthogonal b ecause
A=5
a=1
C
a,1
C
a,2
= (2 × 2) + (−3 × 0) + (2 × −1) + (2 × −1) + (−3 × 0) + (0 × 0) = 0 .
2.5.4 F test
The sum of squares and F
ψ
for a contrast are computed from Equations 8 and 10. For
example, the steps for the computations of SS
ψ1
are given in Table 5:
ABDI & WILLIAMS 9
Table 5: Steps for the computation of SS
ψ1
of Smith (1979).
Group M
a
C
a
C
a
M
a
C
2
a
1 18.00 +2 +36.00 4
2 11.00 −3 −33.00 9
3 17.00 +2 +34.00 4
4 19.00 +2 +38.00 4
5 10.00 −3 −30.00 9
0 45.00 30
SS
ψ1
=
S(
∑
C
a
M
a.
)
2
∑
C
2
a
=
10 × (45.00)
2
30
= 675.00
MS
ψ1
= 675.00
F
ψ1
=
MS
ψ1
MS
error
=
675.00
32.00
= 21.094 . (11)
The significance of a contrast is evaluated with a Fisher distribution with 1 and A(S −1) =
45 degrees of freedom, which gives a critical value of 4.06 for α = .05 (7.23 for α = .01). The
sum of squares for the remaining contrasts are SS
ψ.2
= 0, SS
ψ.3
= 20, and SS
ψ.4
= 5 with 1
and A(S − 1) = 45 degrees of freedom. Therefore, ψ
2
, ψ
3
, and ψ
4
are non-significant. Note
that the sums of squares of the contrasts add up to SS
experimental
. That is:
SS
experimental
= SS
ψ.1
+ SS
ψ.2
+ SS
ψ.3
+ SS
ψ.4
= 675.00 + 0.00 + 20.00 + 5.00
= 700.00 .
When the sums of squares are orthogonal, the degrees of freedom are added the same way
as the sums of squares are. This explains why the maximum number of orthogonal contrasts
is equal to number of degrees of freedom of the experimental sum of squares.
3 A priori (planned) non-orthogonal contrasts
So, orthogonal contrasts are relatively straightforward because each contrast can be evalu-
ated on its own. Non-orthogonal contrasts, however, are more complex. The main problem
is to assess the importance of a given contrast conjointly with the other contrasts. There are
10 Contrast Analysis
currently two (main) approaches to this problem. The classical approach corrects for multi-
ple statistical tests (e.g., using a
ˇ
Sid`ak or Bonferroni correction), but essentially evaluates
each contrast as if it were coming from a set of orthogonal contrasts. The multiple regression
(or modern) approach evaluates each contrast as a predictor from a set of non-orthogonal
predictors and estimates its specific contribution to the explanation of the dependent vari-
able. The classical approach evaluates each contrast for itself, whereas the multiple regression
approach evaluates each contrast as a member of a set of contrasts and estimates the spe-
cific contribution of each contrast in this set. For an orthogonal set of contrasts, the two
approaches are equivalent.
3.1 The classical approach
Some problems are created by the use of multiple non-orthogonal contrasts. Recall that the
most important one is that the greater the numb er of contrasts, the greater the risk of a
Type I error. The general strategy adopted by the classical approach to take this problem is
to correct for multiple testing.
3.1.1
ˇ
Sid`ak and Bonferroni corrections for non-orthogonal contrasts
When a family of contrasts are nonorthogonal, Equation 1 gives a lower bound for α[P C]
(cf.
ˇ
Sid`ak, 1967; Games, 1977). So, instead of having the equality, the following inequality,
called the
ˇ
Sid`ak inequality, holds
α[P F ] ≤ 1 − (1 − α[P C])
C
. (12)
This inequality gives an upper bound for α[P F ], therefore the real value of α[P F ] is smaller
than its estimated value.
As previously, we can approximate the
ˇ
Sid`ak inequality by Bonferroni as
α[P F ] < Cα[P C] . (13)
And, as previously,
ˇ
Sid`ak and Bonferroni are linked to each other by the inequality
α[P F ] ≤ 1 − (1 − α[P C])
C
< Cα[P C] . (14)
3.1.2 An example: Classical approach
Let us go back to Smith’s (1979) study (see Table 2). Suppose that Smith wanted to test
these three hypotheses:
– Research Hypothesis 1. Groups for which the context at test matches the context during
learning will p erform better than groups with different contexts;
ABDI & WILLIAMS 11
Table 6: Non-orthogonal contrasts for the replication of Smith (1979).
contrast Gr. 1 Gr. 2 Gr. 3 Gr. 4 Gr. 5
∑
C
a
ψ
1
2 −3 2 2 −3 0
ψ
2
3 3 −2 −2 −2 0
ψ
3
1 −4 1 1 1 0
Table 7: F
ψ
values for the nonorthogonal contrasts from the replication of Smith (1979).
r
Y.ψ
r
2
Y.ψ
F
ψ
p(F
ψ
)
ψ
1
.9820 .9643 21.0937 < .0001
ψ
2
−.1091 .0119 0.2604 .6123
ψ
3
.5345 .2857 6.2500 .0161
– Research Hypothesis 2. Groups with real contexts will perform better than those with
imagined contexts;
– Research Hypothesis 3. Groups with any context will perform better than those with no
context.
These hypotheses can easily be transformed into the set of contrasts given in Table 6.
The values of F
ψ
were computed with Equation 10 (see also Table 3) and are in shown in
Table 7 along with their p values. If we adopt a value of α[P F ] = .05, a
ˇ
Sid`ak correction
(from Equation 2) will entail evaluating each contrast at the α level of α[P C] = .0170
(Bonferroni will give the approximate value of α[P C] = .0167). So, with a correction for
multiple comparisons we will conclude that Contrasts 1 and 3 are significant.
3.2 Multiple regression approach
Anova and multiple regression are equivalent if we use as many predictors for the multiple
regression analysis as the number of degrees of freedom of the independent variable. An
obvious choice for the predictors is to use a set of contrasts coefficients. Doing so makes
contrast analysis a particular case of multiple regression analysis. When used with a set of
orthogonal contrasts, the multiple regression approach gives the same results as the anova
based approach previously described. When used with a set of non-orthogonal contrasts,
multiple regression quantifies the specific contribution of each contrast as the semi-partial
coefficient of correlation between the contrast coefficients and the dependent variable. We
can use the multiple regression approach for non-orthogonal contrasts as long as the following
constraints are satisfied:
1. There are no more contrasts than the number of degrees of freedom of the independent
variable;
2. The set of contrasts is linearly independent ( i.e., not multicollinear). That is, no contrast
can be obtained by combining the other contrasts.
12 Contrast Analysis
3.2.1 An example: Multiple regression approach
Let us go back once again to Smith’s (1979) study of learning and recall contexts. Suppose
we take our three contrasts (see Table 6) and use them as predictors with a standard multiple
regression program. We will find the following values for the semi-partial correlation between
the contrasts and the dependent variable:
ψ
1
∶ r
2
Y.C
a,1
∣C
a,2
C
a,3
= .1994
ψ
2
∶ r
2
Y.C
a,2
∣C
a,1
C
a,3
= .0000
ψ
3
∶ r
2
Y.C
a,3
∣C
a,1
C
a,2
= .0013 ,
with r
2
Y.C
a,1
∣C
a,2
C
a,3
being the squared correlation of ψ
1
and the dependent variable with the
effects of ψ
2
and ψ
3
partialled out. To evaluate the significance of each contrast, we compute
an F ratio for the corresponding semi-partial coefficients of correlation. This is done using
the following formula:
F
Y.C
a,i
∣C
a,k
C
a,`
=
r
2
Y.C
a,i
∣C
a,k
C
a,`
1 − r
2
Y.A
× (df
residual
) . (15)
This results in the following F ratios for the Smith example:
ψ
1
∶ F
Y.C
a,1
∣C
a,2
C
a,3
= 13.3333, p = 0.0007;
ψ
2
∶ F
Y.C
a,2
∣C
a,1
C
a,3
= 0.0000, p = 1.0000;
ψ
3
∶ F
Y.C
a,3
∣C
a,1
C
a,2
= 0.0893, p = 0.7665.
These F ratios follow a Fisher distribution with ν
1
= 1 and ν
2
= 45 degrees of freedom.
F
critical
= 4.06 when α = .05. In this case, ψ
1
is the only contrast reaching significance (i.e.,
with F
ψ
> F
critical
). The comparison with the classic approach shows the drastic differences
between the two approaches.
4 A posteriori (post-hoc) contrasts
For a posteriori contrasts, the family of contrasts is composed of all the possible contrasts
even if they are not explicitly made. Indeed, because we choose the contrasts to be made
a posteriori, this implies that we have implicitly made and judged uninteresting all the
possible contrasts that have not been made. Hence, whatever the number of contrasts actually
performed, the family is composed of all the possible contrasts. This number grows very fast:
A conservative estimate indicates that the number of contrasts which can be made on A
groups is equal to
1 + {[(3
A
− 1)/2] − 2
A
} . (16)
ABDI & WILLIAMS 13
So, using a
ˇ
Sid`ak or Bonferroni approach will not have enough power to be useful.
4.1 Scheff´e’s test
Scheff´e’s test was devised to test all possible contrasts a posteriori while maintaining the
overall Type I error level for the family at a reasonable level, as well as trying to have a
conservative but relatively powerful test. The general principle is to insure that no discrepant
statistical decision can occur. A discrepant decision would occur if the omnibus test would
fail to reject the null hypothesis, but one a posteriori contrast could be declared significant.
In order to avoid such a discrepant decision, the Scheff´e approach first tests any contrast as
if it were the largest possible contrast whose sum of squares is equal to the experimental sum
of squares (this contrast is obtained when the contrast coefficients are equal to the deviations
of the group means to their grand mean); and second makes the test of the largest contrast
equivalent to the anova omnibus test. So, if we denote by F
critical,omnibus
the critical value
for the anova omnibus test (performed on A groups), the largest contrast is equivalent to
the omnibus test if its F
ψ
is tested against a critical value equal to
F
critical, Scheff´e
= (A − 1) × F
critical,omnibus
. (17)
Equivalently, F
ψ
can be divided by (A−1) and its probability can be evaluated with a Fisher
distribution with ν
1
= (A − 1) and ν
2
being equal to the number of degrees of freedom of the
mean square error. Doing so makes it impossible to reach a discrepant decision.
4.1.1 An example: Scheff´e
Suppose that the F
ψ
ratios for the contrasts computed in Table 7 were obtained a posteriori.
The critical value for the anova is obtained from a Fisher distribution with ν
1
= A − 1 = 4
and ν
2
= A(S − 1 ) = 45. For α = .05 this value is equal to F
critical,omnibus
= 2.58. In order
to evaluate if any of these contrasts reaches significance, we need to compare them to the
critical value of
F
critical,Scheff´e
= (A − 1) × F
critical,omnibus
= 4 × 2.58 = 10.32.
With this approach, only the first contrast is considered significant.
Related entries
Analysis of Variance, Bonferonni correction, Post-Hoc comparisons.
14 Contrast Analysis
Further Readings
1. Abdi, H., Edelman, B., Valentin, D., & Dowling, W.J. (2009). Experimental Design and Analysis for Psychology. Oxford:
Oxford University Press.
2. Rosenthal, R., & Rosnow, R.L. (2003). Contrasts and effect sizes in behavioral research: A correlational approach. Boston:
Cambridge University Press.
