Table 19. SCALE SCORES Regression for Language Arts for 199798 FirstGrade Students
Block 1 Block 2 Block 3
Variable b t b t b t
PreTest Score .56 24.87* .55 24.18* .55 23.94*
Days Absent .37 3.31* .35 3.06* .33 2.90*
Subsidized Lunch Eligibility 2.63 2.46* 1.95 1.72 2.06 1.82
African American 2.78 1.02 2.50 .92
White 5.60 2.37* 5.80 2.47*
SAGE 7.25 3.64*
Constant 292.26 23.33* 291.59 23.06* 304.36 23.28*
Adjusted R Squared .33 .33 .34
Standard Error of Estimate 36.49 36.45 36.30
*significant at .05 level
Table 20. SCALE SCORES Regression for Reading for 199798 FirstGrade Students
Block 1 Block 2 Block 3
Variable b t b t b t
PreTest Score .56 20.88* .55 20.33* .54 20.20*
Days Absent .21 1.96* .18 1.60 .16 1.43
Subsidized Lunch Eligibility 3.01 2.85* 2.06 1.85 2.15 1.94*
African American 3.07 1.15 2.80 1.05
White 7.21 3.14* 7.37 3.22*
SAGE 6.98 3.59*
Constant 285.03 19.25* 284.30 19.08* 296.11 19.48*
Adjusted R Squared .26 .27 .27
Standard Error of Estimate 35.73 35.64 35.50
*significant at .05 level
Table 21. SCALE SCORES Regression for Mathematics for 199798 FirstGrade Students
Block 1 Block 2 Block 3
Variable b t b t b t
PreTest Score .62 30.65* .62 29.21 .61 29.13*
Days Absent .21 1.96* .18 1.60 2.15 1.94*
Subsidized Lunch Eligibility 2.75 3.08* 2.44 2.62* 2.56 2.76*
African American .1.66 .74 1.44 .64
White 3.05 1.56 3.24 1.66
SAGE 7.06 4.31*
Constant 235.05 22.32* 235.69 21.88* 247.05 22.40
Adjusted R Squared .43 .43 .44
Standard Error of Estimate 30.05 30.05 29.88
*significant at .05 level
28
Table 22. SCALE SCORES Regression for Total for 199798 FirstGrade Students
Block 1 Block 2 Block 3
Variable b t b t b t
PreTest Score .77 38.80* .77 37.58* .76 37.46*
Days Absent .34 3.08* 2.44 2.62* 2.56 2.76*
Subsidized Lunch Eligibility .63 .86 .53 .68 .62 .81
African American 3.91 2.1* 3.72 2.01*
White 3.00 1.86 3.20 2.00*
SAGE 6.33 4.68*
Constant 167.12 15.51* 165.11 15.04* 176.57 15.80*
Adjusted R Squared .54 .54 .55
Standard Error of Estimate 24.53 24.51 24.34
*significant at .05 level
AfricanAmerican Students
Among minority students in SAGE and comparison schools, African Americans clearly
comprise the largest group of valid test scores – roughly 25% percent of SAGE students and 28%
percent of comparison school students. In the analyses to follow, AfricanAmerican students are
first compared across SAGE and comparison schools on CTBS subtest and total scale scores.
Second, AfricanAmerican students are compared to white students across SAGE and
comparison schools on CTBS total scale scores.
SAGE vs. Comparison. Table 23 provides comparisons of means on the CTBS posttest,
and change scores from pretest to posttest. On the posttest, AfricanAmerican SAGE students
scored higher than AfricanAmerican comparison school students on every subtest and on total
scale score. The differences between SAGE and comparison schools on posttest scores are all
statistically significant. In addition, the differences between SAGE and comparison schools on
mean change scores from pretest to posttest scores are statistically significant. In other words,
AfricanAmerican SAGE students scored lower on the CTBS pretest than AfricanAmerican
comparison school students, but made significantly larger gains than comparison school students
from pre to posttest, and surpassed AfricanAmerican comparison school students on the posttest.
29
Table 23. African American PostTest and Change Scores, by SAGE or Comparison School for
199798 FirstGrade Students
SCALE SCORE SAGE COMPARISON F
Language Arts
Mean PostTest 572.80 558.32 11.59*
Mean Change Pre to Post 56.05 38.27 16.41*
Reading
Mean PostTest 573.82 554.11 25.31*
Mean Change Pre to Post 50.55 25.79 31.67*
Mathematics
Mean PostTest 522.01 506.22 20.74*
Mean Change Pre to Post 49.06 27.50 41.99*
Total
Mean Post Test 556.72 539.73 25.48*
Mean Change, Pre to Post 52.15 32.78 43.51*
*significant at .05 level
AfricanAmerican Males. Concern over the minority achievement gap on standardized
tests has occasionally focused on AfricanAmerican male students. Table 24 further
distinguishes AfricanAmerican SAGE and comparison school students by gender. The 199697
results showed that AfricanAmerican male SAGE students attained comparable or higher
change scores from pretest to posttest when compared to AfricanAmerican female SAGE
students. The 199798 results show that AfricanAmerican male SAGE students attained
comparable or higher change scores from pretest to posttest on the language arts subtest, the
mathematics subtest, and the total score. However, none of these results is statistically
significant.
Table 24. AfricanAmerican PostTest and Change Scores by Gender
COMPARISON SAGE
Male Female Male Female
Language Arts
Mean PostTest Scale Score 554.74 562.18 570.99 574.27
Mean Change Pre to Post 33.96 42.69 58.64 53.65
Reading
Mean PostTest Scale Score 556.06 552.25 570.40 576.77
Mean Change Pre to Post 35.75 15.19 50.37 50.49
Mathematics
Mean PostTest Scale Score 511.48 501.24 522.82 520.93
Mean Change Pre to Post 28.90 26.08 53.37 45.01
Total
Mean PostTest Scale Score 540.94 538.70 554.85 558.23
Mean Change Pre to Post 34.78 30.65 53.47 50.93
*significant at .05 level
30
AfricanAmerican and White Achievement. AfricanAmerican students scored
significantly lower than white students on the CTBS pretest total scale score, as shown in table
25. This result holds for both SAGE and comparison schools, though the gap between African
Americans and whites is larger in SAGE schools. When all cases are analyzed, African
American SAGE students achieved greater gains on the CTBS total scale score than white SAGE
students from pre to posttest, closing the achievement gap (though the gap remains statistically
significant). In contrast, African Americans in comparison schools achieved lesser gains and in
the comparison schools the achievement gap widened.
Table 25. AfricanAmerican and White Achievement in SAGE and Comparison Schools
on Total Scale Scores for 199798 FirstGrade Students
PRETEST POSTTEST CHANGE
SAGE SCHOOLS
African American 502.79 556.72 52.15
White 531.38 579.94 45.99
F 170.61* 96.09* 10.50*
COMPARISON SCHOOLS
African American 510.07 539.73 32.78
White 528.60 569.02 41.14
F 52.21 90.15* 10.72*
*significant at .05 level
Hierarchical Linear Modeling
Explanation. Many social science research analyses involve hierarchical data structures.
Hierarchical data structures are those in which individual units are nested within larger units, the
latter being the unit of interest. The SAGE data are a prime example: students are nested within
classrooms, and it is the classroom effect that is of particular interest to the SAGE project.
Hierarchical data structures pose special analytical challenges in that data analysis at the
individual level may result in a biased impression of the effect of the nesting unit (in the SAGE
case, the classroom). At the origin of this problem is the fact that different classrooms often
contain different numbers of students, thus those classrooms that contain greater numbers of
31
students have greater influence over the results of analyses done at the individual level. An
analytical approach known as “hierarchical linear modeling” (Bryk & Raudenbush, 1992) was
specifically designed to accommodate these types of data structures. Essentially hierarchical
linear modeling (HLM) estimates individual effects by analyzing data within each class and then
provides a weighted average of these effects. The effects of the class are then estimated as if all
classes contained the same number of students. HLM was used with the SAGE data to provide
an alternative and less biased account of the effects of SAGE experience on test scores. In these
models, variables associated with individual students are referred to as level1 variables and
those associated with the classrooms are referred to as level2 variables.
HLM Analyses. Analyses were conducted for each of the relevant criterion posttest
scores: reading, mathematics, language arts, and total. For all analyses, the level1 variables
were pretest scores and socioeconomic status (SES) measured as eligibility for subsidized lunch.
The posttest scores were adjusted for these two variables at the individual level, therefore the
effects may be thought of as being statistically independent of the effects of these variables. A
number of different level2 models, each containing different level2 variables, was specified for
each variable of interest. It is important to note that the “class size” variable used in these
analyses measures the studentteacher ratio.
HLM Results. Table 26 provides a summary of the effects of each of the level1 and
level2 variables for each of these analyses. Level1 effects can be interpreted as the weighted
average of the withinclassroom effects of the level1 variables. Level2 effects can be
interpreted as the classroom effects of the level2 variables. Level1 coefficients may be thought
of as the average effect of the modeling variable on the criterion score at the individual level.
The level1 results indicate that lower SES is related to lower posttest scores and higher pretest
scores are related to higher posttest scores.
32
The coefficients associated with the level2 variables can be thought of as classroom
effects. For example, in the Model A total score, an increase of one student in class size resulted
in a drop of .828 points for the class average. Likewise, SAGE participation resulted in a 8.909
point gain in the class average on total score for Model B. A discussion of each model follows:
Model A. Class Size. These models examined the effect of class size on the adjusted
criterion score. Class size equals the number of students divided by the number of teachers.
Depending on the test, an increase in class size of one person can be expected to produce a .29 to
1.12 loss in average posttest performance. The results for all scores show this effect to be
significant.
Model B. SAGE. These models examined the effect of SAGE participation on the
adjusted criterion score. Participation in SAGE shows statistically significant class average
increases in all posttest scores as well. These score increases range from 7 points (reading) to
13 points (mathematics).
Model C. Class Size, SAGE. These models examined the effect of SAGE participation
on the adjusted criterion score after the classrooms were class size adjusted, viewed as the effect
of SAGE participation beyond the class size effect. Combining class size and SAGE
participation in a single analysis isolates the effects that SAGE might have beyond those
produced by lower class size. The results show that once class size has been accounted for,
SAGE has no significant effect on class average performance. This may suggest that the other
SAGE interventions (i.e., rigorous curriculum, lighted school house, and staff development) are
not having a significant impact on achievement in SAGE classrooms.
Model D. Class SES, Class Size. These models examined the effect of class size on the
adjusted criterion score after the classrooms were SES adjusted, viewed as the effect of class size
once the effects of the classroom SES are removed. Since socioeconomic status is known to
33
have an influence on academic test scores, a replacement for this variable was used as both a
level1 and level2 predictor. The level2 variable was the average SES for the class and
estimates the effect of the overall class SES level beyond that associated with the individual,
which is accounted for in the level1 model. This model combines class SES and class size. The
results indicate that class SES has a significant effect on the class average posttest performance.
The effect of a 1 point class average gain in SES equates to between a 10 point and 13 point gain
on the average posttest score, depending on the test. SES was measured on a threepoint family
income scale, thus a one point difference on average would be quite pronounced. Class size still
has a significant effect on the posttest scores once SES has been accounted for.
Model E. Class SES, SAGE. These models examined the effect of SAGE participation
on the adjusted criterion score after the classrooms were SES adjusted; viewed as the effect of
SAGE participation once the effects of classroom SES are removed. When class SES and SAGE
participation are entered in the same level2 model, class SES has a significant effect on class
average posttest performance. In addition, SAGE has a significant effect on class average posttest
performance. In other words, the effects of SAGE participation on class average posttest
scores, beyond those produced by SES differences, are significant on all posttest scores. In
general, these effects are roughly the same as when SAGE is the only variable in the model (see
model B), suggesting that SAGE classrooms and control classrooms are about equal on class
SES.
Model F. Class SES, Class Size, SAGE. These models examined the effect of SAGE
participation on the adjusted criterion score after the classrooms were adjusted for class size and
SES; viewed as the effect of SAGE participation beyond the class size and SES effects. This
model combines SES, SAGE participation, and class size in a single analysis. For all subtests,
class SES once again has a significant effect on the class average posttest score. Class size has
34
no significant effect on the class average posttest score. Finally, SAGE had significant effects
only on the mathematics subtest.
Table 26. HLM Results for 199798 FirstGrade Students
Source Total Reading Language Arts Mathematics
Level 1
PreTest 0.870 0.627 0.625 0.712
SES 0.784 3.733 1.612 3.202
Level 2
A. Class Size 0.828* 0.289* 0.899* 1.115*
B. SAGE 8.909* 7.009* 10.148* 13.090*
C. Class Size 0.647 0.734 0.639 0.722
SAGE 2.990 0.195 4.228 6.409
D. Class SES 12.959* 10.410* 12.971* 13.389*
Class Size 0.599* 0.574* 0.698* 0.883*
E. Class SES 14.707* 12.215* 15.201* 16.298*
SAGE 9.354* 7.320* 10.661* 13.428*
F. Class SES 14.883* 11.446* 15.168* 16.211*
Class Size 0.015 .252 0.011 0.027
SAGE 9.074 4.957 10.556 13.172*
*significant at .05 level
SecondGrade Results 199798
Descriptive Statistics
Valid Test Scores. Analyses were conducted to assess the impact of SAGE on the 1997
98 secondgrade CTBS Complete Battery, Terra Nova Level 13 posttest results. There were
1702 persisting students (i.e., students present in both the 199697 SAGE and comparison firstgrade
classrooms and in the 199798 SAGE and comparison secondgrade classrooms), while
there were 482 new secondgrade students (students who were not in the program last year).
However, secondgrade posttest results are compared to the firstgrade pretest, as well as first
grade posttest. Therefore, only those students who took both the firstgrade pretest and posttest,
as well as the secondgrade posttest, were used in the 199798 secondgrade analysis. As
would be expected, the number of secondgrade students having all three valid test scores was
substantially less than the total number of students. The number of valid test scores for the
Fal1996 firstgrade pretest, the Spring 1997 firstgrade posttest, and the Spring 1998 second
35
grade posttest are presented in Table 27.
Table 27. Number of 199798 SecondGrade Students with Valid Test Scores
SAGE Comparison Total
Fall 1996 First
Grade Pretest
Reading 1033 562 1595
Language Arts 1033 562 1595
Mathematics 1020 559 1579
Total 1008 448 1456
Spring 1997 First
Grade PostTest
Reading 1011 545 1556
Language Arts 1011 545 1556
Mathematics 1007 538 1545
Total 1001 534 1535
Spring 1997
Second Grade
Reading 1037 561 1598
Language Arts 1037 562 1599
Mathematics 1043 559 1602
Total 1033 549 1582
PreTest (Baseline) Results. Both the firstgrade pretest and the firstgrade posttest
served as a baseline. Table 28 provides descriptive statistics on the scale scores from the firstgrade
pretest as well as the firstgrade posttest.
Table 28. Descriptive Statistics on CTBS FirstGrade PreTest and PostTest (SAGE and
Comparison)
FIRSTGRADE PRETEST FIRSTGRADE POSTTEST
SCALE
SCORES
NORMAL CURVE
EQUIVALENT
SCALE
SCORES
NORMAL CURVE
EQUIVALENT
Reading 535.20 36.83 45.31 19.78 584.17 35.65 54.44 18.50
Language
Arts
532.70 42.03 45.03 20.74 583.05 37.78 54.53 17.89
Mathematics 494.55 38.27 44.11 18.05 546.59 41.56 55.88 20.41
Total 521.03 33.34 44.33 18.28 571.43 31.94 55.65 17.81
Difference of Means Test. The results from the difference of means tests between SAGE
and comparison student scale scores from the Fall 1996 firstgrade pretest and Spring 1997 firstgrade
posttest are reported in Tables 2932. The differences between SAGE schools and
36
comparison schools on the firstgrade pretest are not found to be statistically significant at the
.05 level. Therefore, any differences between the firstgrade pretest and the secondgrade test
can be more confidently attributed to the studentteacher ratio of 15:1 in the SAGE classrooms.
The differences between SAGE schools and comparison schools on the firstgrade posttest are
found to be significant on the total score and on all subscores. Therefore, any conclusions
discussed regarding secondgrade results must take into account the effects of the SAGE
program while these students were in first grade.
Table 29. Differences of Means Test on FirstGrade PreTest and PostTest:
Language Arts Scale Scores
FIRSTGRADE
PRETEST
FIRSTGRADE
POSTTEST
N MEAN STANDARD
DEVIATION
N MEAN STANDARD
DEVIATION
Comparison Schools 562 530.69 43.09 545 579.01 39.75
SAGE Schools 1033 533.80 41.42 1011 586.07* 36.06
*significant at .05 level
Table 30. Differences of Means Test on FirstGrade PreTest and PostTest: Reading Scale
Scores
FIRSTGRADE
PRETEST
FIRSTGRADE
POSTTEST
N MEAN STANDARD
DEVIATION
N MEAN STANDARD
DEVIATION
Comparison Schools 562 534.62 38.77 545 582.01 36.50
SAGE Schools 1033 535.52 35.75 1011 586.07* 36.06
*significant at .05 level
Table 31. Differences of Means Test on FirstGrade PreTest and PostTest: Mathematics Scale
Scores
FIRSTGRADE
PRETEST
FIRSTGRADE
POSTTEST
N MEAN STANDARD
DEVIATION
N MEAN STANDARD
DEVIATION
Comparison Schools 559 493.70 38.26 538 541.88 40.75
SAGE Schools 1020 495.01 38.29 1007 550.67* 41.17
*significant at .05 level
37
Table 32. Differences of Means Test on FirstGrade PreTest and PostTest: Total Scale Score
FIRSTGRADE
PRETEST
FIRSTGRADE
POSTTEST
N MEAN STANDARD
DEVIATION
N MEAN STANDARD
DEVIATION
Comparison Schools 548 519.96 33.59 534 567.64 32.29
SAGE Schools 1008 521.61 33.20 1001 574.72* 30.97
*significant at .05 level
As noted above, student populations varied in SAGE and comparison schools due to
withdrawals and withinyear enrollments. The posttest results are based only on those second
graders who remained in class the entire 199697 first grade and 199798 second grade school
years.
Results of the difference of means test between SAGE and comparison schools on
the secondgrade posttest can be seen in Table 33. Table 34 shows that when the firstgrade
pretest is used as the baseline score, significant results are found on the language arts subscale,
mathematics subscale, and total score. However, when the firstgrade posttest is used as the
baseline score, no significant results are found. This suggests that the statistically significant
positive effects of SAGE occurred in the first grade. These positive effects were maintained, but
did not significantly increase in second grade.
Table 33. Difference of Means Test – SecondGrade Scale Scores
SAGE Schools Comparison Schools
N MEAN STANDARD
DEVIATION
N MEAN STANDARD
DEVIATION
Language Arts 1037 610.91* 41.10 562 602.70 41.38
Reading 1037 608.17 36.11 561 604.63 37.07
Mathematics 1043 572.11* 41.69 559 564.36 39.10
Total 1033 597.14* 34.29 549 591.25 34.10
*significant at .05 level
The largest gain in SAGE student scores from firstgrade pretest to the secondgrade
posttest was on the mathematics subtest, as shown in Table 34. The smallest relative gain for
38
SAGE students from pretest to posttest was on the reading subscale; this gain was not
statistically significant.
