Table 34. SAGE and Comparison Gain
From FirstGrade PreTest to Second Grade From FirstGrade PostTest to Second Grade
SCALE
SCORE
SAGE Gain Comparison
Gain
Gain
Difference
SAGE Gain Comparison
Gain
Gain
Difference
Language Arts 77.07 71.74 5.33* 25.67 22.76 2.91
Reading 72.78 69.62 3.16 22.33 22.01 3.32
Mathematics 77.54 70.07 7.47* 22.86 21.97 0.89
Total 75.90 70.80 5.1* 23.67 22.36 1.31
*significant at .05 level
Regression Analysis
Regression Models. The effect of the SAGE program on student achievement for second
graders was also tested through a series of ordinary least squares regression models for each subtest
and total scale score. Control variables were again entered into the models in blocks, with
the SAGE/comparison student variable entered into the models last. In addition, two different
regressions were done for each subtest and total scale score. The first regression used the firstgrade
pretest as a predictor variable and the second regression used the firstgrade posttest as a
predictor variable.
The first block of control variables included student score on the firstgrade pretest or
posttest and eligibility for subsidized lunch as an indicator of family income. Because
attendance data were not reliably reported by districts for second graders during 199798,
attendance was not included in the analytical model for second graders. As with the first graders
(discussed earlier), the second block of control variables included dummy variables for
race/ethnicity. Finally, a dummy variable for SAGE or comparison school student was entered
on the third block. As with the first graders, this variable is coded 0 if a student is from a
comparison school and 1 if a student is from a SAGE school.
Regression Results. Results of the regression analyses are presented in Tables 3542.
When either the firstgrade pretest or the firstgrade posttest is used as the predictor variable,
39
membership in SAGE emerges as a significant predictor of student achievement on the total
score and for all subtests except reading. The magnitude of the effect of SAGE on student
achievement, as denoted by the “b” coefficient, varies depending on the CTBS subtest.
The largest effects of SAGE are found when the firstgrade mathematics pretest is used
to predict the secondgrade test. When all cases are analyzed, the goodnessoffit of the models
(as denoted by the adjusted R square statistic), ranges from .19 to .47. Most of the variance, as
was the case with the first graders, is explained by the baseline scores (either the firstgrade pretest
or the firstgrade posttest). “Family Income” and “Race” show some relatively large effects
(as denoted by the b coefficients) and these effects are usually statistically significant. Most
importantly, membership in SAGE schools has a consistently statistically significant positive
effect on the language arts subtest, the mathematics subtest, and the total score.
Table 35. SCALE SCORES OLS Regression for Language Arts: PreTest as Control
Block 1 Block 2 Block 3
Variable b t b t b t
PreTest Score .45 20.24* .43 19.22* .43 19.18*
Subsidized Lunch Eligibility 3.82 3.71* 2.22 2.09* 2.25 2.12*
African American 14.58 5.60* 14.22 5.47*
White 2.48 1.07 2.55 1.11
SAGE 6.44 3.46*
Constant 370.29 30.15* 382.62 31.39* 392.46 31.46*
Adjusted R Squared .24 .26 .27
Standard Error of Estimate 36.03 35.43 35.31
*significant at .05 level
Table 36. SCALE SCORES OLS Regression for Language Arts: PostTest as Control
Block 1 Block 2 Block 3
Variable b t b t b t
PostTest Score .49 20.08* .46 18.62* .45 18.36*
Subsidized Lunch Eligibility 4.63 4.60* 3.08 2.96* 3.14 3.02*
African American 12.97 5.12* 12.60 4.99*
White 3.18 1.41 3.27 1.46
SAGE 6.18 3.36*
Constant 328.73 22.72* 347.62 23.92* 359.60 24.10*
Adjusted R Squared .23 .25 .25
Standard Error of Estimate 36.29 35.77 35.66
*significant at .05 level
40
Table 37. SCALE SCORES OLS Regression for Reading: PreTest as Control
Block 1 Block 2 Block 3
Variable b t b t b t
PreTest Score .37 15.98* .35 14.77* .35 14.77*
Subsidized Lunch Eligibility 6.46 6.89* 5.14 5.27* 5.14 5.28*
African American 6.02 2.50* 5.84 2.42*
White 5.19 2.44* 5.22 2.45*
SAGE 3.18 1.85
Constant 416.18 32.96* 426.13 33.47* 430.56 33.26*
Adjusted R Squared .19 .20 .20
Standard Error of Estimate 32.93 32.68 32.65
*significant at .05 level
Table 38. SCALE SCORES OLS Regression for Reading: PostTest as Control
Block 1 Block 2 Block 3
Variable b t b t b t
PostTest .46 20.98* .44 19.95* .44 19.85*
Subsidized Lunch Eligibility 6.07 7.06* 4.55 5.08* 4.57 5.10*
African American 5.46 2.51* 5.31 2.43*
White 6.23 3.23* 6.26 3.25*
SAGE 2.42 1.53
Constant 344.12 26.28* 352.98 26.84* 357.25 26.58*
Adjusted R Squared .26 .27 .27
Standard Error of Estimate 31.08 30.76 30.75
*significant at .05 level
Table 39. SCALE SCORES OLS Regression for Mathematics: PreTest as Control
Block 1 Block 2 Block 3
Variable b t b t b t
PreTest Score .63 2.80* .58 25.04* .58 25.23*
Subsidized Lunch Eligibility 2.66 2.80* 1.21 1.26 1.22 1.27
African American 17.55 7.35* 17.34 7.32*
White 1.50 .70 1.45 .68
SAGE 7.94 4.73*
Constant 262.50 22.76* 288.48 24.88* 299.02 25.49*
Adjusted R Squared .36 .39 .40
Standard Error of Estimate 32.80 31.96 31.74
*significant at .05 level
41
Table 40. SCALE SCORES OLS Regression for Mathematics: PostTest as Control
Block 1 Block 2 Block 3
Variable b t b t b t
PostTest Score .55 28.17* .50 25.38* .50 25.14*
Subsidized Lunch Eligibility 4.78 5.36* 3.01 3.30* 3.05 3.35*
African American 15.14 6.73* 14.90 6.63*
White 4.10 2.06* 4.20 2.12*
SAGE 3.97 2.44*
Constant 274.87 25.18* 300.38 27.30* 307.76 27.01*
Adjusted R Squared .36 .39 .39
Standard Error of Estimate 32.22 31.41 31.37
*significant at .05 level
Table 41. SCALE SCORES OLS Regression for Total: PreTest as Control
Block 1 Block 2 Block 3
Variable b t b t b t
PreTest Score .65 30.92* .62 28.85* .62 28.93*
Subsidized Lunch Eligibility 2.63 3.42* 1.82 2.32* 1.84 2.36*
African American 12.04 6.19* 11.86 6.12*
White .01 .01 .04 .02
SAGE 5.21 3.80*
Constant 258.65 22.99* 276.79 24.39* 284.31 24.78*
Adjusted R Squared .42 .44 .45
Standard Error of Estimate 26.10 25.66 25.55
*significant at .05 level
Table 42. SCALE SCORES OLS Regression for Total: PostTest as Control
Block 1 Block 2 Block 3
Variable b t b t b t
PostTest Score .69 33.44* .65 31.12* .65 30.88*
Subsidized Lunch Eligibility 3.31* 4.60* 2.29 3.11* 2.33 3.16*
African American 9.75 5.42* 9.58 5.32*
White 2.40 1.51 2.47 1.56
SAGE 2.95 2.26*
Constant 204 16.93* 225.24 18.47* 231.32 18.54*
Adjusted R Squared .45 .46 .47
Standard Error of Estimate 25.31 24.91 24.87
*significant at .05 level
AfricanAmerican Students
Like the first graders, AfricanAmerican secondgrade students definitely comprise the
largest subgroup of valid test scores – roughly 21% of all SAGE students and 25% of all
comparison students. In the analyses to follow, AfricanAmerican students are first compared
across SAGE and comparison schools on the CTBS subtests and total scale score. Second,
42
AfricanAmerican students are compared to white students across SAGE and comparison
schools on the CTBS Total Scale Score.
SAGE vs. Comparison. Table 43 provides comparisons of means on the CTBS secondgrade
test, as well as change scores from the firstgrade pretest to the secondgrade test and
from the firstgrade posttest to the secondgrade test. On the secondgrade test, African
American SAGE students scored higher than comparison school students on every subtest and
on the total scale score. However, the differences between SAGE and comparison students on
the secondgrade test scores are not statistically significant. When using the firstgrade pretest
as the baseline score, statistically significant change scores are found on all scores except for
reading. However, using the firstgrade posttest as the baseline score shows no statistically
significant differences between SAGE and comparison schools.
43
Table 43. AfricanAmerican PostTest and Change Scores, by SAGE or Comparison
SCORE SAGE COMPARISON
Language Arts
Mean SecondGrade Score 591.09 587.19
Mean Change From FirstGrade
PreTest to Second Grade
72.41* 62.26
Mean Change From FirstGrade
PostTest to Second Grade
19.41 20.62
Reading
Mean SecondGrade Score 594.54 543.48
Mean Change From FirstGrade
PreTest to Second Grade
73.07 68.48
Mean Change From FirstGrade
PostTest to Second Grade
19.30 20.91
Mathematics
Mean SecondGrade Score 545.44 543.94
Mean Change From FirstGrade
PreTest to Second Grade
72.59* 61.08
Mean Change From FirstGrade
PostTest to Second Grade
16.46 20.58
Total
Mean SecondGrade Score 577.04 575.94
Mean Change From FirstGrade
PreTest to Second Grade
72.44* 65.44
Mean Change From FirstGrade
PostTest to Second Grade
17.99 21.14
*significant at .05 level
AfricanAmerican Males. Table 44 further distinguishes AfricanAmerican SAGE and
comparison school students by gender. These results show no significant genderrelated gain
differences when using either the firstgrade pretest or the firstgrade posttest as a baseline
measure.
44
Table 44. AfricanAmerican PostTest and Change Scores by Gender
COMPARISON SAGE
Male Female Male Female
Language Arts
Mean PostTest Scale Score 579.63 591.49 590.74 592.22
Change (PreTest as Baseline) 62.03 62.52 73.74 72.09
Change (PostTest as Baseline) 23.97 18.31 19.48 19.35
Reading
Mean PostTest Scale Score 584.84 596.65 590.50 598.16
Change (PreTest as Baseline) 66.23 69.74 71.13 74.34
Change (PostTest as Baseline) 25.24 17.92 13.62 24.21
Mathematics
Mean PostTest Scale Score 538.82 547.33 550.29 540.34
Change (PreTest as Baseline) 55.68 65.23 74.99 69.35
Change (PostTest as Baseline) 21.91 25.46 21.65 30.79
Total
Mean PostTest Scale Score 568.54 580.32 577.36 576.87
Change (PreTest as Baseline) 62.09 67.74 73.17 72.11
Change (PostTest as Baseline) 22.24 20.36 16.60 19.21
*significant at .05 level
AfricanAmerican and White Achievement. AfricanAmerican students scored lower
than white students on the firstgrade pretest total scale score, as shown in table 45. This result
is statistically significant for both SAGE and comparison schools, though the gap between
African Americans and whites is larger in the SAGE schools. The change from firstgrade posttest
to the secondgrade test shows that the SAGE African Americans kept pace with white
students, but did not further close the achievement gap in second grade.
Table 45. AfricanAmerican versus White Achievement on Total Scale
FirstGrade
PreTest
FirstGrade
PostTest
Second
Grade
Change From
Pre to Second
Change From
Post to Second
SAGE
African
American
504.09 559.02 577.04 72.44 17.99
White 533.21 582.56 606.81 73.61 24.50
F 142.06* 94.70* 132.69* .291 8.46*
Comparison
African
American
510.22 553.48 575.94 65.44 21.14
White 528.79 578.28 602.67 73.60 23.80
F 32.24* 61.80* 66.28* 7.46* .925
*significant at .05 level
45
Hierarchical Linear Modeling
Hierarchical linear models were used to evaluate the secondgrade achievement results
using the same series of models used to assess firstgrade results. Two sets of analyses were
done. The first, shown in Table 45 used firstgrade pretest as the initial achievement level of the
students. The second, shown in Table 46, used firstgrade posttest as the initial achievement
level of the students.
HLM Results. Tables 46 and 47 provide a summary of the effects of each of the level1
and level2 variables for each of these analyses. Level1 effects can be interpreted as the
weighted average of the within classroom effects of the level1 variables. Level2 effects can be
interpreted as the classroom effects of the level2 variables. Level1 coefficients may be thought
of as the average effect of the modeling variable on the criterion score at the individual level.
The level1 results indicate that lower SES is related to lower posttest scores and higher pretest
scores are related to higher posttest scores.
The coefficients associated with the level2 variables can be thought of as classroom
effects. For example, in the Model A total score, an increase of one student in class size resulted
in a drop of .764 points for the class average. Likewise, SAGE participation resulted in a 6.322
point gain in the class average on total score for Model B. A discussion of each model follows.
Model A. Depending on the test, an increase in class size of one person can be expected
to produce a .66 to 1.01 loss in average posttest performance. The results for all scores show
this effect to be significant. It is noteworthy that the results of Tables 41 and 42 for the Model A
are similar, indicating that the secondgrade class size has little effect on the two year gain (first
grade and second grade) versus the one year gain (second grade only).
Model B. Participation in SAGE shows no statistically significant class average
increases in all posttest scores with the exception of mathematics in the pretest condition. A
46
comparison of these results with the firstgrade only results indicates secondgrade SAGE effects
of lesser magnitude than the firstgrade effects. These results tend to support the hypothesis that
SAGE is not continuing to produce the same advantage through second grade. The advantage of
SAGE students is, however, maintained.
Model C. Combining class size and SAGE participation in a single analysis isolates the
effects that SAGE might have beyond those produced by lower class size. The results show that
once class size has been accounted for, SAGE has no significant effect on class average
performance.
Model D. Since socioeconomic status is known to have an influence on academic test
scores, a replacement for this variable was used as both a level1 and level2 predictor. The
level2 variable was the average SES for the class and estimates the effect of the overall class
SES level beyond that associated with the individual, which is accounted for in the level1
model. This model combines class SES and class size. The results indicate that class SES has a
significant effect on the class average posttest performance. The effect of a 1 point class
average gain in SES equates to between a 10 point and 13 point gain on the average posttest
score, depending on the test. SES was measured on a threepoint family income scale; thus a one
point difference on average would be quite pronounced. Class size still has a significant effect
on the posttest scores once SES has been accounted for.
Model E. When class SES and SAGE participation are entered in the same level2
model, class SES has a significant effect on class average posttest performance. However, for
the most part, SAGE has no significant effect on class average posttest performance once class
SES is controlled. In other words, the effects of SAGE participation on class average posttest
scores, beyond those produced by SES differences, are not significant on posttest scores. The
single exception is for the mathematics score across the twoyear time span. In general, the
47
SAGE effects are roughly the same as when SAGE is the only variable in the model (see Model
B), suggesting that SAGE classrooms and comparison classrooms are about equal on class SES.
Model F. This model combines SES, SAGE participation, and class size in a single
analysis. For most subtests, class SES once again has a significant effect on the class average
posttest score. Again, class size has a significant effect on the class average posttest score for
all scores except language. Finally, SAGE has no significant effect on any subtest, once class
size is accounted for.
Table 46. HLM Results for 199798 SecondGrade Students – PreTest as Initial Achievement
Source Total Reading Language Arts Mathematics
Level 1
PreTest 0.703 0.440 0.509 .662
SES 1.929 4.503 1.069 3.196
Level 2
A. Class Size 0.764* 0.630* 0.738* 1.007*
B. SAGE 6.322* 3.302* 7.354* 8.612*
C. Class Size 0.840* 0.886* 0.703 1.073*
SAGE 1.533 5.152 0.731 1.372
D. Class SES 9.129* 12.214* 7.932 6.661
Class Size 0.678* 0.523* 0.666* 0.946*
E. Class SES 10.232* 13.214* 8.944* 8.251*
SAGE 5.939 3.024 7.042 8.392*
F. Class SES 9.086* 12.018* 8.017 6.613
Class Size 0.714* 0.715* 0.593 0.979*
SAGE 0.708 3.825 1.487 0.681
*significant at .05 level
Table 47. HLM Results for 199798 SecondGrade Students – Post Test as Initial Achievement
Source Total Reading Language Arts Mathematics
Level 1
PreTest 0.750 0.506 0.551 .574
SES 2.742 4.142 4.640 4.703
Level 2
A. Class Size 0.811* 0.669* 0.732* 0.959*
B. SAGE 6.296 3.651 7.432 7.741
C. Class Size 0.916* 0.916* 0.683 1.038*
SAGE 2.177 5.043 1.004 1.622
D. Class SES 9.106* 12.888* 9.069* 5.906*
Class Size 0.726* 0.553* 0.646* 0.911*
E. Class SES 10.339* 13.836* 10.079* 7.254
SAGE 6.007 3.300 7.040 7.616
F. Class SES 9.023* 12.637* 9.172* 5.835*
Class Size 0.787* .739* 0.556 0.962*
SAGE 1.243 3.737 1.825 1.029
*significant at .05 level
48
Effects Within SAGE Classrooms
It is of some interest to examine achievement gains in light of various classroom and
teacher characteristics, teacher behaviors and student behaviors. Except for the “Teacher
Experience and Achievement” analyses, data for these questions were collected only for SAGE
firstgrade classrooms.
Teacher Experience and Achievement
Student gains (at the classroom level) were correlated with teacher experience. In those
classrooms with more than one teacher, teacher experience was averaged to obtain an appropriate
experience variable for that classroom. Correlations between teacher experience and
achievement gains ranged from .019 to .140, but none were significant for either the firstgrade
cohort or the secondgrade cohort.
Student Participation and Achievement
The student participation questionnaire factored into two scales as was noted earlier.
These scales were “active learning” and “ontask behavior.” Interest here centered on the
relationship between these variables and the classroom characteristics of class size and teacher
experience as well as achievement gains. Composite active learning and ontask variables were
formed for each classroom by averaging over the administrations and across students in each
class. (Note, there were no significant changes in either of these variables across time). These
variables were then correlated with class size, teacher experience and achievement gain in each
of the three achievement subscore areas. The ontask variable showed no significant
relationships with any of the classroom variables. For the 199697 firstgrade cohort, active
learning was significantly correlated with class size (r=.204; p<.05). For the 199798 firstgrade
cohort, active learning correlated significantly with teacher experience (r=.307; p<.05). Since
these correlations did not replicate across cohorts, little practical significance should be placed in
49
these findings.
Proximity to the Curriculum and Achievement
From the teacher questionnaire, variables associated with proximity to the curriculum
were constructed. The only significant correlation was the language arts proximity score with
reading gain (r=.270) for the 199798 firstgrade cohort. Again, since these results did not
replicate across cohorts, little practical significance should be placed in this finding.
Class Organization and Achievement
SAGE students are organized into different types of classrooms as discussed earlier in the
report. These classrooms include 15:1 Regular classrooms, 30:2 2Team Teaching classrooms,
15:1 SharedSpace classrooms, and 30:2 Floating Teacher classrooms. Available data consisted
of fiftynine 15:1 classrooms, and thirtyone 30:2 classrooms. Neither the 15:1 Shared Space
nor the 30:2 Floating Teacher class organization contained enough classrooms to analyze each of
these four types separately. This analysis uses Hierarchical Linear Modeling to make this
comparison. The HLM results show no statistically significant relationship between type of
classroom organization and achievement on any of the subtests or the total score.
50
ANALYSES OF SAGE TEACHERS, CLASSROOMS, AND SCHOOLS 199798
