**Table 19. **SCALE SCORES Regression for Language Arts for 1997-98 First-Grade Students
**Block 1 Block 2 Block 3**
**Variable **b t b t b t
**Pre-Test Score **.56 24.87* .55 24.18* .55 23.94*
**Days Absent **-.37 -3.31* -.35 -3.06* -.33 -2.90*
**Subsidized Lunch Eligibility **-2.63 -2.46* -1.95 -1.72 -2.06 -1.82
**African American **2.78 1.02 2.50 .92
**White **5.60 2.37* 5.80 2.47*
**SAGE **7.25 3.64*
**Constant **292.26 23.33* 291.59 23.06* 304.36 23.28*
**Adjusted R Squared **.33 .33 .34
**Standard Error of Estimate **36.49 36.45 36.30
*significant at .05 level
**Table 20. **SCALE SCORES Regression for Reading for 1997-98 First-Grade Students
**Block 1 Block 2 Block 3**
**Variable **b t b t b t
**Pre-Test Score **.56 20.88* .55 20.33* .54 20.20*
**Days Absent **-.21 -1.96* -.18 -1.60 -.16 -1.43
**Subsidized Lunch Eligibility **-3.01 -2.85* -2.06 -1.85 -2.15 -1.94*
**African American **3.07 1.15 2.80 1.05
**White **7.21 3.14* 7.37 3.22*
**SAGE **6.98 3.59*
**Constant **285.03 19.25* 284.30 19.08* 296.11 19.48*
**Adjusted R Squared **.26 .27 .27
**Standard Error of Estimate **35.73 35.64 35.50
*significant at .05 level
**Table 21. **SCALE SCORES Regression for Mathematics for 1997-98 First-Grade Students
**Block 1 Block 2 Block 3**
**Variable **b t b t b t
**Pre-Test Score **.62 30.65* .62 29.21 .61 29.13*
**Days Absent **-.21 -1.96* -.18 -1.60 -2.15 -1.94*
**Subsidized Lunch Eligibility **-2.75 -3.08* -2.44 -2.62* -2.56 -2.76*
**African American **.1.66 .74 1.44 .64
**White **3.05 1.56 3.24 1.66
**SAGE **7.06 4.31*
**Constant **235.05 22.32* 235.69 21.88* 247.05 22.40
**Adjusted R Squared **.43 .43 .44
**Standard Error of Estimate **30.05 30.05 29.88
*significant at .05 level
28
**Table 22. **SCALE SCORES Regression for Total for 1997-98 First-Grade Students
**Block 1 Block 2 Block 3**
**Variable **b t b t b t
**Pre-Test Score **.77 38.80* .77 37.58* .76 37.46*
**Days Absent **-.34 -3.08* -2.44 -2.62* -2.56 -2.76*
**Subsidized Lunch Eligibility **-.63 -.86 -.53 -.68 -.62 -.81
**African American **3.91 2.1* 3.72 2.01*
**White **3.00 1.86 3.20 2.00*
**SAGE **6.33 4.68*
**Constant **167.12 15.51* 165.11 15.04* 176.57 15.80*
**Adjusted R Squared **.54 .54 .55
**Standard Error of Estimate **24.53 24.51 24.34
*significant at .05 level
African-American Students
Among minority students in SAGE and comparison schools, African Americans clearly
comprise the largest group of valid test scores – roughly 25% percent of SAGE students and 28%
percent of comparison school students. In the analyses to follow, African-American students are
first compared across SAGE and comparison schools on CTBS sub-test and total scale scores.
Second, African-American students are compared to white students across SAGE and
comparison schools on CTBS total scale scores.
SAGE vs. Comparison. Table 23 provides comparisons of means on the CTBS post-test,
and change scores from pre-test to post-test. On the post-test, African-American SAGE students
scored higher than African-American comparison school students on every sub-test and on total
scale score. The differences between SAGE and comparison schools on post-test scores are all
statistically significant. In addition, the differences between SAGE and comparison schools on
mean change scores from pre-test to post-test scores are statistically significant. In other words,
African-American SAGE students scored lower on the CTBS pre-test than African-American
comparison school students, but made significantly larger gains than comparison school students
from pre- to post-test, and surpassed African-American comparison school students on the posttest.
29
**Table 23. **African American Post-Test and Change Scores, by SAGE or Comparison School for
1997-98 First-Grade Students
**SCALE SCORE SAGE COMPARISON F**
**Language Arts**
Mean Post-Test 572.80 558.32 11.59*
Mean Change Pre to Post 56.05 38.27 16.41*
**Reading**
Mean Post-Test 573.82 554.11 25.31*
Mean Change Pre to Post 50.55 25.79 31.67*
**Mathematics**
Mean Post-Test 522.01 506.22 20.74*
Mean Change Pre to Post 49.06 27.50 41.99*
**Total**
Mean Post Test 556.72 539.73 25.48*
Mean Change, Pre to Post 52.15 32.78 43.51*
*significant at .05 level
African-American Males. Concern over the minority achievement gap on standardized
tests has occasionally focused on African-American male students. Table 24 further
distinguishes African-American SAGE and comparison school students by gender. The 1996-97
results showed that African-American male SAGE students attained comparable or higher
change scores from pre-test to post-test when compared to African-American female SAGE
students. The 1997-98 results show that African-American male SAGE students attained
comparable or higher change scores from pre-test to post-test on the language arts sub-test, the
mathematics sub-test, and the total score. However, none of these results is statistically
significant.
**Table 24. **African-American Post-Test and Change Scores by Gender
**COMPARISON SAGE**
Male Female Male Female
**Language Arts**
Mean Post-Test Scale Score 554.74 562.18 570.99 574.27
Mean Change Pre to Post 33.96 42.69 58.64 53.65
**Reading**
Mean Post-Test Scale Score 556.06 552.25 570.40 576.77
Mean Change Pre to Post 35.75 15.19 50.37 50.49
**Mathematics**
Mean Post-Test Scale Score 511.48 501.24 522.82 520.93
Mean Change Pre to Post 28.90 26.08 53.37 45.01
**Total**
Mean Post-Test Scale Score 540.94 538.70 554.85 558.23
Mean Change Pre to Post 34.78 30.65 53.47 50.93
*significant at .05 level
30
African-American and White Achievement. African-American students scored
significantly lower than white students on the CTBS pre-test total scale score, as shown in table
25. This result holds for both SAGE and comparison schools, though the gap between African
Americans and whites is larger in SAGE schools. When all cases are analyzed, African-
American SAGE students achieved greater gains on the CTBS total scale score than white SAGE
students from pre- to post-test, closing the achievement gap (though the gap remains statistically
significant). In contrast, African Americans in comparison schools achieved lesser gains and in
the comparison schools the achievement gap widened.
**Table 25. **African-American and White Achievement in SAGE and Comparison Schools
on Total Scale Scores for 1997-98 First-Grade Students
**PRE-TEST POST-TEST CHANGE**
**SAGE SCHOOLS**
**African American **502.79 556.72 52.15
White 531.38 579.94 45.99
F 170.61* 96.09* 10.50*
**COMPARISON SCHOOLS**
**African American **510.07 539.73 32.78
White 528.60 569.02 41.14
F 52.21 90.15* 10.72*
*significant at .05 level
Hierarchical Linear Modeling
Explanation. Many social science research analyses involve hierarchical data structures.
Hierarchical data structures are those in which individual units are nested within larger units, the
latter being the unit of interest. The SAGE data are a prime example: students are nested within
classrooms, and it is the classroom effect that is of particular interest to the SAGE project.
Hierarchical data structures pose special analytical challenges in that data analysis at the
individual level may result in a biased impression of the effect of the nesting unit (in the SAGE
case, the classroom). At the origin of this problem is the fact that different classrooms often
contain different numbers of students, thus those classrooms that contain greater numbers of
31
students have greater influence over the results of analyses done at the individual level. An
analytical approach known as “hierarchical linear modeling” (Bryk & Raudenbush, 1992) was
specifically designed to accommodate these types of data structures. Essentially hierarchical
linear modeling (HLM) estimates individual effects by analyzing data within each class and then
provides a weighted average of these effects. The effects of the class are then estimated as if all
classes contained the same number of students. HLM was used with the SAGE data to provide
an alternative and less biased account of the effects of SAGE experience on test scores. In these
models, variables associated with individual students are referred to as level-1 variables and
those associated with the classrooms are referred to as level-2 variables.
HLM Analyses. Analyses were conducted for each of the relevant criterion post-test
scores: reading, mathematics, language arts, and total. For all analyses, the level-1 variables
were pre-test scores and socioeconomic status (SES) measured as eligibility for subsidized lunch.
The post-test scores were adjusted for these two variables at the individual level, therefore the
effects may be thought of as being statistically independent of the effects of these variables. A
number of different level-2 models, each containing different level-2 variables, was specified for
each variable of interest. It is important to note that the “class size” variable used in these
analyses measures the student-teacher ratio.
HLM Results. Table 26 provides a summary of the effects of each of the level-1 and
level-2 variables for each of these analyses. Level-1 effects can be interpreted as the weighted
average of the within-classroom effects of the level-1 variables. Level-2 effects can be
interpreted as the classroom effects of the level-2 variables. Level-1 coefficients may be thought
of as the average effect of the modeling variable on the criterion score at the individual level.
The level-1 results indicate that lower SES is related to lower post-test scores and higher pre-test
scores are related to higher post-test scores.
32
The coefficients associated with the level-2 variables can be thought of as classroom
effects. For example, in the Model A total score, an increase of one student in class size resulted
in a drop of .828 points for the class average. Likewise, SAGE participation resulted in a 8.909
point gain in the class average on total score for Model B. A discussion of each model follows:
**Model A. ***Class Size. *These models examined the effect of class size on the adjusted
criterion score. Class size equals the number of students divided by the number of teachers.
Depending on the test, an increase in class size of one person can be expected to produce a .29 to
1.12 loss in average post-test performance. The results for all scores show this effect to be
significant.
**Model B. ***SAGE*. These models examined the effect of SAGE participation on the
adjusted criterion score. Participation in SAGE shows statistically significant class average
increases in all post-test scores as well. These score increases range from 7 points (reading) to
13 points (mathematics).
**Model C. ***Class Size, SAGE*. These models examined the effect of SAGE participation
on the adjusted criterion score after the classrooms were class size adjusted, viewed as the effect
of SAGE participation beyond the class size effect. Combining class size and SAGE
participation in a single analysis isolates the effects that SAGE might have beyond those
produced by lower class size. The results show that once class size has been accounted for,
SAGE has no significant effect on class average performance. This may suggest that the other
SAGE interventions (i.e., rigorous curriculum, lighted school house, and staff development) are
not having a significant impact on achievement in SAGE classrooms.
**Model D. ***Class SES, Class Size*. These models examined the effect of class size on the
adjusted criterion score after the classrooms were SES adjusted, viewed as the effect of class size
once the effects of the classroom SES are removed. Since socioeconomic status is known to
33
have an influence on academic test scores, a replacement for this variable was used as both a
level-1 and level-2 predictor. The level-2 variable was the average SES for the class and
estimates the effect of the overall class SES level beyond that associated with the individual,
which is accounted for in the level-1 model. This model combines class SES and class size. The
results indicate that class SES has a significant effect on the class average post-test performance.
The effect of a 1 point class average gain in SES equates to between a 10 point and 13 point gain
on the average post-test score, depending on the test. SES was measured on a three-point family
income scale, thus a one point difference on average would be quite pronounced. Class size still
has a significant effect on the post-test scores once SES has been accounted for.
**Model E. ***Class SES, SAGE*. These models examined the effect of SAGE participation
on the adjusted criterion score after the classrooms were SES adjusted; viewed as the effect of
SAGE participation once the effects of classroom SES are removed. When class SES and SAGE
participation are entered in the same level-2 model, class SES has a significant effect on class
average post-test performance. In addition, SAGE has a significant effect on class average posttest
performance. In other words, the effects of SAGE participation on class average post-test
scores, beyond those produced by SES differences, are significant on all post-test scores. In
general, these effects are roughly the same as when SAGE is the only variable in the model (see
model B), suggesting that SAGE classrooms and control classrooms are about equal on class
SES.
**Model F. ***Class SES, Class Size, SAGE*. These models examined the effect of SAGE
participation on the adjusted criterion score after the classrooms were adjusted for class size and
SES; viewed as the effect of SAGE participation beyond the class size and SES effects. This
model combines SES, SAGE participation, and class size in a single analysis. For all sub-tests,
class SES once again has a significant effect on the class average post-test score. Class size has
34
no significant effect on the class average post-test score. Finally, SAGE had significant effects
only on the mathematics sub-test.
**Table 26. **HLM Results for 1997-98 First-Grade Students
**Source Total Reading Language Arts Mathematics**
**Level 1**
Pre-Test 0.870 0.627 0.625 0.712
SES -0.784 -3.733 -1.612 -3.202
**Level 2**
A. Class Size -0.828* -0.289* -0.899* -1.115*
B. SAGE 8.909* 7.009* 10.148* 13.090*
C. Class Size -0.647 -0.734 -0.639 -0.722
SAGE 2.990 0.195 4.228 6.409
D. Class SES -12.959* -10.410* -12.971* -13.389*
Class Size 0.599* -0.574* -0.698* 0.883*
E. Class SES -14.707* -12.215* -15.201* -16.298*
SAGE 9.354* 7.320* 10.661* 13.428*
F. Class SES -14.883* -11.446* -15.168* -16.211*
Class Size -0.015 -.252 -0.011 -0.027
SAGE 9.074 4.957 10.556 13.172*
*significant at .05 level
Second-Grade Results 1997-98
Descriptive Statistics
Valid Test Scores. Analyses were conducted to assess the impact of SAGE on the 1997-
98 second-grade CTBS Complete Battery, Terra Nova Level 13 post-test results. There were
1702 persisting students (i.e., students present in both the 1996-97 SAGE and comparison firstgrade
classrooms and in the 1997-98 SAGE and comparison second-grade classrooms), while
there were 482 new second-grade students (students who were not in the program last year).
However, second-grade post-test results are compared to the first-grade pre-test, as well as first
grade post-test. Therefore, only those students who took both the first-grade pre-test and posttest,
as well as the second-grade post-test, were used in the 1997-98 second-grade analysis. As
would be expected, the number of second-grade students having all three valid test scores was
substantially less than the total number of students. The number of valid test scores for the
Fal1996 first-grade pre-test, the Spring 1997 first-grade post-test, and the Spring 1998 second-
35
grade post-test are presented in Table 27.
**Table 27. **Number of 1997-98 Second-Grade Students with Valid Test Scores
**SAGE Comparison Total**
**Fall 1996 First-**
**Grade Pre-test**
Reading 1033 562 1595
Language Arts 1033 562 1595
Mathematics 1020 559 1579
Total 1008 448 1456
**Spring 1997 First-**
**Grade Post-Test**
Reading 1011 545 1556
Language Arts 1011 545 1556
Mathematics 1007 538 1545
Total 1001 534 1535
**Spring 1997**
**Second Grade**
Reading 1037 561 1598
Language Arts 1037 562 1599
Mathematics 1043 559 1602
Total 1033 549 1582
Pre-Test (Baseline) Results. Both the first-grade pre-test and the first-grade post-test
served as a baseline. Table 28 provides descriptive statistics on the scale scores from the firstgrade
pre-test as well as the first-grade post-test.
**Table 28. **Descriptive Statistics on CTBS First-Grade Pre-Test and Post-Test (SAGE and
Comparison)
**FIRST-GRADE PRE-TEST FIRST-GRADE POST-TEST**
**SCALE**
**SCORES**
**NORMAL CURVE**
**EQUIVALENT**
**SCALE**
**SCORES**
**NORMAL CURVE**
**EQUIVALENT**
Reading 535.20 36.83 45.31 19.78 584.17 35.65 54.44 18.50
Language
Arts
532.70 42.03 45.03 20.74 583.05 37.78 54.53 17.89
Mathematics 494.55 38.27 44.11 18.05 546.59 41.56 55.88 20.41
Total 521.03 33.34 44.33 18.28 571.43 31.94 55.65 17.81
Difference of Means Test. The results from the difference of means tests between SAGE
and comparison student scale scores from the Fall 1996 first-grade pre-test and Spring 1997 firstgrade
post-test are reported in Tables 29-32. The differences between SAGE schools and
36
comparison schools on the first-grade pre-test are not found to be statistically significant at the
.05 level. Therefore, any differences between the first-grade pre-test and the second-grade test
can be more confidently attributed to the student-teacher ratio of 15:1 in the SAGE classrooms.
The differences between SAGE schools and comparison schools on the first-grade post-test are
found to be significant on the total score and on all sub-scores. Therefore, any conclusions
discussed regarding second-grade results must take into account the effects of the SAGE
program while these students were in first grade.
**Table 29. **Differences of Means Test on First-Grade Pre-Test and Post-Test:
Language Arts Scale Scores
**FIRST-GRADE**
**PRE-TEST**
**FIRST-GRADE**
**POST-TEST**
**N MEAN STANDARD**
**DEVIATION**
**N MEAN STANDARD**
**DEVIATION**
Comparison Schools 562 530.69 43.09 545 579.01 39.75
SAGE Schools 1033 533.80 41.42 1011 586.07* 36.06
*significant at .05 level
**Table 30. **Differences of Means Test on First-Grade Pre-Test and Post-Test: Reading Scale
Scores
**FIRST-GRADE**
**PRE-TEST**
**FIRST-GRADE**
**POST-TEST**
**N MEAN STANDARD**
**DEVIATION**
**N MEAN STANDARD**
**DEVIATION**
Comparison Schools 562 534.62 38.77 545 582.01 36.50
SAGE Schools 1033 535.52 35.75 1011 586.07* 36.06
*significant at .05 level
**Table 31. **Differences of Means Test on First-Grade Pre-Test and Post-Test: Mathematics Scale
Scores
**FIRST-GRADE**
**PRE-TEST**
**FIRST-GRADE**
**POST-TEST**
**N MEAN STANDARD**
**DEVIATION**
**N MEAN STANDARD**
**DEVIATION**
Comparison Schools 559 493.70 38.26 538 541.88 40.75
SAGE Schools 1020 495.01 38.29 1007 550.67* 41.17
*significant at .05 level
37
**Table 32. **Differences of Means Test on First-Grade Pre-Test and Post-Test: Total Scale Score
**FIRST-GRADE**
**PRE-TEST**
**FIRST-GRADE**
**POST-TEST**
**N MEAN STANDARD**
**DEVIATION**
**N MEAN STANDARD**
**DEVIATION**
Comparison Schools 548 519.96 33.59 534 567.64 32.29
SAGE Schools 1008 521.61 33.20 1001 574.72* 30.97
*significant at .05 level
As noted above, student populations varied in SAGE and comparison schools due to
withdrawals and within-year enrollments. The post-test results are based only on those second
graders who remained in class the entire 1996-97 first grade and 1997-98 second grade school
years.
Results of the difference of means test between SAGE and comparison schools on
the second-grade post-test can be seen in Table 33. Table 34 shows that when the first-grade
pre-test is used as the baseline score, significant results are found on the language arts sub-scale,
mathematics sub-scale, and total score. However, when the first-grade post-test is used as the
baseline score, no significant results are found. This suggests that the statistically significant
positive effects of SAGE occurred in the first grade. These positive effects were maintained, but
did not significantly increase in second grade.
**Table 33. **Difference of Means Test – Second-Grade Scale Scores
**SAGE Schools Comparison Schools**
**N MEAN STANDARD**
**DEVIATION**
**N MEAN STANDARD**
**DEVIATION**
Language Arts 1037 610.91* 41.10 562 602.70 41.38
Reading 1037 608.17 36.11 561 604.63 37.07
Mathematics 1043 572.11* 41.69 559 564.36 39.10
Total 1033 597.14* 34.29 549 591.25 34.10
*significant at .05 level
The largest gain in SAGE student scores from first-grade pre-test to the second-grade
post-test was on the mathematics sub-test, as shown in Table 34. The smallest relative gain for
38
SAGE students from pre-test to post-test was on the reading sub-scale; this gain was not
statistically significant.
**Share with your friends:** |