1
1997-98 EVALUATION RESULTS OF
THE STUDENT ACHIEVEMENT GUARANTEE
IN EDUCATION (SAGE) PROGRAM
Submitted by the SAGE Evaluation Team
School of Education
University of Wisconsin–Milwaukee
Alex Molnar
Philip Smith
John Zahorik
Research Assistants
Lee Breese
Karen Ehrle
Anke Halbach
Amanda Palmer
Alan Silverman
Project Administrator
William Harvey
DECEMBER 1998
For further information, contact Alex Molnar, School of Education
University of Wisconsin–Milwaukee, P.O. Box 413, Milwaukee, WI 53201, (414) 229-2220.
This document is available on the SAGE Website:
http://www.uwm.edu/SOE/centers&projects/sage/
2
TABLE OF CONTENTS
INTRODUCTION
SAGE Program................................. ........... .......... ........... ................................ 3
Goals of SAGE Evaluation ............... ........... .......... ........... ................................ 4
Class Size Research Background ...... ........... .......... ........... ................................ 5
Summary of 1996-97 Findings ......... ........... .......... ........... ................................ 7
EVALUATION DESIGN
Descriptions and Definitions............. ........... .......... ........... ............................... 11
Data Collection Instruments ............. ........... .......... ........... ............................... 14
ANALYSES OF STUDENT ACHIEVEMENT OUTCOMES 1997-98
Methods Introduction ....................... ........... .......... ........... ............................... 18
SAGE School/Classroom vs. Comparison School/Classroom Analyses ............... 21
Effects Within SAGE Schools/Classrooms ... .......... ........... ............................... 48
ANALYSES OF SAGE TEACHERS, CLASSROOMS, AND SCHOOLS 1997-98
Teacher and Classroom Analysis ...... ........... .......... ........... ............................... 50
Schools............................................. ........... .......... ........... ............................... 72
DISCUSSION: MAJOR FINDINGS, LIMITATIONS, AND FUTURE SAGE REPORTS
Major Findings................................. ........... .......... ........... ............................... 81
Limitations ....................................... ........... .......... ........... ............................... 83
Future SAGE Evaluation Reports ..... ........... .......... ........... ............................... 85
3
INTRODUCTION
SAGE Program
The Student Achievement Guarantee in Education (SAGE) evaluation is being conducted
under contract with the Department of Public Instruction by the School of Education at the
University of Wisconsin–Milwaukee. The purpose of the SAGE evaluation is to determine the
effectiveness of the Student Achievement Guarantee in Education (SAGE) program in promoting
academic achievement of students in kindergarten through third-grade classrooms in schools
serving low-income children.
The SAGE statute [s. 118.43] requires participating schools to (1) reduce class size to 15
in kindergarten and grade one in 1996–97, grades kindergarten through two in 1997–98, and
grades kindergarten through three in 1998–99 to 2000–2001; (2) stay open from early in the
morning to late in the day and collaborate with community organizations to provide educational,
recreational, community, and social services (i.e., the "lighted schoolhouse"); (3) provide a
rigorous academic curriculum to improve academic achievement; and (4) establish staff
development and accountability mechanisms.
The SAGE evaluation involves the 30 schools in 21 school districts that launched the
SAGE program in 1996-97 in kindergarten and first grade. Second grade was added in 1997-98,
and third grade in 1998-99. The SAGE evaluation compares SAGE schools to a group of 14-16
non-SAGE comparison schools located in SAGE districts. The results of the 1996-97 and 1997-
98 evaluations are consistent with Tennessee’s Student Teacher Achievement Ratio (STAR)
Project (1985-1989), the largest and best-controlled study on class size reduction to this point.
However, it is worth noting two significant design differences. First, the Tennessee STAR
Project used a true experimental design. The SAGE project uses a quasi-experimental design.
4
This has two implications. The SAGE project evaluation uses naturally occurring classrooms
(the most realistic conditions) while STAR employed random assignment of students to
classroom types which were held constant for the duration of the study. Secondly, the SAGE
evaluation uses a control or comparison group for the purpose of assessing the impact of SAGE
interventions.
Goals of SAGE Evaluation
The SAGE evaluation is intended to determine the impact that the four interventions of
the SAGE program have on student achievement. To ascertain and to explain this impact, the
evaluation addresses the following questions:
SAGE vs. Comparison School – Achievement Outcome Questions
1. What are the achievement levels of SAGE classrooms compared to achievement levels of
classrooms in selected comparison schools?
2. Does attendance in a SAGE classroom have a differential impact on the achievement of
minority students and white students?
3. Do different types of SAGE classrooms (e.g. one teacher with 15 students vs. two teachers
with 30 students) have different impacts on student achievement?
4. Does the impact on achievement of participation in a SAGE classroom change from year to
year as students progress from first through third grade?
SAGE Schools – Classroom and School Questions
1. What are the instructional characteristics of SAGE classrooms?
2. How are SAGE classrooms organized?
3. Does the type and extent of student participation in SAGE classrooms correlate with
achievement outcomes?
4. Does the style of teaching in SAGE classrooms correlate with achievement outcomes?
5. Does the degree of congruence between SAGE classroom curricula and national professional
curriculum standards in reading/language arts and mathematics correlate with the
achievement levels in SAGE classrooms?
6. Does participation in the SAGE program result in an increase in the number or change in the
type of school and/or community activities housed in the school before and after school
hours?
7. What is the structure and focus of professional development activities in SAGE schools?
8. Does the number of years of teaching experience of SAGE teachers correlate with student
achievement?
5
Class Size Research Background
The principal SAGE intervention is a reduction in class size to 15:1 in kindergarten
through third-grade classrooms. Class size reduction in the early elementary grades has been an
increasingly popular policy in recent years. Class size reduction has been credited with more
learning opportunities for students, increased opportunities for teachers to meet children’s
individual needs, and less time spent on discipline problems. Parents and teachers like the idea
and policymakers are embracing it. Several states, among them California, Florida, Indiana,
Nevada, Tennessee, and Wisconsin, have launched class size reduction efforts (Pardini, 1998;
Viadero, 1998).
Decades of research on class size reduction have shown small achievement gains for
students when, for example, class size was reduced from 25 to 20 students. In general, though,
reductions of just a few students per class do not seem to significantly raise academic
achievement. However, in the late seventies, an analysis by Glass suggested that larger
reductions produced greater achievement gains (Glass, 1978; Pate-Bain, Achilles, Boyd-
Zaharias, & McKenna, 1992).
A statewide experiment in Tennessee, the largest and best-controlled study on class size
reduction to this point, assigned kindergarten students on a random basis to classes of 15, 25 with
an aide, or 25 with no aide. The same configurations were maintained through third grade.
Tennessee’s Project STAR (Student/Teacher Achievement Ratio) analyzed student achievement
in relation to class size over a four-year period (1985-1989). The project included 17 inner-city,
16 suburban, 8 urban, and 39 rural schools. Findings showed that students in the smaller classes
scored higher on the Stanford Achievement Test and on the Basic Skills First (BSF) Test in all
four years (K-3) and in all locations. The greatest gains on the Stanford Test were made by
6
inner-city small classes. While all students benefited, disadvantaged minority students seemed to
benefit more from smaller class sizes than their peers (Pate-Bain, Achilles, Boyd-Zaharias, &
McKenna, 1992).
Studies such as STAR and SAGE can provide crucial information for policymakers. For
example, a review of the research literature conducted by Bingham (1993) on white-minority
achievement gap reduction and small class size, which included the STAR Project, suggests that
small class sizes in the early grades represent an effective strategy to reduce the achievement
gap. Bingham proposes that class size reduction may offer an early intervention strategy that
serves to prevent rather than to reduce the achievement gap between white and minority students.
Wenglinsky (1997) studied the relationship between spending and student achievement
by analyzing data from three separate sources: The National Assessment of Educational
Progress, the Common Core of Data, and the Teacher’s Cost Index of the National Center for
Education Statistics. Wenglinsky’s research suggests that increased spending to reduce class
size has a direct positive effect on mathematics achievement for fourth grade students. Further
support for small classes in lower elementary grades is produced by the Lasting Benefits Study
(LBS). The LBS tracked students who participated in Project STAR in order to determine
whether achievement advantages of students from small classes were maintained after students
returned to regular-sized classes in fourth grade. Data from 1990-1994 indicate that students
who were originally in smaller classes continued to perform better than their peers from classes
of 25 with or without a teacher’s aide (Mosteller, 1995).
In Wisconsin, most class sizes exceed the class size standards set by the STAR Project.
A study completed by Allen (1997) of K-6 class sizes in Wisconsin’s public schools reported
that 92% of Wisconsin’s kindergarten classes exceed the lower class size standard established by
7
Project STAR of 15 or fewer students per teacher. Twenty-seven percent of Wisconsin
kindergarten teachers reported class sizes that exceeded 25 students per teacher.
Summary of 1996-97 Findings
Achievement Outcome Findings 1996-97
To measure academic achievement, first grade students in SAGE schools and in a group
of comparison schools were tested in October 1996, and again in May 1997, using the
Comprehensive Test of Basic Skills (CTBS) Complete Battery, Terra Nova edition, Level 10.
After one year, students in SAGE first-grade classrooms scored higher on the CTBS than
first-grade students in comparison schools. As a group, SAGE students scored significantly
higher on the post-test in reading, language arts, and mathematics sub-tests of the CTBS. The
total score of SAGE students was also significantly higher than the total score of comparison
group students. The achievement advantage associated with participation in the SAGE program
was revealed both in the analysis of individual student scores and in the analysis of averaged
classroom scores.
At the individual level of analysis, after controlling for pre-test scores, income,
absenteeism, and race and ethnicity, SAGE first-grade students scored higher than comparison
school first-grade students on the CTBS post-test in reading, language arts, mathematics and
total score. The results were statistically significant for all but the reading scores. On averaged
classroom scores, the post-test performance of SAGE first-grade students was 4 scale score
points higher in language arts, 4.3 scale score points higher in reading, 4.6 scale score points
higher in mathematics and 4.6 scale score points higher in the total test score than comparison
school students. Each of these findings was statistically significant.
After adjusting for individual pre-test results, socioeconomic status (SES) as defined by
eligibility for subsidized lunch, and student attendance, participation in SAGE shows a
8
statistically significant advantage of 6.4 scale score points in the total score and 8.1 scale score
points on the mathematics sub-test.
The data on the average performance of students in SAGE classrooms suggest that the
lower student-teacher ratio in SAGE classrooms mitigates the negative achievement
consequences of poverty. SAGE classrooms achieved at a higher level than comparison school
classrooms despite the fact that, as a group, SAGE classrooms enrolled more students who were
eligible for subsidized lunch. Furthermore, after adjusting for individual pre-test results and SES
as defined by lunch status and student attendance, the post-test scale score advantage increased
to 9.8 for SAGE first-grade classrooms. The advantage was 7.1 on the reading sub-test, 9.0 on
the language arts sub-test, and 12.3 on the mathematics sub-test. These results were all
statistically significant.
School and Classroom Findings 1996-97
To more fully understand the SAGE program, it is important to understand how SAGE
schools structured classrooms and implemented a reduced student-teacher ratio, rigorous
curriculum, staff development, and lighted schoolhouse. Together, that information provides a
within SAGE school and classroom data description of life in SAGE classrooms and schools.
Classroom Level Findings 1996-97
Data from 1996-97 suggested that after one year of the SAGE program classroom
discipline problems are greatly reduced, and when classroom management was needed, it was
overwhelmingly positive. The direct benefit of having to spend less time managing the class was
increased instructional time, i.e., actual time spent on teaching. Further, the increased
instructional time available to teachers was used to attend to the learning needs of individual
students.
The type of instruction that students encountered in SAGE classrooms was
predominantly teacher centered. Listening, practicing, receiving help, and answering accounted
for between 50 to 75 percent of the teaching-learning that occurred. Although teachers indicate
that their use of more student-centered activities such as creating, manipulating, and problem
9
solving increased because of reduced class size, student-centered teaching played only a
supplemental role in most SAGE classrooms.
Several anticipated outcomes did not emerge from a composite of the interviews,
observations, logs, and questionnaires in 1996-97. While all teachers reported some changes in
their teaching, a large swing to student-centered teaching, a change that some might expect as a
result of reduced class size, was not observed. Lastly, a clear difference in teaching and learning
among the four main types of SAGE classrooms was not apparent.
School Level Findings 1996-97
The Teacher Questionnaire and Principal Interviews, both completed in May 1997, were
the sources of data regarding rigorous curriculum implementation. The Teacher Questionnaire
contained a section on classroom curriculum designed to determine the congruence of SAGE
classroom curricula with professional curriculum standards. First-grade and mixed-grade teacher
responses indicated that their reading/language arts curricula were more congruent with
professional standards than SAGE kindergarten curricula. Teacher responses suggested no
important differences in the degree to which their curricula were congruent with professional
standards in the area of mathematics. Principal responses to curriculum-related questions
suggested that a rigorous curriculum included basic skills, problem solving, and higher-level
thinking. Only a handful of principals seemed to believe that the curriculum of their school was
rigorous. However, most SAGE principals regarded parts of their curriculum as strong.
A section of the Teacher Questionnaire contained staff development questions. Teachers
are asked about their individual level of professional development as well as the extent to which
their school district provided staff development programs. Despite contractual requirements,
roughly 60 percent responded that they had no "personal, formal, written development plan."
Twenty-one percent rated their school district’s staff development program at the “initialization”
phase, about 66 percent rated it at the “implementation” stage, and about 9 percent felt the
district was “institutionalizing” the staff development program.
10
Data regarding implementation of lighted schoolhouse activities for 1996-97 were
obtained from the Principal Interviews and year-end reports required by DPI. In addition, data
regarding lighted schoolhouse activities existing prior to schools’ participation in SAGE were
obtained from the Baseline Data Questionnaire administered in May 1996 and the school
contracts completed for DPI prior to enrollment in the SAGE program. Principal Interview data
suggested that because of SAGE schools took responsibility for the conception and operation of
the lighted schoolhouse activities. However, the schools tended not to focus heavily on their
lighted schoolhouse activities in the first year of SAGE implementation.
11
EVALUATION DESIGN .
Descriptions and Definitions
Schools
During 1997–98, the SAGE program was implemented in 30 schools located in 21 school
districts throughout the state, as shown in Table 1. In addition, the SAGE program consisted of
14 Comparison schools located in 7 school districts.
Table 1. SAGE Schools 1997-98
SAGE DISTRICTS AND SCHOOLS
DISTRICT
School
DISTRICT
School
ADAMS-FRIENDSHIP AREA
Adams Elementary
MENOMONEE INDIAN
Keshena Primary
BELOIT
Robinson Elementary
MENOMONEE AREA
River Heights Elementary
CUDAHY
Parkview Elementary
GILMAN
Gilman Elementary
GLIDDEN
Glidden Elementary
GREEN BAY AREA
Jefferson Elementary
MILWAUKEE PUBLIC SCHOOLS
Carleton Elementary
Fairview Elementary
Longfellow Elementary
Maple Tree Elementary
Maryland Avenue Elementary
Sherman Elementary
Wisconsin Conservatory of Lifelong Learning
JANESVILLE
Wilson Elementary
PRENTICE
Ogema Elementary
Tripoli Elementary
KENOSHA
Durkee Elementary
SIREN
Siren Elementary
LAC DU FLAMBEAU #1
Lac Du Flambeau Elementary
SOUTH SHORE
South Shore Elementary (Port Wing)
LACROSSE
Franklin Elementary
Hamilton Elementary
SUPERIOR
Blaine Elementary
Cooper Elementary
LAONA
Robinson Elementary
SURING
Mountain Elementary
MADISON METROPOLITAN
Glendale Elementary
WEBSTER
Webster Elementary
12
Students
In 1997-98, the SAGE evaluation involved 7,161 active students in 117 kindergarten, 118
first-grade, and 113 second-grade classrooms. The gender, race, and other characteristics of
students in SAGE schools are displayed in Table 2:
Table 2. Characteristics of SAGE Students 1996-97 and 1997-98
Characteristic Percent of Students
1996-97
Percent of Students
1997-98
Gender
Female 48.6 49.3
Male 51.4 50.7
Race/Ethnicity
African American 24.8 24.3
Asian 5.7 5.3
Hispanic 6.6 7.8
Native American 11.7 7.9
White 48.8 46.9
Other 1.6 2.0
Subsidized Lunch Eligibility
Free 57.7 67.4
Reduced 10.9 11.0
Not Eligible 31.4 21.7
Repeating Grade 3.2 3.2
English as Second Language 8.2 7.1
Referred to M-Team 13.6 13.1
Exceptional Education Need 13.1 13.2
Individualized Education Plan 8.2 7.6
During the course of the 1997-98 school year, records were compiled on 8,843 students.
Many students withdrew from SAGE and comparison schools during the year, while others
enrolled. Those students who remained in their schools for the entire year are labeled
“persisters”. As Table 3 shows, enrollment in comparison schools was slightly more stable than
in SAGE schools. Moreover, in both SAGE and comparison schools, the number of students
withdrawing exceeded the number of students enrolling for the year. Thus, the number of
persisters plus newly enrolled students recorded during spring data collection totals 7161,
distributed across schools and grades as shown in Table 4.
13
Table 3. Enrollment Changes in SAGE and Comparison Schools by School Year (Number of
Students and Percentage of Students)
SAGE COMPARISON ALL SCHOOLS
1996-97 1997-98 1996-97 1997-98 1996-97 1997-98
# % # % # % # % # % # %
Ongoing 2943 81.4 2455 42.3 1706 85.3 1402 44.3 4649 82.8 3857 43
Withdrew 397 11 1093 18.8 178 8.9 589 18.6 575 10.2 1.682 18.7
Enrolled 274 7.6 2262 39 115 5.8 1175 37 389 6.9 3437 38
Table 4. Number of Students in SAGE and Comparison Schools by Grade and School Year
SAGE COMPARISON TOTAL
1996-97 1997-98 1996-97 1997-98 1996-97 1997-98
Kindergarten 1494 1524 820 676 2314 2200
first Grade 1723 1567 1001 985 2724 2552
second Grade NA 1541 NA 868 NA 2409
Totals 3217 4632 1821 2529 5038 7161
Classrooms
SAGE schools reduced class size in several ways in order to meet statutory requirements.
The SAGE legislation defines class size as "the number of pupils assigned to a regular classroom
teacher." In practice, reduced class size has been interpreted as a 15:1 student-teacher ratio
(number of students per teacher in one classroom). Implementation occurs in the following
ways:
• A Regular classroom refers to a classroom with one teacher. Most regular
classrooms have 15 or fewer students, but a few exceed 15.
• A 2-Teacher Team classroom is a class where two teachers work collaboratively to
teach as many as 30 students.
• A Shared-Space classroom is a classroom that has been fitted with a temporary wall
that creates two teaching spaces, each with one teacher and about 15 students.
• A Floating Teacher classroom is a room consisting of one teacher and about 30
students, except during reading, language arts, and mathematics instruction when
another teacher joins the class to reduce the ratio to 15:1.
14
Two other types of classroom organization were also utilized in the SAGE program, but
to a limited extent. They are the Split Day classroom consisting of 15 students and two teachers,
one who teaches in the morning and one who teaches in the afternoon, and the 3-Teacher Team
classroom where there are 45 students taught collaboratively by three teachers.
The types of classrooms are displayed in Table 5. SAGE classes range in number of
students from 7 to 38. A few SAGE classrooms exceed the 15:1 student-teacher ratio, but only
by a few students.
Share with your friends: |