The student achievement guarantee

THE STUDENT ACHIEVEMENT GUARANTEE

Submitted by the SAGE Evaluation Team

School of Education

University of Wisconsin–Milwaukee

Alex Molnar

Philip Smith

John Zahorik

Research Assistants

Lee Breese

Karen Ehrle

Anke Halbach

Amanda Palmer

Alan Silverman

Project Administrator

William Harvey

DECEMBER 1998

2

TABLE OF CONTENTS

INTRODUCTION

SAGE Program................................. ........... .......... ........... ................................ 3

Goals of SAGE Evaluation ............... ........... .......... ........... ................................ 4

Class Size Research Background ...... ........... .......... ........... ................................ 5

Summary of 1996-97 Findings ......... ........... .......... ........... ................................ 7

EVALUATION DESIGN

Descriptions and Definitions............. ........... .......... ........... ............................... 11

Data Collection Instruments ............. ........... .......... ........... ............................... 14

ANALYSES OF STUDENT ACHIEVEMENT OUTCOMES 1997-98

Methods Introduction ....................... ........... .......... ........... ............................... 18

SAGE School/Classroom vs. Comparison School/Classroom Analyses ............... 21

Effects Within SAGE Schools/Classrooms ... .......... ........... ............................... 48

ANALYSES OF SAGE TEACHERS, CLASSROOMS, AND SCHOOLS 1997-98

Teacher and Classroom Analysis ...... ........... .......... ........... ............................... 50

Schools............................................. ........... .......... ........... ............................... 72

DISCUSSION: MAJOR FINDINGS, LIMITATIONS, AND FUTURE SAGE REPORTS

Major Findings................................. ........... .......... ........... ............................... 81

Limitations ....................................... ........... .......... ........... ............................... 83

Future SAGE Evaluation Reports ..... ........... .......... ........... ............................... 85

3

INTRODUCTION

The Student Achievement Guarantee in Education (SAGE) evaluation is being conducted

under contract with the Department of Public Instruction by the School of Education at the

University of Wisconsin–Milwaukee. The purpose of the SAGE evaluation is to determine the

effectiveness of the Student Achievement Guarantee in Education (SAGE) program in promoting

academic achievement of students in kindergarten through third-grade classrooms in schools

serving low-income children.

The SAGE statute [s. 118.43] requires participating schools to (1) reduce class size to 15

in kindergarten and grade one in 1996–97, grades kindergarten through two in 1997–98, and

grades kindergarten through three in 1998–99 to 2000–2001; (2) stay open from early in the

morning to late in the day and collaborate with community organizations to provide educational,

recreational, community, and social services (i.e., the "lighted schoolhouse"); (3) provide a

rigorous academic curriculum to improve academic achievement; and (4) establish staff

development and accountability mechanisms.

The SAGE evaluation involves the 30 schools in 21 school districts that launched the

SAGE program in 1996-97 in kindergarten and first grade. Second grade was added in 1997-98,

and third grade in 1998-99. The SAGE evaluation compares SAGE schools to a group of 14-16

non-SAGE comparison schools located in SAGE districts. The results of the 1996-97 and 1997-

98 evaluations are consistent with Tennessee’s Student Teacher Achievement Ratio (STAR)

Project (1985-1989), the largest and best-controlled study on class size reduction to this point.

However, it is worth noting two significant design differences. First, the Tennessee STAR

Project used a true experimental design. The SAGE project uses a quasi-experimental design.

4

This has two implications. The SAGE project evaluation uses naturally occurring classrooms

classroom types which were held constant for the duration of the study. Secondly, the SAGE

evaluation uses a control or comparison group for the purpose of assessing the impact of SAGE

interventions.

The SAGE evaluation is intended to determine the impact that the four interventions of

the SAGE program have on student achievement. To ascertain and to explain this impact, the

evaluation addresses the following questions:

SAGE vs. Comparison School – Achievement Outcome Questions

1. What are the achievement levels of SAGE classrooms compared to achievement levels of

classrooms in selected comparison schools?

2. Does attendance in a SAGE classroom have a differential impact on the achievement of

minority students and white students?

3. Do different types of SAGE classrooms (e.g. one teacher with 15 students vs. two teachers

with 30 students) have different impacts on student achievement?

4. Does the impact on achievement of participation in a SAGE classroom change from year to

year as students progress from first through third grade?

SAGE Schools – Classroom and School Questions

1. What are the instructional characteristics of SAGE classrooms?

2. How are SAGE classrooms organized?

3. Does the type and extent of student participation in SAGE classrooms correlate with

achievement outcomes?

4. Does the style of teaching in SAGE classrooms correlate with achievement outcomes?

5. Does the degree of congruence between SAGE classroom curricula and national professional

curriculum standards in reading/language arts and mathematics correlate with the

achievement levels in SAGE classrooms?

6. Does participation in the SAGE program result in an increase in the number or change in the

type of school and/or community activities housed in the school before and after school

hours?

7. What is the structure and focus of professional development activities in SAGE schools?

achievement?

5

Class Size Research Background

The principal SAGE intervention is a reduction in class size to 15:1 in kindergarten

through third-grade classrooms. Class size reduction in the early elementary grades has been an

increasingly popular policy in recent years. Class size reduction has been credited with more

learning opportunities for students, increased opportunities for teachers to meet children’s

individual needs, and less time spent on discipline problems. Parents and teachers like the idea

and policymakers are embracing it. Several states, among them California, Florida, Indiana,

Nevada, Tennessee, and Wisconsin, have launched class size reduction efforts (Pardini, 1998;

Viadero, 1998).

Decades of research on class size reduction have shown small achievement gains for

students when, for example, class size was reduced from 25 to 20 students. In general, though,

reductions of just a few students per class do not seem to significantly raise academic

achievement. However, in the late seventies, an analysis by Glass suggested that larger

reductions produced greater achievement gains (Glass, 1978; Pate-Bain, Achilles, Boyd-

Zaharias, & McKenna, 1992).

A statewide experiment in Tennessee, the largest and best-controlled study on class size

reduction to this point, assigned kindergarten students on a random basis to classes of 15, 25 with

an aide, or 25 with no aide. The same configurations were maintained through third grade.

Tennessee’s Project STAR (Student/Teacher Achievement Ratio) analyzed student achievement

in relation to class size over a four-year period (1985-1989). The project included 17 inner-city,

16 suburban, 8 urban, and 39 rural schools. Findings showed that students in the smaller classes

scored higher on the Stanford Achievement Test and on the Basic Skills First (BSF) Test in all

four years (K-3) and in all locations. The greatest gains on the Stanford Test were made by

6

inner-city small classes. While all students benefited, disadvantaged minority students seemed to

McKenna, 1992).

Studies such as STAR and SAGE can provide crucial information for policymakers. For

example, a review of the research literature conducted by Bingham (1993) on white-minority

achievement gap reduction and small class size, which included the STAR Project, suggests that

small class sizes in the early grades represent an effective strategy to reduce the achievement

gap. Bingham proposes that class size reduction may offer an early intervention strategy that

serves to prevent rather than to reduce the achievement gap between white and minority students.

Wenglinsky (1997) studied the relationship between spending and student achievement

by analyzing data from three separate sources: The National Assessment of Educational

Progress, the Common Core of Data, and the Teacher’s Cost Index of the National Center for

Education Statistics. Wenglinsky’s research suggests that increased spending to reduce class

size has a direct positive effect on mathematics achievement for fourth grade students. Further

support for small classes in lower elementary grades is produced by the Lasting Benefits Study

(LBS). The LBS tracked students who participated in Project STAR in order to determine

whether achievement advantages of students from small classes were maintained after students

returned to regular-sized classes in fourth grade. Data from 1990-1994 indicate that students

who were originally in smaller classes continued to perform better than their peers from classes

of 25 with or without a teacher’s aide (Mosteller, 1995).

In Wisconsin, most class sizes exceed the class size standards set by the STAR Project.

A study completed by Allen (1997) of K-6 class sizes in Wisconsin’s public schools reported

that 92% of Wisconsin’s kindergarten classes exceed the lower class size standard established by

7

Project STAR of 15 or fewer students per teacher. Twenty-seven percent of Wisconsin

Achievement Outcome Findings 1996-97

To measure academic achievement, first grade students in SAGE schools and in a group

of comparison schools were tested in October 1996, and again in May 1997, using the

Comprehensive Test of Basic Skills (CTBS) Complete Battery, Terra Nova edition, Level 10.

After one year, students in SAGE first-grade classrooms scored higher on the CTBS than

first-grade students in comparison schools. As a group, SAGE students scored significantly

higher on the post-test in reading, language arts, and mathematics sub-tests of the CTBS. The

total score of SAGE students was also significantly higher than the total score of comparison

group students. The achievement advantage associated with participation in the SAGE program

was revealed both in the analysis of individual student scores and in the analysis of averaged

classroom scores.

At the individual level of analysis, after controlling for pre-test scores, income,

absenteeism, and race and ethnicity, SAGE first-grade students scored higher than comparison

school first-grade students on the CTBS post-test in reading, language arts, mathematics and

total score. The results were statistically significant for all but the reading scores. On averaged

classroom scores, the post-test performance of SAGE first-grade students was 4 scale score

points higher in language arts, 4.3 scale score points higher in reading, 4.6 scale score points

higher in mathematics and 4.6 scale score points higher in the total test score than comparison

school students. Each of these findings was statistically significant.

After adjusting for individual pre-test results, socioeconomic status (SES) as defined by

eligibility for subsidized lunch, and student attendance, participation in SAGE shows a

8

statistically significant advantage of 6.4 scale score points in the total score and 8.1 scale score

The data on the average performance of students in SAGE classrooms suggest that the

lower student-teacher ratio in SAGE classrooms mitigates the negative achievement

consequences of poverty. SAGE classrooms achieved at a higher level than comparison school

classrooms despite the fact that, as a group, SAGE classrooms enrolled more students who were

eligible for subsidized lunch. Furthermore, after adjusting for individual pre-test results and SES

as defined by lunch status and student attendance, the post-test scale score advantage increased

to 9.8 for SAGE first-grade classrooms. The advantage was 7.1 on the reading sub-test, 9.0 on

the language arts sub-test, and 12.3 on the mathematics sub-test. These results were all

statistically significant.

School and Classroom Findings 1996-97

To more fully understand the SAGE program, it is important to understand how SAGE

schools structured classrooms and implemented a reduced student-teacher ratio, rigorous

curriculum, staff development, and lighted schoolhouse. Together, that information provides a

within SAGE school and classroom data description of life in SAGE classrooms and schools.

Classroom Level Findings 1996-97

Data from 1996-97 suggested that after one year of the SAGE program classroom

discipline problems are greatly reduced, and when classroom management was needed, it was

overwhelmingly positive. The direct benefit of having to spend less time managing the class was

increased instructional time, i.e., actual time spent on teaching. Further, the increased

instructional time available to teachers was used to attend to the learning needs of individual

students.

The type of instruction that students encountered in SAGE classrooms was

predominantly teacher centered. Listening, practicing, receiving help, and answering accounted

for between 50 to 75 percent of the teaching-learning that occurred. Although teachers indicate

that their use of more student-centered activities such as creating, manipulating, and problem

9

solving increased because of reduced class size, student-centered teaching played only a

Several anticipated outcomes did not emerge from a composite of the interviews,

observations, logs, and questionnaires in 1996-97. While all teachers reported some changes in

their teaching, a large swing to student-centered teaching, a change that some might expect as a

result of reduced class size, was not observed. Lastly, a clear difference in teaching and learning

among the four main types of SAGE classrooms was not apparent.

School Level Findings 1996-97

The Teacher Questionnaire and Principal Interviews, both completed in May 1997, were

the sources of data regarding rigorous curriculum implementation. The Teacher Questionnaire

contained a section on classroom curriculum designed to determine the congruence of SAGE

classroom curricula with professional curriculum standards. First-grade and mixed-grade teacher

responses indicated that their reading/language arts curricula were more congruent with

professional standards than SAGE kindergarten curricula. Teacher responses suggested no

important differences in the degree to which their curricula were congruent with professional

standards in the area of mathematics. Principal responses to curriculum-related questions

suggested that a rigorous curriculum included basic skills, problem solving, and higher-level

thinking. Only a handful of principals seemed to believe that the curriculum of their school was

rigorous. However, most SAGE principals regarded parts of their curriculum as strong.

A section of the Teacher Questionnaire contained staff development questions. Teachers

are asked about their individual level of professional development as well as the extent to which

their school district provided staff development programs. Despite contractual requirements,

roughly 60 percent responded that they had no "personal, formal, written development plan."

Twenty-one percent rated their school district’s staff development program at the “initialization”

phase, about 66 percent rated it at the “implementation” stage, and about 9 percent felt the

district was “institutionalizing” the staff development program.

10

Data regarding implementation of lighted schoolhouse activities for 1996-97 were

regarding lighted schoolhouse activities existing prior to schools’ participation in SAGE were

obtained from the Baseline Data Questionnaire administered in May 1996 and the school

contracts completed for DPI prior to enrollment in the SAGE program. Principal Interview data

suggested that because of SAGE schools took responsibility for the conception and operation of

the lighted schoolhouse activities. However, the schools tended not to focus heavily on their

lighted schoolhouse activities in the first year of SAGE implementation.

11

EVALUATION DESIGN .

Schools

districts throughout the state, as shown in Table 1. In addition, the SAGE program consisted of

14 Comparison schools located in 7 school districts.

Table 1. SAGE Schools 1997-98

SAGE DISTRICTS AND SCHOOLS

DISTRICT

School

School

Adams Elementary

Keshena Primary

Robinson Elementary

River Heights Elementary

Parkview Elementary

Gilman Elementary

Glidden Elementary

Jefferson Elementary

Carleton Elementary

Fairview Elementary

Longfellow Elementary

Maple Tree Elementary

Maryland Avenue Elementary

Sherman Elementary

Wisconsin Conservatory of Lifelong Learning

Wilson Elementary

Ogema Elementary

Tripoli Elementary

KENOSHA

Durkee Elementary

Siren Elementary

Lac Du Flambeau Elementary

South Shore Elementary (Port Wing)

Franklin Elementary

Hamilton Elementary

SUPERIOR

Blaine Elementary

Cooper Elementary

LAONA

Robinson Elementary

Mountain Elementary

Glendale Elementary

Webster Elementary

12

Students

first-grade, and 113 second-grade classrooms. The gender, race, and other characteristics of

students in SAGE schools are displayed in Table 2:

Table 2. Characteristics of SAGE Students 1996-97 and 1997-98

Characteristic Percent of Students

1996-97

Percent of Students

1997-98

Gender

Female 48.6 49.3

Male 51.4 50.7

Race/Ethnicity

African American 24.8 24.3

Asian 5.7 5.3

Hispanic 6.6 7.8

Native American 11.7 7.9

White 48.8 46.9

Other 1.6 2.0

Subsidized Lunch Eligibility

Free 57.7 67.4

Reduced 10.9 11.0

Not Eligible 31.4 21.7

During the course of the 1997-98 school year, records were compiled on 8,843 students.

Many students withdrew from SAGE and comparison schools during the year, while others

enrolled. Those students who remained in their schools for the entire year are labeled

“persisters”. As Table 3 shows, enrollment in comparison schools was slightly more stable than

in SAGE schools. Moreover, in both SAGE and comparison schools, the number of students

withdrawing exceeded the number of students enrolling for the year. Thus, the number of

persisters plus newly enrolled students recorded during spring data collection totals 7161,

distributed across schools and grades as shown in Table 4.

13

Table 3. Enrollment Changes in SAGE and Comparison Schools by School Year (Number of

Students and Percentage of Students)

SAGE COMPARISON ALL SCHOOLS

1996-97 1997-98 1996-97 1997-98 1996-97 1997-98

# % # % # % # % # % # %

Ongoing 2943 81.4 2455 42.3 1706 85.3 1402 44.3 4649 82.8 3857 43

Withdrew 397 11 1093 18.8 178 8.9 589 18.6 575 10.2 1.682 18.7

Enrolled 274 7.6 2262 39 115 5.8 1175 37 389 6.9 3437 38

Table 4. Number of Students in SAGE and Comparison Schools by Grade and School Year

SAGE COMPARISON TOTAL

1996-97 1997-98 1996-97 1997-98 1996-97 1997-98

Kindergarten 1494 1524 820 676 2314 2200

first Grade 1723 1567 1001 985 2724 2552

second Grade NA 1541 NA 868 NA 2409

Totals 3217 4632 1821 2529 5038 7161

Classrooms

SAGE schools reduced class size in several ways in order to meet statutory requirements.

The SAGE legislation defines class size as "the number of pupils assigned to a regular classroom

teacher." In practice, reduced class size has been interpreted as a 15:1 student-teacher ratio

(number of students per teacher in one classroom). Implementation occurs in the following

ways:

• A Regular classroom refers to a classroom with one teacher. Most regular

• A 2-Teacher Team classroom is a class where two teachers work collaboratively to

teach as many as 30 students.

• A Shared-Space classroom is a classroom that has been fitted with a temporary wall

that creates two teaching spaces, each with one teacher and about 15 students.

• A Floating Teacher classroom is a room consisting of one teacher and about 30

students, except during reading, language arts, and mathematics instruction when

another teacher joins the class to reduce the ratio to 15:1.

14

Two other types of classroom organization were also utilized in the SAGE program, but

one who teaches in the morning and one who teaches in the afternoon, and the 3-Teacher Team

classroom where there are 45 students taught collaboratively by three teachers.

The types of classrooms are displayed in Table 5. SAGE classes range in number of

students from 7 to 38. A few SAGE classrooms exceed the 15:1 student-teacher ratio, but only

by a few students.