Keywords: Participatory evaluation; Organizational learning; Science enrichment; Appreciative inquiry
Abstract: A participatory evaluation (PE) of an inner-city science enrichment program for elementary school youth was conducted using an evaluation team consisting of staff, board members, students, parents, and representatives of the agency's major funder. This evaluation team designed and implemented the entire evaluation with guidance from an evaluator-consultant and researchers from a local university. Data gathered from surveys of alumni, parents, students, and teachers revealed high satisfaction with the program and a validation of the hands-on teaching model. Alumni reported that the program impacted their scientific and social competence. The highly participative design process was seen as very successful by all participants, and offers useful guidelines for other PE, including active participation of all stakeholders, commitment to a shared vision, and a good match between the organization and the evaluator.
Reference Type: Film or Broadcast
Record Number: 94
Director: Respaud, Stéphane
Year Released: 2004
Title: Current status on science teaching in Ariège primary schools
Series Title: La main à la pâte seminars
Date Released: July, 16th, 2004
Short Title: Current status on science teaching in Ariège primary schools
Synopsis: Since four years now, the department of Ariège began its PRESTE (Plan of Renovation of Sciences and Technology Teaching at Primary School) by creating and equipping a science classroom, an Internet site and by organizing many training activities for teachers. First outcomes of the inquiry held on those actions by La main à la pâte Pilot Site of Pamiers. Questionnaire proposed to teachers also available on line.
Reference Type: Report
Record Number: 97
Author: Rosenquist, Anders; Shavelson, Richard J.; Ruiz-Primo, Maria Araceli
Title: On the “Exchangeability” of Hands-On and Computer-Simulated Science Performance Assessments
Institution: CRESST/Stanford University
Type: CSE Technical Report
Short Title: On the “Exchangeability” of Hands-On and Computer-Simulated Science Performance Assessments
Report Number: 531
Abstract: Inconsistencies in scores from computer-simulated and “hands-on” science performance assessments have led to questions about the exchangeability of these two methods (e.g., Baxter & Shavelson, 1994), in spite of the former’s highly touted potential (e.g., Bennett, 1999). Five possible explanations of students’ inconsistent performances were considered: (1) inadequate exposure to computers and simulations, (2) differential views of computer-simulated (2-dimensional icons) and hands-on tasks, (3) different methods tapping different aspects of achievement, (4) partial or incomplete knowledge, and (5) a combination of partial knowledge and method differences. The first explanation was ruled out by the fact that students had computers in their classes and used them for a variety of purposes, including simulation. The second explanation was ruled out using talk-aloud data, randomized experiments, and student questionnaire responses. If explanation 3 were tenable, the correlation between Electric Mysteries scores at time 1 and time 2 for either hands-on or computer simulation should be higher than the correlation between hands-on scores and computer simulation scores at either point in time. Shavelson, Ruiz-Primo, and Wiley (1999) provided correlations that did not jibe with this expectation. To explore the remaining two possible explanations dealing with student expertise, we compared the performance of high school physics students (“experts”) to that of Baxter and Shavelson’s elementary school students and found, somewhat surprisingly, that these “experts” were far from expert. Indeed, they were no more expert than the elementary students. Consequently, we have narrowed the possible explanations for the lack of exchangeability between computer-simulated and hands-on performance assessments to one of two choices: partial knowledge or the interaction of partial knowledge with method. The jury is still out.
Reference Type: Journal Article
Record Number: 98
Author: Ruiz-Primo, Maria; Li, Min; Ayala, Carlos; Shavelson, Richard
Title: Evaluating Students' Science Notebooks as an Assessment Tool. Research Report
Journal: International Journal of Science Education
Date: Oct 2004
Short Title: Evaluating Students' Science Notebooks as an Assessment Tool. Research Report
Keywords: Instructional Effectiveness; Performance Based Assessment; Teacher Effectiveness; Feedback; Communication Skills; Student Journals; Science Education; Evaluation Methods; Academic Achievement; Scores;
Abstract: The idea of using science notebooks as a classroom assessment tool is not new. There is general agreement that science notebooks allow teachers to assess students' conceptual and procedural understanding and to provide the feedback students need for improving their performance. In this study we examined the use of science notebooks as an unobtrusive assessment tool that can also be used by individuals outside the classroom (for example, school district personnel), and as a means for obtaining information about students' learning and their opportunities to learn. More specifically, in this study students' science notebooks were used as a source of data about the (a) implementation of a curriculum's intended activities, (b) students' performance, and (c) quality of teachers' feedback. Our results indicated that: (1) Students' science notebooks can be reliably scored. Unit implementation, student performance, and teacher feedback scores were highly consistent across raters and units. (2) High and positive correlations with other performance assessment scores indicated that the student performance score can be considered as an achievement indicator. And (3) low performance scores across the two units revealed that students' communication skills and understanding were far away from the maximum score and did not improve over the course of instruction during the school year. This result may be due, in part, to the fact that no teacher feedback was found in any of the students' notebooks across the six classrooms studied. This may reflect some characteristics of the teachers' assessment practices that may require further professional development.
Reference Type: Book Section
Record Number: 133
Author: Russell, Terry; McGuigan, Linda
Title: Making formative use of a national summative assessment regime
Editor: Behrendt, H., Dahncke, H., Duit, R., Gräber, W., Komorek, M., Kross, A. and Reiska, P.
Book Title: Research in science education-Past, present and future
Publisher: Kluwer academic publishers
Short Title: Making formative use of a national summative assessment regime
Keywords: formative asessment, summative assessment, UK
Abstract: While the impact of formative assessment practices on learning outcomes is receiving increasing attention, the dominant function of the statutory assessment system in England and Wales is summative. As authors of the statutory end of Key Stage 2 tests between 1995 and 1999, we are aware of a volume of test items and pupil performance data that has been generated in the course of the summative regime. Summative performance data are available on the cohort of 600,000 pupils at age 11 assessed annually. We have further illuminated these data by an annual qualitative re-marking of a sub-sample stratified by three overall achievement levels.We suggest that many test items are analogous to concept probes within the constructivist paradigm. Qualitative re-marking using a cross-sectional sample yields diagnostic data which invite developmental inferences about conceptual progression. When pupils’ assessed understanding can be mapped onto such lines of progression, assessment can have a powerful formative capability in informing classroom teaching and learning practices. The characteristics of test items which may combine summative and formative utility are discussed in the context of pupils’ ideas about Forces.
Reference Type: Report
Record Number: 100
Author: Sarmant, Jean-Pierre
Title: Rapport d’étape sur l’opération "La main à la pâte" et l’enseignement des sciences à l’école primaire
(General intermediate report of La main à la pâte and the science teaching at primary school.)
Institution: Inspection générale de l'Éducation nationale
Date: June 1999
Short Title: Rapport d’étape sur l’opération "La main à la pâte" et l’enseignement des sciences à l’école primaire
(General intermediate report of La main à la pâte and the science teaching at primary school.)
Abstract: This report was established, in 1999, by a request of the minister of Education. It was drawn up by the general Inspector of Ministry of Education in order to determine the effects of La main à la pâte approach (this because La main à la pâte launched a pilot experience over France in 1997. On 1995, only 5% of the French classes got sciences sessions – whatever the approach).
Three types of data collection were employed: meetings (institutional actors, project managers), visits of classes (27) followed by interviews (192 persons in total).
The sample was divided into 2 groups (= 2 separated investigations): The first one linked to 3 most implied departments: Loire Atlantique, Haute Savoie and the Rhone. It aimed at measuring the intensity and the nature of engagement on La main à la pâte approach. The observations were carried out in the classes by the IEN (National Education Inspector), they were focused on the application of the ten principles of la main à la pâte. The second investigation aimed at specifying the characteristics (quantitative and qualitative) of the sciences teaching in two departments very few or not at all implied on this type of approach (Vienne and Indre).
- In general, there is a better science teaching (inquiry teaching) (in terms of N° of hours devoted to scientific teaching) and in terms of N° of training sessions for teachers.
- At the level of the non implied departments, the teachers approach was descriptive and “affirmative”
- On implied departments, the activity of pupils during the classroom was higher and their attitude more engaged in reflection. A better teacher’s attitude towards science (felt as a pedagogical renewal) was noticed.
- The effects are very positive in social and moral behavior fields, the speaking of the mother tongue and the general culture but not necessary at knowledge level.
- The most obvious results were noticed under the sociological and specifically on behavioral field: children are more attentive, they have a better collective behavior. They used to speak each other and listen themselves by testifying a mutual respect.
- In multicultural contexts, a unifying effect of scientific activities was observed.
- La main à la pâte approach has also a benefic effect for a higher proportion of children having school difficulties = low achievers (They are more confident and better perceived by other students)
- The improvement of expression skills was also observed, as well oral as written: particularly for the immigrant children.
- Better effects on logic: pupils are able to reinvest in other fields, apart from the sciences
- Benefits are evident for classes engaged for at least two years
- The effects obtained on behaviour and the expression skills are enough to justify the attention of the ministry of education
-Examination of scientific notebooks shows that there is also a gain in basic scientific knowledge required by the standards.
Some observed difficulties:
- In certain classes, the acquisition of knowledge is a minor objective, even non-existent (and this also on most implied departments). Sometimes an exclusively technological activity could be also a reducing activity (realization of an object, without any other aim).
- Sometimes teachers have problems stating conclusions and confronting them to the scientific knowledge (Sometimes there is no scientific knowledge confrontation at all).
- A strong piloting structure is then needed followed by the implementation of coaching tools for teachers.
- In spite of the prestige of its founders and the mediatisation of the approach, teachers are inclined to grant value only to the ministry instructions coming through the school territorial inspection.
- Teachers having a scientific background are of help for those starting and that need to feel confident respecting to science teaching.
- For some teachers la main à la pâte approach seems to be so time consuming bringing difficult to cover the standards
- About the units “insights “opinions collected are very divergent. Opinions go from enthusiasm without reserve to a frank hostility because those considered that they provide a too assisted pedagogy. They often say "I would like to know the insights, to help me in my beginnings, but I will probably leave them once I feel more confident. Then there is a special need to propose documents coming from different sources. Internet website may play an important role.
- It is important to consider the generalization of the approach even if not possible at medium term.
- The objectives of the institution (ministry) should overlap those of La main à la pâte by the drawing up lawful texts implementing a science teaching based on la main à la pâte approach
- La main à la pâte must keep its own dynamics and preserve its own specificity based on the association of scientific partners
(Written by P. Lucas)
Reference Type: Report
Record Number: 138
Author: Schoultz, J.; Hultman, G.
Title: NTA is a great idea. We don’t do things just to get bored but because we want to learn.
Institution: Department of Educational Science, University of Linköping
Short Title: NTA is a great idea. We don’t do things just to get bored but because we want to learn.
Keywords: NTA, primary science, evaluation, learning
Abstract: The evaluation has focussed on three central aspects:
- pupils’ ability and opportunities to participate and take initiatives in the classroom teaching
- to what extent NTA stimulates pupils’ curiosity about and interest in science
- pupils’ ability to observe, experiment, predict, argue, discuss and document
During our work with evaluation we have mainly been focussed on pupils’ learning and development in NTA. But during our observation visits and interviews we have more and more come to understand the great importance of the teacher for the pupils’ development and learning progress. We feel then that it is not possible to evaluate pupils’ development and learning without at the same time looking at the constraints and opportunities inherent in the teaching situation. These constraints can differ greatly. The teacher here has an important task to organise the teaching situation and adapt teaching material to the group of pupils. The role of the teacher is extremely diverse, with the teacher’s strategies, thoughts, values and actions jointly creating the learning situation. What happens when a new project makes its debut in a complex classroom situation?
We conclude in our evaluation that there is a great potential for development in the NTA concept, which is not always achieved in the concrete school situation. The pupils learn about science and many become interested in the subject area. But the NTA material is not sufficient in itself. The role of the teacher, his/her attitude and knowledge are extremely important and are a precondition for success. Professional development for the teachers in the form of working-team meetings, theme meetings, contact with universities, experienced teachers are all very important factors in the NTA concept. The concept has its limitations but also great possibilities. Properly used it will provide a platform for both pupils and teachers to develop their scientific knowledge.
Reference Type: Conference Proceedings
Record Number: 134
Author: Scott, Rowena H.; Fisher, Darrell L.
Year of Conference: 2002
Title: The impact of an in-servive course for primary teachers
Conference Name: annual conference of the Australian Association for Research in Education
Conference Location: Brisbane, Australia
Date: 1-5 December
Short Title: The impact of an in-servive course for primary teachers
Keywords: hands on programs, in-service training, Australia
Abstract: Reform is now common, with people from nearly every nation engaging in improving science teaching as political leaders recognise that knowledge of science has economic consequences (Gallagher, 2000). A purpose of this study was to preserve the voices of teachers who had participated in a university course as a "special treatment" program with those of "control" teachers. A program was launched for the training of Specialist primary science teachers in twelve schools. These 17 teachers attended a unique, four-week, in-service course (Zaitun, 1999) and regular school-based workshops. It was hoped that teaching methods would become less traditional and less teacher-directed. Also, it was anticipated that the ability of participants to use questioning techniques to probe students' understanding would improve (Zaitun, 1999). Three male and three female Specialist science teachers, experienced and less experienced, were chosen from various Project schools for interview including one male and one female teacher from the same school. Four teachers from Project schools, who were not involved in the Specialist science teaching Project, including two from the same school, were also interviewed to seek out and capture the essence of different participants' experience.
Reference Type: Unpublished Work
Record Number: 102
Author: Shavelson, Richard J.; Gao, Xiaohong; Baxter, Gail P.
Title of Work: Sampling variability of performance assessments
Institution: National Center for Research on Evaluation, Standards, and Student Testing (CRESST) - UCLA
Keywords: measurement, facets, methods, data bases, mathematics and science achievement, performance, elementary
Abstract: In this paper, performance assessments are cast within a sampling framework. A performance assessment score is viewed as a sample of student performance drawn from a complex universe defined by a combination of all possible tasks, occasions, raters, and measurement methods. Using generalizability theory, we present evidence bearing on the generalizability (reliability) and convergent validity of performance assessments sampled from a range of measurement facets, measurement methods, and data bases. Results at both the individual and school level indicate that rater-sampling variability is not an issue: raters (e.g., teachers, job incumbents) can be trained to consistently judge performance on complex tasks. Rather, tasksampling variability is the major source of measurement error. Large numbers of tasks are needed to get a reliable measure of mathematics and science achievement at the elementary level, or to get a reliable measure of job performance in the military. With respect to convergent validity, results suggest that methods do not converge. Performance scores, then, are dependent on both the task and method sampled.
Reference Type: Report
Record Number: 152
Author: Smith, Laura M.; Others, And
Title: Assessment of Student Learning in Science
Institution: South Carolina Univ., Columbia. South Carolina Center for Excellence in the Assessment of Student Learning.
Short Title: Assessment of Student Learning in Science
Keywords: Academic Achievement; Alternative Assessment; Computer Simulation; Curriculum Development; Educational Assessment; Educational Change; Elementary Secondary Education; Evaluation Methods; Hands on Science; Performance Based Assessment; Portfolios (Background Materials); Science Achievement; Science Education; State Programs; Student Evaluation; Student Journals
Abstract: As part of a series of documents describing assessment of student learning in various curriculum areas in South Carolina, this document reviews the assessment of student learning in the science classroom. The report begins with a discussion of effective science learning, and outlines the knowledge, skills, and dispositions that encompass science education. Reform efforts currently underway to transform school science are described. Alternative methods of assessment that support curriculum and instruction and promote exemplary science learning are explored, and a brief summary of activities in science assessment in selected states is provided. Performance assessments that are discussed include: (1) performance tests and tasks; (2) open-ended questions; (3) student journals; and (4) computer simulations. Portfolios and concept maps are other forms of authentic assessment that are discussed. Appendix A presents an example of a hands-on science assessment, and Appendix B presents a problem and a concept map. (SLD)
Reference Type: Conference Proceedings
Record Number: 110
Author: Songer, Nancy Butler; Gotwals, Amelia Wenk
Year of Conference: 2004
Title: What Constitutes Evidence of Complex Reasoning in Science?
Conference Name: Sixth International Conference of the Learning Sciences (ICLS)
Conference Location: Mahwah, NJ: Erlbaum
Short Title: What Constitutes Evidence of Complex Reasoning in Science?
Keywords: review, investigation activities, assessment methodologies
Abstract: The laboratory has been given a central and distinctive role in science education, and science educators have suggested that rich benefits in learning accrue from using laboratory activities. Twenty years have been elapsed since we published a frequently cited, critical review of the research on the school science laboratory (Hofstein & Lunetta, Rev. Educ. Res. 52(2), 201-217, 1982). Twenty years later, we are living in an era of dramatic new technology resources and new standards in science education in which learning by inquiry has been given renewed central status. Methodologies for research and assessment that have developed in the last 20 years can help researchers seeking to understand how science laboratory resources are used, how students' work in the laboratory is assessed, and how science laboratory activities can be used by teachers to enhance intended learning outcomes. In that context, we take another look at the school laboratory in the light of contemporary practices and scholarship. This analysis examines scholarship that has emerged in the past 20 years in the context of earlier scholarship, contemporary goals for science learning, current models of how students construct knowledge, and information about how teachers and students engage in science laboratory activities.