This research was supported by Chais Research Center for the Integration of Technology in Education, The Open University of Israel.
About the Author
Dr. Tali Heiman is a senior lecturer at the Department of Education and Psychology at The Open University of Israel. Dr. Heiman is the Head of The Open University Diagnostic Center for students with LD. Her fields of research include Learning Disabilities and ADHD: Learning Style, assistive technology, adjustment and coping in higher education; emotional and social coping of students with learning disabilities; families with a child with special needs: coping, adjustment and expectations. She can be reached by email at: firstname.lastname@example.org.
Assessment of Campus Climate to Enhance Student Success
Susan A. Vogel
Janet K. Holt
Northern Illinois University
East Carolina University
Northern Illinois University
This article describes the development, content, and use of four questionnaires that comprise the Assessment of Campus Climate to Enhance Student Success with the focus on the Faculty Questionnaire. Faculty development activities are described as an example of how the questionnaires can be used to enhance knowledge and change attitudes and practices. The results showed significant increase in faculty knowledge and changes in practices. Questionnaire findings were compared to the results of analyzing the wording and location of a welcoming paragraph in syllabi as an example of changes in faculty practices and as a method to validate questionnaire findings. Lastly, a service/research partnership with the Association on Higher Education And Disability (AHEAD) organization is described, which will provide access to the questionnaires to facilitate improving campus climate to enhance the academic success of students with disabilities.
Concern about academic success for students with disabilities in higher education has increased as the proportion of students with disabilities has increased. The proportion of first-year full-time students with disabilities increased almost four-fold from 2.3% in 1978 to 9.8% 20 years later (Henderson, 1999). This finding was corroborated by the U.S. Department of Education (USDO, 2000) (USDO, 2003) when students with disabilities in all years of undergraduate education were found to represent 9% of the total college population (Horn, Peter, & Rooney, 2002).
Although entrance to college or university is a major first step, the ultimate goal is degree completion. Limited research regarding graduation rates of students with disabilities in general has reported a bleak picture. Students with disabilities were less likely to complete their undergraduate degree than students without disabilities (Horn & Berktold, 1999; Murray, Goldstein, Nourse, & Edgar, 2000; National Center for Education Statistics, 1999; Rath & Royer, 2002). However, there were exceptions to these findings, and graduation rates were the same for those with and without disabilities when the former had access to comprehensive support services (Cowles & Keim, 1995; Vogel & Adelman, 2000). Nevertheless, in most cases, students with disabilities graduated at a lower rate, and those who did not graduate were more often unemployed, employed part-time, or held jobs in occupations that were not of equivalent status or salary to that of their nondisabled peers (Dickinson & Verbeck, 2002; U.S. Department of Education, 2000; Vogel & Reder, 1998).
These findings have led to growing concern and inquiry regarding barriers to academic success that create a chilly classroom climate for students with disabilities in higher education. Hall and Sandler (1982) and Beilke (1999) characterized the behaviors of faculty that contribute to this atmosphere as ignoring, interrupting, distancing, avoiding eye contact, criticizing, offering limited guidance, and attributing success, when it did occur, to factors other than students’ ability or hard work. These behaviors resulted in students’ loss of self-confidence, feelings of second-class status, disempowerment, and marginalization. Because faculty knowledge, attitude, and behaviors are considered to have the most significant impact on students’ academic success (Kurth & Mellard, 2006; Wilson, Getzel, & Brown, 2000), the focus of this article and literature review is mainly on faculty knowledge, attitude, and behaviors.
Although some researchers reported that faculty were in general willing to provide accommodations and often did so (Houck, Asselin, Troutman, & Arrington, 1992; Leyser, Vogel, Brulle, & Wyland, 1998; Nelson, Dodd, & Smith, 1990), many reported they lacked basic knowledge regarding disabilities. This lack of basic knowledge included not only knowledge regarding disabilities, but also knowledge regarding legal mandates and provision of reasonable accommodations (Dona & Edmister, 2001; Kurth & Mellard, 2006; Leyser et al., 1998; Thompson, Bethea, & Turner, 1997).
When faculty lacked knowledge about disabilities (especially nonvisible disabilities such as learning disabilities, attention deficit hyperactivity disorder, and psychiatric disabilities), they sometimes exhibited behaviors that students described as skeptical or suspicious regarding the existence of their disability. Students also reported that faculty made negative comments about them, their disability, and their need for accommodations (Beilke, 1999; Jensen, McCrary, Krampe, & Cooper, 2004; Kurth & Mellard, 2006; Perry & Franklin, 2006; Wilson, Getzel et al., 2000).
Such perceived negative attitudes were identified as one of the most significant barriers to student academic success (Deshler, Ellis, & Lenz, 1996; Dona & Edmister, 2001; Hill, 1996; Kruse, Elacqua, & Rapaport, 1998; Kurth & Mellard, 2006; Lehmann, Davies, & Laurin, 2000; Wilson et al., 2000). When faced with negative attitudes, students reported feeling intimidated and reluctant to disclose their disability and request accommodations (Bourke, Strehorn, & Silver, 2000; Norton, 1997; Perry & Franklin, 2006) and were at increased risk of lower grades and academic failure.
In spite of the importance of faculty attitude, limited research has been conducted regarding the characteristics of faculty with more positive attitudes. Ibrahim and Herr (1982), Junco and Salter (2004), Leyser et al. (1998), Rao (2004), and Salzberg et al. (2002) noted consistently that female faculty, faculty with more contact (whether personal or in teaching), and faculty in fields of education and social science had more positive attitudes. However, much more important were the few studies that confirmed that faculty who had more knowledge about legal mandates and disabilities were more likely to have positive attitudes (Leyser et al., 1998; McGee, 1989; Rao, 2004). Yet, Leyser also reported that in spite of willingness to provide accommodations, 82% of faculty reported that they had little or no knowledge about providing accommodations. These studies led Salzberg (2003) to the strong recommendation that all faculty attend 2-3 hours of mandatory training. However, because Disabled Support Services Directors have typically reported poor attendance at such workshops, Burgstahler and Doe (2006), Scott and Gregg (2000), and Vogel, Leyser, Wyland, and Brulle, (1999) recommended offering alternative strategies of staff development for infusion of information such as short workshops, speakers, online delivery of presentations/videos/PowerPoint™ presentations on demand, and making online and/or hard copy of well designed information, and web-based information available 24/7.
Not surprising, even more faculty lacked in-depth understanding of more recent instructional innovations such as the principles of universal design and accessible electronic instructional materials (McGuire & Scott, 2006; McGuire, Scott, & Shaw, 2004; Vogel, Holt, Sligar, & Er, 2005; Vogel et al., 1999). Lack of such information makes it even more critical for each institution to assess faculty knowledge periodically and provide an infusion of information customized to faculty’s interests and needs. Moreover, identification of faculty members’ preferred method of delivery, time frame, media, and format is a critical first step in faculty willingness to collaborate and partake in staff development (Burgstahler, 2001, 2002, 2007; Burgstahler & Doe, 2006; Debrand & Salzberg, 2005; Junco & Salter, 2004; Salzberg, et al., 2002; Scott & Gregg, 2000; Vasek, 2005; Vogel et al., 2005).
Recently, awareness and concern regarding faculty attitude, knowledge, and practices have been broadened to include attitude and knowledge of students without disabilities. While we would expect that classmates without disabilities would have a positive attitude toward students with disabilities receiving accommodations, from the very limited research completed to date, we know this is not always the case. For example, Perry and Franklin (2006) reported that some students with disabilities experienced negative nonverbal communication from students without disabilities who may reflect the faculty’s perception that accommodations penalize students without disabilities. If students without disabilities lack information about disabilities and the law, they may be of the opinion that accommodations are unnecessary, discriminatory, or both, and mistakenly believe that they give students with disabilities an unfair advantage. Once they become aware of students with disabilities receiving accommodations, students without disabilities may be less willing to collaborate with them on in-class or out-of-class assignments.
These negative attitudes and behaviors may be a double-edged sword, making students with disabilities less likely to disclose and/or request accommodations so their disability or an accommodation does not become apparent to students without disabilities. The resulting impact of the lack of accommodations will, inevitably have a significant negative impact on their academic success.
History of the Development of the Faculty Questionnaire
In the 1980s, researchers began to express interest in exploring the impact on faculty of the increase in the number of students with disabilities in higher education. One of the first to study this issue was Leyser (1989), who developed an initial questionnaire on the topic. His survey instrument was limited to students with learning disabilities because this disability was one of the most complex, and accommodations in higher education were in the infancy stage of development. Moreover, the number of college students with learning disabilities was rapidly increasing at the time, and there was concern about faculty willingness to provide accommodations as well as attitudes toward students with learning disabilities.
Ten years later, Leyser, Vogel, Wyland, and Brulle (1998) updated Leyser’s questionnaire to determine if the passage of time since implementation of Section 504 and continuing increases in the number of students with LD in higher education had resulted in changes in knowledge, practices, and willingness to provide accommodations. Also investigated were how faculty in professional preparation programs like teaching were impacted by the increase in requests for accommodations, especially in entrance examinations and clinical experiences (Vogel et al., 1999; Wertheim, Vogel, & Brulle, 1998).
In 2001, the faculty questionnaire was further refined, updated, and broadened to include all disabilities. In addition, item-level analyses and reliability analyses were conducted to ensure that credible inferences could be made from the surveys. The purpose of the current study was twofold. First, the study evaluated faculty knowledge, attitudes, practices, and topics of interests regarding students with disabilities and assessed change in these characteristics after interventions to increase knowledge-base and improve campus climate for students with disabilities. Second, the study assessed the effectiveness of the faculty questionnaire to evaluate campus climate for students with disabilities.
All faculty received an e-mailed letter from the project director inviting them to respond and providing the URL to find the questionnaire online or to print a PDF version. A reminder e-mail was sent three weeks later. Faculty included all full-time and part-time tenured and tenure-track faculty, instructors, and teaching assistants, based on the assumption that all of the above individuals have a direct impact on student academic success.
In Year One, 271 faculty replied, yielding a 28% response rate. In Year Three, 109 faculty responded, yielding an 8.9% response rate. Faculty were 93% (Year One) and 94% (Year Three) full-time, 79% (Year One) and 77% (Year Three) were tenured or tenure track, and they were equally divided between males and females in Year One with slightly more females (59%) than males in Year Three. For the most part, the respondents had considerable teaching experience, with 60% (Year One) and 69% (Year Three) having 11 or more years of experience. Fifty-two percent (Year One) and 55% (Year Three) of the faculty were either associate or full professors, and 69% (Year One) and 61% (Year Three) were between the ages of 36 and 55. With the exception of gender distribution, the demographic information indicated that the respondents were representative of the demographics of the faculty as a whole in that more than half had longevity and seniority in the institution and were seasoned teachers.
This overview provides findings from the faculty questionnaire used over a three-year period at Northern Illinois University, a large midwestern doctoral-degree-granting public university (Vogel et al., 2005). Faculty are associated with seven colleges in the university: Business, Education, Engineering and Engineering Technology, Health and Human Sciences, Law, Liberal Arts and Sciences, and Visual and Performing Arts.
In this longitudinal study, data were collected at two times, 2002 and 2004 from the same population of faculty. Faculty were re-sampled in 2004 and, therefore, were not necessarily the same individuals who responded in 2002 and 2004. At both data collection points, a web survey was administered online with the option to print and return the hard copy of the survey. During the time between the 2002 and 2004, alternative staff development interventions were provided to the campus community as a whole, designed to increase knowledge base, thereby creating a more positive climate toward students with disabilities.
Development of questionnaire. The faculty questionnaire used in this study was developed within a global higher education context because faculty members are not the only ones who play an important role in student academic success. A suite of questionnaires was developed for assessing campus climate among four campus groups: faculty, administrators and staff, students with disabilities, and students without disabilities.
The administrators and staff questionnaire was developed because these constituents influence the campus climate, especially in students’ initial contact with the institution; for example, in meeting with admissions, financial aid, work-study, advisors, registration and records, housing, and transportation staff. Many administrators and staff also play a major role in providing direct support in meeting the needs of students with disabilities such as the disability services staff, those in special admissions and outreach to applicants with disabilities, legal counsel, ADA, Sections 504 and 508 compliance officers, affirmative action, diversity, housing, transportation, financial aid, counseling, advising, library and assistive technology, and information technology for students with disabilities. Administrators also have traditionally had a major responsibility in determining policies and procedures that impact students with disabilities, such as determination of major and graduation requirements, policies pertaining to course substitutions, modification of requirements, grievance procedures, and accessibility of online information on the institution’s Web sites.
More recently, faculty have been involved in the development of such policies as well, especially when they are directly affected, as in delivering instruction online or in electronic communication related to learning, to mention only a few. For this reason, the Faculty Questionnaire also included items regarding policies and procedures.
The third questionnaire enabled a critical voice to be heard; namely, that of students with disabilities. Students with disabilities were asked to report their firsthand experiences and to assess faculty, administrators, and staff knowledge, attitudes, and needs for further information. Their input told us the “way it is” and served to cross-validate what we learned about the campus climate from the other three constituents’ self-report. Indeed, the input from students with disabilities is the litmus test regarding campus climate.
A fourth questionnaire allowed students without disabilities to report their knowledge about disabilities, legal mandates, fairness of accommodations, and need for more information. They are important constituents on every college campus because they also contribute significantly to the classroom environment, especially when learning is collaborative. Because many disabilities are visible and are apparent to others when students with visible disabilities receive accommodations, students without disabilities are able to observe when such students are receiving accommodations. Students with visible disabilities may experience disability stigma as a result of the negative attitude of students without disabilities in their classes. In addition, when students with hidden disabilities disclose their disability and needed accommodations to their instructor and receive classroom or examination accommodations, students without disabilities can observe this and may express resentment and be unfriendly, if not worse.
The content of the students without disabilities questionnaire resembles the faculty and administrator/staff questionnaires and includes questions regarding experience, knowledge, attitude, and need for information. For these reasons, students without disabilities are included in the assessment of campus climate and should be included in targeted activities to enhance knowledge about disabilities and change in attitudes.
To date, one or more of the questionnaires have been distributed to faculty, administrators, staff, and students in the United States on seven campuses and to the national membership of the professional organization of occupational therapists (Foss, 2002; Vasek, 2005; Vogel et al., 2005; Vogel, Leyser, Burgstahler, Sligar, & Zecker, 2006). In addition, the faculty questionnaire was distributed to the faculty in an Israeli college of education (Leyser, 2003). The U.S. institutions included three large, doctoral-degree-granting public institutions, two private colleges/universities that offered undergraduate, master’s, and professional degrees, and two associate-degree-granting public colleges in the midwest and northwest of the United States. (Further information about this suite of four updated and expanded questionnaires is available at www.ahead.org.)
Faculty questionnaire content. The faculty questionnaire consists of 35 items divided into five subgroups: (a) knowledge, (b) practices, (c) attitudes, (d) topics of interest, and (e) alternative methods for staff development opportunities. The knowledge subgroup consists of items regarding disabilities, the law germane to disabilities in higher education, accommodations, policies and procedures, universal design of instruction and assessment, accessible online instructional material, and the office of disability services. Items regarding practices include provision or willingness to provide accommodations, inclusion of a paragraph in syllabi regarding needed accommodations, incorporation of principles of universal design and accessibility of electronic instructional materials in teaching, and development and/or dissemination of policies and procedures germane to students with disabilities. The attitude domain is assessed by items regarding the fairness of specific accommodations, policies, and modifications to department or institution requirements vis a vis students without disabilities. In addition, faculty were asked about their assessment of the abilities of students with specific disabilities to complete the requirements of certain professions and to perform satisfactorily once employed. Faculty were asked about their interest in specific topics for future learning opportunities and preferred methods to acquire information such as workshops, speakers, web-based information, or credit and non-credit e-Learning courses. The final section of the questionnaire pertains to demographic information. (Sample items from the faculty questionnaire appear in Figure 1.)
One of the steps in the revision of the 1998 questionnaire was to solicit and incorporate feedback from a representative group of colleagues from various institutional perspectives, including teaching faculty, researchers, administrators, and disability service providers from a variety of institutions (public/private, competitive/open admissions, undergraduate/graduate). These individuals were members of a professional group of colleagues in central and northern Illinois who were interested in students with disabilities in higher education. They were asked for feedback regarding item and directions clarity, possible item bias, and/or additional items needed. Their comments were incorporated, redundant items were deleted, and new items added to reflect cutting-edge issues and developments in the field.
In 2001, the faculty questionnaire was further refined, updated, and broadened to include all disabilities. Because of sensitivity surrounding items of fairness, several new items were developed to assess attitude toward students with disabilities. A group of students with disabilities who regularly participated in a focus group at the University of Hawaii (Stodden, personal communication) were asked to provide feedback on items that may have lacked clarity, were misleading, or possibly biased (with special attention to the fairness items). Respondents’ feedback was subsequently incorporated into the questionnaire.
Figure 1. Sample items from faculty questionnaire.
How knowledgeable are you regarding strategies to make online and electronic instructional materials accessible to students with print disabilities?
Response alternatives are on a Likert scale from 1-6 with 1) To a very limited extent and 6) To a very large extent, plus Not at all, and Not applicable.
How often have you included a statement in your syllabus regarding provision of accommodations for students with documented disabilities?
Response alternatives are on a Likert scale from 1-6 with 1) To a very limited extent and 6) To a very large extent, plus Not at all, and Not applicable.
Fairness Item. Indicate how fair it is for students without disabilities when students with documented disabilities are provided priority registration.
Response alternatives are on a Likert scale from 1-6 with 1) Unfair and 6) Fair
Professionals with disabilities may be as effective on the job as
professionals without disabilities in the same occupation.
Response alternatives are on a Likert scale from 1-6 with 1) Strongly disagree and 6) Strongly agree.
Item construction. The majority of items were structured and used a six-point Likert response scale ranging from 1, indicating low degree of support, unwillingness to accommodate, or strong disagreement with the statement, to 6, reflecting high level of support or willingness to accommodate, or strong agreement with the statement. For some items, there were additional response options, including “Not At All,” “No Need, ” “Not Involved,” “Don’t Know,” “Not Applicable,” or “ No Experience” (see Figure 1). Responses related to these additional response options were treated as missing data in quantitative analyses.
Administration of questionnaires. The questionnaires were administered in Years One and Three of the Enhancing Success for Students with Disabilities in Higher Education Project (http://www.niu.edu/enhancingsuccess), a project designed to enhance the overall campus climate and all campus constituents’ knowledge, willingness to provide accommodations, and practices regarding students with disabilities in order to enhance their academic success.
Reliability. Internal consistency reliability was computed for the four faculty composite variables: knowledge, fair accommodations, fair modifications, and faculty needs and interests across the two doctoral-degree-granting public institutions and one associates-degree-granting public institution that participated in the project. Although reliability of the fairness of providing accommodations and fairness of providing modifications were computed separately, these two constructs together are considered a reflection of attitude toward students with disabilities.
Cronbach’s alpha reliability for the four constructs indicated that the scores provided adequate evidence of the reliability of the items for each construct. The Cronbach alpha reliabilities were well above the threshold of .7 for all composites (see Table 1). Considering that these constructs consisted of between three and eight items, the reliability is considered greater than adequate evidence of the reliability of the scores for each construct in the three questionnaires.
Reliability of Faculty Composite Scores for Years One and Three
Faculty Needs and Interests
Note. Number of items is in parentheses.
Staff development opportunities to acquire more information on a wide array of desired topics were made available in Years Two and Three of the project and included online web-based expertise on the Enhancing Success for Students with Disabilities Web site (http://www.niu.edu/enhancingsuccess). The Web site included information on disabilities, legal mandates and accommodations, links to other university and national Web sites, readings linked to full text, and PowerPoint™ presentations. For those who preferred live speakers, the project sponsored a series of motivational speakers and workshops on learning disabilities, visual disabilities, and hearing impairments. In addition, a collection of media and readings in hard copy were donated to the main library, and department chairs were asked to inform faculty of their availability.
We also offered a tuition-free, e-learning doctoral-level graduate course (Disabilities and Higher Education) in Years Two and Three. The course presented information about the legal mandates pertaining to disabilities in higher education, types of disabilities, and accommodations. Also included were the concepts of universal design for learning and examinations, common policies and procedures germane to higher education, and alternative strategies to provide staff development, A prerequisite for the course was that participants were to be employed in a higher education setting so they could apply the knowledge they acquired and design and provide a staff development activity so they would become change agents in their own department and/or administrative unit.
Quantitative analysis. Frequencies were generated for demographic data. Descriptive statistics were calculated for all six-point Likert scale items. Frequencies were tallied and reported for those who responded “None At All (“NAA”), Likert scale responses 1 and 2 combined, and Likert scale responses 5 and 6 combined, since those responses were considered of special significance and utilized to plan staff development activities. Year One and Year Three item knowledge composite means as well as individual items were compared using independent samples t-tests. The significance level was set at .01 to avoid inflated Type I error and to maintain the family-wise error rate at a standard significance level.
The means and standard deviation were computed for Likert scale items to create four constructs of similar items (i.e., knowledge, fairness of accommodations, fairness of modifications, and need for further information). In this overview, only the subgroup of knowledge items for faculty were compared for Year One and Year Three to serve as a model of how the questionnaires can be used in summative evaluation. The mean scores on four knowledge items: knowledge about accommodations, knowledge about federal laws, knowledge about the office of disability services, and general knowledge about disabilities were statistically compared between Years One and Three. Independent-samples t-tests were used to compare means from the Year One and the Year Three samples. Effect sizes of mean differences were calculated and classified as small (i.e., .2 < d < .5), medium (i.e., .5 < d < .8), or large (i.e., d > .8; Cohen, 1988).
Qualitative analysis. The analysis of the open-ended questions followed a “general analytical strategy” (Yin, 1994, p. 103) that was applied in this within- and cross-case study with a descriptive approach that used terms from the questionnaires and quotes from the respondents to develop themes and nodes. These may be free (stand-alone) or hierarchical tree nodes with subcategories of children (Richards & Richards, 1994). The nodes allowed for quantification of occurrence and subsequent analysis to determine any relationship(s) between the nodes (Yin, 1994). The latter approach was completed through the use of the constant comparison method of data analysis with open and axial coding (Creswell, 1998). In Year One qualitative analysis of 846 comments from 413 faculty-generated three free nodes, five tree nodes, and nine children nodes. Illustrative quotes are included in the Results section.
We compared findings from Year One and Year Three and briefly described the staff development activities in the interim year. The focus was on: (a) Year One compared to Year Three information gaps and practices among faculty; (b) in-depth findings and supplementary research regarding the inclusion of a welcoming paragraph in syllabi; and (c) the use of the questionnaires in summative evaluation by comparing findings on the items that comprised the knowledge construct for faculty in Year One and Year Three.