IR Standard 2

During the last decade, the professional preparation of educators has emerged as a national priority and an object of vigorous accountability mandates. Presidents, federal legislators, state and local governments, business and industry, professional associations, and parents demand greater accountability for educator preparation programs through assessments that demonstrate the impact of graduates on PK-12 student learning (National Commission on Teaching and America’s Future, 1996). Adding greater complexity to these calls for increased accountability are state and national mandates for schools, colleges, and departments of education to address new academic standards for PK-12 students and to prepare individuals to work in diverse classrooms and schools. Anticipating the issues surrounding the debate over accountability, we initiated a transformation of our educator preparation programs to incorporate multifaceted assessment systems linking the performance of our candidates to national and state standards and student learning. With the involvement of our professional community, we are implementing assessment systems to assess candidate performance and our educator preparation programs relative to our Standards for Beginning Teachers and Standards for Advanced Professionals delineated in our conceptual framework and the Idaho Standards for the Certification of School Personnel.

Assessment Systems

Using the research base on assessment and knowledge of best practices, we worked with our partners in PK-12 schools to develop an assessment philosophy and principles that guide our design, development, implementation, and use of the components of our assessment systems. Our assessment systems include comprehensive and integrated sets of evaluation measures that are used to monitor candidate performance and manage and improve operations and programs. These evaluation measures include (1) course evaluations that assess candidate perceptions of the quality of teaching and learning in program courses; (2) an annual University Graduate Survey conducted by the Office of Institutional Research to gather perceptions of graduates regarding the quality of University programs and services; (3) College of Education Student Advising Center exit surveys that gather candidate perceptions of the quality and accessibility of advising in the College of Education; (4) annual alumni and employer surveys conducted by the College of Education to gather data on program effectiveness; and (5) program assessments of candidate knowledge, skills, and dispositions. In terms of program assessments, our teacher education assessment system is used to assess candidate performance of all candidates enrolled in the program (e.g, early childhood education, elementary, secondary, special education, etc.). Each of our programs for the continuing preparation of teachers and the preparation of other school personnel (i.e., instructional technology, educational administration, school psychology, etc.) has a unique assessment system tailored to the goals of the program. All of the assessment systems in the College of Education share a common framework with the following requirements:

Candidate knowledge, skills, and dispositions are assessed relative to institutional and state standards.
Program faculty collaborate with PK-12 educators and faculty in other programs (including arts and sciences, where appropriate) to design and implement the assessment system.
The program assessment system includes multiple assessments that assess candidate performance in a variety of ways and in a variety of contexts.
All assessments have scoring rubrics for determining levels of candidate performance.
Candidate performance is assessed at multiple points during the program and data is used to make decisions regarding program admission, retention, and completion.
Graduate program assessment systems must include College-wide measures of candidate performance and program effectiveness (e.g., GRE scores, admission grade point average, practicum/internship evaluations, grade point average for retention in program, final scholarly activity, oral examination, course evaluations, follow-up studies with graduates) in addition to program specific assessments.
The assessment system includes external assessments, such as technology competency assessments or the Praxis II, as mandated by the state.

Our program assessments of candidate performance are based on the concept of assessment-as-learning (Alverno College Faculty, 1994). As such, we envision assessment as a process integral to learning that involves observing performances of candidates in context, judging those performances on the basis of our program standards, connecting educator work to PK-12 student learning, and then providing the resulting feedback to candidates. Our gradual development of the meaning of assessment was part of designing a process that would ensure candidates of direction, resources, and coherence of instruction throughout our educator preparation programs.

The complex nature of our Standards for Beginning Teachers and Standards for Advanced Professionals demand assessment systems that are multiple in mode and context. Therefore, our assessment systems incorporate traditional forms of testing as performance tasks and self-assessments that are completed in both simulated and authentic contexts through the course work and clinical experiences of our educator preparation programs. For example, to assess candidate professional and pedagogical knowledge and skills we use course-based examinations, case study analyses, teacher work samples, teaching performance evaluations, and interviews. As indicated in our admission, retention, and graduation criteria, we use data from these assessments to make decisions about candidate performance at multiple points during the program.

Our assessment systems rest on the assumption that assessment should be developmental and continuous; that is, candidates should have opportunities to learn by building on what they already know and are able to do and to carry forward these skills and knowledge to expanded and more complex uses. Within this framework, through assessments at each stage of our educator preparation programs, candidates are expected to demonstrate at increasingly higher levels of performance and complexity, the knowledge, dispositions, and skills set forth in program standards. For example, in our teacher education program, we use our teaching performance evaluations and teacher work samples for assessing candidate performance during both the pre-internship and the internship. The assessments employ the same scoring rubric and evaluation criteria and we expect to see growth in candidate performance relative to the criteria as they progress from the pre-internship to the internship. Through these assessments, candidates receive focused feedback on their performance and work with faculty to identify ways to develop knowledge, dispositions, and skills relative to program standards.

As an essential element of the implementation of our assessment systems, we have embarked on validity and reliability studies to establish the fairness, accuracy, and consistency of our assessments and assessment procedures and to investigate the relationship between our performance assessments and candidate success in the program and the profession. One set of studies involves the ongoing tracking of candidate performance from program admission to program completion and the first year of professional practice. We have created the infrastructure for these studies through development of our electronic databases. As more of our candidates complete programs using the newly implemented assessment systems, we will have data showing the relationship between their performance on various assessments during the program and their performance following program completion. A second set of studies involves gathering credibility evidence of specific assessments including the teacher education program admission interview, the Idaho Technology Portfolio Assessment, and the teacher work sample. Through these studies we are gathering data regarding the validity of the assessments and the inter-rater reliability of scorers. Preliminary results of these studies have been presented at recent AACTE meetings.

Data Collection, Analysis, and Evaluation

As we have implemented our assessment systems, we have developed processes for collecting, analyzing, and reporting data on candidate performance, program effectiveness, and unit operations. Critical components of our data management system include two electronic tracking databases – the Teacher Education Tracking Database and the Graduate Program Database. We have worked with University Computer Services and the Registrar’s Office to connect our databases directly to the university student information database so that information from the university system is automatically downloaded to our databases. The Teacher Education Tracking Database was finished and online in fall of 2000, and we subsequently began entering candidate assessment data. The Graduate Program Database was completed in July of 2001 and is currently being used to track the graduate program admission process and to generate statistical reports on GRE and course grades; other candidate assessment data will be entered beginning in fall of 2001. Through our tracking databases, we are able to manage data on candidate performance; track candidate progress relative to program admission, retention, and graduation; and conduct data analyses to prepare summary reports for internal and public dissemination.

Assessment of candidate performance and program effectiveness is a college-wide activity coordinated through the Office of the Associate Dean. Course evaluations and alumni and employer surveys are conducted by the office, and the Associate Dean prepares the annual outcomes assessment reports for the Office of Institutional Research, summarizes data for accountability reports (e.g.,Title II, accreditation, etc.), and works with University Computer Services and the University Registrar to manage the teacher education and graduate databases. The Associate Dean also prepares periodic assessment reports that are forwarded to the Dean, Assistant Dean, and department/division chairs.

While the Office of the Associate Dean provides direction and support for the College of Education assessment program, the faculty are responsible for developing and implementing the assessment systems in individual programs. Assessment data are reported to faculty via the College of Education Assessment Committee comprised of faculty members from all programs and chaired by the Associate Dean. The Assessment Committee develops policies relative to assessment activities in the College, assists programs in developing and implementing assessments, reviews candidate performance data and other information, and makes recommendations on the basis of assessment data for the improvement of programs and services.

Use of Assessment Data for Program Improvement

Through regular summary reports of assessment data, the College of Education Assessment Committee, and a network of program and course planning and coordinating committees, we constantly use assessment data for reflection on candidate and faculty performance and to initiate program changes where needed. These mechanisms also ensure that we constantly monitor changes to ensure that intended program strengthening occurred and that there are no adverse consequences.

Through our assessment systems, candidates are required to reflect on their performance and develop plans for improvement. For example, in their teacher work samples teacher education candidates must plan and deliver an instructional sequence, assess student performance before and after instruction, analyze student learning, and then reflect on their practice in light of the learning of their students. As part of their reflection, candidates must consider areas of their practice that should be developed and improved. Another example of how we guide our candidates to reflect on their practice for improvement is the performance evaluations used during the clinical experiences in both our initial and advanced programs. Candidates receive explicit standards-based feedback regarding their performance and then confer with the university and onsite supervisors to identify areas for improvement. These areas then become the targets for subsequent performance evaluations. Data from the performance evaluations are also used to inform faculty of the efficacy of courses and clinical experiences in developing candidate knowledge, skills, and dispositions relative to program and state standards.

We have established additional policies and procedures through which we systematically use data on candidate performance to judge the quality of program components and instruction. For example, the Assessment Committee routinely reviews data from surveys with graduates and employers to evaluate the extent to which our programs are addressing institutional and state standards. Another instance of our ongoing use of candidate performance data for program improvement is the work of the teacher education professional education core course planning and coordinating committees. One of these committees, comprised of the EDUC 309 instructors, reviews candidate performance data on the teacher work samples completed in their courses to determine if, as instructors, they are providing course content and instruction that develops the knowledge and skills necessary for successful completion of the teacher work sample. Through data from the assessment of teacher work samples, individual faculty members and programs are informed of candidate attainment of professional and pedagogical knowledge and skills and demonstration of teaching that leads to PK-12 student learning.

We have also established processes through which faculty continuously and systematically use data to reflect on and improve their own practice. Our course evaluations completed by candidates for every course every semester serve as an important source of data for the annual evaluation of individual faculty members. Our faculty mentoring system and annual professional development plans ensure that faculty continuously work with their peers and supervisor to reflect on performance and develop plans for improvement.

An important part of our assessment systems is program assessment involving both practice and scholarship to support ongoing improvement and demonstrate accountability. Critical elements at the heart of our assessment systems include explicit standards for candidate performance and a clear feedback process through which assessment is tied to ongoing revision and improvement of the curriculum, instruction, and assessments. Following are some selected examples of our use of assessment data for program improvement:

Modification of the Teacher Education Professional Education Core. From data gathered via course-based assessments and follow-up surveys with program graduates, we learned that the teacher education program needed to include more specific course work on adaptations for students with special needs and students with diverse backgrounds and abilities. Based on this information, the professional education core was redesigned to include three courses targeting the knowledge, dispositions, and skills required for accommodating instruction, curriculum, and assessment to meet the diverse needs of students.

Increase in Teacher Education Program Field Experiences. Data from the student teaching intern survey, the follow-up survey with program graduates, and evaluation of candidate teaching performance during the student teaching internship indicated that candidates, especially those preparing to be secondary school teachers, needed more experience in authentic teaching contexts. As a result of this assessment data, we revised the teacher education program to include a 150 hour pre-internship during the junior year.

Revision of Assessment Protocols and Scoring Rubrics. On the basis of our benchmarking studies with the teacher work sample, we revised the protocol and scoring rubrics for the assessment. Revisions included clearer directions on the types of student learning data that must be included in the student learning analysis section of the work sample. We also worked with Ann Harmon from the National Board for Professional Teaching Standards to develop an holistic scoring rubric that categorizes candidate performance on a developmental continuum from beginning to exemplary.

Creation of the College of Education Advising Center. Data from both the student teaching intern survey and the graduate follow-up survey indicated that candidates felt the need for more accessible and better quality academic advising. Based on this data, we established the College of Education Student Advising Center staffed by a full-time advising coordinator, graduate assistant, and undergraduate peer advisor.

Addition of Diversity Course Work to Graduate Programs. Data from course-based assessments and performance evaluations completed during internships indicated that candidates in our graduate programs lacked the necessary knowledge and skills to effectively address diversity and exceptionality in their professional practice. As a result of this information, the school psychology faculty added a new course, CFS 631 Family Diversity, to the required course work for the masters program in school psychology and the literacy faculty added a new course, EDUC 634 Literacy: Multicultural Views, to the required course work for the masters program in literacy.

Faculty Professional Development. Data from faculty professional development plans and surveys conducted by the Diversity Committee and the Technology Committee indicated that faculty needed opportunities to develop their knowledge and understanding of diversity issues and to develop the skills to effectively integrate technology into teaching and learning. As a result of this data, we initiated a series of monthly diversity awareness activities that included lectures and presentations specific to cultural groups in southeast Idaho, and we began a Visiting Scholar program that brought a scholar in multicultural and diversity education to the College of Education to model instructional strategies for faculty. To address the development of technology integration skills, we held a faculty inservice on the use of technology resources, created a website to disseminate information about technology use, and hired a full-time technology support specialist.