Assessment System and Unit Evaluation

 The School of Education at The College of William and Mary has developed an assessment plan that will result in an assessment system to provide information that can be used to assess individual student’s progress, program outcomes, and unit performance. The initial draft of the assessment plan was developed primarily through the interaction of the Conceptual Framework Committee and the Assessment Committee. Both of these committees were comprised of two representatives from each of the School’s three recognized Areas. Following the adoption of an initial draft, the Assessment Plan was reviewed and adapted by the Faculty of the School, the Teacher Education Advisory Committee, and the members of the Clinical Faculty. The review and adaptations by these groups insured that the assessment plan was developed collaboratively by School of Education Faculty, Arts and Sciences Faculty, and field-based practitioners including teachers, counselors, school psychologists and administrators.

The assessment system centers on the Conceptual Framework of the School and incorporates state and national standards that are appropriate to each program. An essential feature of the assessment system is that it manifests differently for each program in the School of Education. Thus, while there is a general assessment plan for the School of Education, there are multiple assessment systems that reflect the different foci and needs of our programs.

The information that follows is a description of the general plan for the School of Education. The specific plans for each Area are included in Program Folios in the Evidence Room.

General Assessment Plan Elements

Transition points at the unit and/or program level.  The School of Education has decided on transition points at entry, during the program (major points prior to entering clinical practice and exiting clinical practice when appropriate), immediately prior to program completion, and post graduation follow up for all of the programs within the School. The specific timing and nature of the evaluations does differ by program. The specifics for each program are detailed in each program’s assessment plan.

Major assessments to be used at the stated transition points.  The major assessments that are to be used by programs are delineated in the specific program assessment plans. The table that follows includes a summary of major assessments for the School of Education unit plan.

Table 2.1. Major Assessments for the School of Education

^a Standards Addressed: NCATE, Specialty Organization, State, School of Education, Program Specific

Timeline for the development and implementation of the major assessments.  The timelines for the development and implementation of the major assessments are detailed in the specific program assessment plans. Table 2.1 indicates the general stage of development for the School of Education as a unit.

The Level 1 areas of admissions and graduation are well developed for individual decision making and program evaluation purposes. The processes in these areas are well defined, and data are recorded and aggregated at the program level. The data from these two assessment points are reported to various groups within and outside of the School of Education. The enhancements that are planned for these two assessment points are related to the use of a new rubric for admissions, the new requirement of Praxis II scores in certain programs, and the incorporation of aggregate data for the portfolio reviews.

The assessments within the preparation programs have been substantially changed and defined through the specialty area review and self-study process. The programs have developed and reformulated many of the assignments, projects, and rubrics in order to ensure alignment with professional standards and the Conceptual Framework of the School of Education. Since this assessment point includes the majority of the preparation programs including field experiences, many of the instruments were adapted and are being field tested with university faculty and field supervisors. An example of the reconstruction process is the Student Teaching Evaluation Form. The newest version of this form was derived from INTASC Standards for Beginning Teachers and William and Mary program objectives. It was constructed in conjunction with partnership schools and Clinical Faculty to reflect both the university and field perspectives. The form is organized around the twenty competencies that were identified through the discussions of standards and the Conceptual Framework. The reconstructed form arranged around the twenty competencies was just adopted in the Spring of 2003. Thus, there are no data for the competencies that were added. (Note that there are four years of data based on the Competencies and Student Teaching Evaluation Form in effect from 1999-2003.) The plan is to employ the new form in the Fall of 2003, make adaptations for the Spring of 2004, and conduct an outcome study in the Summer of 2004.

The Assessment System as planned will allow the collection and analysis of comprehensive and systematic data that will be used for candidate and program evaluation. Even at this stage of development and implementation, the data that have been collected has been utilized for program review and revision. Several examples of how the data available from the current Assessment System have been used meaningfully are described below.

 The Program Profiles (see Program Profiles in the Evidence Room) that were produced during the 2001-2002 academic year contain aggregates of many of the data points in the Assessment System. Administrative and Faculty review of these data led to several important decisions and initiatives including:

1. The allocation of open tenure-line  positions to need areas identified in the curriculum analysis;
2. A study of expanding the focus on student assessment and disposition assessments in early clinical experiences in Teacher Preparation programs;
3. An examination of overlapping curricular content in and across program areas;
4. Refinements of master schedules of program course offerings; and
5. Proposals for reducing reliance on part-time faculty in core content courses.

The discussions related to expanding the focus on student assessment in Teacher Preparation programs and overlapping curriculum across programs initiated a special taskforce on Psychoeducational Assessment that had a School-wide focus. The report produced from that taskforce (see Psychoeducational Assessment Report in the Evidence Room) is in consideration by the program Areas and may result in restructuring the manner Psychoeducational Assessment content is delivered. The report of the taskforce makes recommendations that would deliver the assessment content to classes with mixed program representation to strengthen the focus on cross-disciplinary collaboration. In addition, the proposals in the report would extend the instruction on assessment for Teacher Candidates to a level comparable to the specialists in other areas. The additional understanding of measurement and assessment should position the Teacher Candidates to become leaders in the interpretation of assessment data.

Student and faculty evaluation data on the master’s Teacher Preparation program indicated that during the first summer there were issues related to one of the capstone courses in the final summer sequence. Current Curriculum, Instructional, and Assessment Issues in Schools (CRIN 591) was perceived by the Teacher Candidates to be too preparatory and exclusive of their recent student teaching experience. Teacher Candidates also believed that the course should be taught in the early part of the summer term so that the it would be easier to link to their student teaching experience and would be complete in time for them to devote attention to their culminating projects. Program faculty conducted a year-long analysis and planning on the CRIN 591 course and made several adjustments. The faculty decided that a full-time faculty member would be responsible for the course; in fact, it was co-taught by two core faculty in 2002 to pilot the revised curriculum. The course would address advanced issues of curriculum development and assessment and be moved to an earlier session of the summer term. Recent data from Teacher Candidates indicate that the changes to structure and delivery are favorable.

The EPPL area utilized the data from admissions, exit interviews, and focus group interviews to make significant modifications to their Ed.D track. Admissions trend data indicated that the doctoral emphasis in General Administration was attracting fewer applicants than in past years and was in danger of graduating fewer than the targeted goal for the program. The EPPL program faculty incorporated questions into their individual exit interviews and focus group interviews that attempted to discover the reasons for the downward trend in applications. The data from these processes indicated that the full-time residency requirement and total time until graduation were perceived as impediments even to individuals in the program. Based on this information, a new Ed.D. track was constructed that does not require consecutive semesters of full-time residency, has reduced coursework that still meets state endorsement requirements, and offers the opportunity for part-time study. Data for the October, 2003 admissions cycle indicate that applications to General Administration are increased from last year.

Design for the collection, analysis, summarization, and use of data.  The specific unit assessment plans indicate the use(s) of the collected data. Data will be used to assess individual student’s progress, program outcomes, and unit performance. Data will be aggregated and summarized at least at the program level for specific program evaluation, planning, and feedback. Several core elements that relate most directly to the Conceptual Framework and unit functioning will be summarized at the unit level and used by various groups for program evaluation, planning, and feedback at the unit and College level. The School of Education has several standing committees that review data on Unit functioning.

The Admissions and Financial Aid Committee is tasked to monitor the selection decisions that are made by each program area. The Associate Dean for Academic Programs currently reports to this committee annually on the admissions decisions made during the prior year. The committee is responsible to make recommendations to the Areas and the School on admissions topics. This Committee and the Associate Dean for Academic Programs will continue to be responsible for monitoring the first key assessment point.

The Curriculum Committee currently approves all changes to the curriculum and grants exceptions to curriculum-related regulations. The Committee will be the conduit to the unit for data related to the second key assessment point, preparation program. Programs will be responsible for preparing reports of the aggregate data from the assessments collected as part of course work, semester reviews, and portfolios. The Associate Dean for Academic Programs and the Associate Dean for Professional Services will be responsible for aggregating data from field supervisors and test data. These data will be presented to the Areas and Curriculum Committee for their review and recommendations.

The Associate Dean for Academic Programs and the Associate Dean for Professional Services will be responsible for reporting to the Areas, the Faculty, and the College on the data related to the Level 3 assessment point, graduation. The Director of Records and Matriculation currently produces reports on graduation that are utilized by the Areas, School, and the College. These reports will continue to be produced and the data used for evaluation purposes. A parallel process is in place for license and Praxis test data through the Director of Professional Services. Reports will continue to be produced and shared at the Area, School, and College levels.

The Associate Dean for Academic Programs and the Associate Dean for Professional Services in cooperation with the Areas will be responsible for conducting the follow up studies related to the Level 4 assessment point, post graduation. Follow up studies will be conducted on a three year cycle with one Area being targeted each year. Follow up studies will employ a combination of mail and on-line surveys as well as focus groups. Program areas will be encouraged to conduct their own follow up studies and to share those data with the school. The School conducted a follow up study of graduates from the C&I area during the Spring of 2003. The questionnaire and results of this can be accessed in the Evidence Room (See C&I Follow Up Survey in the Evidence Room). An on-line version of the survey can be accessed at SOE.CAMPUS.WM.EDU/SURVEY.HTM. The methodology for the current survey was successful and will be utilized in subsequent surveys. Addresses of graduates of C&I programs since 1997 were obtained from University and Alumni/Development records. Notification of the survey was pre-mailed to 675 graduates who were asked to verify and update their addresses through the Alumni Association’s on-line system. A transmittal letter and survey were then sent to 634 graduates. The letter explained that the graduate could respond by answering on-line at a given web address or by returning the enclosed survey by mail or fax. Two hundred twenty one (35 percent) useable surveys were received and encoded for analysis.

Program Areas will review available evidence in September of each year. Based on their review each Area will produce a report to the faculty of the school that will detail the results of their review, the recommended actions, and a summary of the actions taken during the previous year.

Aspects of the system that address unit operations.  The School of Education collects data from applicants, candidates, recent graduates, faculty and members of the professional community. Data on the applicants are currently used by the programs, the Admissions and Financial Aid Committee, the Executive Committee, the Associate Dean for Academic Programs, and the Provost. These groups and individuals use the data to assess the individual applicants, track the admissions decisions of the programs, monitor admissions policies, and assess the quality of applicants, admits, and matriculated students.

Monitoring the progress of candidates has taken place primarily through the Offices of Academic Programs, Professional Services, the Registrar, and individual advisors. Recent changes brought on by the new Assessment Plan have introduced program level evaluations of candidate progress. Within all of the programs, data collection instruments and/or the rubrics have been adapted to match with mission, philosophy, and Conceptual Framework of the School. Program and unit level analyses of candidate progress have examined completion numbers, academic failures, and grades. As new information based on the transition point data is becoming available, it is being incorporated at the program, Area, and School levels. Programs are currently producing transition point data and using those data for individual candidate monitoring. Although there is not enough data on the new elements for complete, meaningful program or Unit analysis, the Assessment Plan incorporates available data into the Program Profiles that are used at both the program and Unit levels for planning and policy decisions.

Other aspects of unit operations that could also be evaluated as part of the unit assessment system include the effectiveness of advisement, record keeping, the admissions system, student teaching placement, and governance structures. In addition, provisions should be made to study the extent to which the requirements at the various transition points are adequate predictors of candidate success. The types of questions that might be posed by these studies include: Are candidates who barely met admissions requirements scoring as well on the assessments as those who exceeded the requirements? Do candidates with high scores on the assessments have higher scores on employer surveys than those who had weaker scores?

The use of information technology.  The College of William and Mary is in the process of implementing a new student information system (Banner). The Banner system replaces an out-dated legacy system that is no longer compatible with current technologies, including the web. Banner also enables the integration of recruiting, admissions, matriculation, registration, degree auditing, and graduation certification in one database. The installation of this new system is expected to be complete by Fall 2005.

To this point, the College has finished student information system business process analyses, installed and upgraded Banner and Oracle software, and received a range of training prior to going “live” with Admissions Prospects in August 2002. The Admissions go-live, the College's first major milestone in the Banner implementation, was accomplished successfully. It was followed by the first on-line registration in the Spring, 2003. The Summer of 2003 will mark the first semester where registration, grading, and student academic history will all be recorded in the Banner system. Future developments include the degree tracking and audit capabilities, as well as an ability to track students longitudinally.

The implementation of the Banner system at the College has direct implications for the School of Education’s ability to utilize technology in the Assessment System. School of Education staff members are playing critical roles in the development of the Banner system, where we are working to incorporate as many of our data needs into the Banner system as possible. The goal is to incorporate all of the standardized test data, including Praxis exams, into the academic history components of Banner. Even when the Banner is fully implemented in 2005, the School of Education will need to track certain student and faculty outcomes in separate databases.

While it is hoped that much data and progress tracking will occur through the Banner system, the current development of the assessment system is being done within the School of Education. Currently, the School uses Access, a relational database tool, to collect, store, and import data on students. The areas related to admissions, degree progress, graduation, standardized test results, and certifications are all highly developed. These areas have sophisticated linkages to the legacy student information system of the College as well as intra-school connections. The data are used to assess individual student progress and to provide feedback to the programs, relevant School and College Committees, and the unit.

We have recently (Spring 2003) moved some of our data operations (collection, storage, and analysis) to a web-enabled dedicated server. The move to a dedicated system outside of the general student information system and shared network of the College was a necessity because of the inability of the general system to accommodate the detailed data required in the planned assessment system even when Banner is brought completely on-line. The security necessary for data that identify students complicates the implementation of a data system that is able to be accessed by multiple users over the web. Thus, fully implementing the Assessment Plan in an efficient and secure manner is dependent upon the ability of School of Education staff to develop and maintain the system internally. To this point, administrative access to all data and secure online data input have been accomplished. We have completed a successful online follow-up survey of graduates from the C&I area and are developing a web-based interface to enter data on practicum students and student teachers that will be tested in the Spring of 2004. Continued expansion of data entry and wider access to entered data are planned for the coming years.

While the College is in the process of implementing the Banner system, it has suspended any modifications to the legacy student information system and has stopped adding new information to the system. The challenge for the School of Education is to convert existing database functions from the legacy system to the new Banner system. In the best scenario, data flow will be disrupted only until existing database connections can be mapped to the new system. A more probable outcome is that data operations will be substantially disrupted as School of Education staff are required to work on the implementation of the new system to the exclusion of existing internal data systems. The Banner system will need substantial attention to initialize the system for basic operations, but the long term outcome is a College-wide student information system that enables better data collection and reporting.