![[Case Western Reserve University -- Toolbar]](/pix/lowpro.gif)
Self-Study, Volume I, Chapter VII: Assessment
The university has long been committed to assessing the degree to which we succeed in our overall missions of teaching, research, and service, and we have used a variety of techniques over the years to measure the various relevant aspects of these missions. However, these efforts have often been unsystematic, and the university realizes that it is now time to frame the issues within a comprehensive set of plans to achieve full, continuous assessment and the improvements that can result from such an approach.
The first component, that of student learning, has three stratified sub-sections: undergraduate, graduate, and professional. In previous studies conducted by ad hoc faculty committees on assessment established by the provost in the late 1980s, five outcome criteria were identified as essential. They are mentioned in a footnote in the current subcommittee report on undergraduate education, but they need repeating and highlighting here because they apply with equal force to the assessment of graduate and professional learning outcomes as well. These five are:
The university is committed to the systematic measuring of these criteria. Both traditional and new methods will continue to be developed. Successful, empirically tested programs that have been employed in parts of the university will be generalized as appropriate into and as part of the university-wide plan.
As a major part of its charge, the subcommittee on undergraduate education had responsibility for describing techniques of assessing learning outcomes currently applied in the university, and for generating a set of recommendations for future consideration and action. The subcommittee's response to this charge is contained in its report to the Steering Committee (see appendix), and has the Steering Committee's strong endorsement. The discussion here incorporates sections from the subcommittee's report to provide a summary of existing assessment measures.
In proceeding with its charge, the subcommittee required every department offering an undergraduate major to examine and report on its assessment methods, in this way ensuring the involvement - at least indirectly - of the entire faculty. The subcommittee's findings describe both internal and external measures of effectiveness.
Department-Based Assessments. CWRU currently has no mandated or formalized outcome assessment structure at the departmental level, and faculty continue to rely largely on conventional methods of evaluating student success and faculty (teaching) performance. Nonetheless, in the documents prepared for the subcommittee nearly all departments addressed the subject seriously and thoughtfully. (This discussion focuses primarily on majors in the arts, sciences, and management; assessment in engineering is reviewed separately.)
Traditionally, faculty have relied on indicators of student performance and progress through the curriculum to help them judge the effectiveness of their efforts. Nearly all departments mentioned such mechanisms as grades, formal and informal departmental advising and counseling, and required seminars, projects, performances, and internships. Several departments specified a more elaborate formal structure for continuous student monitoring and evaluation; art education, music education, and communication sciences, for example, have a formal multipartite evaluation structure, including capstone experiences and exit interviews. The English department is considering requiring a writing portfolio as a means of assessment. Other departments rely almost entirely on grades and conventional advising.
Acceptance of graduates into post-baccalaureate education and job placement are other measures of undergraduate educational outcome. Those departments whose majors need advanced education for career qualification measured their outcomes partly by the success of their graduates in entering first-rate graduate and professional schools, their success at winning fellowships, and their performance there. Some departments keep careful records of such outcomes: for example, 71 percent of CWRU astronomy majors have been admitted to graduate schools since 1960, 60 percent of biochemistry majors since that program began ten years ago, and 100 percent of the communication disorders majors in communication sciences. All nutrition graduates in the last three years have passed the National Dietetic Examination (well above the national average), and two-thirds of the accountancy graduates during the years 1981-85 have succeeded in becoming CPAs. Overall, about 37 percent of CWRU graduates go on to advanced studies immediately; many more proceed to graduate or professional education later.
Most departments also mentioned placement rates as an indicator of success. Statistics over the last eight years indicate that only about 9 percent of our graduates are still seeking employment six months after graduation. A few individual departments keep records. Placement success is at or near 100 percent in some departments that normally provide B.A./B.S. career paths (e.g., accountancy, music education, nutrition, and medical technology), and placement of graduates of the management program remains 'above acceptable levels.' Departments that mostly feed students into graduate programs also were able to cite favorable statistics, in the form of lists of ultimate positions taken (such as at elite universities).
Department-Based Assessments: Engineering. Overall evaluation of the effectiveness of Case School of Engineering degree programs is provided from several sources: corporations and graduate schools; performance statistics on the fundamentals of engineering licensing examination; and internal and external evaluation of senior design projects by faculty and engineering design competition judges, respectively. The senior design project of each graduating engineering student is evaluated in each degree program and provides the most comprehensive and systematic internal measure of the effectiveness of the program to the responsible department faculty. Statistics from the Office of Career Planning and Placement provide each department with information on employment and graduate school admissions. Other assessments are based on performance of a subset of the graduating engineering students:
Taken together, these assessment mechanisms provide reassuring indications that the Case School of Engineering is providing its students with an educational experience of high quality. Nonetheless, there is a need for continuing robust outcome assessment measures at the university level, as discussed later in this chapter.
Other Internal Measures. For the past six years, graduating seniors have been asked to complete a detailed senior survey, designed by the Higher Education Data Sharing Consortium and administered at commencement rehearsal every May, which provides valuable data for self-assessment. The overall response rate is usually around 60 percent (though much lower, unfortunately, for the class of 1994). Eighty percent or more of students completing the survey have expressed satisfaction with their education, and a decided majority indicated that they would encourage high school students to attend CWRU. The detailed results of the senior survey are studied in various university offices, including collegiate affairs, student affairs, and by the various deans. Several initiatives undertaken by the university over the last few years to improve both curricular and extracurricular life have begun from recognition of deficiencies identified in these exit surveys. These senior surveys should continue to be re-examined for depth and focus, adjusted for local terminology and conditions, and ways be devised to ensure even fuller response from the graduating class.
The Freshman Year Task Force was appointed by the provost in the fall of 1992 to discuss issues related to the beginning of the student's college career. Although there was much good news in the report (only 13 percent of freshmen in April 1993 were dissatisfied overall with CWRU, whereas 87 percent were satisfied or very satisfied), many areas for improvement were identified; detailed recommendations were offered under the headings of orientation, peer relationships, academics, advisor relations, faculty involvement, campus community, and facilities. Some of these recommendations have already been acted upon. The Freshman Year Task Force has since become a standing committee, and is proceeding actively toward implementing the recommendations. A questionnaire similar to that given to seniors is now being administered to the freshman class at the end of each academic year. In addition, a separate Quality of Life Survey has recently been instituted in order to provide continuous review of the university's residential programs. To the extent possible, similar information should also be obtained from all who leave the university without graduating.
Instructional Assessment and Quality Control. An additional method of evaluating instruction is provided by a variety of review mechanisms for maintaining and improving the quality of classroom teaching. Numbers of majors and enrollment statistics represent one kind of indicator of instructional success, and these are followed carefully by all departments. Since fall 1988, faculty in departments in the arts and sciences, engineering, and management are required by university policy to carry out detailed student course evaluations at the end of every undergraduate and graduate course with an enrollment greater than four. After grades are submitted, these anonymous evaluations are studied by the instructor and department chair, and also (in aggregated form) by the dean. In addition, some departments (e.g., biochemistry and history) have additional formal structures for regular peer review of both curriculum and teaching effectiveness, for every instructor and every core course.
The student evaluations enter materially into not only every promotion and tenure procedure, but also the annual performance reports of every faculty member regardless of rank. The evaluation statistics for each course are made available in hard copy and on CWRUnet for all to view. Instructional deficiencies identified by this mechanism are further addressed by chair and peer review (e.g., classroom visitation), peer advising, and departmental tracking. University policy also mandates the collection of additional evidence of quality of classroom teaching - such as blind letters from former students or from colleagues who have had direct experience with the candidate's classroom performance - for all promotion and tenure actions.
External Outcome Measures. External independent measures of the quality and effectiveness of undergraduate programs are few, and the ones that do exist are generally (and justly) regarded as problematical. Nonetheless, such data are at least suggestive. Because of the extraordinary degree of overlap of courses, programs, and extracurricular experiences in college, it can well be argued that departmentally-based outcome assessments are uncertain, or even illegitimate; assigning identifiable 'value-added' outcomes to specific educational events may not be valid. Moreover, departments themselves do not directly control access to job markets. Consequently, more general measures of the entire college experience may be a better approach. Some data on the entry point are ready to hand: SAT, ACT, and Advanced Placement scores, high school GPAs, and so on. Statistics on exit are also available, in the form of standardized advanced examination scores - GRE, MCAT, LSAT, etc. - as well as professional credentialing examinations in a variety of fields (engineering, accounting, nursing, etc., some of which are mentioned in the departmental assessment sections above, and some below).
All of these measures tend to indicate that CWRU graduates are well equipped to compete successfully in the markets they enter after graduation. But CWRU is highly competitive in admissions policy; excellent entering freshmen tend to be excellent departing seniors. What these data fail to measure, then, is the value-added educational increment attributable to the four years students spend on our campus. There is also no structural feedback loop connected to these measures, to provide substantive policy responses when deficiencies are identified. There are private companies (such as the Educational Testing Service and the American College Testing program) that are beginning to provide instruments that could be used for this purpose. The Undergraduate Assessment Program and College-Level Examination Program of ETS, or the College Outcomes Measures Project at ACT could be used to provide the incremental value-added data which remains hidden by the more traditional measures.
Many of the measures discussed above have long been established at CWRU and elsewhere; what has been lacking has been an approach that integrates them into a unified and structured program of long-term assessment. The subcommittee's report emphasizes that most measures focus on criteria (1) through (4) of the five outcome criteria identified at the beginning of this chapter, since in most cases CWRU undergraduates continue their education and training in a graduate or professional context. Indeed, their placement in quality graduate or professional schools constitutes the basic way in which our undergraduate assessment effort can best address criterion (5) of the five outcome criteria.
A measure that applies to all CWRU students, drawing on observations and evaluations by graduates of the institution, takes the form of reports by alumni themselves of personal and career development that appear in the various alumni magazines issued by CWRU and several of its departments and schools. These reports provide useful information in evaluating program effectiveness and have proved to be of particular value at the graduate and professional level, where career paths are somewhat more homogenous than among undergraduate alumni. Beyond that highly impressionistic measure, however, plans and programs for assessment have been developed independently by the various schools for their discrete needs and objectives.
School of Graduate Studies. The school has traditionally deferred to the autonomy of individual departments offering graduate programs, but it has now developed a standardized evaluation form to be completed by the members of each doctoral candidate's Ph.D. oral defense committee, as well as standardized guidelines for evaluating quality and progress, to be transmitted to the students and to their research advisors. The school is also actively planning to develop an exit interview instrument.
Mandel School of Applied Social Sciences. The Mandel School currently tracks the post-graduate employment of M.S.S.A. program alumni through its annual development efforts. The information compiled indicates that graduates hold positions from entry to executive levels in organizations that span the full range of health, education, and social welfare services. The university's 1992 alumni survey, described later in this chapter, gathered information on employment, salary, involvement with professional associations, advanced study, and licensure and certification. In connection with the school's self-study for re-accreditation by the Council on Social Work Education, recent M.S.S.A. graduates (1990-92) were also surveyed for their evaluation of how well the program met its stated educational objectives.
The Mandel School has appointed a task force on social work assessment with a charge to become familiar with the theory and philosophy of assessment, identify the major components of assessment programs, and develop several models of assessment programs that would be relevant for the school. The task force will examine social work programs that have implemented assessment components; the program of the Weatherhead School of Management will also be studied. The task force will then develop a specific proposal for a social work assessment program based on faculty discussion and feedback; it hopes to have a proposal ready for consideration by the end of this academic year.
School of Dentistry. As part of its self-study for re-accreditation, the school has developed and is presently using a survey instrument for alumni, the professional community, and patients (in addition to current faculty and students) to assess program effectiveness. Outcome measures also include the results of state licensing examinations, which graduates must pass before entering into practice.
The school's Dental Education Committee is re-examining the curriculum to place new emphasis on criterion-based mastery of the core and is promoting faculty development programs to enhance mentoring of student research projects, problem-solving teaching approaches, and greater emphasis on critical thinking outcomes.
School of Law. At present the basic measures relied on by the school are course evaluations, the self-study process periodically required by the ABA re-accreditation review, data on student placement, and the comparative annual performance of graduates on the bar examinations of Ohio and other jurisdictions. Sixty to seventy percent of graduates take the Ohio bar exam, and the pass rate for CWRU graduates has been consistently higher than the state-wide rate (92 percent of 1994 graduates who took the July exam passed it.) A proposal to develop an exit interview instrument might be considered in the near future.
Weatherhead School of Management. As discussed in Chapter IV, the Weatherhead School introduced a competency-based M.B.A. curriculum in 1990 that drew national attention for its emphasis on adding value to the individual skills and knowledge that students bring to the program. Systematic outcomes assessment instruments are used extensively, including course evaluations, exit interviews, and a special two-phase (or value-added) program in which a representative sample of each graduating M.B.A. class engages in exercises designed to measure improvement in competencies and skills in 22 specified areas that were initially assessed at the time of matriculation. (Weatherhead faculty discuss the development of the program and its assessment of outcomes in Innovation in Professional Education: Steps on a Journey from Teaching to Learning, published recently by Jossey-Bass.) This two-phase program has been in place since 1992, and it is a priority of the school to develop similar measures for its other academic programs and degree tracks as well.
School of Medicine. The medical faculty's Committee on Students is charged with evaluating all aspects of student performance (including knowledge, skills, and personal characteristics) that are important to the development of a responsible, competent, and humane physician. The committee reviews the performance of every student after completion of each of the four years of the M.D. program, determines each student's continuing status, and makes recommendations of candidates for graduation.
External outcome measures include the results of the U.S. Medical Licensing Examinations, taken in the third and fourth years of the program; each student must achieve a passing score, set by the National Board of Medical Examiners, in order to graduate. Both the pass rate and mean score of CWRU students on USMLE Step 1 and 2 are generally higher than the national pass rate and mean. Student performance on USMLE Step 1 has resulted in curricular and other changes, including revision of evaluation mechanisms in the first two years and in-depth evaluation of basic science teaching. The graduation questionnaire of the Association of American Medical Colleges, completed by all graduates, provides feedback on the curriculum; changes in physiology and anatomy instruction were supported by data from this questionnaire.
CWRU medical students have also been successful in obtaining the residencies of their choice in the field of their choice. For the 1993 National Resident Matching Program, 73 percent of CWRU students gained their first choice of program, and 92 percent one of their first three choices. Residency match data indicate an excellent outcome, and quality is inferred from the high percent of CWRU students receiving their first choice. The school recognizes the need for information on licensure and career outcomes of its graduates, and is exploring sources of available data.
A Continuous Quality Improvement working group of students, faculty, and administrators has focused attention on problem-solving and problem-based learning (thus de-emphasizing lecturing and emphasizing small group interactive learning), the development of self-directed and personal-learning student programs, and a comprehensive and standardized system of student evaluation and remediation of their courses and hospital clerkships.
Frances Payne Bolton School of Nursing. External accreditation reviews are done by the National League for Nursing and the State Board of Nursing, and the Council on Certification of Nurse Anesthetists. A faculty evaluation committee exists to implement and monitor course and program evaluations. In response to a directive from the NLN, the evaluation committee is currently developing an assessment questionnaire that will be sent to all nursing graduates both one year and five years after graduation. The May 1995 graduating class is expected to be the first to receive the questionnaire.
A global survey instrument, developed in 1987 by the Office of Development and Alumni Affairs, has provided a periodic (every half decade) report of alumni attitudes toward their CWRU learning experience and its relationship to patterns of alumni involvement. Results from the 1987 and 1992 surveys show that nearly 68 percent of 1987 respondents and 78 percent of 1992 respondents reported positive or very positive attitudes when asked about their experience at CWRU. Fewer than 4 percent of respondents to both surveys offered negative or very negative responses to the same question.
The Office of Development and Alumni Affairs will continue to survey alumni on a regular basis. However, for logistical and economic reasons, the office plans to survey only groups of alumni (reunion classes, for example) each year, instead of all alumni every five years. The development office is also interested in modifying its current survey to assist with the university's outcome assessment goals.
It is clear from the above review that every CWRU academic unit is, or is beginning to be, actively focused on developing adequate and appropriate assessment instruments. The most frequently adopted approaches include course evaluations, exit interviews, alumni tracking (i.e. placement, career development, etc.) and, in the case of the professional schools, the search for school-specific self-study strategies necessitated by professional credentialing and re-accreditation. Thus, despite the obvious and proper differences among their missions and objectives, the university's undergraduate, graduate, and professional programs share a common concern for accountability and effectiveness. The Steering Committee believes the opportunity exists to unite these disparate efforts in a common effort to develop a comprehensive university-wide program. We propose the following overall guidelines and timetable.
The Steering Committee strongly endorses the appointment by the provost this fall of a standing University Committee on Assessment. The committee includes faculty, student, and administrative representatives, with persons from each of the constituent academic units (applied social sciences, arts and sciences, dentistry, engineering, graduate studies, law, management, medicine, and nursing). The committee's preliminary charge, to be defined in more detail as its work progresses, is to review and utilize information on all existing or planned assessment measures throughout the university, including the steps achieved by the Provost's Assessment Task Force of 1987-1989; to survey the steps taken at other comparable private comprehensive research universities; to test and to build on the existing assessment programs and to encourage additional pilot or experimental programs; and, finally, to present a comprehensive and systematic plan to the entire university community for its consideration, approval, and implementation. The Steering Committee envisions a time frame to accomplish these tasks that would span five academic years:
In short, the plan to be developed by the Assessment Committee will address these critical needs:
Among the university's efforts to improve the quality of the undergraduate student experience has been a renewed emphasis on the importance of undergraduate teaching, often overlooked in the culture of a research university. The quality of teaching is receiving greater attention in the evaluation of faculty for promotion and tenure, as noted earlier in this chapter, and a number of other initiatives are underway to foster innovative undergraduate teaching.
Lilly Fellowships. In 1991 the Lilly Endowment awarded CWRU a three-year, $136,000 grant to support undergraduate teaching. The fellowships were awarded on a competitive basis by a faculty committee to non-tenured, tenure-track faculty to create or redesign undergraduate courses using creative teaching approaches. The awards covered teaching assistants, supplies, and salaries for replacement lecturers to allow the Lilly fellows release time from teaching one course while working on their project. Senior faculty mentors were appointed for each of the Lilly fellows, who met regularly to share ideas and teaching techniques. The Lilly project has also sponsored several conferences on teaching at CWRU, open to all faculty, that have brought national experts to campus.
Fifteen Lilly Endowment Teaching Fellows have been named over the grant period. Course initiatives supported by the Lilly program range from collecting oral histories and exploring how religious views can affect attitudes toward the environment, to using negotiation theory to teach business French and growing 'fast plants' to enable students to study more of a plant's life cycle in one semester. Though not required by the terms of the award, a number of the Lilly fellows have used the capabilities of CWRUnet in designing their courses; examples are projects to integrate computers into courses in psychology and chemistry, and to develop software that uses animation to help students understand the dynamic behavior of computer data structures.
Courses developed through the Lilly fellowships will be integrated into the curriculum. The program has influenced how the Lilly fellows and their colleagues teach in other classes as well, and has created a group of faculty who will serve as leaders for future teaching initiatives.
Hewlett and Nord Initiatives. Discretionary funds from the Hewlett Foundation were made available by President Pytte for senior faculty to develop innovative undergraduate teaching projects. Among the projects supported by Nord funds were tutoring workshops in introductory math and physics. The workshops were based on national models that have been successful in increasing faculty participation in tutoring, encouraging collaboration and cooperative learning, and reaching students who might otherwise drop out of math and science courses.
Awards from the university's Nord Endowment Fund for Innovative Academic Programming will support five new teaching projects proposed by faculty in the College of Arts and Sciences and Case School of Engineering. Projects must meet three criteria: an interdisciplinary approach to complex human problems, global orientation, and enrichment of the Northern Ohio community.
University Center for Innovation in Teaching and Education. Building on the Lilly, Hewlett, and Nord programs, CWRU has created a faculty teaching center to stimulate university-wide innovations and improvements in teaching and educational programs at all levels. The center will offer services to help individual faculty improve classroom teaching, including videotaping, observation, self-evaluation materials, and presentations by experts in education; award grants for innovative programs proposed by faculty; and serve as an advocate for a holistic approach to scholarship in which learning and teaching are viewed as a single enterprise. Among the center's early activities will be a seminar for new faculty to discuss issues, approaches, and problems of first-year teaching. The center director, a tenured faculty member in the Department of Biology, will meet regularly with an internal advisory committee of senior faculty from all CWRU schools. The advisory committee will help develop ideas for use of faculty support funds, and will in general provide the first level of faculty input on all activities sponsored or proposed by the center.
Graduate Teaching Assistant Training. With the support of the provost and the Dean of Graduate Studies, teaching assistant training has become an accepted and necessary component in the professional development of new graduate teaching assistants. The university depends on TAs as lab assistants, recitation leaders, graders, tutors, and, in some cases, classroom instructors. These graduate students contribute significantly to the quality of undergraduate life, and the effectiveness of their interactions with undergraduates is of considerable concern to the university.
Training of graduate teaching assistants has evolved over the past seven years from a series of voluntary seminars to a non-credit course, University 400, required of all new TAs. The training is designed not only to improve instructional skills but also to introduce graduate students to the academy and to provide a forum where communication skills, essential in all professional arenas, can be modelled and discussed. The training is coordinated by the Office of Educational Support Services, with the guidance of a faculty advisory committee.
University 400 consists of an orientation for all new TAs, an additional orientation for new international TAs, and a series of eight seminar options, all of which include faculty as presenters or discussants. Successful completion of UNIV 400 requires attending one or, for international students, both of the orientations and at least three different seminars. International teaching assistants are also evaluated for spoken English language proficiency using the SPEAK test. Students who do not demonstrate adequate spoken English skills are further required to enroll in UNIV 400(C), a twice-weekly non-credit English conversation class. For fall 1994, 67 students are enrolled in UNIV 400(A), Professional Development for Graduate Teaching Assistants; 53 are enrolled in UNIV 400(B), Professional Development for International Graduate Teaching Assistants; and 36 students are enrolled in UNIV 400(C), ITA Communication Skill Development.
The university is continuously engaged in evaluating the research conducted on the campus, as well as the environment for this activity. The special emphasis report on research, discussed in Chapter VI of this self-study, describes the various approaches used recently in this effort, including:
In addition, faculty members submit annual reports to their deans with information on individual accomplishments, including research and scholarship, honors and awards, editorships, leadership roles in professional organizations, and similar contributions to their discipline.
Next Chapter - Conclusion
Previous Chapter - Areas of Special Emphasis
Table of Contents
![[Toolbar]](/pix/ribbon.gif){kind=small}