Recommendations for the Content of a Program Review Self-Study

Introduction
Of the various components of the program review process, the written self-study is without question the most demanding, but also potentially the most rewarding. Because the primary purpose of a program review is to provide program faculty and academic administrators the information they need for effective academic decision making, program faculty have a great deal of discretion in what they choose to include in their self-study. The Dean of the College and the Associate Deans have not established specific College-wide requirements for the content of a self-study; instead, they have encouraged program faculty, in consultation with their Associate Dean, to highlight the most pressing issues and seek the information they most need to sustain what they are doing well, address what needs improving, and plan for the future.

The purpose of this document is to suggest some guidelines for the content of a self-study, based on what other programs have included in self-studies conducted at the College to date and informed by the new criteria for institutional accreditation established by the Higher Learning Commission of the North Central Association (NCA), the professional organization that accredits St. Olaf as a whole. These are offered as recommendations, not requirements. Programs that are separately accredited by other professional associations prepare their self-studies in accordance with the requirements of their accreditors, which may be more extensive than what is recommended here. In general, though, program faculty should feel free to include only what is useful to them and their readers, to add anything that is not listed below, and to omit those things that do not support their particular self-study learning objectives.

Factors to consider in determining the content of a self-study
1. Who is your audience? At present, the primary audiences for the self-study report are the program faculty themselves, the Dean of the College and Associate Deans, and the external reviewers and/or accreditors (in the case of programs which have their own external accrediting agencies). However, self-studies can have a number of secondary audiences, including potential funding agencies for future grant applications; prospective or newly-appointed faculty; other St. Olaf programs or faculty with whom the program may currently, or wish to, partner; potential collaborators at other institutions; or others who may play a role in future program activities. Keep the information needs of your audience at the forefront of your decisions about what to include in your self-study.

2. What questions do you most want to answer at this point in the life of your program? Like any inquiry, a self-study agenda should be governed by what you want to learn and convey to your audiences and how the results will be used (Patton, 1997). Many departments have found it helpful to establish clear learning objectives for their self-study before deciding what information to track down. There should be a reason for each item you decide to include in the final report. This can help you avoid collecting information that, in retrospect, you really didn’t need to spend time collecting (and that your audiences didn’t need to spend time reading). It may be useful to state the governing questions explicitly in the self-study.

3. What aspects of the larger context of the College are important for your program? The website of the Office of Academic Research and Planning has compiled a list of key College documents that most self-studies replicate or reference (see “Information resources to support the preparation of the self-study” below). Links to each document are provided on the E&A website for ease of access and printing. This is helpful material for the external reviewers to have in hand. It is also helpful for the program faculty to consider and reference explicitly in developing or revisiting their mission, goals, resources, and priorities.

4. What national trends in the disciplines and/or professional fields served by your program have a bearing on your program? Information about similar programs in comparable undergraduate institutions, the expectations of the graduate and professional schools to which program majors may apply, and trends in the professions program graduates enter, may have important implications for current and future program activities.

5. What criteria or benchmarks will you use to evaluate the strengths and weaknesses of your program? Choosing or developing appropriate criteria for the evaluation of your program, and articulating them explicitly in the self-study, is one of the most important dimensions of the program review process. There are a number of helpful resources, both internal and external to the College, to support the selection of criteria for academic program evaluation. Here are some examples:

· Principles for Appraising the Contributions of Departments and Programs to the St. Olaf Curriculum as a Whole, prepared by the St. Olaf Curriculum and Educational Policies Committee (CEPC) in May 2004.

· The standards for faculty evaluation established in the St. Olaf College Faculty Manual, Section 4.VII, pp. 32-33, available at:

http://www.stolaf.edu/offices/doc/media/new_faculty_manual__06-04-04_-1.pdf

Although these standards are used to evaluate the contributions of individual faculty members for purposes of reappointment, tenure, or promotion, they may be adapted for use in evaluating the collective contributions of a program as well.

· The accreditation criteria established by the North Central Association (NCA) Higher Learning Commission, the agency which accredits St. Olaf as a whole. These criteria, recently revised and effective as of January 2005, were designed to apply to an entire institution of higher education, but portions or versions of the criteria may be helpful in evaluating a program within the institution. The five criteria, the components of each criterion, and examples of the kinds of evidence institutions are invited to provide to demonstrate that they meet a given criterion, are available at:

http://www.ncahlc.org/download/2003overview.pdf

· Discipline- or profession-specific professional association(s), which often publish reports, recommendations, position statements, or other documents on teaching, scholarship, and/or professional service and outreach.

· Professional associations in higher education more generally, such as the American Association of Higher Education or the Association of American Colleges and Universities (www.aacu-edu.org), which also release reports, recommendations, position statements, and other documents on teaching, scholarship, and/or professional service.

Major areas of inquiry
Below is an outline of the major issues and questions that programs may wish to address in their self-studies. Most self-studies to date have referenced most of these topics, although programs have varied in the ways in which they have sequenced them.

I. Articulating program mission, goals, and historical context

A. Program mission statement

A program mission statement helps give identity, coherence and focus to the activities and outcomes of a program. Some programs already have mission statements, and use the self-study as an opportunity to revisit and sometimes revise it. Others develop one in the course of the self-study. Some programs link their mission statements to statements prepared by relevant professional associations. It is also helpful to discuss the program mission statement in relation to the mission of the College as a whole. Program mission statements often make their way to program websites.

B. Program goals and objectives

(1) Teaching

Programs are strongly encouraged to include in their discussion of their teaching goals a list of desired student learning outcomes. Some of these outcomes may reflect college-wide objectives, while others may be distinctive to more specialized study within the program. Clear statements of intended student learning outcomes can foster continued reflection on curriculum and pedagogy, and can also help shape an agenda for assessment. The college-wide statement of General Education Student Learning Outcomes (forthcoming) is likely to be a helpful resource as programs develop or revisit their own intended learning outcomes. Like the program mission statements, program learning goals may be useful additions to program websites.

(2) Professional activity

Many programs articulate goals for the collective professional activity of their department. Some revise their Statement of Significant Professional Activity in the course of discussing this dimension of program goals.

(3) Leadership, service, and outreach

Some programs also reflect on their intended contributions to the well-being of the College community, relevant professional associations, higher education more generally, the professions served by the program, or public life. Establishing goals in this area can help departments be more intentional in the nature and extent of their involvement in campus governance, service and leadership in the wider community, and other kinds of faculty work.

C. Relevant program history

Programs vary in the attention they give to this dimension of the self-study. Use the above factors (audience, governing questions, relevant College-wide context, and evaluation criteria) to determine what aspects of program history to include in the self-study.

II. Identifying program resources

A. Program budget. It is also common to include non-compensation budget information, generally over a period of several years. The time frame should be selected with the above factors in mind.

B. Physical resources. Many programs describe an array of physical resources for the support of instruction: library holdings, technology, labs, special equipment, physical space, etc.

C. Grant funding. Support received through external grants may be an important complement to internal college resources for both instruction and professional activity.

III. Describing the instructional program.

Most self-studies include a fairly extensive but primarily descriptive discussion of the instructional program. Different programs organize this material in different ways, but most self-studies address the following in one way or another:

A. Information about the students who major and/or concentrate in the program. Self-studies typically include longitudinal data on the number of majors, with the time frame determined by the above factors. Many discuss the additional majors, concentrations and co-curricular activities their students choose to complement their studies in the program.

B. Structure of and rationale for program requirements for a major and/or a concentration. Every self-study includes a description of the requirements for its major(s) and concentration(s). Many also provide a rationale for each requirement, often linked to their evaluation criteria.

C. Contributions of the program to the College’s general education curriculum and/or learning objectives. The 2004 report of the General Education Task Force recommends that self-studies include a careful examination of the program’s contributions to the GE curriculum –“where they are offering GE courses, where they are not, and what the optimum mix should be, for the mutual enrichment of the GE curriculum and major studies in the department or program.”

D. Course offerings. A comprehensive discussion of program courses includes not only semester-length courses but also interim courses, independent study and research projects, and summer courses. Self-studies address this dimension of program review in different ways; some examine course offerings by disciplinary subfield, while others group courses by level. Some programs have found it helpful to put together a program teaching portfolio, including course syllabi, assignments, and samples of student work, not only for reference by the external evaluators but also to share ideas and accomplishments with one another. Some longitudinal data on course enrollments are usually included as well. Programs vary in the time frame they select, but it is probably advisable to include at least three years of data.

E. Off-campus study, service learning, student-faculty research projects, and other learning opportunities outside the classroom. Self-studies typically describe the available opportunities and level of participation in domestic and international programs for which students may earn credit in the program. Some also describe the integration of experiential or service learning in courses offered on campus.

F. Contributions to interdisciplinary programs. The strong emphasis on interdisciplinary study is a distinctive feature of a St. Olaf education, and has fostered rich collaborations among both disciplinary and interdisciplinary programs. Most self-studies describe program contributions to other programs and the activities of program faculty in developing interdisciplinary programs, linked courses, and other opportunities for integrative study.

G. Pedagogy and faculty development. Some programs give attention not just to what they teach, but how they teach, addressing both long-standing and more recently-developed practices. Samples of course activities and assignments included in the program teaching portfolio are helpful here as well. Self-studies may also discuss the participation of program faculty in various faculty development programs supporting innovation and improvement in instructional strategies (e.g., the use of technology, the incorporation of oral communication or writing as a means of learning, just-in-time teaching, etc.).

H. Other contributions to student learning and development. Many self-studies describe their academic advising practices, student organizations, honors houses, speaker series, graduate school and career planning programs, alumni networks, and other opportunities for student development outside the classroom.

IV. Examining the other dimensions of faculty work.

A. Statement of Significant Professional Activity. As with the program mission statement, the program’s Statement of Significant Professional Activity is often revisited and sometimes revised in the course of the self-study. It is generally included in an appendix and referenced in the discussion of faculty professional activity.

B. Faculty professional expertise, activities and accomplishments. Self-studies always describe the principal areas of scholarly and/or artistic expertise represented by program faculty and summarize their professional activities and accomplishments (publications, performances and showings, grants, etc.). Copies of each faculty member’s CV are included in an appendix.

C. Faculty leadership and service. Faculty members at St. Olaf commit significant time to leadership and service within and beyond the College (of which the conduct of a self-study is but one example). They model the kind of engagement they seek to foster in their students. Giving attention to this dimension of faculty work in self-studies not only affords recognition of faculty investments and accomplishments but also makes collective planning possible.

V. Evaluating program outcomes

Programs are encouraged to evaluate all three dimensions of faculty work in the context of the program, including teaching and learning, scholarship and artistic activity, and leadership and service.

A. Assessment of student learning outcomes. The College has defined assessment, or “inquiry in support of student learning,” as follows:

“Assessment ('inquiry in support of student learning') at St. Olaf College is the ongoing collection and analysis of systematic evidence concerning the relationship between educational practices and student learning, for purposes of program improvement. ‘Educational practices’ may include curriculum requirements, instructional strategies, programs of study, and other dimensions of the educational programs and services offered by the College. ‘Student learning’ may include a wide array of both cognitive and affective outcomes. ‘Systematic evidence’ may be gathered through a variety of methods, both qualitative and quantitative.”

Note that simply conducting a survey of current students or alumni does not necessarily constitute assessment; it depends on the content of the instrument and the degree to which it provides information about student learning processes and outcomes within the program.

The College’s vision of assessment at all levels is that it should be mission-driven, manageable, and meaningful. Consequently, programs are NOT expected to conduct a comprehensive assessment of all their intended student outcomes. Rather, they are encouraged to be selective, choosing just a few key areas to assess. These choices should be governed by the program mission and learning outcomes identified earlier in the self-study, and by the above factors shaping the contents of the self-study as a whole (audience, key questions, national trends, evaluation criteria, etc.). Resources to support the assessment component of a self-study are discussed below.

B. Feedback from current students and program alumni. Many programs conduct interviews, surveys, or focus groups with current students and alumni, asking for feedback on their experiences in the program – program strengths, weaknesses, suggestions for improvement, etc. This kind of feedback does not necessarily constitute assessment, since it may not provide valid or reliable evidence of student outcomes. But program faculty have nevertheless found student and alumni feedback to be a valuable component of program review. The capacity to conduct surveys online through the FormCreator website (www.fusion.stolaf.edu/formcreator/), even with off-campus respondents, has made it even easier to gather this kind of feedback. Programs can use their agreed-upon evaluation criteria to shape their student and alumni survey or interview questions.

C. Feedback from St. Olaf colleagues outside the program. A few programs have solicited evaluation letters from colleagues in other departments, programs, or academic units (such as the library, the Center for Experiential Learning, or the Academic Support Center) who are familiar with the program or its faculty. The process is modeled on the process of soliciting letters on behalf of an individual faculty member who is a candidate for tenure or promotion.

D. Evaluation of faculty professional activity. The professional activities and accomplishments of the faculty, as well as recommendations for future activity, can be evaluated in light of the program’s Statement of Significant Professional Activity, the evaluation criteria established up front as part of the self-study process, and the other factors shaping the agenda and content of the self-study.

E. Evaluation of faculty leadership and service. Similar criteria can be brought to bear on the contributions of the faculty to the well-being of the College, relevant professions, and the wider community.

VI. Planning for the future.

Programs vary a great deal in their discussion of future plans, which is understandable in view of the variation in the agenda and evaluation criteria for each self-study. Nevertheless, most self-studies include some discussion of short-term (3-5 years) and longer-term (6-10 years) plans. All reference the instructional program; most reference faculty professional activity; and some reference service and leadership as well. Explicit objectives, specific strategies, clear identification of needed resources, and proposed timelines will provide useful benchmarks against which future program decisions can be evaluated. Some discussion of constraints and challenges may also be helpful.

VII. Inviting specific feedback.

While the external reviewers are free to comment on any dimension of the self-study, programs can ensure that they receive feedback on the areas of greatest concern to them by listing specific questions or issues they would like the reviewers to address in their report.

Information resources to support the preparation of the self-study
In addition to the program’s own records and archives, the following resources are available to assist in the preparation of the self-study.

1. Links to college-wide documents referenced in a program review. These documents are available on the E&A website for ease of access and printing, and include the following:

· The St. Olaf College Mission Statement

· Identity and Mission for the 21st Century

· Faculty Manual, particularly the Standards for Faculty Evaluation (Section 4. VII, pp. 32-33)

· The Academic Catalog 2004-06

· Requirements for B.A. Degree Candidates (excerpted from the Academic Catalog)

· General Education Requirements (excerpted from the Academic Catalog)

· General Education Curriculum: Detailed Descriptions and Guidelines for Each GE Requirement

2. Self-study information available from other administrative offices. This link on the E&A website describes program-specific and college-wide information available from the Office of the Registrar and Institutional Research (IR). Data on course offerings, enrollments, numbers and names of students who completed majors or concentrations in the program, grades given, college-wide and program-level FTE, student-faculty ratios, and other indicators of program resources and activity are available. Links are provided to data or information that is readily available on the web. For information that must be requested from the Office of the Registrar, the names and contents of the relevant data source are clearly described, so program faculty know what to reference in their request.

3. Resources to support assessment at the program level. More detailed suggestions for conducting meaningful and manageable inquiry in support of student learning at the program level are available in the following documents and links on the E&A website:

· Inquiry in Support of Student Learning: Chapter IV of the St. Olaf College 2003 Self-Study submitted to the Higher Learning Commission of the North Central Association

· Mission-Driven, Manageable, and Meaningful: Practical Suggestions for Assessment at the Program Level. This document addresses the process of establishing an agenda for inquiry and gathering and interpreting data.

· Tools for Assessment at the Program Level. This document suggests some of the available instruments for assessing student learning and briefly describes the strengths and limitations of each alternative.

· Additional assessment resources and links.. The E&A website includes links to additional websites and resources supporting assessment in general.

The Office of Academic Research and Planning is available to consult with program faculty in designing the assessment component of their self-studies.

4. Self-Study Document Distribution

1. E&A: Send one complete copy of the self-study to the Office of Academic Research and Planning.

2. Office of the Dean of the College: Send three complete copies of the self-study to the Office of the Dean of the College (one for the Provost, one for the department’s associate dean and one for Dean’s Council to peruse); one copy may be returned to you after the visit if you request it.

3. Dean’s Council: Send seven copies of the narrative only (one for each member of Dean’s Council) to the Office of the Dean of the College OR e-mail the narrative to deans-council@stolaf.edu.

Concluding thoughts and practical suggestions
Programs will differ in their styles of collaboration and division of labor. The following suggestions may be helpful as program faculty consider how to conduct the inquiry and report their findings.

· Work collectively to articulate the mission, identify program goals, and select appropriate evaluation criteria. Since these will provide benchmarks for evaluating program outcomes, it is important to build consensus around them by collectively drafting or revising them.

· Know what you want to know and why. Your agenda for gathering information should be determined by your audience, governing questions, and evaluation criteria. Make sure your collection of information is purpose-driven.

· Consult with other programs that have completed self-studies. E&A does not distribute self-studies, as some contain information the program wishes to keep confidential. However, many programs are willing to share complete or partial copies of their self-studies. If you would like a copy of another program's self-study, please direct your request to the program chair or director.

· Enlist help from your students. Student assistants, of course, can provide invaluable help in gathering and presenting data. But there are other ways to involve students as well. Course assignments or class activities can be built around data-gathering or data-interpretation in support of the self-study. One or more recognition internships involving data gathering, survey construction, assessment of student learning, or other features of the self-study could be supervised by one or more program faculty. Not only will this mitigate the practical problems of limited time, it will also engage students meaningfully in the evaluation of their own educations.

Although the preparation of a self-study constitutes a substantial addition to faculty work, it is also a substantial accomplishment. Most program faculty have found the opportunity for collaboration, clarification, feedback and planning to be well worth the effort. The Office of Academic Research and Planning is available to consult with program faculty at any point in the planning and conduct of the self-study.