Guidelines for program review self-studies
Introduction
Of the various components of the program review process, the
written self-study is without question the most demanding, but also
potentially the most rewarding. Because the primary purpose of a program
review is to provide program faculty and academic administrators the
information they need for effective academic decision making, program
faculty have a great deal of discretion in what they choose to include
in their self-study. The Dean of the College and the Associate Deans
have not established specific College-wide requirements for the content
of a self-study; instead, they have encouraged program faculty, in consultation
with their Associate Dean, to highlight the most pressing issues and
seek the information they most need to sustain what they are doing well,
address what needs improving, and plan for the future.
The purpose of this document is to suggest some guidelines for the content of a self-study, based on what other programs have found useful in preparing self-studies and informed by the new criteria for institutional accreditation established by the Higher Learning Commission of the North Central Association (NCA), the professional organization that accredits St. Olaf as a whole. These are offered as recommendations, not requirements. Programs that are separately accredited by other professional associations prepare their self-studies in accordance with the requirements of their accreditors, which may be more extensive than what is recommended here. In general, though, program faculty should feel free to include only what is useful to them and their readers, to add anything that is not listed below, and to omit those things that do not support their particular self-study learning objectives.
Establishing the self-study agenda
Program mission, goals and historical context
Evaluation of program outcomes (including the assessment of student learning)
Links to documents and data typically included in a self-study
Concluding thoughts and practical suggestions
_________________________________________________________________________
Establishing the self-study agenda
1. Who is your audience?
At present, the primary audiences for the self-study report
are the program faculty themselves, the Dean of the College and Associate
Deans, and the external reviewers and/or accreditors (in the case of
programs which have their own external accrediting agencies). However,
self-studies can have a number of secondary audiences, including potential
funding agencies for future grant applications; prospective or newly-appointed
faculty; other St. Olaf programs or faculty with whom the program may
currently, or wish to, partner; potential collaborators at other institutions;
or others who may play a role in future program activities. Keep the
information needs of your audience at the forefront of your decisions
about what to include in your self-study.
2. What questions do you most want to answer at this point
in the life of your program?
Like any inquiry, a self-study agenda should be governed by
what you want to learn and convey to your audiences and how the results
will be used (Patton, 1997). Many departments have found it helpful
to establish clear learning objectives for their self-study before deciding
what information to track down. There should be a reason for each item
you decide to include in the final report. This can help you avoid collecting
information that, in retrospect, you really didn’t need to spend
time collecting (and that your audiences didn’t need to spend
time reading). It may be useful to state the governing questions explicitly
in the self-study.
3. What aspects of the larger context of the College are important
for your program?
The website of the Office of Academic Research and Planning
has compiled a list of key College documents that most self-studies
replicate or reference (see “Information resources to support
the preparation of the self-study” below). Links to each document
are provided on the E&A website for ease of access and printing.
This is helpful material for the external reviewers to have in hand.
It is also helpful for the program faculty to consider and reference
explicitly in developing or revisiting their mission, goals, resources,
and priorities.
4. What national trends in the disciplines and/or professional
fields served by your program have a bearing on your program?
Information about similar programs in comparable undergraduate
institutions, the expectations of the graduate and professional schools
to which program majors may apply, and trends in the professions program
graduates enter, may have important implications for current and future
program activities.
5. What criteria or benchmarks will you use to evaluate the
strengths and weaknesses of your program?
Articulating appropriate criteria for the evaluation of your
program, is one of the most important dimensions of the program review
process. There are a number of helpful resources, both internal and
external to the College, to support the selection of criteria for academic
program evaluation. Here are some examples:
Principles for Appraising the Contributions of Departments and Programs to the St. Olaf Curriculum as a Whole, prepared by the St. Olaf Curriculum and Educational Policies Committee (CEPC) in May 2004.
The standards for faculty evaluation established in the St. Olaf College Faculty Manual, Section 4.VII, pp. 32-33. Although these standards are used to evaluate the contributions of individual faculty members for purposes of reappointment, tenure, or promotion, they may be adapted for use in evaluating the collective contributions of a program as well.
The accreditation criteria established by the North Central Association (NCA) Higher Learning Commission, the agency which accredits St. Olaf as a whole. These criteria, recently revised and effective as of January 2005, were designed to apply to an entire institution of higher education, but portions or versions of the criteria may be helpful in evaluating a program within the institution.
Discipline- or profession-specific professional association(s), which often publish reports, recommendations, position statements, or other documents on teaching, scholarship, and/or professional service and outreach.
Professional associations in higher education more generally, such as the Association of American Colleges and Universities and the American Council on Education, which also release reports, recommendations, position statements, and other documents on teaching, scholarship, and/or professional service.
Below is an outline of the major issues and questions that programs are encouraged to address in their self-studies. Most self-studies to date have referenced most of these topics, although not necessarily in the same sequence.
I. Program mission, goals, and historical context
- Program mission statement
- Program goals and objectives
- Relevant program history
- Budget
- Physical resources
- Grant funding
III. Program activities
- Instructional program
- Faculty professional activity
- Faculty leadership and service
IV. Evaluation of program outcomes
- Assessment of student learning
- F eedback from current students and program alumni
- Feedback from St. Olaf colleagues outside the program
- Evaluation of faculty professional activity
- Evaluation of faculty leadership and service
- Planning for the future
- Inviting feedback from external reviewers
Program mission, goals and historical context
Program mission statement
A program mission statement helps give identity, coherence and focus to the activities and outcomes of a program. Some programs already have mission statements, and use the self-study as an opportunity to revisit and sometimes revise it. Others develop one in the course of the self-study. Some programs link their mission statements to statements prepared by relevant professional associations. It is also helpful to discuss the program mission statement in relation to the mission of the College as a whole. Program mission statements often make their way to program websites.
Program goals and objectives
Teaching
Programs are strongly encouraged to include in their discussion of their teaching goals a list of intended student learning outcomes. Some of these outcomes may reflect college-wide objectives, while others may be distinctive to more specialized study within the program. Clear statements of intended student learning outcomes can foster continued reflection on curriculum and pedagogy, and can also help shape an agenda for assessment. The college-wide statement of General Education Student Learning Outcomes is likely to be a helpful resource as programs develop or revisit their own intended learning outcomes. Like the program mission statements, program learning goals may be useful additions to program websites.
Professional activity
Many programs articulate goals for the collective professional activity of their department. Some revise their Statement of Significant Professional Activity in the course of discussing this dimension of program goals.
Leadership, service, and outreach
Some programs also reflect on their intended contributions to the well-being of the College community, relevant professional associations, higher education more generally, the professions served by the program, or public life. Establishing goals in this area can help departments be more intentional in the nature and extent of their involvement in campus governance, service and leadership in the wider community, and other kinds of faculty work.
Relevant program history
Programs vary in the attention they give to this dimension of the self-study. Use the factors considered in establishing the self-study agenda when determining the content of a self-study (audience, governing questions, relevant College-wide context, and evaluation criteria) to determine what aspects of program history to include.
Program budget
It is common to include non-compensation budget information, generally over a period of several years. The time frame should be selected with the factors to consider in establishing the self-study agenda in mind.
Physical resources
Many programs describe an array of physical resources for the support of instruction: library holdings, technology, labs, special equipment, physical space, etc.
Grant funding
Support received through external grants may be an important complement to internal college resources for both instruction and professional activity.
The instructional program
Most self-studies include a fairly extensive, though primarily descriptive, discussion of the instructional program. Different programs organize this material in different ways, but most self-studies address the following in one way or another:
Information about the students who major and/or concentrate in the program. Self-studies typically include longitudinal data on the number of majors, with the time frame determined by the above factors. Many discuss the additional majors, concentrations and co-curricular activities their students choose to complement their studies in the program.
Structure of and rationale for program requirements for a major and/or a concentration. Every self-study includes a description of the requirements for its major(s) and concentration(s). Many also provide a rationale for each requirement, often linked to their evaluation criteria.
Contributions of the program to the College’s general education curriculum and/or learning objectives. The 2004 report of the General Education Task Force recommends that self-studies include a careful examination of the program’s contributions to the GE curriculum –“where they are offering GE courses, where they are not, and what the optimum mix should be, for the mutual enrichment of the GE curriculum and major studies in the department or program.”
Course offerings. A comprehensive discussion of program courses includes not only semester-length courses but also interim courses, independent study and research projects, and summer courses. Self-studies address this dimension of program review in different ways; some examine course offerings by disciplinary subfield, while others group courses by level. Some programs have found it helpful to put together a program teaching portfolio, including course syllabi, assignments, and samples of student work, not only for reference by the external evaluators but also to share ideas and accomplishments with one another. Some longitudinal data on course enrollments are usually included as well. Programs vary in the time frame they select, but it is probably advisable to include at least three years of data.
Off-campus study, service learning, student-faculty research projects, and other learning opportunities outside the classroom. Self-studies typically describe the available opportunities and level of participation in domestic and international programs for which students may earn credit in the program. Some also describe the integration of experiential or service learning in courses offered on campus.
Contributions to interdisciplinary programs. The strong emphasis on interdisciplinary study is a distinctive feature of a St. Olaf education, and has fostered rich collaborations among both disciplinary and interdisciplinary programs. Most self-studies describe program contributions to other programs and the activities of program faculty in developing interdisciplinary programs, linked courses, and other opportunities for integrative study.
Pedagogy and faculty development. Some programs give attention not just to what they teach, but how they teach, addressing both long-standing and more recently-developed practices. Samples of course activities and assignments included in the program teaching portfolio are helpful here as well. Self-studies may also discuss the participation of program faculty in various faculty development programs supporting innovation and improvement in instructional strategies (e.g., the use of technology, the incorporation of oral communication or writing as a means of learning, just-in-time teaching, etc.).
Other contributions to student learning and development. Many self-studies describe their academic advising practices, student organizations, honors houses, speaker series, graduate school and career planning programs, alumni networks, and other opportunities for student development outside the classroom.
Faculty professional activity
Statement of on Significant Professional Activities. As with the program mission statement, the program’s Statement on Significant Professional Activities is often revisited and sometimes revised in the course of the self-study. It is generally included in an appendix and referenced in the discussion of faculty professional activity.
Faculty professional expertise, activities and accomplishments. Self-studies always describe the principal areas of scholarly and/or artistic expertise represented by program faculty and summarize their professional activities and accomplishments (publications, performances and showings, grants, etc.). Copies of each faculty member’s CV are included in an appendix.
Faculty leadership and service
Faculty members at St. Olaf commit significant time to leadership and service within and beyond the College (of which the conduct of a self-study is but one example). They model the kind of engagement they seek to foster in their students. Giving attention to this dimension of faculty work in self-studies not only affords recognition of faculty investments and accomplishments but also makes collective planning possible.
Evaluation of program outcomes (including
the assessment
of student learning)
Programs are encouraged to evaluate all three dimensions of faculty work in the context of the program, including teaching and learning, scholarship and artistic activity, and leadership and service.
Assessment of student learning outcomes
Academic programs benefit when faculty gather evidence of student learning outcomes and consider the implications of that evidence for teaching and learning in the program. Consequently, departments and programs are asked to include some assessment of student learning as one component of their program review, and to reflect on what they learn from the results in their self-study reports. It is important to note that simply conducting student surveys, interviews, or focus groups does not necessarily constitute assessment; it depends entirely on the content of the questions and the nature of the evidence students are asked to provide. To be sure, student feedback can provide important insights on the effects of a program on student learning, but student feedback alone is unlikely to provide direct evidence of the nature and extent of that learning.
Gathering direct evidence of the learning that results from a major, concentration, general education program, or other program of academic study is not a simple matter. However, there is a growing array of instruments and approaches programs can use to learn about their impact on student learning. The Office of Academic Research and Planning is available to assist programs in developing a mission-driven, meaningful, and manageable assessment strategy.
Mission-driven, meaningful, and manageable assessment at the program level
Developing effective statements of intended learning outcomes
Instruments for assessing student learning at the program level
Feedback from current students and program alumni
Many programs conduct interviews, surveys, or focus groups with current students and alumni, asking for feedback on their experiences in the program – program strengths, weaknesses, suggestions for improvement, etc. As noted above, this kind of feedback does not necessarily constitute assessment, since it may not provide valid or reliable evidence of student outcomes. But program faculty have nevertheless found student and alumni feedback to be a valuable component of program review. The capacity to conduct surveys online through the FormCreator website, even with off-campus respondents, has made it even easier to gather this kind of feedback. Programs can use their agreed-upon evaluation criteria to shape their student and alumni survey or interview questions.
Feedback from St. Olaf colleagues outside the program
A few programs have solicited evaluation letters from colleagues in other departments, programs, or academic units (such as the library, the Center for Experiential Learning, or the Academic Support Center) who are familiar with the program or its faculty. The process is modeled on the process of soliciting letters on behalf of an individual faculty member who is a candidate for tenure or promotion.
Evaluation of faculty professional activity
The professional activities and accomplishments of the faculty, as well as recommendations for future activity, can be evaluated in light of the program’s Statement on Significant Professional Activities, the evaluation criteria established up front as part of the self-study process, and the other factors shaping the agenda and content of the self-study.
Evaluation of faculty leadership and service
Similar criteria can be brought to bear on the contributions of the faculty to the well-being of the College, relevant professions, and the wider community.
Planning for the future
Programs vary a great deal in their discussion of future plans, which is understandable in view of the variation in the agenda and evaluation criteria for each self-study. Nevertheless, most self-studies include some discussion of short-term (3-5 years) and longer-term (6-10 years) plans. All reference the instructional program; most reference faculty professional activity; and some reference service and leadership as well. Explicit objectives, specific strategies, clear identification of needed resources, and proposed timelines will provide useful benchmarks against which future program decisions can be evaluated. Some discussion of constraints and challenges may also be helpful.
Inviting specific feedback
While the external reviewers are free to comment on any dimension of
the self-study, programs can ensure that they receive feedback on the
areas of greatest concern to them by listing specific questions or issues
they would like the reviewers to address in their report.
Links to documents and data typically included in a self-study
In addition to the program’s own records and archives, the following resources are available to assist in the preparation of the self-study.
Links to college-wide documents
Most self-studies include several of the documents listed below. You can link to these documents here and include them either electronically or in paper form in your self-study.
The St. Olaf College Mission Statement
Identity and Mission for the 21st Century
Faculty Manual, particularly the Standards for Faculty Evaluation (Section 4. VII, pp. 32-33)
Requirements for B.A. Degree Candidates (excerpted from the Academic Catalog)
General Education Requirements (excerpted from the Academic Catalog)
General Education Curriculum: Detailed Descriptions and Guidelines for Each GE Requirement
Self-study
data available from other administrative offices
This link on the E&A website describes program-specific
and college-wide information available from the Office of the Registrar
and Institutional Research (IR). Data on course offerings, enrollments,
numbers and names of students who completed majors or concentrations
in the program, grades given, college-wide and program-level FTE, student-faculty
ratios, and other indicators of program resources and activity are available.
Links are provided to data or information that is readily available
on the web. For information that must be requested from the Office of
the Registrar, the names and contents of the relevant data source are
clearly described, so program faculty know what to reference in their
request.
Concluding thoughts and practical suggestions
Programs will differ in their styles of collaboration and division of labor. The following suggestions may be helpful as program faculty consider how to conduct the inquiry and report their findings.
Work collectively to articulate the mission, identify program goals, and select appropriate evaluation criteria.
Since these will provide benchmarks for evaluating program outcomes, it is important to build consensus around them by collectively drafting or revising them.Know what you want to know and why.
Your agenda for gathering information should be determined by your audience, governing questions, and evaluation criteria. Make sure your collection of information is purpose-driven.Consult with other programs that have completed self-studies.
E&A does not distribute self-studies, as some contain information the program wishes to keep confidential. However, many programs are willing to share complete or partial copies of their self-studies. If you would like a copy of another program's self-study, please direct your request to the program chair or director.Enlist help from your students.
Student assistants, of course, can provide invaluable help in gathering and presenting data. But there are other ways to involve students as well. Course assignments or class activities can be built around data-gathering or data-interpretation in support of the self-study. One or more recognition internships involving data gathering, survey construction, assessment of student learning, or other features of the self-study could be supervised by one or more program faculty. Not only will this mitigate the practical problems of limited time, it will also engage students meaningfully in the evaluation of their own educations.
Although the preparation of a self-study constitutes a substantial addition to faculty work, it is also a substantial accomplishment. Most program faculty have found the opportunity for collaboration, clarification, feedback and planning to be well worth the effort. The Office of Academic Research and Planning is available to consult with program faculty at any point in the planning and conduct of the self-study.
