Institutional Research and Planning (IRP) 2005-2006 Annual Report

Susan Canon, Director
May 2006


The Office of Institutional Research and Planning (IRP) was established in 1998 to provide relevant and accurate information about St. Olaf College. The office monitors strategic characteristics of the College, provides public information about the College, and aids senior administrators and program managers in assessing progress and making decisions to meet institutional goals. The IRP office has two full-time staff members: a Director and a Research Associate. IRP reports to the Assistant Provost.

The work of the office is focused on five major kinds of activities:

  1. Overseeing a system of regular surveys of students, faculty, staff, and alumni.
  2. Preparing and validating college data for submission to external agencies.
  3. Providing a variety of reports and data summaries for internal use by committees and managers.
  4. Helping guide the college data procedures to ensure consistency and utility.
  5. Maintaining an informational website as well as providing data as requested from various internal and external constituencies.

This annual report highlights IRP activities in each of the five major areas, followed (in section F) by a summary of internal activities required to maintain the office itself. A final section (G) outlines potential new projects and looks at priorities, goals, challenges, and opportunities for the future.

A. Surveys

A chief activity of IRP is to conduct surveys - of students, faculty, staff, parents, and alumni - and to communicate insights from these surveys to various college constituencies. Most surveys are conducted on a regular basis – annually, triennially or less frequently as appropriate. Most surveys (e.g., the HERI (Higher Education Research Institute) Faculty Survey and the NSSE (National Survey of Student Engagement) are externally prepared and compiled, while a few (e.g., Recent Graduates Survey) are locally developed. Externally prepared surveys, while generic in character, offer the advantage of comparative benchmarks that, when properly used, can be of help in improving the College's educational program.

Most surveys stretch over two years - one for administration, the next for interpretation. Thus this section includes both surveys conducted in 2004-05 whose analysis and interpretation took place in 2005-06 as well as surveys conducted during 2005-06. Results from some of these surveys will not be available until next year.

  • Beginning College Student Survey (BCSS). During Week One, 2004, St. Olaf College participated in a pilot study from the National Survey of Student Engagement (NSSE). This study provided baseline data for all incoming freshmen, who were later administered the NSSE survey (see below). The data was analyzed with the NSSE data and presented for a Center for Innovation in the Liberal Arts (CILA) Lunch Conversation, in conjunction with the First Year Experience work group. That summary was posted to the IRP website.
  • Higher Education Research Institute (HERI) Faculty Survey. During winter 2004-05 St. Olaf participated in this triennial national survey of faculty and data was received in Fall 2005. Two IRP Notes (reports sent via e-mail to faculty, administrators, staff, and student leaders) were sent to the community, comparing St. Olaf data with two comparison groups and highlighting some trends over time at St. Olaf. Data charts and summaries were posted to the IRP website.
  • National Survey of Student Engagement (NSSE). St. Olaf participated in this survey in Spring 2005, surveying all first year students, as a follow-up to the BCSS, and almost all seniors, as we participated in the on-line version for the first time. The phenomenal response rate was 77% of first-years and 73% of seniors. IRP worked together with the Academic Research & Planning (ARP) staff to make various presentations to the community, through two Informed Workplace Seminars, a CILA Lunch Conversation (see above), Boldt Seminar participants, and the Academic Leadership group. An overview IRP Note was sent to the community and various data summaries posted to the IRP website.
  • Recent Graduates. The IRP office conducts an annual survey of the previous year's graduates for the Minnesota Private College Council (MPCC) and the St. Olaf Center for Experiential Learning (CEL). This survey collects information about the primary activity of our graduates (graduate school/employment/volunteer work). The IRP staff finalized reports on the Class of 2004 for MPCC and CEL last summer and posted summaries to the IRP website. The MPCC report on the class of 2005, which had a response rate of 49%, will be forthcoming this summer.
  • Cooperative Institutional Research Program (CIRP). During Week One 2005, St. Olaf participated in the national survey of incoming students conducted by the Higher Education Research Institute (HERI). This spring we posted the data for this survey on our the IRP web site and e-mailed two IRP Notes highlighting trends at St. Olaf as compared with national norms as well as comparisons with two smaller peer groups.
  • HEDS Alumni Survey. In Spring 2006 IRP administered a “ten-years out” follow-up alumni survey as well as a “five-years out” survey. Return rates for these surveys were 26% for the Class of 1996 and 33% for the Class of 2000. Data will become available for these survey next fall.
  • Parent Survey. Also this spring, the IRP office conducted an internally designed survey of parents of students in the classes of 2005-2009. This is the same survey that was conducted in 2001-02. With a response rate of 38%, the office is in the process of analyzing the data and comments, and IRP Notes as well as data posting to the website will be forthcoming.
  • Campus Climate Survey. Although this survey was initiated by the Dean of Community Life and Diversity (CLD), the Director of IRP was actively involved with the survey, from design through implementation and data summaries. She worked together with the Dean of CLD and members of the President’s Advisory Committee on CLD Concerns to prepare a preliminary report for the Board of Regents, President’s Cabinet, and Dean’s Council. Work on presenting this data to the community will continue through the fall semester. Overall return rate for the survey was 43%, although not all respondents completed the entire survey.

B. External Reports

Every year IRP works with the relevant offices and databases on campus to prepare and review reports for many external organizations and agencies. Data for numerous topics are reported, including general institutional characteristics, admissions, financial aid, enrollment, graduation, retention, faculty/staff compensation and demographics, endowment, finance, tuition and fees, and voluntary support. Most reporting is in summary form; some require submission of unit-record data for individual students.

  • Publications
    o Common Data Set (CDS). The completed CDS is sent to many college guide publishers. IRP also posts it on its website, together with longitudinal summaries of the data and other data asked for regularly by publishers.
    o College Guides’ Supplemental Surveys. In addition to the CDS, U.S. News and World Report, Peterson's, College Board, Princeton Review, and many other college guides request much additional data.
  • Institutional Associations
    o American Association of University Professors (AAUP)
    o Associated Colleges of the Midwest (ACM)
    o College & University Professional Association for Human Resources (CUPA-HR)
    o Evangelical Lutheran Church of America (ELCA)
    o Higher Education Data Sharing Consortium (HEDS)
    o Minnesota Private Colleges Research Foundation (MPCRF)
    o National Collegiate Athletic Association (NCAA)
  • State & Federal Government
    o Minnesota Office of Higher Education (MOHE)
    o National Center for Educational Statistics (NCES): Integrated Postsecondary Education Data System (IPEDS)

Various other reports are submitted to institutional and departmental accreditation agencies, grant agencies, auditors and other organizations. These reports are either reviewed and supplemented by the IRP office and then sent, or a final copy is sent to the IRP office as the central repository for institutional reports.

In addition to the above data submissions, the IRP Director has continued discussions with ACM staff regarding changes to the ACM data collection schedules and content. Both the Director and the Research Associate participate in ongoing electronic discussions regarding the Common Data Set and various data issues with HEDS members.

C. Internal Data Analyses, Collaborations and Committee Involvement

IRP provides a variety of reports and data summaries for internal use by committees and managers. In addition, the IRP Director continues involvement in many campus committees and collaborations this year. Involvement with these groups provides access to existing data and information, an institution-wide perspective on various issues, and allows the IRP office more opportunity to offer its relevant services.

Data collections, analyses and reports:

  • Center for Integrative Research (CIR) Cohort Retention Analysis. Worked with a student from the CIR for an analysis of student characteristics as they relate to retention and graduation.
  • Grades Study. Prepared data for Special Assistant to the Provost to present at CILA session on grade inflation.
  • Student Samples for Tenure & Promotion reviews. Worked with the new ARP Associate to devise electronic random sampling for forming student evaluation groups for faculty tenure & promotion reviews.
  • External data comparisons. IRP provides external benchmarking information to the St. Olaf leadership in support of institutional planning. This year included providing detailed information from AAUP and CUPA faculty compensation studies, providing a preliminary analysis of our peer groups’ IPEDS staffing data, using the IPEDS Data Feedback report to determine the peer groups of our own peers, and updating the baccalaureate origins of PhDs ranking data.
  • Internal data collections. IRP has continued its service as data-provider, especially when it involves integrating data from two or more internal systems. This year data collections included a review of sequence of courses taken/grades earned in the Economics Department, grades earned/Advanced Placement data for an admissions study in conjunction with the CIR, and ethnic/gender distribution of participants in off-campus programs for IDOCS.
  • Advising on Data. The IRP staff frequently works with other campus offices (e.g., Admissions, Advancement, Communications, Financial Aid, Government & Foundation Relations) on collecting and verifying data.

Campus Collaborations:

  • Student Learning Item Catalog. In collaboration with Academic Research & Planning (ARP), we finalized the survey item classification catalog. We also assisted ARP student workers to input actual survey results data into the grid. This allows users to find survey items related to teaching and learning for use in various assessment efforts
  • Teagle Foundation grant. The IRP Director serves on the steering committee for this three-year grant, together with the St. Olaf Director of ARP, as well as administrators and faculty from Carleton, Macalester, and Grinnell. The purpose of the grant is to engage faculty in assessment of four student-learning outcomes: effective writing, critical thinking, quantitative reasoning, and global understanding. As a consortium, the group:
    • Established inter-institutional Collaborative Assessment for Liberal Learning (CALL) faculty teams for each learning outcome.
    • Planned and held a conference for team members in early April.
    • Three colleges, including St. Olaf, implemented the first year of the Collegiate Learning Assessment (CLA), with four-year funding from the Lumina Foundation.
  • Biology/Chemistry Initiative. The IRP Director worked with members of this grant-funded group to identify data needed to study the relationship between introductory Biology and Chemistry courses. She then collected and provided preliminary analysis of the required data.
  • Study Abroad/Financial Aid analysis. As part of the college’s ongoing commitment to incorporating a global perspective, IRP began work this year with Financial Aid and International and Off-campus Studies to examine the financial profile of study abroad participants. The purpose was to help determine if the cost of such programs is prohibitive – or perceived to be prohibitive – to lower-income students. However, that project has languished for some months and needs to be revitalized.

Committee Membership:

  • Inquiry in Support of Student Learning (ISSL). The work of the committee this year focused primarily on implementation of the CLA (see above) and the Intentional General Education Initiative.
  • Institutional Review Board. The IRP Director has continued her work with the IRB this year, primarily reviewing applications for human subjects research.
  • Staff Development Task Force. The IRP Director served on this committee to help organize, advertise and host campus-wide staff development workshops, seminars and presentations.
  • Registrar’s Academic Advisory Group (RAAG). This group is a resource-sharing/learning group that meets monthly to understand more about other offices within the institution that work with the Registrar’s office.

D. Data Coordination

A responsibility of IRP is to help establish and maintain the integrity and consistency of data that is stored, transformed, and reported from several different computer systems. Most of this activity is on-going:

  • Policy and Procedures. The IRP Director continues her oversight of policy and procedures to ensure data integrity and appropriate relationships between the faculty database in the Provost's Office and the Lawson financial system in the Human Resources Office. This year included assisting the Information Management Coordinator in the Provost’s Office to design a data-checking information sheet sent to each faculty member for confirmation in October.
  • Student Information System. The IRP Director continued to work with IIT and Registrar staff with final touches on the new student record system, including the preparation of data to upload for the integration between student enrollment records and faculty instructional records. As she has begun working with the data, she continues to provide input to the IIT and Registrar staff about the new system.

E. Public Communication

Another function of the IRP office is to respond to questions about data from faculty, staff, and students. Requests are quite varied; this year they included requests for data for Manitou Messenger articles, student requests for data for class projects, Regents’ Student Committee, Office of Student Leadership and the Admissions office. IRP also reviews and updates data for use in many campus and admissions publications and websites. Summer 2006 involved major revisions to the college Viewbook, with much input and fact-checking from the IRP staff.

Most broad communication from IRP on campus takes place in the form of e-mail "IRP Notes" that are sent to all faculty, staff, emeriti, and student leaders. Topics of the 2005-06 notes are:
09/30/05 Retention, Graduation & Recent Graduates
11/04/05 HERI Faculty Survey: St. Olaf and Two Comparison Groups
01/13/06 Overview of NSSE 2005
01/26/06 CIRP Survey of Incoming Freshmen: Compare to National Norms
02/01/06 HERI Faculty Survey: Comparisons over time
04/19/06 CIRP Survey of Incoming Freshmen and HEDS peer group comparisons

These IRP Notes, as well as all other IRP reports and data summaries are posted on the IRP website. Updates to web pages are ongoing. In addition, the IRP Director has made presentations to various groups and open forums on a variety of topics, including presentations for CILA Lunch Conversations, Informed Workplace seminars, President’s Cabinet, Academic Leadership meetings, RAAG and Boldt Seminar participants.

Information sharing also takes place outside the campus with queries and studies initiated by members of HEDS. The IRP Research Associate responds regularly to various e-mail queries from HEDS institutions. We receive summary reports from every query that we answer, and each summary is sent to the relevant office or department on campus. This year St. Olaf responded to approximately 65 requests, covering such diverse topics as commencement “walkers,” institutional payments in lieu of taxes, parking, tuition rate guarantees, endowment spending, student indebtedness, employee maternity & adoption leave policies, insurance coverage for board members, and background checks on faculty.

Finally, the IRP Office responds to various ad hoc requests from off-campus. This year these included inquiries from:

  • a student at Winona State doing research on “suitcase colleges”
  • a Doane College Honors Seminar needing data on admissions requirements, enrollments, foreign language requirements, and study abroad programs
  • a professor at University of New Hampshire writing a book about “the American college town” and wanting St. Olaf enrollment data since our founding
  • the IR Director at College of St. Scholastica inquiring about our budget and policies regarding faculty development


F. IRP Office Activities

Internal activities of the IRP Office for 2005-06 included:

  • IRP Office Move. IRP was requested to move into the Administration Building 100 suite to accommodate a closer proximity to Admissions for the Financial Aid Office. Although initial plans were to move in the summer 2005, the staff gave up its student office in the fall and the final move was not completed until January 2006. The new suite also houses the ARP Office, and the shared space has enhanced the interaction and combined work of the two offices.
  • Director, Professional Development. During late summer and early fall Susan Canon completed an online course entitled Foundations and Fundamentals of Institutional Research through Penn State’s World Campus Graduate Certificate in Institutional Research program. Susan attended the HEDS Summer conference in Philadelphia, PA and the January Forum in Miami, FL. At the summer conference, Susan made a presentation on the ARP/IRP Survey Item Classification catalog and jointly presented an Excel Tips & Tricks demonstration with another IR colleague. In June 2006 Susan will end her term on the HEDS Technology Committee, but was just nominated for the Board of Directors (elections will be held in June 2006). She also attended the AIRUM fall conference in Bloomington, MN. On campus, Susan participated in several Informed Workplace sessions and CILA lunch conversation groups.
  • Research Associate, Professional Development. Joy Johnson attended the AIR Forum in San Diego (June 2005) where she completed two Access classes on information organization and analysis in IR applications, participated in a small group discussion on current trends and methods of survey implementation, and focused on presentations in the survey design, analysis and reporting areas.

G. Looking Ahead

Our highest priority this summer is to plan and prepare for an external program review. Given the close interaction between ARP and IRP, and overlap of mission in many areas, the two offices will be coordinating an external review with a site visit planned for this fall. This will provide us with the opportunity to examine the overall structure and distribution of work between the two offices, as well as the efficiencies that may be obtained by working more closely together. Outcomes from this review, in addition to input from our new college president, will help establish priorities for both offices.

The staff must maintain the reporting of required data and information consistently and accurately to external and internal constituencies. Additionally, the staff will continue to provide analysis of survey data, work on specific requested projects, and contribute valuable exploration of institutional and comparative data. IRP is uniquely positioned in the institution to coordinate information-sharing and work on research that crosses all divisions of the college. Therefore, there are several projects of significant size and potential benefit to the College that remain high on our list of potential projects. Depending on results from our program review, we will hopefully be able to focus on a few of these in the coming year or two.

  • Strategic or Key Indicators. As directed in the 2004-09 Strategic Plan, the IRP Director established a peer group of colleges in order to “make regular and consistent use of the peer group to evaluate the work of the college.” However, an ongoing challenge for IRP is to encourage the use of a quarterly or annual report tracking key strategic indicators about the college. This report could include data from admissions, financial aid, advancement, enrollment, graduation, and faculty and administrative staffing. Benchmarking data could be provided on many indicators as a systematic way to monitor the health of the college in various ways, to provide information for planning purposes, and to keep college constituencies well informed.
  • Alumni Outcomes Tracking Database. Several years ago the former IR Director proposed the creation of an alumni database to systematically collect information on alumni achievements after graduation (graduate degrees, employment, etc.) and connect that information to St. Olaf data about admissions, demographics, majors, grades, etc. The database could be used as the source of many different investigations, such as (a) an often-requested study of the relation between entering students' major interests, their graduation majors, and their initial post-baccalaureate pursuits and (b) a longitudinal retention study to determine how different factors contribute to students finishing or leaving St. Olaf and, for those who transfer or drop out, whether they complete their studies at other institutions. Although some preliminary discussions have taken place, no work has been done to further this project.
  • Need-sensitive admissions analysis. With the completion of at least three admissions cycles using need-sensitive admissions, IRP could work with the Treasurer, Registrar, Admissions, and Financial Aid offices to determine the impact of this change for the institution, both financially and on the profile of the student body.
  • Study Abroad/Financial Aid analysis. As noted above, this project needs to be revitalized and completed.
  • Institutional Review Board. Both directors of ARP and IRP serve on this board, and together will be working with a student this summer to improve the IRB website, application forms, and presentation of existing policies. While not directly a function of either office, both offices have a vested interest in improving the processes involved in conducting human subjects research at St. Olaf. Although this summer’s work will be a step in that direction, the IRB will need more attention in the near future.