2001-2002 Annual Report

Institutional Research and Planning (IRP)

Lynn Steen, Director
May 2002

The Office of Institutional Research and Planning (IRP) was established in 1998 to provide relevant and accurate information about St. Olaf College in order to monitor strategic characteristics of the College, provide public web-based information about the College, and aid senior administrators and program managers in making decisions and assessing progress in meeting institutional goals. The office has a staff of 2.25 FTE--a part-time faculty director, a full time associate director, and a full time research associate. Its work is focussed on four major kinds of activities:

A. Overseeing a system of regular surveys of students, faculty, staff, and alumni.

B. Preparing and validating college data for submission to external agencies.

C. Providing a variety of reports and advice for internal use by committees and managers.

D. Helping guide the college data procedures to ensure consistency and utility.

E. Maintaining an informational website and communicating with various constituencies.

This report will highlight IRP activities in each of these areas, followed (in section F) by some internal activities necessary to maintain the office itself. In a final section G, we outline examples of projects that are important for the IRP mission and the College but which are unlikely to be undertaken without additional resources.

A. Surveys

A chief activity of IRP is to carry out surveys of students, faculty, staff, parents, and alumni. Whereas these used to be always conducted on paper using a bubble scanner, the IRP staff has essentially completed a two-year transition to web-based surveys as the preferred form for campus surveys, although paper forms will always be needed as an alternative for those who prefer them.

The following describe this year's surveys which, in anticipation of the North Central review, have been especially numerous:

• College Student Survey. Conducted the HERI College Student Survey (CSS) for sophomores and juniors. Comparative data from this survey will be returned to St. Olaf early next fall in the next few weeks.
• National Survey of Student Engagement (NSSE). St. Olaf students (400 first-years, 400 seniors) participated in this survey in the spring of 2001. Results from the survey were returned to St. Olaf last fall, and were summarized in an IRP Note and posted on the internal web site.
• Senior Survey. Conducted the HEDS Senior Survey this spring, completing data collection just before commencement. Data and preliminary analysis of comparative data will arrive from HEDS in early fall
• Teagle Diversity Survey. Completed the analysis and executive summary of a survey of students, faculty, and staff conducted in the spring of 1999 to assess the campus climate for racial and ethnic diversity. (This was part of a four-college project sponsored by the Teagle Foundation.)
• Faculty Survey. Conducted HERI faculty survey in late October, including development of additional questions to follow up on prior faculty surveys. Data from this survey is analyzed by the HERI staff at UCLA and will be returned to St. Olaf in the fall of 2002.
• Staff Survey. Completed the analysis of last spring's staff survey, prepared an executive summary and associated data templates, posted the report on the web, and summarized it in an IRP note for faculty, staff, and student leaders.
• Parent Survey. For the first time in anyone's memory, the IRP office prepared and conducted an internally designed survey of parents of students in the classes of 2001-2005. Data has been received and recorded, and will be analyzed in the next few months.
• Recent Graduates. Following long-standing custom, the IRP office conducted a survey of last year's graduates for the Minnesota Private College Council (MPCC) and the St. Olaf Career Connections office (formerly Placement Office). Results of this survey are used by Career Connections and are supplied to other campus departments on request .
• Alumni Surveys. Conducted HEDS survey for alumni classes of 1990 and 1996. This involved preparing supplementary questions and all carrying out all details associated with mailing and follow-ups. Data from these surveys is analyzed by HEDS and will be returned to St. Olaf in the fall of 2002.

Most external surveys stretch over two years--one for administration, and one for interpreting results once the data have been analyzed. Thus some items on the previous list about analysis or last year's surveys, others are about collecting data for this year's surveys. Locally developed surveys such as the parent survey can be administered, analyzed and interpreted in a shorter cycle-but at a price, namely, no inter-institutional comparisons. Surveys conducted under external auspices such as HEDS or HERI generally provide comparative data on broad classes of institutions (e.g., private liberal arts colleges) as well as permitting St. Olaf to identify a smaller comparison group, which we generally do.

Finally, for various reasons some surveys were accelerated and others delayed in comparison with the survey schedule that IRP had adopted only a year ago. Thus this summer we need to re-examine the survey schedule in light of these changes. One question may merit wider discussion: do we need to repeat our own internal faculty survey (as had been planned) or is the triennial HERI survey sufficient for taking the pulse of the community.

B. External Reports

Most of the reports listed below recur every year:

• College Guides. Completed surveys for U.S. News and World Report, Peterson's, College Board, Barron's, Princeton Review, and other college guide surveys.
• Common Data Set. Completed the 2001-02 Common Data Set (CDS) with appendices, posted them on the web, and updated the web summaries that IRP has prepared to make the CDS data easier to access.
• ELCA. Completed and submitted reports to the ELCA on admissions, financial aid, enrollment and retention.
• HEDS. Completed various data reports for the Higher Education Data Sharing (HEDS) collaborative: Admissions, Freshman Financial Aid, EADA Athletics (DOE), Endowment (NACUBO), Graduation & Retention, Fall Enrollment (IPEDS), Faculty Compensation, Faculty Demographics, Voluntary Support of Education (CAE), FTE & Student Faculty Ratios, Tuition & Fees, Finance (NCES).
• IPEDS. Submitted Institutional Characteristics Survey for the federal government's Integrated Postsecondary Education Data System (IPEDS) in Fall 2001. Winter data collection included Employees by Assigned Positions, Fall Staff Survey and Faculty Salaries. Spring 2002 surveys completed were Student Financial Aid, Finance, Graduation Rate data and Fall Enrollment.
• Recent Graduates. Reported status of 2000 graduates to Minnesota Private College Council, including cross tabs of majors with jobs, further education, and volunteer organizations.
• Salary Reports. Compiled and submitted the AAUP, IPEDS, CUPA and ACM reports on faculty salary and benefits,
• Staff & Employee Reports. Prepared data, in conjunction with the HR staff, for the biennial IPEDS winter data collection reports on fall staff, faculty salaries & benefits, and employees by assigned position as well as for the ELCA Faculty, Staff & Administration report. Also supplied HEDS with supplemental demographic information on the AAUP faculty salary report.

One IRP goal for the coming months is to prepare and post to the web a master inventory and timetable of these kinds of regular external requests for institutional information.

The following two activities were undertaken as pilot projects to help us learn how we can best link data to serve as an aid to instructional planning. One is on-going, the other will be restarted this coming year:

• NCES Instructional Activity Pilot. St. Olaf accepted an invitation to participate in a pilot project of the National Center on Educational Statistics (NCES) to determine feasibility and usefulness of more refined methods of federal data collection for higher education. Susan Canon has compiled all the data and will participate in an on-line focus group this summer to complete our part of the project.
• Instructional Costs. We also began preparing data for St. Olaf to participate in the National Study of Instructional Costs and Productivity being carried out by the University of Delaware. Although much progress was made to link our academic and financial records--which is the central purpose of this project--we ultimately withdraw from the study because the staff work required in accessing the required data was excessive. (We hope to join next year's effort, building on work already completed.)

C. Internal Reports

• Assessing the Strategic Plan. IRP helped gather and analyze data to prepare for a special May meeting of faculty and staff to discuss the degree to which the current strategic plan has been assessed. (The goal of this exercise was not to assess the strategic plan but to learn from experience how to write a new strategic plan that may be more readily assessed.)
• Advising on Data. The IRP staff frequently advises other campus offices (e.g., Admissions, Communications, Education) on collecting and verifying data.
• Data Collaborative. With help from ACC, created a web interface for changes in student and parent addresses, linked via e-mail to all offices with a need to know.
• Diversity Reports. Identified and copied for the North Central diversity review committee all studies, surveys, and reports related to diversity at St. Olaf that have been completed in the last decade. In particular, secured release of an unpublished report on diversity at St. Olaf prepared by the U.S. Department of Education.
• Diversity Study. In anticipation of the North Central review, IRP commissioned a special report on diversity based on an analysis of data and evidence available from prior IRP surveys and external data sources. This survey has been received, formatted, printed and distributed. An adaptation was posted on the internal web site, and a summary was distributed as an IRP Note.
• Faculty Characteristics. Posted on the IRP web page data and graphs summarizing faculty age, rank, tenure, and promotion characteristics.
• Faculty Computer Use. Prepared a report for IIT on the St. Olaf Faculty Computer Use Survey for 2000-2001.
• Faculty Salaries. Prepared a report on faculty salary norms for the Board of Regents and advised RPC in their development of salary and compensation guidelines.
• On-Line Professional Activity Form. Monitored the design and development of an on-line system for recording and disseminating faculty professional activity. (This will become an ARP project).
• Retention Data. Posted on the IRP web page data from the registrar's office on retention and graduation rates.
• Student-Faculty Ratios. Prepared and placed on the web graphs with supporting data on student enrollments and faculty FTE for the period 1969-70 through 2000-01.

D. Data Coordination

A chief responsibility of IRP is to help establish and maintain the integrity and consistency of data which is stored, transformed, and reported from several different compute systems. Much progress has been made during the past year, but most of this activity is never-ending:

• Policy and Procedures. Susan Canon continues her oversight of policy and procedures to ensure data integrity and appropriate relationships between the faculty database in the Provost's Office and the Lawson financial system in the Human Resources Office.
• Survey Policy. Working with Jo Beld, Chuck Huff, Arnie Ostebee, the IRP staff revised and issued a new policy on surveys that was approved by the Cabinet last fall, including an approval form and guidelines for approval. We received many questions and issues during the pilot phase that suggests the need for clarification or reconsideration of some aspects of the policy. This is a matter that will have to be addressed before the beginning of the fall term.
• Student Records. Susan Canon continues to advise on the redesign of the student record system with a goal of making consistent and transparent the relation between student enrollment records and faculty instructional records.
• Faculty Database. As a result of recent changes in policy, Susan:

  (a) redesigned the faculty database to provide an addition to include faculty appointment history that is no longer maintained by the Lawson system;

  (b) extracted part-time coaches from the faculty database and designed a new database for them; and

  (c) refined data entry procedure in the faculty database to eliminate the distinction between "instructional" and "non-instructional" administrative assignments and provide more detailed data on actual administrative assignments.

These changes will create discontinuities in the way faculty data is reported on the IRP website, so for one year both the old and new systems will be used.

• Staff Data. This spring we used IPEDS data to begin the study of staff and administrative positions analogous to those already completed for faculty. Continuing and extending this work will be a priority for 2002-03.

E. Public Communication

A chief function of the IRP office is to respond to questions about data from faculty, staff, and students. Requests are quite varied, and many involve formal committees; this year the latter included requests from the student observer to the Board of Regents for information on diversity and student engagement; from the chair of the Review and Planning Committee (RPC) for information about faculty salaries and demographics; and from the Provost's office for things ranging from assistance with Access databases to reports to the College Council.

Another type of information sharing takes place with queries and studies initiated by members of HEDS--the Higher Education Data Sharing collaborative. Joy Johnson responds regularly to various e-mail queries from HEDS institutions. We receive summary reports from every query that we answer, and each summary is sent to the relevant office or department. This year St. Olaf responded to approximately 50 requests, covering topics such as dorm space, science and music fees, health insurance, facilities, registrar and treasurer's procedures, athletics, car usage, etc. In addition, St. Olaf participated in two ad hoc HEDS surveys on Deposits and Admissions.

On campus, most broad communication from IRP takes place in the form of e-mail messages called "IRP Notes" that, are sent to all faculty and, starting last fall, also to staff and student leaders. Topics of the 2001-02 notes are:

7/11/01	Report on Teagle Diversity Survey, Part I
7/11/01	Report on Teagle Diversity Survey, Part II
9/4/01	What is IRP?
9/5/01	National Survey of Student Engagement (NSSE)
9/6/01	US News College Rankings
9/13/01	Faculty Salaries and Compensation
9/14/01	Geographic Distribution of Students
9/27/01	Faculty Age and Tenure Data
10/28/01	Retention and Graduation Rates
11/6/01	Report on Last Spring's Staff Survey
11/19/01	NSSE "Student Engagement" Benchmarks
11/26/01	Tenure and Promotion, 1989-01
12/21/01	Faculty Salaries, Age Distribution, and Teaching Loads
1/14/02	Open Comments from Last Spring's Staff Survey
3/24/02	St. Olaf Employee Numbers, 1995-2002
4/9/02	New Policy on Surveys
4/14/02	Grade Point Averages
4/19/02	Comparisons with ELCA Colleges
5/19/02	Majors in MN Private Colleges
5/21/02	Diversity at St. Olaf in the 1990s
5/21/02	Demographic Climate for Diversity

The IRP web page is the main means by which the office communicates its reports. Most data is posted in the open section where it is available to everyone; some sensitive information and all survey results, being subjective, are posted for on-campus viewing only. During 2001-02 the organization, appearance, and content of the IRP web site was significantly improved. In addition, old files in the IRP UNIX account were sorted out, with important old reports relocated for web access. In parallel, last summer student workers prepared an inventory of past IRP studies and data.

The IRP web site contains approximately two-thirds of the reports and data that is included in the current plan. The chief area that is underdeveloped concerns college financial reports, and this will be a priority in coming months. But now that the site contains many reports, each of them must be updated (and occasionally corrected), usually on an annual basis. The time required merely to maintain a current website grows as the site grows.

Examples of planned and pending improvements to the IRP website include:

• Adding summaries of IPEDS reports to IRP web site.
• Preparing and posting staff salary histograms from IPEDS data.
• Completing a report begun this spring on tracing the history of IPEDS staff numbers.
• Updating various tables and improving the match of titles to data;
• Preparing and posting summaries of college income and expenses.
• Add appropriate links from other St. Olaf offices as well as archived data.
• Completing the separation of ARP and IRP files and data storage.
• Inventory contents of storage drives IRP network storage
• Prepare an on-line inventory of both paper and on-line data and reports

F. IRP Office Activities

• Space. Susan and Joy moved the IRP office into three offices on the lower level of the administration building.
• Job Classification. Completed the PDQ and job evaluation forms for HR office.
• Personnel Change. IRP Senior Consultant Jim Tallon, former director, retired last summer. He worked this spring as a consultant to write a special review of diversity issues.
• Research Associate. The joint request from ARP and IRP for a shared research associate to aid in special studies associated with the North Central review was not acted on for lack of funds.
• Computer Upgrades. IRP software was upgraded (Windows 98, SPSS Level 10, MacOS X, and Word for OS X.
• Professional Development (SC). Susan Canon gave a plenary presentation at the HEDS summer conference on St. Olaf's system of tracking and planning faculty FTE. She also attended the AIRUM conference in Bloomington, an MPCRF meeting in Minneapolis, and a HEDS conference in January in San Antonio. Susan has been appointed to the Technology Committee of HEDS.
• Professional Development (JJ). Joy Johnson finished the first part--a logistic regression study of the admission data of the class of 1998--of a planned statistical education project prepared under Bill Carlson's leadership prepared as a follow-up to auditing Statistics 263, Statistical Problem Solving. Joy completed the Intermediate SPSS course in late April and will be attending two additional SPSS courses in July.

G. Potential Projects

The following projects are of significant size and potential benefit and are listed as examples of what might be done with additional resources. Each represents unfulfilled potential--information that has been gathered and paid for, but which will lie underutilized unless a determination is made to learn as much as we can from the data.

• Alumni Trends. Prepare a summary of jobs, salaries, and further education from previously completed alumni surveys. Comparisons with similar data on recent graduates (six months out) would be valuable and also relatively easy to prepare.
• Analyze CSS Surveys. This year's CSS survey should be analyzed in relation to (a) comparative data provided with the survey itself; (b) related information from other current surveys (e.g., NSSE); and (c) trends based on previous CSS surveys of St. Olaf students.
• Analyze HERI Faculty Surveys. When we receive the results of this year's HERI faculty survey, we should be prepared to analyze it in relation to earlier faculty surveys and data from other colleges.
• Data Warehouse. Transfer important IRP archive files to IRP Web site to make them available as a source of data internally (for college offices, departments, and programs) as well as for external constituencies.
• Longitudinal Retention Study. A continual study that tracks each entering class to determine how different factors (e.g., GPA, SAT scores, ethnicity, finances) contribute to students finishing, leaving, or dropping out of St. Olaf.
• Majors Study. Several faculty have expressed interest in a study of the relation between entering students' major interests and their graduation majors--not just in aggregate numbers but by tracking individual students. (This might as well be an ARP project.)
• Narrative Comments. Analyze comments from various recent surveys (seniors, parents, alumni, etc.) to seek patterns in areas respondents have concerns about. Ideally, this information would be used to generate improvements, which would be highlighted in college mailings to parents, students, and admissions counselors as evidence that St. Olaf listens and responds to constituent concerns.
• Personnel Reviews. Conduct staff reviews (none have been done in several years).
• Web Site Index. As the IRP web page grows, it become important to prepare a site map or index.