#CountAllStudents and the Move from Graduation Rate to Outcomes Measures

As we make our way through the final few weeks of the traditional college graduation season, it makes me reflect on the flawed first-time, full-time federal graduation rate used by the Department of Education. There’s been news on this front in the last few weeks. Let me update you on how these stories might affect you.

The IPEDS Workshop
A few weeks ago I was fortunate enough to attend the 2016 IPEDS Coordinator Workshop and State Data Conference. Yes, it takes a certain type of data nerd to enjoy room full of equally nerdy people. Wait! Wait! Don’t stop reading! Rest assured that I will focus on the highlights.

IPEDS is the Integrated Postsecondary Education Data System, which is a set of surveys conducted by the U.S. Department of Education and that every institution receiving federal funds is expected to complete. We use IPEDS data for our reports on distance education enrollments.

Thank you very much to the U.S. Department of Education for allowing me to participate. The meeting is focused on the IPEDS Coordinators (they help the institutions with questions they have) and a few others doing interesting things with data. I was pleased that the Department responded to a request from Demi Michelau, WICHE policy director, to include me.

New Outcomes Measures to Accompany the Much-maligned Graduation Rate
The IPEDS Graduation Rate has come under fire in recent years because the measure is based on a few students. The process begins with an institution identifying all of the students in the freshman class who are “full-time” (no part-timers need apply) and “first-time” (first time that they have gone to any college). [IPEDS definitions, choose F to find both first-time and full-time]

You can see how this is troublesome for community colleges and other adult-serving institutions as only a small percentage of their enrollments fit this category. The increase in high school dual and concurrent enrollments also calls the usefulness of the measure into question. Likewise, students transferring into an institution have no home in this counting method.

In response, IPEDS recently started collecting data for a new Outcomes Measure (OM) to accompany, but not replace the Graduation Rate. The first set of results should be released later this year. Instead of the one “full time, first time” cohort, the Outcomes Measure includes four cohorts:

• Full time, first time,
• Part Time, first time,
• Full time, not first time (not first-time means that they attended another college before coming to yours),
• Part time, not first time.

Outcomes will be reported for both six and eight years after each cohort has entered a college. The six-year report will focus on those students who are awarded a certificate or degree in that time. The eight-year report will report completions and will additionally report: students still enrolled at the original reporting institution, students who are now enrolled at another institution, and students whose current status is unknown.

The data will be available in the downloadable datasets provided by IPEDS. It will also be included in the College Navigator tool, which is aimed at people engaged in the college search.

In the reporting, the Outcomes Measures will report the “total number of students who did not receive an award from your institution” from each cohort. An attendee at the IPEDS Workshop asked if there is any way to report students who completed elsewhere. The answer is no because they did not want to double-count graduates at both the original institution and the institution that granted the degree. They also did not want to mandate use of the National Student Clearinghouse, which tracks such activity.

In my opinion, adding the Outcomes Measure is a fantastic first step. It is finally a recognition that the traditional full time, first time group of students is an increasingly smaller set of overall higher education enrollments. I do worry that they will collect this information and still highlight the “students who did not receive an award from your institution” number. A suggested remedy was to add all students who obtain a certificate or degree with transfer students. That’s a nice suggestion, but there is no indication that they will implement it.

The Student Achievement Measure Urges Us to “#CountAllStudents”
In the last few weeks, proponents of the Student Achievement Measure (SAM) have launched a media campaign in conjunction with spring graduations. They urge the Department to “Count All Students” using an alternative graduation measure jointly-developed by several colleges.

It’s an effective campaign highlighted by stories of real students, who are not counted as graduates by any institution in the current Graduation Rate. Among those not being counted is President Obama. He transferred from Occidental College to earn a bachelor’s degree at Columbia College, Columbia University. I also don’t count as I attended three institutions before earning my undergraduate degree.

The Department of Education highlights the differences between the Outcomes Measure (OM) and SAM in an FAQ:

“OM is similar to SAM in that both have the same goal of measuring postsecondary success and progression of undergraduate students. However, the methodologies used to measure the outcomes are different. First, OM is part of the mandatory IPEDS collection compared to SAM’s voluntary participation. Second, OM has 4 cohorts and SAM has up to 7 cohorts. Third, SAM captures student progress and success at the award level (bachelor’s and associate’s/certificates awards), whereas OM does not make a distinction between award levels. Lastly, the time points for SAM varies depending on the cohorts compared to OM’s standard use of 6-year and 8-year time points across all cohorts.”

Not Everyone Thinks Change is Necessary

Meanwhile, one former Department of Education official discounted the call for improved graduation rates by referring to an article he wrote a few years ago, in which he said, “Officials at colleges with low graduation rates have for years defended their rates by falsely asserting, perhaps mistakenly believing, that their part-time students who eventually graduate are ‘counted as failures.’ There might be a small few participating institutions for which that is true, but according to an Inside Higher Ed article, roughly three-quarters of the nearly 600 institutions supporting SAM are four-year, public institutions. Even though the rest of the narrative explains how the rate is calculated, there is little understanding shown of what actually happens in community colleges, urban universities, or other adult-serving colleges.

The Proof is in the Data…and the Display
From the IPEDS Workshop, they did not seem ready to entertain many changes to the new Outcomes Measure. It will be good to see the real data and how it is displayed. Those factors will help us determine the usefulness of the data.

For our members, I think we need to watch this closely. Since the Outcomes Measure merely supplements the Graduation Rate, will this cause more confusion for your potential students. While we will have finer data, will students want to delve into the different cohorts and figure out what it means? There is a heavy burden in how this is displayed by the Department. Additionally, an increasing number of third-party college rating sites (ugh) use the Graduation Rate with no explanation what its limitations are. Will they use, ignore, or botch the Outcomes Measures?

Finally, I must acknowledge that the IPEDS staff have a thankless task in trying to create understandable and comparable data when colleges can be so different. And colleges OF ALL TYPES are constantly looking for ways to game the numbers.

Understandably, IPEDS staff want to be careful about changes to survey procedures. Since this data release will be the first round of Outcomes Measures, I do hope that they will be open to reasonable improvements.

Although in the current political climate, I’ll admit that I am hard-pressed to say what the definition of “reasonable” is any more.

Russell Poulin
Director, Policy & Analysis
WCET – WICHE Cooperative for Educational Technologies
rpoulin@wiche.edu
wcet.wiche.testing.brossgroup.com
Twitter:  RussPoulin