The Benefits and Challenges of Course Learning Outcome Assessment in Every Course Every Term

How does your college or program handle learning outcome assessments? Today we’re thrilled to be joined by several representatives from the University of Western States, Bernadette Howlett, Denise Dallmann, Dana Sims-Barbarick, and Susan Donoff, to discuss the University’s Every Course Every Term (ECET) initiative. This continuous improvement program helps faculty collect and act on course learning outcome data. Read on to learn more about this great initiative and their lessons learned, in case you’re interested in developing a similar program.

Thank you to Bernadette, Denise, Dana, and Susan for sharing this program with us!

Enjoy the read and enjoy your day,

Lindsey Downs, WCET

---

Questions regarding the quality and value of a college education are stimulating calls for accountability through measures of student outcomes. Much of this attention has been focused on graduation rates, loan repayment, debt burden, and gainful employment. And, as WCET has recently reported, the federal government is preparing to alter the regulations that govern higher education accreditation. Among other changes the Department of Education is considering is to increase the emphasis on outcomes (degrees conferred, licensure exam results, etc.) for purposes of determining access to Title IV funding. While metrics associated with graduation and employment offer high-level perspectives on how institutions and programs perform, they have two important limitations:

1. There is no opportunity to intervene for individual students by the time they experience the outcome; and,
2. The measures are one-dimensional.

Most institutions engage in robust assessment processes focused on measurement of mission and goals (or Core Themes for those accredited by the Northwest Commission on Colleges and Universities) attainment as part of their self-assessment of quality.

Institutions collect a variety of focused indicators of students’ performance and progress at multiple levels as represented in Figure 1.

Though there is debate about the impact of learning outcome assessment, one of the most useful quality indicators can be course learning outcome assessment data, because it enables early intervention and continuous course and program improvement.

Every Course Every Term (ECET)

In order to utilize early performance data about students and programs as part of its self-assessment system, University of Western States (UWS) in Portland, Oregon, has undertaken a bold continuous improvement initiative to collect and act on course learning outcome (CLO) data every time each course is offered. This initiative is called “Every Course Every Term, or ECET.”

After a successful pilot phase, UWS launched its ECET initiative in the summer term of 2018. The cycle for the process, represented in Figure 2, utilizes a rapid instructional design system.  Each course proceeds through a three-step process:

1. mapping CLOs in the Learning Management System (LMS) to their corresponding assessments and teaching the course;
2. extracting and analyzing the CLO data from the LMS; and,
3. initiating course improvements based on that analysis.

The process is then repeated and documented in a course ECET report that is used to track attainment of student learning outcomes over time.

The data are used to inform revisions at multiple levels: course design/outcomes, assessments, program design/outcomes, admissions requirements, policies, identify professional development opportunities, and more. UWS faculty and staff initiated greater frequency of CLO review than the common practice of once every few offerings due to a strong interest in continuous improvement to support student success.

Student success and teaching excellence are central values to the institution. As articulated by Dr. Denise Dallmann (Dean, Center for Teaching and Learning):

“We always come back to the mission.” The mission of UWS is to advance the science and art of integrated health care through excellence in education and patient care. Further, the university’s first Core Theme is Student Success. “Excellence in education is really about continuous improvement and doing everything we can to support student achievement of learning outcomes.”

The Project

UWS anticipates it will take three academic years to “onboard” every course to this project.

The plan is to include an additional 10 courses per term (two per program) for a total of 120 courses, including all electives. Not every course is offered every term, but by the end of the three-year onboarding period, all courses will be in one of the three phases of evaluation and that evaluation will be repeated each term that course is taught.

Challenges

The College of Graduate Studies at UWS offers primarily fully online programs and therefore was selected to pilot ECET. The following are examples of challenges identified during the pilot. We have also identified possible solutions to each challenge:

Initial Challenges	Possible Solutions
Some courses need to use assessments that involve observation of students performing skills in settings that involve interactions with clients or patients. Data capture in these situations is technologically challenging.	Using recorded videos of student-client interactions along with assignment rubrics to assist with accurate grading of clinical skills.
The first round of data collection revealed there were some learning outcomes that lacked assessments and some instances of mismatch between learning outcome level (Bloom’s Taxonomy) and the assessment strategy.	Faculty may choose to improve these areas as part of their action plans in the ECET reports.
Volume of data and reporting required to track every course every term – and finding efficient technology to support the process.	The project leads have been using Excel spreadsheets for the ECET reports. Basecamp, a web-based project management software application is used to track due dates and assign steps in the process. UWS is evaluating more efficient tracking and reporting options going forward.

Lessons Learned

1. This initiative has been very faculty-centered from the beginning, and that’s a major reason why it is proving to be a success! Faculty are in the best position to initiate course changes in order to improve student achievement of learning outcomes.
2. Start with the “low-hanging fruit.” Certain courses were more readily adapted to the new system. Starting with them allowed us to learn how to run and improve this process.
3. Get started, even if everything isn’t perfect. Learning outcomes and assessments don’t have to be perfectly written or aligned in order for similar initiatives to succeed. This process is intended to allow for growth and improvement. As we have affirmed along the way, “Perfect is the enemy of good.”

While this project is still in its beginning stages, we have had some important wins such as engaging and getting buy-in from all UWS’s graduate program directors and developing a good system to keep the project on track. We look forward to reaping the rewards of systematically reviewing student CLO data and seeing stronger student achievement of learning outcomes.

Bernadette Howlett
Affiliate Faculty
Idaho State University
(worked for UWS until recently as Provost and VPAA)

 

Denise Dallmann
Dean, Center for Teaching & Learning
University of Western States

 

Dana Sims-Barbarick
Dean, College of Graduate Studies
University of Western States

 

 

Susan Donoff
Director of Institutional Appraisal & Accreditation
University of Western States

 

---