What Faculty Designers Can Tell Us about Online Course Reviews

In 2019, I asked our readers the question “how do you ensure quality in online courses?” This question was part of my introduction to a fantastic article written by Aimee deNoyelles from the University of Central Florida. Aimee discussed UCF’s quality assurance program – an online course review process that acknowledges the achievement of quality online course design.

Well, after several years (including those disrupted by you know what) of implementing said course review process, Aimee joins us again today to give us an update on additional results of the course review process and discuss recommendations for improving the program based on recent faculty surveys and focus groups. Thank you to Aimee (and the team at UCF!) for your hard work supporting quality online learning and for sharing your journey with us here at Frontiers.

Enjoy the read,

Lindsey Downs, WCET

---

In a previous Frontiers post, I shared my university’s approach to promoting online course reviews to teaching faculty. Main recommendations included in that post were for online course reviewers to:

1. engage in dialogue with the faculty designer;
2. emphasize guiding philosophies of the course review process;
3. amplify student and faculty voices; and,
4. celebrate the successes.

Once reviewed, courses could potentially be designated as quality online courses, which highlights that the course meets certain characteristics and expectations.

Three unpredictable years have passed since that post was written. Since then, another 300+ course designations have been earned by nearly 300 faculty members. In addition to the basic online course designation (“Quality”) that was developed in 2017, an advanced designation called “High Quality” began being offered in 2018. Due to demand from faculty and the success with blended courses, courses offered in blended modalities became eligible for the “Quality” review in 2020, with the “High Quality” version following in 2021.

Having last gathered formal feedback from faculty in 2019, we thought it was high time to reengage in dialogue in 2021. Faculty who had ever earned a course designation were invited to complete surveys and participate in focus groups in order to better understand the dynamics of the online course review. Four themes emerged:

Theme 1 – Identifying the Motivations to Participate

When asked about their motivations to participate in course reviews, the majority (74%) of survey respondents said the primary reason was “to improve the learning experiences for students.” All other reasons (“I wanted a digital badge to display,” “I enjoy working with my instructional designer,” “This supports my promotion and tenure”) were much lower (under 10%).

This supports one of the underlying guiding philosophies posed in my earlier blog post: “Ultimately, this is about the success of your students.” 90% agreed or strongly agreed that the course review process improved the design of the course. Some respondents mentioned these specific improvements:

• Better course organization,
• Ease of use and improved accessibility, and
• Increased student performance.

Theme 2 – Articulating the Nature and Evolution of the Reviews

The review items address effective online course design rather than teaching. Understanding the nature of the review remained a challenge for faculty participants. As one respondent declared in the survey, “Quality can’t be done in a checklist – it’s the knowledge of the instructor and the skill working with students.” Faculty in the focus group were asked to interpret this declaration. This led to a discussion about how faculty saw the items in the review (the “checklist”) as a means to set up the conditions for a quality online learning experience, with the role of the instructor crucial in actually bringing it to fruition.

Survey respondents were asked how well they understood aspects of the review process. Some aspects were not as understood as well as we would have hoped, namely “when the designation expires,” “what course modalities are eligible for a review,” and “how a High Quality designation is achieved.” In hindsight, this should not have been a surprise, since all three of those items have either been updated or introduced since the inception of the review process in 2017.

Theme 3 – Clarifying Faculty and Reviewer Roles

One of the guiding philosophies of the course review process is the engagement in a collaborative review with the course reviewer, which is typically an instructional designer (ID). The relationship with the ID was positively regarded. 94% of survey respondents agreed or strongly agreed that the collaboration helped to generate strategies to improve course design.

A concern, however, was when an ID would recommend course revisions that the faculty did not personally agree with. As one faculty noted in the focus group, “I did things my ID told me to do to get the badge even though I didn’t really want to….” Another faculty member suggested reframing the initial dialogue from the ID; she wanted to hear, “tell me what you’re trying to achieve,” rather than “here’s what to do earn the designation.” 

Theme 4 – Notifying the Student

During the focus group sessions, questions about student understanding of the course designation organically emerged. Questions included, “How do students know about it?”, “Can they see it when they sign up for the course?”, “How do they know to click on the badge?”, “How many students have clicked on the badge?” One faculty member lamented, “No students ask me about it.”

What’s Next?

Deeply listening to faculty members about the course review process is critical to the continued success of this evolving initiative. Based on their feedback, we developed the following key recommendations:

• Explicitly relate faculty motivations to participate with the cited benefits of engaging in a review. We learned that faculty were engaging in the review to improve learning for students, therefore this is a relatable way to frame the reviews to people who are reluctant. To encourage faculty not yet participating, publicize the proven benefits; for instance, our course review home page shares direct quotes from participating faculty about improvements in course organization, accessibility, and student performance.

• Clearly articulate the intention of the reviews to all parties involved: faculty, course reviewers, and students. Acknowledge that “quality can’t be [fully] done in a checklist” – that the faculty member is the expert and facilitation is important; however, these items have been proven to help design the initial foundation. This will require modifications to our website and training.
• Clarify the aspects that are not well understood. Talking with faculty allowed us to understand the aspects of the process that were not well understood such as when the designation expires, what modalities were eligible, and how High Quality was achieved – all opportunities to strengthen the communication regarding the program. Modifications to the website and training are needed.
• Emphasize the collaborative element of the review process. Positive feedback was given with regards to working with an instructional designer. We think that has a lot to do with the guiding philosophy of the review, and the front-end quality approach of pairing with instructional designers throughout the process. However, with instructional designers joining the team since the inception of the review process, along with growing workloads, it seems an appropriate time to reengage in dialogue about the balanced nature of “collaboration.”
• Involve students more actively. In the previous blog post, one of the recommendations concerning promoting online course reviews was to amplify student voices. In several cases, students have been surveyed to provide their feedback about the course design before the review takes place, offering a “pre” and “post” comparison of sorts. Frankly, this has not happened as much as we would have liked. One effort to make students more aware of the designation came from a faculty member – he let us use his course announcement that describes the nature of the designation.

As  discussed in my first post, quality can be a tense topic for many in the online education realm. However, by taking the time to listen to those most involved in course development and creation, we can, ultimately, ensure the success of our program, our faculty, and, most importantly, our students.