ACE's First Review of MOOCs for Academic Credit

Over the next few weeks, WCET will publish a series of blog posts on Massively Open Online Courses.  Given the amount of press that MOOCs have received over the past year, you must be wondering why we need more writings on the subject.  Our guest blog posts will focus on implementation issues for institutions, including: the credit conundrum (are MOOCs credit-worthy?), lessons learned from those implementing MOOCs, and we’ll conclude with some overall observations and some (sometimes uncomfortable) questions.  Thank you to Pat Book for starting the series in clarifying many of the misconceptions around ACE’s credit recommendations for five MOOCs.

Russ Poulin
Deputy Director, Research and Analysis
WCET

ACE as Academic Credit Reviewer–Adjustment, Accommodation, and Acceptance
In retrospect, it seems like a foregone conclusion that the courses Coursera self-selected for review would be highly likely to receive an ACE’s CREDIT® recommendation.  And it is just that, a recommendation.  Credit decisions remain the sole discretion of each educational institution that receives an American Council on Education (ACE) transcript.

ACE’s history in reviewing training for credit recommendations began in 1945 with the review of military training.   It expanded in 1974 to include training occurring in corporations, federal agencies, associations, and other non-profit organizations.  More recently, ACE has seen a real shift in the emergence of new, alternative educational vendors in the ACE review process, largely focused on subjects commonly included in lower division baccalaureate studies.  These include the College Network, StraighterLine, DreamDegree Academics LLC, SOPHIA Learning, LLC, etc. While working at ACE, I was surprised to see that these unaccredited educational vendors now comprise a third of ACE’s CREDIT® clients and provide largely low-cost, accessible alternatives to college curricula.  Clients pay for the ACE review and students not only pay the vendor but also pay ACE to receive their transcript for submission to the institution of their choice.  ACE’s Center currently conducts close to 150 reviews yearly for the military and non-military clients.

This activity is largely self-supporting within the ACE organizational structure and has not been very visible in the institution’s priorities until recently.  MOOCs had the potential to catalyze ACE CREDIT’s expanded role and to generate substantial revenue from the envisioned sale of millions of transcripts worldwide.

ACE finds itself in the awkward position of advocating for the best interests of their institutional members while at the same time serving as a shadow accrediting body distributing the ACE imprimatur (defined by those very member institutions) to a host of newly emerging for-profit ventures whose mission and goals are very different.  While ACE seeks to fend off federal policy requiring the acceptance of the kind of Prior Learning Assessment ACE itself provides to these competitors and others, its staff also advocate on behalf of students receiving ACE transcripts for their acceptance at member and non-member institutions, working with state policy makers around the country to inform them about the ACE CREDIT® and ACE military reviews.  Given all this anomy, my experience at ACE suggests three rubrics to define their strategy—“adjustment” of their CREDIT review processes, “accommodation” to the demands of new providers, and “acceptance” of just about every paying client that walks in the door without regard to how it benefits its members.

The First MOOC  Review:  Adjustment and Accommodation
While serving as Assistant Vice President at the American Council on Education this past year, I led the unit that conducted the first review of MOOC courses offered through Coursera‘s platform.  The universities involved included the University of California Irvine, Penn, and Duke.  This was a pilot project tied to a Bill and Melinda Gates Foundation Grant I wrote for ACE to study MOOCs.  The review was conducted using the process that ACE currently uses in its reviews of workplace training.

Reviewing academic courses taught by faculty at top tier universities was a new venture for ACE as its CREDIT recommendation service was not designed for nor (to my knowledge) ever had been deployed for this type of review.  ACE leadership was anxious, as was Coursera, to address the major topic of discussion last year about whether or not MOOCs were credit-worthy.  My team conducted the review in January 2013, after conducting an onsite review of Coursera’s operation in the Fall of 2012.   The review was conducted virtually with four faculty reviewers over a one week period.  While we typically guarantee results within 30 days from the review, in this case, we produced the results within 15 days and this became a major news story.

The rules that ACE had to bend to accommodate the review included the requirement that the provider be in business for at least three years.  The review was conducted on Coursera’s one-year anniversary.  Since the courses were developed and taught by faculty at universities with considerable stature and history, this waiver was made.  Another rule that needed to be adjusted was whether or not the course had been offered at least once in the past.  In some cases, the courses had not yet been offered or were not fully developed.

Who are the Evaluators?
Key to the ACE review process is that all reviews are conducted by faculty, not ACE staff.  ACE staff support the process logistically and assist the provider in preparing for the faculty evaluators review according to the ACE Handbook, but ACE staff do not participate in the review itself.  Over the decades in which these reviews have been conducted, a substantial database of faculty reviewers has been developed with the supposed expertise to review the type of training you would expect to see in the military or in a corporate or agency setting.

To become a faculty evaluator an individual must be actively teaching college-level courses at an accredited institution—they can be adjuncts—and have a minimum of five years teaching experience.  They also must be receptive to recommending academic credit for learning that occurs in an extra-institutional setting.  Faculty apply online and ACE staff enter them into the faculty database if they meet the minimum requirements.  As reviews get scheduled, staff search the database for the content expertise and select faculty to participate in a review whose schedules match the date of the review.   An effort is made to pair a faculty member new to the ACE review process with a more experienced faculty evaluator since no formal training program is provided for these faculty evaluators.  Faculty are provided with a detailed Handbook and become part of a Faculty Review Team led by a National Coordinator who oversees and guides the faculty review process.  Sometimes that National Coordinator is an ACE staff member and sometimes an experienced faculty member.

The compensation for conducting these reviews is $400 for the faculty reviewer plus all travel expenses if an onsite review.   The National Coordinator, if a faculty member, receives an extra day of compensation for their responsibility in preparing the final report.  The reviews typically are conducted onsite where the organization under review has prepared materials for the faculty to review, including training materials, syllabi, assessments, and facilitator resumes.

Piloting ACE CREDIT Review on MOOCs
To engage this process in a review of courses taught by faculty at top tier academic institutions made demands on ACE’s Credit Review Process that had not been encountered in the past.  My prominent concern was finding peer faculty reviewers in the faculty database and, as one would expect, those were far and few between.  Therefore, I personally made a nationwide call for faculty at top tier institutions to encourage them to apply to be faculty reviewers and we scoured the database for faculty who were full-time professors, or were at top tier institutions or who had received their academic preparation at top tier institutions.  I used my continuing education colleagues as the contact point at institutions.

The response was fairly low and typically included faculty not at the main campus but at a branch campus of a top tier institution.  Having worked for 12 years at a top tier institution, I realized that there was virtually no incentive for faculty at these types of institutions to participate in this type of review.  First, the compensation is not competitive.  Reviews typically require three days for onsite reviews, two involving travel days, but the faculty are only paid for one day plus all expenses.   The time commitment is a second major disincentive, taking faculty away from higher value research activities.

Even with those barriers, reviewers were identified.

The initial courses subject to ACE review were selected by Coursera in consultation with their partner universities.  Coursera and the partner universities chose courses that were already offered on campus or were using content similar to an on-campus course.  ACE required that the universities provide a written statement confirming for ACE that they agreed to submit their courses to the ACE CREDIT review process since the review was conducted with the course hosting platform, Coursera.

ACE Review Process
As part of the ACE review process, faculty are charged to consider recommendations based not on their institution, locale, or region, but to critically appraise materials from a national and professional perspective.

The Faculty Review Team was led by the director of ACE CREDIT® and included four faculty with teaching experience.  Generally, faculty review teams consist of faculty across institutional types who are current in the relevant academic disciplines.  In this case, faculty from public and private four-year institutions were used, one an adjunct professor of math.

Depending on the course content, a minimum of two faculty reviewers are used for each course, sometimes expanding to three or more if the content includes substantial material from multiple disciplines.  In this case, there were two faculty reviewers per course.  Faculty on the team worked together to review the “content, scope, and rigor” of the course.  They reviewed the course syllabi, any texbooks, assessment methods, any student or instructor guides, laboratory projects, if relevant, instructional materials and curriculum vitae of instructors involved in developing and delivering the course to assess their qualifications.

ACE staff conduct the pre-site review for organizations seeking an ACE review.  Since these organizations are not primarily in the education business, they often need considerable coaching on what to expect in an academic review, including basics about development of a syllabus.  The pre-site review in the case of MOOCs was a review of the course platform, Coursera (at their Mountain View, California site), paying particular attention to student records and their security.

In the review process faculty focus on course content and learning objectives.  As they examine the depth and breadth of material and level of difficulty, they consider the applicability of the content to postsecondary programs.  Importantly, faculty examine whether the assessments of learning match the learning objectives in terms of depth, breadth, and level of difficulty.  One of the issues faculty frequently encounter in these reviews is that the learning objectives are at a higher level than the assessments resulting in a recommendation for lower division work.

Reviews are typically conducted in a face-to-face format with the organization at considerable cost due to the travel and hosting expenses of the review team.  Most organizations do not have the internet capacity to host an online review.  ACE created a virtual course review capability recently for those who do have the capacity and now some 20% of reviews are conducted with faculty distributed across the country and using a SharePoint platform to conduct the review.  In the Review, faculty are instructed to consider learning outcomes, course intensity, pre- and post-course assignments, instructor qualifications, and participants’ academic/work-related experience.  The faculty use standard and consistent review guidelines and ultimately must find consensus on any credit recommendations they provide, including the level of the course (vocational, undergraduate lower division, undergraduate upper division, or graduate) and the amount of credit to recommend.

Faculty reviewers are also responsible after the review is conducted to write the course exhibits for ACE’s online guides—The National Guide and Military  Guide Online.  Institutions rely on these guides when considering ACE recommendations as transfer credits for enrolling students.

Acceptance of Coursera’s MOOCs
Faculty used ACE review criteria to evaluate MOOCs offered through the Coursera platform.  All five courses reviewed received credit recommendations based on ACE’s review criteria.  The five courses received math and science recommendations, one at the developmental math level, that is, three-credits of pre-college, three at the lower division baccalaureate level, all three credits, and one two-credit recommendation at the  upper division baccalaureate level.  Faculty reviewed all course exhibits including learning outcomes, competencies, and assessment methods.  Faculty made suggestions regarding perquisites and offered other notes.  While ACE has recommended academic credit, it is up to each university or college to review these credit recommendations and determine how they may align with their general education requirements or degree programs.  There is no guarantee that any university of college will accept the ACE credit recommendations.

What does this mean?
In retrospect, as noted in the beginning, it seems like a foregone conclusion that the courses Coursera self-selected for review would be highly likely to receive an ACE CREDIT® recommendation.  They were courses developed by faculty and already reviewed for credit in their university system in some cases and just being offered in a new delivery method albeit to a massively scaled audience.

The review process doesn’t evaluate learning outcomes, but is a course content focused review thus obviating all the questions about effectiveness of the pedagogy in terms of learning outcomes.  The fact that they can be credit worthy, therefore, may not be a surprise, but it is useful to know that experienced, disciplinary faculty perceived them as credit worthy.  ACE announced in their original press release that they plan to continue to work with Coursera and other MOOC providers to evaluate additional courses in the coming months.  As a result of reaching out to post-traditional learners to promote educational attainment, ACE should see a substantial increase in their transcript revenue.

Another component of the Gates Foundation grant is underway with the University Professional and Continuing Education Association as they pilot acceptance of ACE CREDIT® from MOOCs by universities and colleges, especially those serving the demographic of low income, young adults.  Credit recommendations are one thing, their acceptance in meaningful ways by institutions to further the academic aspirations of vulnerable students is another.  MOOCs currently serve largely an international audience who already hold college degrees and have reasons other than degree attainment motivating them.   The jury is still out on the value for the vast majority of American students who need developmental education and/or are seeking affordable access to college credentials.  The integrity of the ACE CREDIT® review process, grounded in peer faculty review, is of critical importance to those institutions who depend on their recommendations as well as to students whose academic goals can be accelerated by Prior Learning Assessment.

Patricia Book
WCET Fellow
(Former Assistant Vice President at the
American Council on Education)

Photo credit:  Morgue File.