MOOCs – A Question of Credit

John Ebersole, president of Excelsior College, gives his views on where MOOCs currently stand on their trek to become avenues for students to receive college credit.  This guest blog posting is part of a short series on MOOCs and follow last week’s “ACE’s First Review of MOOCs for Academic Credit.”  We are pleased to welcome President Ebersole and thank him for his contribution.
Russ Poulin

Massive open online courses (MOOCs) have captured the attention of higher education and the political community in a way seldom seen. To many, the idea of access to instruction from world class institutions for free is the answer to a prayer. They offer the promise of a new form of instructional delivery that could take some of the heat off of higher education, and its ever rising tuition, as well as for the politicians whose cuts to state support have made necessary those increases. Hopefully, with MOOCs, state governors, legislators, and college administrators can find ways to promote universal access to higher education without having to worry about cost.

On the other hand, many within higher education are critical of MOOCs as vehicles for the creation of learning. They see any rush to recommend credit that can be used to satisfy degree requirements as premature. Partly as a result of this growing concern, only five MOOCs have been deemed credit-worthy by the American Council on Education, so far.

What Have MOOC Students Learned and How was It Measured?
Even where credit has been recommended, questions remain as to the extent that participants actually learned anything. While few question the capabilities of the sponsoring institutions or their faculty, the degree to which these reputational factors translate into learning is not clear. To date, very little attention has been given to the measurement of learning outcomes by MOOC providers.

Additionally, breathtaking attrition has called into question whether MOOCs are more akin to entertainment, or a 21^st century text book, than to a serious learning experience. According to those in a position to know, one recent Coursera offering, initially enrolling more than one hundred thousand, had but seven take the final exam. When such a large number fall along the way, one must wonder about the quality of the journey according to those administering the exam (who have asked to remain anonymous).

Historically, a recommendation of credit for learning outside of an accredited institution, and the oversight of its faculty, has been held to higher standards than we see with MOOCs. Whether it be The College Board and its College Level Examination Program (CLEP), Educational Testing Service and its DSST portfolio, or the UExcel exams of Excelsior’s Center for Educational Measurement (CEM), college-level credit comes from passage of a psychometrically valid assessment, administered in a secure facility, operated by a third party. Those colleges and universities that accept such credit recommendations have come to expect nothing less.

At Excelsior, a typical subject assessment (i.e. college algebra) is created with input from subject matter experts drawn from multiple institutions. Working with CEM’s doctorally prepared psychometricians, exams are prepared for field testing. This process assures statistical reliability, predictability, and overall validity. Such exams are a far cry from those drawn from a text book publishers “test bank” or an instructor’s notes.

Assuring Academic Integrity is Essential  
In addition to the validity of an assessment, it is also expected that it will be administered with attention to student identity and security for the instrument itself. While some MOOC providers do verify the identity of those taking their course, and proctor their end-of-course examinations, little attention is being given the security of the assessment instruments.  This could become a problem.

With the potential for thousands of exam takers from around the world, question banks need to be larger than for a homegrown instrument or a smaller population of test takers. ETS and Pearson learned this some years ago when the Test of English as a Foreign Language (TOEFL) was compromised by well-organized Asian students with near photographic memories.

To be clear, for an assessment’s results to be worthy of transcription and acceptance toward degree requirements, it should be:

1. Prepared with input from multiple subject matter experts who agree as to what constitutes an acceptable level of knowledge to warrant college credit;
2. Created and field tested by qualified psychometricians;
3. Based upon question/item banks sized to insure random item ordering (i.e., no two examinees receive an identical exam). [The fact that this is not yet a problem has more to do with high attrition than attention to scalability.];
4. Administered in a secure facility, where multiple forms of identity verification can be employed, including biometrical data.

MOOCs often Fall Short in their Assessments
This contrasts with what is seen from the major MOOC providers. A final examination, or assessment of learning, if provided, is typically prepared by the offering instructor. As one edX executive stated, “The fact that the exam comes from an MIT faculty member is the only validity we need.” This person is usually not trained in the preparation of statistically valid assessments and, in the eyes of some, has an inherent conflict of interest in evaluating his own work. Administration of those exams now offered is understood to involve remote proctoring employing an internet camera and/or various forms of identity assuring software.

At a time when the federal government, regional accreditors, and much of the academic world has shifted from the measurement of inputs (number of PhD faculty, ratio of students to instructors, books in the library, etc.) to the assessment of outputs, MOOC providers and their sponsors must increase their attention to the learning side of the ledger. No longer can or should we assume that a sufficient level of learning takes place merely because the instruction comes from a brand name institution, with a credentialed instructor. Today, there is an expectation that we measure actual learning. This is, admittedly, new, both for the instructional provider and for those who are asked to determine possible equivalency – the American Council on Education (ACE) and the National College Credit Recommendation Service (NCCRS).

But There is Hope for MOOCs
The good news is that this need not be a particularly difficult problem to resolve, assuming of course that learning is occurring and that we just need to do a better job of measuring. ETS, The College Board, and Excelsior have a combined inventory of exams that nears one hundred undergraduate subjects (upper and lower division). While most MOOCs have so far featured topics for which there isn’t an appropriate exam, the process to create one is not substantially longer or more complex than what is required to create and offer a course in MOOC format. In return for paying more attention to learning outcomes, and how they are measured, MOOC providers can not only help enhance the credibility of this pioneering effort, but also gain credibility in the eyes of critics. It can also provide those who complete their offerings with the credit they seek.

John Ebersole
President
Excelsior College