What MOOCs can teach us
about teaching

GS902 Research Project

Trevor Bekolay

Hello, my name is Trevor. I'm a PhD candidate in Computer Science, and today I want to talk about Massive Open Online Courses, or MOOCs. Specifically, I want to try to convince you that, whatever you think about how useful or transformative MOOCs are, they are undenaibly an amazing resource for advancing education research, because they provide a platform through which we can do empirical research about different teaching interventions.

What is a MOOC?

Platforms: edX, Coursera, Udacity

First, I want to make sure we're all on the same page about what a MOOC is. So, let me ask you, by a show of hands, how many people here have heard of MOOCs? How many people know about at least one of these MOOC platforms? How many people have actually taken a course through one of these platforms?

If not many have then... Okay, so to give you a very brief introduction, a Massive Open Online Course is a university-level course that is delivered through the internet, typically at no cost at all to the learner. Just to briefly show you what one of these courses look like, we can see this philosophy course, Think 101, that I took last year. It contains videos, similar to in-class lectures; quizes, similar to in-class quizes and assignments, only graded automatically; as well as many ways to track your progress and communicate with the instructors and fellow students.

The rise of MOOCs

Higher education is now being disrupted; our MP3 is the massive open online course (or MOOC), and our Napster is Udacity, the education startup.

— Shirkey, 2012

The fall of MOOCs

Compared to formal learning, there tends to be much higher rates of drop-out, and steeply unequal patterns of participation. This is probably an almost-inevitable consequence of any open, online activity. [...] The phenomenon shows that MOOCs alone cannot replace degrees or most other formal qualifications.

— Clow, 2013

For a bit of historical context, MOOCs are a very recent development. They evolved from traditional distance education and learning management systems like Moodle. But they're really a very different entity. The first MOOCs started rolling out in around 2008, and they were heralded as a transformative, disruptive technology. Quotes

But, since about 2014, the MOOC hype died down significantly, and people are much more metered and reserved in their view on MOOCs. I wouldn't say that anything has really changed in this time period, it's primarily the perception of MOOCs that have changed. Quotes

Questions

1. What completion rates should we expect?
2. Can we raise MOOC completion rates?
3. Can we raise traditional completion rates?

I'm going to address three questions with studies using quantitative research methods. My secondary goal in this presentation is to convince you that MOOCs give us an unprecedented ability to do large-scale quantitative education research.

Q1. What completion rates should we expect?

A: Most people are sampling

• Learner subpopulations in MOOCs.
  Kizilcec, Piech & Schneider, 2013
• Dataset: 94,000 student activity logs from 3 Coursera courses

Example trajectory: $[T, T, T, T, T, B, A, A, A]$

Q2. How do we raise MOOC completion rates?

A1. Students watch short, well-produced videos

• How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos. Guo, Kim & Rubin, 2014
• Dataset: 6.9 million viewing sessions from 4 edX courses

A2: Superposters foster healthy forums

• Superposter behavior in MOOC forums.
  Huang, Dasgupta, Ghosh, Manning & Sanders, 2014
• Dataset: 325,000 posts from 44 Coursera courses

Q3. Can we raise traditional completion rates?

1. Are all in-class students "completing"?
2. 6-minute lectures before interactive classes.
3. Provide an ignorable outlet for "superposters".

Recommendations

Teach data literacy
to education researchers

It was surprising to see a low number of proposals that had planned to make use of the techniques and methods of learning analytic and educational data mining (LA/EDM).

[...] Our results indicate that the MRI review panel expressed a strong preference towards the use of the LA/EDM methods.

— Gašević, Kovanović, Joksimović & Siemens, 2014

Discuss the ethics
of huge data collection

Enrolled learners in a MOOC are potentially all over the world and therefore are likely to have different cultural and personal sensitiveness about privacy issues. [...] Learners might feel violated if they saw their posts de-contextualized and highlighted in a publication.

— Esposito, 2012

Follow up
with qualitative studies

It does not require trillions of event logs to demonstrate that effort is correlated with achievement. [...] The next generation of MOOC research needs to adopt a wider range of research designs with greater attention to causal factors promoting student learning.

— Reich, 2015

Takeaways

1. Are all in-class students "completing"?
2. 6-minute lectures before interactive classes.
3. Provide an ignorable outlet for "superposters".

Recommendations

1. Teach data literacy
2. Discuss big data research ethics
3. Follow up with qualitative studies

I hope these studies have convinced you that the wealth of data that MOOC platforms make available to researchers enables us to learn ways to improve our teaching, both in and outside of the MOOC environment.

I want to finish the presentation with what I think are the interesting implications of the rapid data-driven advancement in education research, and these are things that we can talk about in the discussion period.

First, as more data-driven education research is done, education researchers will need to become more data literate in order to review new literature and assimilate it into their own pedagogical ideas. While researching this topic, I found several papers that were using poor statistics, or very strange interpretations of the statistical tools. The fact that these papers passed the peer review process is evidence that we need ways in which to educate education researchers about data. There are some initiatives attempting to do so, such as Software Carpentry and its sister projects Data Carpentry, and Library Carpentry.

Second, we need some clear ethical guidelines as to how much data we can collect on students without their explicit written consent. While we can skirt some of these issues with anonymized user data, sometimes it becomes necessary to actually identify certain users to have a real conversation with them about their experiences in a course.

This leads to my third and final thought, which is that we should be careful not to claim that data-driven research will replace other kinds of research. Data science is just one tool; it can help us generate new research questions to investigate, but in order to answer those research questions, we need to use our whole aresenal of research tools; live interviews, studies with control groups, and so on.

With that, I'll thank you all for listening, and look forward to talking with you about these ideas!