How do audio-based podcasts contribute to learner experience and success in video-based MOOCs?

We foster the following contributions from our studies:

Contribution 1: In a study with more than 900 learners during an online course offered in 2022, we assessed the applicability of audio-based education (Köhler et al., 2022). We identified no statistically significant differences in learning success between video and audio-based education. Hence, we derive that audio-based education could prove to be a suitable addition to video-based online course.¹

Contribution 2: A follow-up study and survey in another 2022 online course with more than 1,400 participants and 250 survey respondents provided answers on learners' expectations for podcasts alongside video-based MOOCs. From that survey, we derived, e.g., the ideal length to be 35–45 min and the desired consumption method of podcasts for our participants.

Once we derived applicability, context, and participants' wishes toward the podcast design, we performed an in-depth study with weekly podcasts provided alongside a 6-week MOOC on cybersecurity.

Contribution 3: Using a Static-Group Comparison, we identified a statistically significant increase in learning success of participants' exposing themselves to podcasts (Köhler et al., 2023). We contextualize the findings by an in-depth discussion of alternative explanations or threats to the validity of our study.²

In this manuscript, we particularly extend our previous publications by providing insights into participants' expectations toward our podcasts (Contribution 2) and discussing the results of Contribution 3, such as investigating threats to validity and potential alternative explanations. In the remainder of this work, we highlight methodologies employed during the different studies (Section 3). In Section 4, we shortly present results already highlighted in the previous publications (Köhler et al., 2022, 2023), and perform an in-depth discussion of Contribution 3 to provide more references for contextualization of the observed results (Section 5).

Podcasts as a medium for information exchange and learning have been rising in popularity in recent years (Koehler et al., 2021). Similarly to video-based MOOCs, they provide free knowledge and are easy to consume by their listeners. To the best of our knowledge, explicit comparison between both types of education formats has been limited in previous publications, with some researchers explicitly excluding the comparison from their work (Drew, 2017). However, only some studies have investigated the impact of different education mediums, concretely comparing audio-based to video-based education. Yet, the respective research is often limited by a small study group size (N < 100) (Daniel and Woody, 2010; Shqaidef et al., 2021).

In other cases, podcasts are used to aid students with disabilities in keeping up with their peers and learning content (Kharade and Peese, 2012). In 2023, Gunderson and Cumming (2023) have performed a literature review on podcasts in higher education teaching and learning examining 17 previous studies. Building on the context of higher education in universities, however, the previous studies are also subject to the limitation of participant size with < 150 students or learners each. The authors report that most studies connect podcasts to favorable student perception and that podcasts increased or aided student's knowledge of the respective subject.

Our work adds to research on the media of podcasts in education (Salomon and Clark, 1977) by evaluating whether current video-based MOOCs can substantially benefit from podcasts. While learners can currently choose to only listen to videos (and thereby treat them as audio content), they would miss out on the video's visual component, including any slides shown. In contrast, if the content is created particularly for a podcast with an audio-first principle, it is designed to be listened to without relying on visual input. Thereby, research in our field could open up possibilities for integrating education into a busy day, such as on a commute, during household chores, or even enhance the MOOC experience for visually impaired learners (Kharade and Peese, 2012; Amponsah and Bekele, 2022).

Various researchers are investigating methods to help learners succeed in online courses. A particularly well-studied topic is, for example, the field of video-based education, in which researchers identify best practices for increasing learning success or content retention and maintaining the interest of students (Prensky, 2006; Xia et al., 2022). Similarly, gamification and game-based learning are topics studied by many researchers, as both can be implemented in various forms to improve learning outcomes for all age groups (Strmečki et al., 2015; Orhan Göksün and Gürsoy, 2019; Hadi Mogavi et al., 2022). Finally, the last years showed increased research on Self-Regulated Learning (SRL) methods, which should support learners to be more focused and achieve the most in their invested time (Zimmerman and Schunk, 2011; Broadbent and Poon, 2015; Kizilcec et al., 2017; Cohen et al., 2022).

Most previous studies on podcasts as an educational medium have reported podcasts to impact learner perception positively (Stoten, 2007; Kay, 2012; Goldman, 2018; Hense and Bernd, 2021; Gunderson and Cumming, 2023). Podcasts as a standalone medium are widely accepted due to their ease of use and flexibility. Our study investigates whether podcasts could be a suitable addition to video-based online courses to help with problems such as video fatigue and limited interactivity in the course, thereby opening up new possibilities for learners to consume the learning content without being tied to a screen.

Within our studies, we investigate the behavior of human subjects who receive or do not receive a specific treatment. All studies were carried out on the online education and research platform openHPI. One of the core goals of the team of researchers behind openHPI is developing and assessing new methods and types of online education. Users agree that pseudonymized data can be evaluated to enhance the services and drive research. This data includes, e.g., users' course performance. Additional data, such as demographic information, can be provided optionally by a user in voluntary surveys, which are not linked to or required for course participation or success. All studies presented within this manuscript are based on additional, optional course content. Participation in the studies and working on the respective course content was not required to complete the courses offered on the platform, and users could opt out of the study at any time.

To the best of our knowledge, the impact of audio-based compared to video-based education is understudied. Some authors have explicitly excluded the comparison from their work (Drew, 2017), or the number of participants in their studies was limited (N < 100) (Daniel and Woody, 2010; Shqaidef et al., 2021). As an initial foundation for potential follow-up studies, we therefore challenged whether podcasts are fit to provide educational content in the context of MOOCs. To assess the differences between audio- and video-based content transmission, we performed a Posttest-Only Control Group (Campbell and Stanley, 1966) study. We randomized (R) learners into three groups, exposing them to three different treatments (X₁, X₂, X₃), and observed the learning outcome through an identical posttest (O). The experiment is repeated three times with three different Deep Dives throughout the course. Assignments to the treatment groups of each user remain identical throughout the three experiments.

Traditionally, our MOOCs feature a presenter and slides to accompany the content presentation. To foster an initial understanding of the applicability of audio-based education, we provided two variants of podcasts for the Deep Dive elements. In one of the podcasts, we employed only a single presenter (Traditional Podcast). In the other, we recorded an interview between two presenters (Interview Podcast). To compare particularly the Interview Podcast against video-based education, we provided Interview Videos to the third cohort of learners. Upon completing each learning item, learners were tasked with a content quiz and asked to provide further feedback in a survey.

We first recorded the Interview Video to ensure content consistency throughout the treatments. Then, we extracted the audio track to provide as our Interview Podcast. Our presenters did not use additional visual aids, such as presentation slides, to help the presentation. This ensured that the supporting visual presentation would not be missing in the audio-only variant. Later, we wrote a script for a single presenter based on the recorded video and audio. We ensured that the Traditional Podcast with one presenter, which we recorded as a follow-up, covered the same content.

In the initial study, we learned that podcasts serve equally well as videos for providing learning content (Contribution 1, Section 4.1). Therefore, our second study aimed to identify how learners of the MOOC platform openHPI would prefer to consume their podcasts. For this, we performed a One-Shot Case Study (Campbell and Stanley, 1966).

Of particular interest in this study were answers by participants provided in a follow-up survey to the Deep Dive in which we questioned on, e.g., the ideal podcast length or what kind of content they prefer to listen to. For this study, we offered all learners of our MOOC two learning items for the Deep Dive, of which they could choose the presentation form freely. On the one hand, we provided an 80-min podcast with three presenters available either using a web player or to be streamed via common platforms, such as Spotify™. On the other hand, we provided participants with a video of the same podcast. For that video, we placed three cameras in the recording room to video-record the speakers while they recorded the podcast. Hence, no additional visual aid, such as presentation slides, was provided in the video, and the audio track is identical to the podcast available for streaming. Participants could freely move back and forth between both forms of presentation.

Building on the knowledge of ideal podcasts for learners on the MOOC platform openHPI, derived from the two previous studies, we performed a third study, analyzing the impact of podcasts in a 6-week course. Based on insights gathered through the previous study (Contribution 2, Section 4.2), we aimed for a podcast length of ~40–45 min. Previous research has shown that this is far too long for single videos in a MOOC (Guo et al., 2014; Renz et al., 2015). Hence, we decided not to offer any video-based alternative to the podcasts. As we wanted to offer the content to all learners, we refrained from applying group randomization in this study. Instead, learners could self-select if they wanted to consume or skip the optional podcasts. We, therefore, employed a Static-Group Comparison (Campbell and Stanley, 1966) as the methodology for our study. We discuss the implications of self-selecting the exposure to the study, such as threats to validity, in our later discussion, i.e., in Section 5.4.

In each of the 6 weeks of our course (as indicated by the annotation [[W1..W6]]), learners self-selected whether they want to listen to the podcast (X). We discuss the bias of self-selection later in the manuscript. As in the previous studies, each podcast is used as a Deep Dive in the course and followed by an optional survey and quiz for knowledge (O_K). Further, each of the 6 weeks is concluded by a graded exam (O_GE). After all six course weeks, learner knowledge was further assessed by a final examination (O_FE). All three sources of knowledge assessment were used in the analysis of the impact provided by podcasts on secondary knowledge throughout the course. The dashed line (- - -) in the semi-formal representation of our study design indicates that without randomization, we cannot ensure that both groups are equal. Table 1 shows an overview of course structures on our platform, notably including the optional course elements of podcasts in this study.

We designed the podcasts for the third study by building on the knowledge derived from the previous two studies (see Section 4.2). Additionally, we investigated whether we can use podcasts to increase interaction in the course. Therefore, we dedicated some time (20%) of the podcast to picking up hot topics from the course forum and adjusted the content distribution accordingly. Generally, our podcasts are aimed at being 35–45 min long and were planned to consist of the following parts (Köhler et al., 2023):

Massive Open Online Courses on the learning platform openHPI are usually prepared and operated by a teaching team of experts on the course topic. A principal lecturer records presentation videos to ensure consistency throughout the videos. For our podcasts, we chose to have teaching team members as speakers. This allows more reflection on the course topics since other perspectives are included. The resulting reactionary content within the course can help foster learner reflection and interaction (Lewis, 2020).

The initial study's research question covered how audio-based education performs compared to video-based education. Specifically, we investigated the learning success and the learner perception of these two education methods at hand.³ We performed the study alongside an English-speaking cybersecurity MOOC in which 2,815 learners actively participated. Throughout the course, learners were randomized into one of the three groups for the treatments we compared. If learners progressed far enough in the course, every learner could experience all three Deep Dive learning items in the respective style. After each item, they could answer a survey and test their knowledge in an optional quiz. A total of 1,186 learners (42% of all active learners) completed the course. Table 2 presents an overview of participation statistics on study content for the respective course.

On average, the new learning items were rated at least Good on a school-grade scale (1: Very Good; 6: Insufficient). The average rating of the Traditional Podcast was worst with a grade of 2.18, followed by the Interview Video with 2.03, and the Interview Podcast rated best at 1.73. Regarding learning success, Figure 1 presents an overview of learner performance in the self-tests that followed the respective learning items, partitioned by the type of Deep Dive learning content they received. Between the three groups, we observe non-significant differences in learning outcomes interpreted as performance across the entire course (measured with a one-way ANOVA, p = 0.68). We, therefore, conclude that the three studied types of content transmission serve equally well at presenting knowledge to learners.

We further questioned participants' perception of whether the content was understandable. We used a 5-point Likert scale (Likert, 1932) on which participants could express their (dis-) agreement with the respective statement. Throughout all three Deep Dives, at least 89% of learners agreed that the content was well understandable for all content types. Generally, we observed that learners better understood the content presented by multiple presenters than that of our Traditional Podcasts with a single presenter.

In our surveys, we also questioned whether learners would prefer video-based or audio-based content and whether they would like to have it presented by single or multiple presenters. For those two variables altered in our study, we observe that regarding number of speakers, participants tend to opt for the variant they have experienced during the study. Regarding the availability of video content, all participants slightly preferred to have a video available. However, indecisiveness among those who previously received the audio-only podcast was higher than among the learners with the Interview Video. Throughout all Deep Dives, users wished to have more content in the form they experienced. However, the learners already showed tiredness when multiple podcasts were offered within a single course week (Köhler et al., 2022).

Contribution 1: In this initial study on the potential use of podcasts in combination with Massive Open Online Courses, we identified that the learning outcome using podcasts showed no significant differences to that of video-based learners. Learners showed great interest in singular podcasts but appeared to get tired once they were used multiple times within a single course week. Hence, the question of how podcasts should be integrated into MOOCs on our platform remains subject to future analysis.

Building on the knowledge obtained with the first study of podcasts on our online education platform, openHPI, we aimed to investigate further how learners would like podcasts to be presented, including the ideal length and topics to be covered. Alongside a 2-week German-speaking course on Cloud Computing, we provided an ~90-min podcast on secondary content, expanding on the content provided in the course videos.

The podcast could be consumed in the course using an embedded audio player, listened to via Spotify™, or watched as a video from the recording studio with the regular video player. Figure 2 provides screenshots of both conditions, which the participants could choose freely. Afterward, learners could test their knowledge in a content quiz and provide additional feedback using a survey. Seven hundred and sixty-one of our learners answered that they only listened to the audio (independently of the player), while 445 opted to watch the video recording. One hundred ninety learners reported to have made use of both methods.

After listening to or watching the podcast, learners could take a content quiz covering content only presented as part of the podcast and not available in the remainder of the online course. Figure 3 depicts learners' performance in the respective content quiz. We refrain from directly comparing the consumption methods in this study, as these are subject to a strong selection bias, and a One-Shot Case Study cannot sufficiently counteract such implications. Instead, we want to draw attention to the fact that throughout all three consumption methods, prolonged exposure to the content results in increased performance in the quiz. We assume that the outliers from this generalization for the groups who used both methods and listened 15–30, or 30–60 min are due to the limited participant group size available (N_{15 − 30} = 28, N_{30 − 60} = 15). We primarily interpret the fact that even in prolonged times of podcast consumption (e.g., >60 min), participants are still able to understand and internalize (new) content.

The main goal of this study, however, was the qualitative exploration of participants' expectations toward podcasts used alongside our Massive Open Online Courses. Two hundred and eighty-three participants provided answers to the questions in the optional follow-up survey. Participation fluctuated slightly between the different questions presented in the following. Regarding ideal podcast length, participants could provide any arbitrary number. The mean of participants' preferences was 34.97 min (N = 234, σ = 30.22).

Regarding wishes for technical implementation of the podcasts (N = 283, multiple answers allowed), 61% of respondents wished for shownotes to be made available alongside the podcast. In qualitative free-text answers, the central argument for this wish was that written shownotes would allow a more accessible recap of specific information from the episodes. This is followed by 56% wishing for chapter marks to be made available for easier podcast navigation. Forty-five percentage of the participants wished for a content overview of the podcast either in the first minute(s) or in the description. Much less important was a better integration into our platform (11%) or the availability of the podcast on other streaming services (4%).

The content presented in our podcast can generally be divided into Repetition of underlying content and knowledge, Secondary Content, which expands on the basic content provided in the online course, and Small Talk, such as personal reflections and stories related to the topic. Using a 5-point Likert scale (Likert, 1932), we questioned whether participants liked how much time in the podcast was spent on the respective types of content. 77.5% of respondents (N = 258) liked the proportion of Repetition we included. 84.5% of 258 participants liked the proportion of Secondary Content as presented in the podcast. Only 58.1% of 248 participants liked the amount of Small Talk present in the podcast. We used the same Likert scale questions to investigate which content types learners would want to be presented more. Figure 4 presents the results of the respective question block for the three content types. Judging from both question blocks on the current length and a potential increase of each content type, the tendency shows that participants appreciate more Secondary Content and wish less Small Talk. At the same time, content repetition can stay as it is but should not become much more. Based on the presented insights from our survey questions, learners' desired consumption method, and the observed learning outcomes, we derive the following requirements for ideal podcasts integrated into our MOOCs.

Contribution 2: Most learners self-selected to consume the audio-only presentation form of the learning content provided. An ideal podcast for our population would be ~35–45 min long and should combine content repetition (15%) with a vast amount of new, secondary content (~80%). Small Talk as a content element should only be used sparsely (remaining 5%), ideally resulting in valuable further perspectives on a certain topic. From a technical perspective, show notes, chapter marks, and an overview of the episode's content should be provided.

In the third study, we provided one podcast for each of the 6 weeks of a MOOC on cybersecurity. Learners who enrolled in the course could self-select whether they want to expose themselves to the treatments, i.e., listen to the podcasts. At any time, learners could decide to start listening to the weekly podcasts. Further, whenever they started to listen to a podcast, they could freely select to pause, continue, or stop listening to it.

Limitation 1: Due to technical limitations, we cannot precisely track a learner's exposure to the podcast in terms of the exact time they have listened to the podcast. Instead, we have to rely on their reflection in the surveys. We discuss this limitation in Section 5.3.

As highlighted in the methodology (Section 3.4), the study design allows us to measure users' learning success in three forms:

Finally, we can report on a few qualitative insights, e.g., if users did other activities while listening to the podcasts, by evaluating their survey responses. One optional survey was provided alongside each of the podcasts.

The previous sections indicated that optional podcasts help to foster in-depth learning and learner's course success. We follow the proposal by previous researcher to enumerate alternative explanations to the observed study outcome (Shadish et al., 2002). In the following, we highlight three potential alternative explanations for the observed effects, providing cues that support the genuinity of the effect derived by the integration of podcasts in MOOCs.

In the third podcast study, we provided additional secondary knowledge through podcasts and observed increased learning success for the graded exercises included in the course. However, the reason for that is not necessarily the medium of podcasts. One could hypothesize that presenting similar additional content as videos or live streams would be a suitable choice. However, increasing the weekly effort for an online course from three to approximately four hours with optional content burdens the learners' schedules. Audio-only education content, therefore, was the obvious choice for our study to provide the learner with as much flexibility for its consumption as possible.

From answers to our surveys, highlighted in Figure 10, we observe that most learners use the flexibility the audio-only educational material provides. When asking the learners which activities they were doing while listening to the podcast, only a third answered Nothing. This is unchanged throughout the course weeks.

As presented in Section 4.3.5, learners were actively interacting and discussing with each other and the teaching team on the content of the podcasts inside our discussion forum. Based on qualitative feedback in the forum and course-end surveys, we judge that this discussion has benefited from the atmosphere of the podcast. As there is no need for a camera to record the podcasts and recording the audio alone is easier, we were less bound to our available time in the (video-) recording studio. Hence, we could take plenty of time to start the podcast recording, ensuring that each teaching team member was in the mood for the recording. This generated a relaxed and friendly atmosphere. With this setting, we enabled our speakers to remain relaxed for the prolonged recording and helped to create a certain feeling of privacy. Reflections by learners in the course forum suggested that it felt like being able to join in on a personal discussion between the speakers. Similar perceptions could be created with fireside-chat-like videos and interviews, but not traditional presentation videos in a professional atmosphere.

In summary, podcasts are easier to create for the teaching team and allow a more relaxed setting than videos. Further, podcasts allow learners to allocate the time required to consume the additional learning material more flexibly per week. Hence, we conclude that while we could have provided videos with the same content, we do not think similarly positive results would have been obtained. A concrete comparison, however, remains subject to future research.

Our Third Study on podcasts in the context of online courses has been performed in a large-scale MOOC spanning 6 weeks. However, as previously discussed, we were limited in evaluating participants' exposure to our podcast. Only 40.2% of learners responded to our surveys throughout the online course, providing information on how much of the podcasts they listened to. As the actual number of users listening to the podcasts is likely higher than those who reported it in the surveys, we had to perform our analysis based on a different proxy variable. For the performance analysis in the final examination (see Section 4.3.4), we therefore considered learner exposure to the treatment based on their visit to the learning item. However, this also includes learners visiting the learning item but not actually listening to the podcast. Both variables we measured in our study are thereby inaccurate, which is an important consideration for future research. In the following, we, therefore, attempt to perform an educated guess for the actual statistics on learner exposure to the podcasts.

As the first indicator, we calculate the upper maximum of podcast plays by our learners. A single learner could theoretically play and listen to a total of six podcasts, one during each course week. The sum of active learners over the six course weeks (cf. Table 3) is, hence, the upper bound of our estimation. As indicated in Table 5, this sum is 10,266 potential plays by course learners. Our learning analytics data allows us to evaluate which learning items a user has visited. Our podcasts generated a total of 8,724 impressions for learners who visited the respective learning items but did not necessarily listen to the podcast. As the final value of the upper bound, we can derive actual access to the podcast through either the web player or any streaming platform. Those podcast plays, however, cannot be annotated to learners in our course. From the hosting platform, we derive the information that ~7,900 podcasts were played throughout the course. However, this includes multiple counts for learners who have opened and listened to (different parts of) the same podcasts multiple times.

As a lower bound for our educated guess, we observed 3,504 reports of learners listening to (different) podcast episodes. These reports were provided in the weekly surveys. Based on previous experience with surveys in our online courses, we know that only ~30–40% of learners answer (weekly) surveys. Hence, we derive that 6,000 actual plays would be an optimistic estimated guess for the actual number of podcasts listened to. This would still be roughly 20% less than the number of plays our hosting platform reported, allocated to learners listening multiple times to that episode. With learner exposure in that range, we would have achieved that around 60% of active learners each week actually listened to the podcast. Even with a more pessimistic estimation of around 4,500 plays, we would see a conversion rate of 45%, which is worth the effort of creating the podcasts. For our future work (cf. Section 6), an improvement in the tracking of podcast consumption is essential to enable a more precise investigation.

The following paragraphs discuss the internal and external validity of the Third Study on podcasts. While we provide reasons to consider our study internally valid, this singular experiment cannot achieve external validity, interpreted as generalizability (McDermott, 2011).

Internal validity refers to the question of whether, for the performed experiment or study, the observed outcome and the inferred causality can be considered fool-proof (Campbell, 1957; McDermott, 2011). Mainly, quasi-experimental or pre-experimental designs such as our study are prone to issues regarding their internal validity (Ross and Morrison, 2004). Campbell and Stanley list eight categories of variables potentially threatening internal validity: (1) History, (2) Maturation, (3) Testing, (4) Instrumentation, (5) Regression, (6) Selection, (7) Mortality, (8) Interaction of the Previous (Campbell and Stanley, 1966). For Static-Group Comparisons, Campbell and Stanley mainly highlight threats for internal validity with four of the previous categories: (2, 6, 7, 8). For our described study, in which we added podcasts to an online course while allowing learners to self-expose themselves to the podcasts in each course week, we consider the following significant points of discussion regarding the internal validity of the study:

External validity questions the generalizability of the findings of a study. Aronson et al. (1989) stated that no single research alone could achieve generalizability as it can only be inferred through systematic, repeated testing and verification of a hypothesis in different contexts. In our research, we compared the impact of podcasts in a study with a total of 2,167participants across all age groups who self-enrolled in the study. We thereby provide a good overview of the average population. Still, our courses educate on cybersecurity, a topic from the niche of IT education.

In our conception of the MOOC and the according study, we chose to allow all learners to consume the podcast. Therefore, we could not employ randomization and control groups for the treatment. Based on the previous discussions of validity, demographic factors, and alternative explanations, we argue that we would have observed similar results if we had experimented with randomization.

Campbell and Stanley (1966) challenge the experimental mortality, which refers to the fact that people receiving (or not receiving) a treatment could drop out precisely due to, e.g., the treatment itself and can hence not be evaluated anymore in comparison after the treatment. When comparing completion rates between the 2020 and the current version of the course in 2022, we observe that fewer enrolled learners completed the current course (77.0% in 2022 compared to 82.4% in 2020). The dropout could be induced by various reasons, from learners who previously took the course and realized that they remember almost everything to learners being tired of computer-based training after 2 years of COVID-19 safety practices. To ensure that experimental mortality alters our results as little as possible, we experimented with six stages of treatment (one podcast in each course week) and repeated evaluations throughout the course. During each week, our learners could seek exposure to the treatment voluntarily. Additionally, in each of the weeks, we evaluated the success of the treatment at two points: with a content quiz immediately after offering the podcast and a graded exam at the end of the week. Hence, we consider that we have counteracted the threat of experimental mortality as well as possible for our context.

To enhance the experience for learners, we further opted against a pretest at the beginning of our course, which would have turned our Static-Group Comparison into a quasi- or pseudo-experiment, such as the Nonequivalent Control Group Design (Campbell and Stanley, 1966). As we had already tasked the learners with weekly surveys on the podcasts, we wanted to keep further effort for participation in the study low. Additionally, learners could answer course-start and –end surveys, which are required for internal analysis of the courses. As we wanted to avoid burdening the learners with another quiz or task before the course began, we opted against the pretest.