In both PIRLS Literacy 2016 and PIRLS 2021, South Africa employed the IEA’s two-stage stratified cluster sampling design to yield nationally representative estimates of Grade 4 students’ reading literacy scores (LaRoche et al., 2017; Almaskut et al., 2023). Regarding the sampling design, schools were first selected with probability proportional to size, stratified by language and province. In the second stage, intact Grade 4 classes were selected via the WinW3S sampling software (Department of Basic Education [DBE], 2023; Almaskut et al., 2023). Using this design ensures that students who have 4 years of formal education drive comparability across languages and provinces.

In 2016, the target population was all Grade 4 students; the sampling frame was stratified by the language of instruction (LoLT) and province so that the sample would represent both language groups and provinces (Howie et al., 2017). The language of the test was selected based on the LoLT used in Grades 1 to 3. In 2021, the study followed a similar approach to 2016 based on the IEA guidelines. In both cycles, the final data were weighted to the national Grade 4 population (LaRoche et al., 2017; von Davier et al., 2023). A total of 12,810 and 12,426 students participated in 2016 and 2021, respectively.

Both cycles in South Africa included two sets of instruments, that is, the reading achievement booklets and the contextual questionnaires. The PIRLS achievement booklets assess Grade 4 students’ reading comprehension with two purposes of reading: (a) reading for literary experience, and (b) reading to acquire and use information (Mullis and Martin, 2019). Each passage was accompanied by a set of multiple-choice and extended response questions. Each question is linked to one of the four processes of comprehension that measure students’ reading levels from emergent to advanced:

The two cycles slightly differ in terms of the organisation of instruments. PIRLS Literacy 2016 was developed based on the need for an easier PIRLS assessment (Mullis and Prendergast, 2017), where less difficult passages were selected for testing. In terms of PIRLS 2021, group adaptive design was selected, where the passages and their accompanying items were divided into three groups: easy, medium and difficult. Countries with mean achievement scores of less than 450 score points were grouped into the ‘easy’ category, where 70% of the passages were considered easy and 30% of the passages were considered difficult (Wry and Mullis, 2023). South Africa was grouped into the ‘easy’ category (Department of Basic Education [DBE], 2023).

The translation of the PIRLS instruments into the 10 other official South African languages proved a challenging task. During PIRLS Literacy 2016, the materials were first adapted from US-English to UK-English and adjusted for local contexts, then translated into the 10 other official languages. This translation process was completed within 6 weeks as the international instruments arrived late (Howie et al., 2017). As part of the IEA standards, an independent verification also took place by a second certified company. To ensure that the translations were appropriate in terms of language used and concepts defined at the Grade 4 level, a panel of Grade 4 language teachers reviewed each version of the test booklets (Howie et al., 2017). The final sets of achievement booklets were sent to the IEA for translation verification, where any inconsistencies were directed back to the National Research Coordinator (NRC) for clarification and correction.

For PIRLS 2021, the translation procedures occurred during the COVID-19 pandemic and its accompanying restrictions (Mullis et al., 2023). The translation procedures evolved from the previous cycle to a five-step procedure where forward and back translation occurred, draft versions were revised by language specialists, DBE consultation and reconciliation against the source language, English, to resolve any discrepancies (Department of Basic Education [DBE], 2023). All changes were documented across both cycles in the National Adaptation Forms (NAFs). The protracted translation process during the COVID-19 pandemic ensured rigour and DBE sign-off on all 11 languages. Both cycles followed the standard operating procedures as set out by the IEA regarding the instrument translation and adaptation. As part of these procedures, South Africa, along with other participating countries, submitted their instruments for translation verification (von Davier et al., 2023).

Before the main data collection could occur, both PIRLS Literacy 2016 and PIRLS 2021 cycles required extensive fieldwork preparation.

Regarding the translation of the PIRLS instruments, both achievement booklets and contextual questionnaires into the remaining 10 official languages are complex. The complexity is attributed to differences in linguistic structures, the necessity of preserving text equivalence, and item difficulties for comparability (see Roux, 2020). During PIRLS Literacy 2016, an external certified translation company was contracted by the CEA to conduct the translations and adaptations from US-English to UK-English. The translation company was responsible for the forward-and back-translations within a six-week deadline in order to meet the international timeframes (Howie et al., 2017). Grade 4 teachers were recruited to review the passages and make suggestions for readability and appropriateness. The final translated instruments were sent to the IEA for international language verification. Comments and suggestions from the IEA were implemented, ensuring strict and transparent translation protocols were observed.

During the 2021 cycle, the translation process took on a more consultative approach where the translators, the CEA, and the DBE worked together to ensure all new passages and items were translated fairly. Due to the pandemic restrictions, the translation process took place online via online consultations and webinars. The online translation sessions included training for translators according to IEA protocol and standards, as well as instructions and guidance to complete the NAFs. The translated instruments, together with the NAFs, were provided to the IEA. Similar to the 2016 cycle, the IEA conducted an international language verification where comments were provided. A five-step translation protocol was adopted: (1) forward translation from the source language (i.e., English) to the target languages (i.e., 10 other official languages), (2) back-translation from the target language back to the source language, (3) professional revision of forward and back-translations, (4) DBE consultations and language experts, and (5) reconciliation against the source text to address any inconsistencies (see Department of Basic Education [DBE], 2023, p. 18). The meticulous translation process entailed that every language version, that is, translated text, underwent extensive quality checks before printing.

The PIRLS Literacy 2016 test administration in South Africa encountered three substantial logistical hurdles: a compressed international timeframe, local socio-political events, and a national assessment. The late release of the international assessments left only 6 weeks for the contextualisation, translation, back-translation, verification, formatting and layout of the achievement booklets and contextual questionnaires in the 10 official languages, a process that normally requires several months. The shorter timeframe was described as “exceptionally challenging” (Howie et al., 2017, p. 39). During October–November 2015, and again in January 2016 the University of Pretoria, including the CEA, was closed due to the ‘Fees-Must-Fall’ student protest action (Luescher et al., 2022), which resulted in a loss of approximately 3 weeks of preparatory work, necessitating the rescheduling of printing, packing and distributing (Howie et al., 2017) of the achievement booklets, contextual questionnaires and other testing materials. Moreover, the forced closure also necessitated rescheduling of testing. The Annual National Assessments (ANAs) were unexpectedly implemented in December 2015, which further prevented the South African team from completing the data collection. This delay resulted in the data collection only being finalised in 2016, however, only after the second wave of ‘Fees-Must-Fall’ protest action took place (Howie et al., 2017). The South African NRC at the time stated, “… this was an exceptionally difficult round of PIRLS to implement given the socio-political conditions, the team was very resilient and managed despite all the problems to complete the study” (Howie et al., 2017, p. XV).

In contrast to the PIRLS Literacy 2016 cycle, where only 6 weeks were provided for translations and finalisation of instruments, the PIRLS 2021 cycle benefited from a pre-pandemic decision to test during August–December 2021. This decision granted the South African team additional time for rigorous translation processes and verification (Department of Basic Education [DBE], 2023). Nevertheless, the COVID-19 pandemic introduced novel logistical challenges, in particular rotational attendance policies, safety measures and school closures (Department of Basic Education [DBE], 2023; Mullis et al., 2023). Rotational attendance policies in schools meant that only half of the students were present on the initial visit to the schools, and this may have contributed to the decline in the average scores between the two PIRLS cycles (Böhmer and Wills, 2023). These policies required the test administrators to schedule follow-up visits to the schools to complete the assessment. In some instances, where the school had available infrastructure to test the group on the same day, but adhering to social distancing, additional fieldworkers were deployed to ensure the test administration was conducted according to set protocols. During the pandemic, the provincial coordinators from the DBE played a crucial role in mediating these rapid changes in the schedule, liaising between the fieldwork service provider (Nexia SAB&T) and school management to ensure high participation rates (Department of Basic Education [DBE], 2023). By using adaptive scheduling practices, such as identifying return visits and maintaining a flexible fieldwork roster, the CEA was able to adhere to international timeframes despite the uncertainty of the pandemic and the disruptions faced by many schools across the country.

When any large-scale assessment is undertaken, the importance of test administrators should be underscored. In South Africa, with its diverse cultural and language groups, the recruitment of test administrators posed another challenge for both PIRLS Literacy 2016 and PIRLS 2021 (Department of Basic Education [DBE], 2023; Howie et al., 2017). For 2016, the pool of data collection companies with experience in educational settings, with fieldworkers who are extensively familiar with the South African school system, was limited. In addition, due to the number of languages assessed, the fieldworkers were also required to be fluent in English and the language of the test. Other than the IEA requirements for test administrators, the CEA also struggled to secure an external data collection company that met all the requirements of the University of Pretoria (Howie et al., 2017). Although the PIRLS Literacy 2016 training procedures are not public, the high level of inter-rater reliability in scoring suggests that fieldworkers and scorers received intense training from the CEA (see Howie et al., 2017, pp. 40–41).

Due to the COVID-19 restrictions, the PIRLS 2021 recruitment and training of test administrators were completed differently than in the previous cycle. For PIRLS 2021, travel restrictions and other safety measures, such as social distancing, mandated a virtual training model. Between 18 and 21 August 2021, DBE officials and fieldworkers attended webinars covering the PIRLS framework, test administrator role, test administrator manual, preparation before testing, different test materials, timing of test sessions and post-test procedures (Department of Basic Education [DBE], 2023). The webinars included practice scenarios and group exercises to ensure that the test administrators are fully trained for the main data collection across the country. These webinars were recorded and made available for review by the data collection service provider (Nexia SAB&T) and DBE officials (Department of Basic Education [DBE], 2023). Using a range of recorded webinars for training, which placed emphasis on practice, ensured a comprehensive coverage of IEA protocols and standards despite the lack of in-person training.