The MMI-EC consists of four components with individual items and descriptive statistics for each listed in Tables 1–4:

The first two MMI-EC components together make up the Observation Checklist. The Observation Checklist includes nine 5 min time sampling observation periods which are rotated across the Engaged Materials (3 rotations) and Teacher Activity components for both the lead teacher (4 rotations) and an assistant teacher (2 rotations), if applicable, for a total of 45 min of observation. Within the Engaged Materials rotations, the observer notes which hands-on Montessori learning materials the children are using within each 5 min period. During the Teacher Activity rotations, observers record teacher behaviors, both supportive of and contrary to Montessori philosophy, as they interact with children and manage the classroom within each 5 min period. The results from each of the time sampling periods were aggregated to form a single measure for each item within each classroom.

The final two components of the MMI-EC make up the Room Inventory which is completed at a time when children are not involved in their free choice learning activities in the classroom, ideally when children were outside at recess or while children were eating lunch or having group instruction in circle time.

Observers participated in an 8 h training program developed by the researchers and completed on the Canvas Learning Management System. The training included videos describing Montessori education and the MMI-EC; specific modules on recognizing Montessori materials including interactive activities; details about using the digital data collection tool and how to score items; and the CITI Social & Behavioral Research human subjects protection tutorial. The training program concluded with a practice classroom observation visit with an experienced, trained Montessori practitioner we called a Montessori mentor. During the visit, the mentor and the trainee observer each completed the MMI-EC and then discussed the experience and compared differences in their observations so that the observer was prepared for their first classroom observation.

In developing the items for the MMI-EC, we considered five criteria for measuring fidelity of implementation as outlined by O'Donnell (2008): adherence, duration, quality of delivery, participant responsiveness, and program differentiation which can broadly be categorized under the headings of structure (adherence, duration) and process (quality of delivery, program differentiation) with participant responsiveness having elements of both. The four components of the MMI-EC can be aligned with O'Donnell's (2008) criteria. While duration information is captured in a separate teacher profile survey which will be analyzed in a separate article, the other structural aspect of fidelity, adherence, is reflected in both the Classroom Index and Teacher Activity observation which assess the degree to which the equipment and types of teacher interactions align with the Montessori method as described in foundational literature. The process aspects of fidelity, made up of quality of delivery and program differentiation, are also reflected in the Teacher Activity observation in ascertaining the manner of teacher implementation of Montessori practices and in the Classroom Index in documenting that key features distinguishing Montessori from other types of educational settings are present. Participant responsiveness is evident in the Engaged Materials observation component which records the specific Montessori materials children are working with during the observation.

To develop individual items and scoring criteria for assessing fidelity using the MMI-EC, we relied on multiple sources including previous work by the authors. First, we referenced the Logic Model for Montessori Education proposed by (Culclasure et al. 2019). Documentation supporting each individual item came from a number of specific sources including Montessori organizational standards, writings of Maria Montessori herself, and other peer reviewed publications as illustrated in Appendix A. Once we developed the initial list of items to be included in the MMI-EC using the sources listed in the Appendix A, we had draft iterations of the tool evaluated while under development by Montessori scholars, Montessori practitioners, and a three-member Technical Advisory and Cultural Relevance Committee from outside of the field of Montessori education. This diverse group of scholars provided input on the MMI-EC as well as the training materials to identify any issues and to recommend revisions.

Analyses were conducted using SPSS v. 29 and R v. 4.3.0 and included descriptive statistics, correlations, item-total correlation analysis, ANOVA comparison of means, and internal reliability consistency scale analysis using Cronbach's alpha (Cronbach, 1951). For the Observation Checklist components which involved multiple time sample rotations, data were combined across time periods. The total score for Teacher Activity was the sum of observations across time periods, and the total score for Engaged Materials was the maximum value observed across time periods for each item. We used the maximum value for Engaged Materials because children could have been engaged in the same work in multiple time rotations. Using the maximum value in any given rotation ensured that we did not confound continued work with materials with new students choosing the same work. The Teacher Activity scale items represented discrete actions, so the sum of these observations across time samples was used. The Material Snapshot items were a count of key Montessori materials available in the classroom, and Classroom Index items were dichotomous measures of present or not present.

When calculating the Cronbach's alpha for the Teacher Activity section, we reverse coded the negative items so that observed negative behaviors were recorded as 0s and absences of negative behaviors were recorded as 1s. Reverse coding allowed us to calculate the internal reliability of the negatively weighted items in the same scale as the positively scored ones. We also treated the negative items as a separate subscale from the positive items because we identified more problematic negative items and wished to examine them separately. After initial analysis, we refined the four scales of the MMI-EC and removed items with item-total correlations below 0.05.

In addition to internal reliability, we also conducted an interrater reliability analysis on a subset of 15 classrooms with paired observers. We calculated the exact percent agreement across each of the four scales. In evaluating the results, we considered percent agreement above the widely used cutoff of 75% to be acceptable for consensus (Graham et al., 2012). Alternative approaches for scoring such as dichotomizing scales were also utilized. In addition, Cohen's Kappa was calculated as a more rigorous measure of interrater reliability. Several interpretations of Cohen's Kappa were found in the literature. Although some suggest values greater than 0.50 are acceptable (Stemler and Tsai, 2008), we consider Landis and Koch's (1977) more detailed ranges for interpretation (< 0 Poor, 0.01–0.20 Slight, 0.21–0.40 Fair, 0.41–0.60 Moderate, 0.61–0.80 Substantial, 0.81–1.00 Almost perfect).

After examining individual items of subscales, we calculated a subscale total scores for each component by taking the sum after eliminating items that showed low internal consistency with other items. We calculated the intercorrelation between each subscale in order to determine how aligned each subscale was to an overarching measurement of fidelity. The sum of these subscale scores can represent a total fidelity score for each observation. We analyzed total fidelity score by whether the school was public or private using an ANOVA test. We also used each individual subscale score in separate ANOVA tests with public or private school as the independent variable.

The MMI-EC pilot study involved 11 observers who all had at least an undergraduate degree in education, psychology, social sciences, or a related field. They may have had some familiarity with Montessori education but not a Montessori teaching credential. To increase the credibility of the MMI-EC, we intentionally avoided using Montessori experts as observers and designed the instrument with low inference items to reduce subjectivity and avoid factors other than the items being measured being used to assess a classroom's Montessori authenticity. With support from an external funder, observers were provided a $100 stipend for the training and for each observation conducted. Observers participated in an 8 h online observer training course using the KU Canvas learning management system and received support from a Montessori Mentor who was an experienced and credentialed early childhood Montessori educator. The Canvas course included 11 modules, ranging from topics on understanding the MMI-EC tool to recognizing Montessori materials (Murray et al., 2024).

Additional data was collected for the purpose of correlating MMI-EC results to external comparison measures. First, teachers were invited to complete the TQMP (Murray et al., 2019), and we obtained 45 completed questionnaires from classrooms we observed. We also invited Montessori mentors to complete an informal rubric assessing key components for a small subset of observed classrooms based on their extensive experience as practitioners. We received eight completed rubrics which were scored on a four-point scale on the following items with each scale point offering a description of what reflects a particular score:

The MMI-EC pilot study involved 81 early childhood classroom observations with 21 public school classrooms participating Further, a subset of observations were conducted by two observers visiting the same 15 classrooms for inter-rater reliability (Hallgren, 2012), leaving 66 unique classroom observations. Criteria for classroom participation included having at least 18 children enrolled across at least a 2-year age range. Classroom teachers were not required to be Montessori trained. The mean number of children in attendance during observations was 19.49 with a standard deviation of 3.245.

This project was approved by the KU Institutional Review Board, and all participant observers and mentors submitted signed consent forms. With support from the Brady Education Foundation, observer/participants and Montessori Mentors were provided a $100 stipend for the training and for each observation conducted.

As shown in Table 5, we found the strongest internal reliability consistency with the Material Snapshot section which is at an acceptable level using the common criteria of a minimum of 0.70 (Tavakol and Dennick, 2011). The Material Snapshot scale is also the most extensive with 72 items initially. Since many items did not contribute to the internal consistency, with item total correlations below 0.05, we removed 18 items to achieve produced a reasonable degree of internal consistency. The Classroom Index scale began with 26 items and after removing 3 items with item total correlation below 0.05 gave us an internal consistency just over the 0.70 threshold. The positive Teacher Activity scale items began with 11 items and did not require removal of any of them for having an item total correlation of below 0.05. The internal reliability for the positive Teacher Activity scale items was acceptable at the 0.70 cutoff. The Engaged Materials scale (even if we dichotomize it) and the negative items on the Teacher Activity scale were less consistent and require additional scrutiny (Taber, 2018). Tables 6–9 show item-total correlations of individual items in the reduced scales for each of the four sections of the MMI-EC with the Teacher Activity scale separated between positive and negative behaviors.

We also examined the correlation among the MMI-EC subscales to better understand the similarities and differences in what each is measuring. The results in Table 10 demonstrate that the overlap for the subscales is rather limited with the highest correlations being between negative and positive Teacher Activity, between Engaged Materials and Material Snapshot, and between Engaged Materials and the Classroom Index. These correlations make intuitive sense because one would expect that positive Teacher Activity that is conducive to Montessori education would be related to an absence of negative Teacher Activity which would be detrimental to Montessori education. Similarly, it is logical to expect that classrooms with a larger number of Montessori materials available in the classroom would be related to classrooms in which more children are actively engaged in working with Montessori materials. And, finally, classrooms that are well equipped with Montessori materials would also be expected to have the resources to provide other important classroom features to support an effective Montessori environment. Correlations among other subscales are negligible. Table 10 also shows the correlation of each of the subscales with the total MMI-EC scores, showing the strongest relationships with a strong positive correlation between total MMI-EC and positive Teacher Activity. Negative Teacher Activity and Material Snapshot are both moderately correlated with the MMI-EC total score. Engaged Materials and Classroom Index are significantly but weakly correlated with the overall total MMI-EC scale score.

We also conducted an analysis of the 15 classrooms with two observers gathering data simultaneously to gauge interrater reliability. We used percentage rater agreement based on how frequently the two raters assigned the exact same rating. Results for the percent agreement calculations for each scale are shown in Table 11. We also reevaluated the percent agreement with dichotomized items for scales that utilized time sample rotations (Teacher Activity and Material Snapshot). In other words, we examined only if the action was observed in any of the time samples rather than using the sum or maximum of each time sample. The Material Snapshot was also dichotomized which resulted in changing the items from counting the number of specific materials available in the classroom to a checklist representing whether or not each material was present in the classroom. For example, instead of the value for dressing frames being the number of individual frames available to being an indicator of whether or not any dressing frames were available in the classroom. The Classroom Index scale was already a dichotomous score, so we did not adjust the scale. With these updated dichotomous percent agreement scores, every subscale satisfied the 75% cutoff with the exception of the negative Teacher Activity items. In fact, with the exception of the negative Teacher Activity scale which only had 36.36% of individual items achieving the cutoff of 75% agreement, more than 70% of all items within each of the other subscales met the 75% exact agreement cutoff, ranging from 72.73% for the positive Teacher Activity items to 90.74% for the Engaged Materials items. The highest and lowest percent agreement items are listed in Table 12.

Cohen's Kappa values were also calculated in order to further examine interrater reliability. Across the 15 classrooms, Teacher Activity (positive and negative items combined) (K = 0.482, SE = 0.176) and Engaged Materials (K = 0.573, SE = 0.128) fall into the Moderate range for interrater reliability while the Material Snapshot (K = 0.713, SE = 0.093) shows good interrater reliability. The Classroom Index (K = 0.370, SE = 0.182) had the lowest interrater reliability according to Cohen's Kappa and the average fell into the Fair range. When investigating which classrooms had lower interrater reliability, we found that there were several instances of straightlining where observers rated a scale with all 1s and no 0s. Since this reduces Cohen's Kappa values to 0 due to the likelihood of random chance causing agreement, Kappa statistics were also calculated excluding these straightlined observations. With this adjustment, the Classroom Index (K = 0.427, SE = 0.210) average Cohen's Kappa was in the Moderate range. Several observers also straightlined the Teacher Activity positive items, so Cohen's Kappa was also examined with those observations excluded. The average positive Teacher Activity Kappa (K = 0.337, SE = 0.270) after excluding straightlined observations was still in the Fair range.

We compared the total MMI-EC score and subscale scores between public and private schools and found significant differences only for Classroom Index and Material Snapshot when using a Bonferroni correction to control for inflation of Type I error rates due to multiple comparisons (Abdi, 2007). Private school classrooms on average had significantly higher scores on the Classroom Index and Material Snapshot (see Table 13). None of the other components of the MMI-EC demonstrated differences between public and private schools. One could explain this result by considering the limited resources and competing requirements of public schools as compared to private Montessori schools with ostensibly greater latitude to arrange classrooms in ways that reflect effective Montessori practices and are likely to have the budgetary resources and supportive administration to allow for purchasing a larger number of Montessori materials.

We analyzed correlations between subscales of the MMI-EC and the TQMP scale for the 45 observed classrooms where teachers completed the questionnaire. Overall, the Pearson correlation between the total MMI-EC scale and two of the three TQMP constructs, Structure and Curriculum, are significant but weak. The relationship of the total MMI-EC scale with the third construct, Freedom, is not significant. These results suggest that while there are some similarities in the aspects of Montessori quality measured by each of these instruments, they do not represent the same constructs. Table 14 illustrates the Pearson correlation for the individual MMI-EC subscales and the TQMP constructs showing that a moderate relationship exists between the Classroom Index and the TQMP Structure and Curriculum constructs. Similar results are evident for Material Snapshot having a moderate degree of correlation with the TQMP constructs of Structure and Curriculum. A small but significant relationship also exists between Engaged Materials and the TQMP Structure construct. Relationships among the other TQMP constructs and MMI-EC subscales are negligible and non-significant.

Finally, we analyzed the small subgroup of eight observed classrooms where a Montessori Mentor also completed a quality rubric. The extremely small number of completed rubrics as well as the relatively narrow range of scores on the rubric scales limits the value of this analysis. However, even at a high level the results showed that the two classrooms that received the highest scores from mentors (19 out of a total of 20 possible points) also scored above the median on the MMI-EC, and the two classrooms that received the lowest scores from mentors (12 and 15 points out of a total of 20 possible) also scored below the median on the MMI-EC. Clearly, these results are preliminary but do point to some support for the MMI-EC's ability to measure Montessori quality.