Worldwide, numerous community and clinical studies have shown that the quality of early parent–child interaction is a significant predictor of later child development, including attachment security (Bakermans-Kranenburg et al., 2004; McElwain and Booth-LaForce, 2006; Mills-Koonce et al., 2007; Wright et al., 2018), socio-emotional development (Murray et al., 2007; Wright et al., 2018; Cooke et al., 2022) and cognitive development (NICHD Early Child Care Research Network and Duncan, 2003; Valcan et al., 2018). This body of research underpins government initiatives such as the UK Best Start for Life Programme: A Vision for the first 1,001 Critical Days (Department of Health and Social Care, 2021) and the US Early Head Start Programme (2013) which emphasize the importance of supporting early infant-caregiver relationships at the earliest opportunity, so as to minimize adverse developmental outcomes for children. A major challenge then, for clinicians and other perinatal and early years professionals, is how best to reliably identify caregiver-infant dyads that most require intervention. Observational tools to assess parenting are widely agreed to be a gold standard approach providing an objective assessment and generating different information to self-reports from parents which are subject to informant biases (Corcoran and Fischer, 2013). However, in research contexts observation periods are typically longer than is feasible in clinical practice, and coding requirements are often more intensive than practicable. We report findings from a study designed to test the reliability and predictive validity of two observational parent-infant interaction assessment tools, over incremental brief observational periods, to establish their potential for use routinely by clinicians in the first year of life.

Compromised parenting at this early stage in development is seen particularly, though not exclusively, in the context of parental mental health problems, and is often indexed by lower maternal sensitivity to infant signals, higher withdrawn or intrusive behaviors, higher negative regard and/or lower levels of positive regard (warmth) expressed towards the infant (e.g., Nicol-Harper et al., 2007; Feldman et al., 2009; Murray et al., 2010; Azak and Raeder, 2013). For instance, studies of postnatal depression have shown lowered behavioral responsivity and sensitivity to infant signals (Stanley et al., 2004) and fewer signs of overt affection (Herrera et al., 2004). These disturbances are most evident when depression is chronic and/or severe (Campbell et al., 1995; Netsi et al., 2018) and are strongest in the context of socio-economic deprivation (Taraban and Shaw, 2018). Importantly, ameliorating depression alone has not been shown to sufficiently improve mother-infant interactions (Poobalan et al., 2007). Longitudinal prospective studies provide evidence that early parenting is an important mediator or moderator of adverse effects of perinatal mental health problems on child cognitive and social–emotional development (e.g., NICHD Early Child Care Research Network, 1999; Campbell et al., 2004; Milgrom et al., 2004; Tomlinson et al., 2005; Murray et al., 2010; Goodman et al., 2020), hence there is an additional focus for clinicians working with parents in the perinatal period on assessment of early parenting and use of evidence based interventions to ameliorate difficulties.

The economic argument for identifying troubled dyads and intervening to improve early parent-infant relationships is strong. For instance, in the context of perinatal mental health problems perinatal depression, anxiety and psychosis carry a total long-term cost to society of about £8.1 billion for each one-year cohort of births in the UK (Bauer et al., 2014) but importantly nearly three-quarters (72%) of these costs are related to adverse lifetime impacts on the child rather than the mother. Similarly, in the US a recent public health report estimated the costs of not treating perinatal mental health conditions was $14.2 billion in 2017, with one third of the costs being attributed to adverse child outcomes in the first five years of life (Luca et al., 2020).

Internationally, clinicians require assessment tools to identify those in need of intervention with proven predictive validity, a robust evidence-base and that can feasibly be implemented in busy clinical practice. However, many observational tools either have unknown predictive validity, have been evaluated in research studies using lengthy observation periods (typically 10–30 min; Mesman and Emmen, 2013), or are deemed too time-intensive to code in routine clinical practice (Royal College of Psychiatry, 2018). There is also a wide range of observational scales to choose from. For instance, Mesman and Emmen’s (2013) systematic review identified 50 observational measures designed to assess parental sensitivity. They highlighted eight measures with the most robust research pedigrees. These measures included a coding scheme developed for the National Institute of Child Health and Human Development study of Early Child Care and Youth Development (NICHD-SECCYD) (Owen, 1992) and Crittenden’s CARE Index (Crittenden, 2001). In this study we focused on evaluating the psychometric properties of two candidate scales, selected for different reasons. The Parent Infant Interaction Observation Scale (PIIOS) (Svanberg et al., 2013) was developed in the UK and cross-sectionally validated against the Maternal Sensitivity subscale of the Crittenden Care Index as a screening tool for universal use. Training on PIIOS had already been rolled out in UK specialist perinatal mental health teams on the basis of this. For this scale, three to 4 min of observation are recommended. However, its predictive validity to later child outcomes was unknown and required testing. By contrast the NICHD-SECCYD system, developed in the US, has repeatedly been shown to have predictive validity to later socio-emotional outcomes including attachment security (e.g., McElwain and Booth-LaForce, 2006; Mills-Koonce et al., 2007; Leerkes et al., 2009; Birmingham et al., 2017), but in research contexts observation periods are typically longer (>10 min) which may limit its routine use in clinical practice.

The full NICHD-SECCYD system (NICHD) and PIIOS systems are similar in that they each involve coding multiple dimensions of parent-infant interaction and both aim to capture parental sensitive-responsiveness. The NICHD system includes ratings of 14 dimensions of parent–child interaction, though research studies have typically used the scales in a hypothesis driven manner to examine specificity of prediction from specific parenting dimensions to later outcomes of interest. Some have used a single coding dimension (e.g., global sensitivity or sensitivity to distress) and others have used subsets of codes. Commonly a 3-item composite has been used to yield a broader ‘sensitivity’ composite, using sensitivity to non-distress or global sensitivity (which includes non-distress and distress episodes during play), positive regard (warmth) and intrusiveness (reverse coded) examining prediction to later child mental health or attachment related outcomes (NICHD Early Child Care Research Network, 1997; Bakermans-Kranenburg et al., 2004; Campbell et al., 2004; Mills-Koonce et al., 2007; Birmingham et al., 2017). In contrast, for PIIOS there is a limited research literature with only one study describing the development of the measure (Svanberg et al., 2013). The manual (Svanberg, 2009) recommends calculation of a total score across 13 parent-infant interaction dimensions to index overall sensitive-responsiveness and then conversion of this score into three categorical domains (no concern, some concern, significant concern), designed to drive clinical decision making and intervention level [e.g., Universal, Universal Plus and Universal Partnership as part of the Healthy Visitor Implementation Plan within the Healthy Child Program (Department of Health, 2009, 2011)]. To our knowledge no research has yet been conducted to examine whether or not the whole scale or a subset of codes within PIIOS has utility in the prediction of later child health outcomes. Finally, since the full NICHD and the full PIIOS system each require coding 13 or 14 dimensions of interaction we were keen to test whether brief forms of each tool (formed using only a subset of codes) could be derived with sufficient reliability and predictive validity, in view of the obvious time and cost-saving implications for implementation in clinical practice.

In summary, we aimed to assess the inter-rater reliability and predictive validity of these two widely used parent-infant interaction observational assessments; The NICHD system and the PIIOS. We rated parent-infant interactions filmed when infants were between 6 and 8 months old as part of an ongoing longitudinal study of child health and development. We felt this film archive was a suitable resource since it represents a common time for assessment in clinical services, once mental health or parent-infant difficulties have come to the attention of professionals. Child outcomes were attachment security and disorganization assessed at mean age 14 months and child externalizing and internalizing problems reported by parents at mean age 14 months and 30 months. In order to determine which measure or combination of codes within each scale had optimum predictive validity and brevity, and thus clinical utility, we examined whole scale performance and item-subset performance, determined a priori from the research literature or from factor analytic or machine learning analyses conducted within this study. Since parents with moderate to severe mental health problems may not tolerate lengthy observation, we further tested whether lengthier observation periods incrementally improved inter-rater reliability and predictive validity to child outcomes or not, examining ratings from 3 min, 5 min and 7 min of observation.

In this study we aimed to evaluate two observational measures (NICHD and PIIOS), used to assess the quality of mother-infant interaction to inform their use in clinical services. We evaluated them in terms of their reliability, and predictive validity to age 1 attachment status and age 1 and age 2 child mental health outcomes.

Regarding raters’ reliability, two naïve raters were trained on both measures and each was able to achieve an excellent level of reliability against a gold-standard rater on both measures when rating 3, 5, and 7 min of observation. There was evidence to suggest that agreement between ratings for 5 and 7 min was more stable than the one gained between 3 and 7 min, suggesting that observation for 5 min was optimal in terms of reliability. We therefore recommend filming for 5 min in clinical practice, a finding that was also supported by the results of validity testing discussed next.

In terms of predictive validity, we aimed to determine whether brief forms of each observation tool might possess sufficient predictive validity for use in routine outcome measurement. Our findings established that the PIIOS needs to be used in its full form (i.e., 13 items) and that the PIIOS_total was optimal with predictive validity to both internalizing and externalizing outcomes at age 2, superior to PIIOS_domain. Instead, the optimal form of the NICHD coding system was the brief 3-item composite measure (NICHD-3); this composite score includes global sensitivity, positive regard (warmth) and intrusiveness (reverse scored), and it was used previously in the literature as an index of maternal sensitivity (e.g., Mills-Koonce et al., 2007). In fact, NICHD-3 predicted externalizing problems at age 1 and age 2, whereas NICHD_total (across 8 codes) predicted age 1 outcome only in ROC analyses. Reassuringly, our factor analytic, lasso regression and machine learning approaches to analysis of each tool confirmed there were no other combinations of codes, including interactions between codes, in either measure that were any better than these a priori selected approaches to combining codes, in terms of predictive validity to the outcomes under test.

ROC analyses confirmed that both the PIIOS_total and the NICHD-3 from 5 min of observation could be used to identify at-risk dyads with fair to good predictive validity to age 2 mental health outcomes. Scores above threshold on the PIIOS_total correctly classified 75% of cases and the scores above threshold on NICHD-3 correctly classified 67% of cases, in relation to later age 2 externalizing problems. On the one hand, this is impressive since in our arguably conservative approach to analysis we were using one construct (parenting) to longitudinally predict a different outcome (child mental health), rather than the conventional use of ROC to evaluate the performance of a screening tool against a diagnostic measure of the same construct often cross-sectionally. In the former case, one might expect the results for AUC to be attenuated compared to the latter. On the other hand, using other indicators of risk in addition to parent-infant interaction quality, such as presence of parental mental health problems, may further improve the accurate identification of dyads who warrant early intervention. In our study, we also showed that the validity attained from 7 min of observation was very similar to that attained from 5 min, whereas for both measures prediction was subthreshold (<=0.70) from 3 min of observation. This finding also supports the use of a five-minute observation period in clinical practice. Although there is no previous published work on the predictive validity of PIIOS, previous work using NICHD in research settings supports the findings that the three-item composite predicts later adverse child outcomes, but longer observation periods have typically been used. Our findings are in line with previous work summarized in a meta-analytic review confirming direct associations between observed maternal sensitivity in the first year of life and childhood emotional and behavior problems (Cooke et al., 2022), and they extend these findings by confirming predictive validity can be achieved with only 5 min of observation.

Neither measure predicted attachment security or disorganization at the high-bar level required for a screening tool in our study, with AUC > 0.70. However, this is in line with published meta-analyses where a combined effect size of around 0.24 from maternal sensitivity to attachment security assessed using the Strange Situation Paradigm has been found in community samples, suggesting that the magnitude of prospective association is typically small to moderate (e.g., De Wolff and Van Ijzendoorn, 1997). Our findings from logistic regression are also concordant with this. We found several parenting dimensions in each measure that were significantly associated with later attachment security. Each unit increase in the NICHD-3 composite score (range 0–15; high scores represent optimal parenting) was associated with a 27% increase in likelihood of secure attachment at age 14 months. Whereas each unit decrease in the PIIOS_total score (range 0–40; low scores represent optimal parenting) was associated with a 3% increased likelihood of secure attachment. In addition, each unit increase in the NICHD-3 composite score was associated with a 16% decrease in likelihood of disorganized attachment at age 14 months. However, none of the parenting dimensions assessed by PIIOS were significant in the prediction of attachment disorganization.

The PIIOS was developed to identify at-risk dyads for insecure attachment and it was originally validated against the maternal sensitivity scale of Infant CARE Index (Crittenden, 2001). Yet our findings from logistic regression provide only weak support for the prediction to attachment security. Certainly, since no other studies have reported on prospective prediction to attachment outcomes from PIIOS, our findings require replication. However, the fact that infants who were identified as at-risk for later externalizing problems using the NICHD-3 index at 6–8 months of age were also found to be nearly three times more likely to be insecure at 14 months (OR 2.7) suggests NICHD-3 may be identifying vulnerability to attachment security, as assessed by the Strange Situation, more effectively than the PIIOS since infants at-risk for externalizing problems on PIIOS were only found to be at slightly increased risk (OR 1.5). Future studies might usefully determine whether PIIOS is more effective at identifying dyads who go on to be deemed at-risk using Crittenden’s Toddler CARE Index.

One of the main strengths of this investigation is the use of rare, longitudinal data from a relatively large high psychosocial risk community sample, with filmed observational archive data and longitudinal follow-up to gold standard methods to assess children’s outcomes such as the Strange Situation. We ensured that independent raters of parenting were blind to attachment status and child mental health outcome and we tested the psychometric properties of two observational schemes, selected for different reasons, alongside one another. We followed contemporary best practice guidance to evaluate the psychometric properties of each tool including examination of confounder effects on the predictive validity of each scale in which we were able to show that key demographic factors did not significantly reduce the measures’ discriminative validity. Finally, we also used a novel approach to testing incremental validity over three different short observation periods in order to make recommendations for routine clinical practice.

In terms of limitations, whilst we established predictive validity for both tools to BITSEA externalizing outcomes at age 2, the cut point selected was arbitrary, set at the top tenth centile of scores, albeit within a high psychosocial-risk community sample. Future work will need to assess prediction to diagnostic outcomes and assess performance of the scales in clinical samples. In relation to internalizing problems at age 2, only PIIOS showed acceptable predictive validity. The internal consistency of the BITSEA internalizing subscale was low in our study and markedly lower that that reported in a slightly older mixed referred/non-referred sample previously (mean age 36.9 months; Briggs-Gowan et al., 2013). The items within this BITSEA subscale assess a broad range of internalizing problems (e.g., distress on separation, sadness, social withdrawal) which may emerge at different stages in the early years, may not necessarily co-occur consistently and may be harder to recognize and endorse by parents. Though, the use of a top decile cut-point as the outcome in the study is still likely to have identified a group of children with high levels of internalizing problems of a varying nature. Also, whilst the WCHADS sample included a mix of families living in mostly deprived neighborhoods but also some affluent areas, the sample has very little ethnic variation which is characteristic of the local population from which it was drawn. PIIOS was originally validated against the Care Index cross-sectionally in a socio-economically deprived sample from the Northeast of England. In contrast, the NICHD-SECCYD system was developed in US and has been used in many studies with more ethnically diverse populations. Replication of the current work is therefore required with more ethnically diverse populations. The WCHADS study filmed interactions between the main caregiver and the infant and for 100% of families this was the mother figure at 6–8 months of age, as a consequence our findings cannot be assumed to extend more broadly to other parental figures or alternative caregivers. Whilst we showed that the discriminative validity of each tool was only marginally affected by maternal demographic covariates in this study, future larger scale studies might consider accounting for other infant-related covariates which may contribute to mental health outcomes such as gestational age at birth (Xia et al., 2021).

Finally, assessing the contribution of parental sensitivity specifically to distress within interactions, which is part of the NICHD full version, was limited since during the standard NICHD-SECCYD assessment 66.3% did not show distress in the play-based task during 3 min observation, although this reduced to 57.5% during 5 min and 48.4% during 7 min of play. In anticipation of this we selected the Global sensitivity code which encompasses distress and non-distress episodes during play for inclusion in NICHD-3 following some previous studies (Mills-Koonce et al., 2007). We used a play-based observation since both the PIIOS and NICHD-SECCYD systems adopt this approach. We also felt that a play-based assessment would be more normative for use in clinical practice and would be most acceptable to women, in contrast to using distress-eliciting tasks sometimes reported in the literature. However, since previous work has shown that sensitivity to distress may be important in the prediction of attachment security (McElwain and Booth-LaForce, 2006) future studies might usefully contrast the predictive valdity of the measures when applied to play interactions with that achieved when filming mildly stressful caregiving tasks which might elicit more naturally occuring distress in young infants and enable coding this element of interactions in a higher proportion of dyads.

This study evaluated two observational systems and confirmed that the PIIOS and a short form of the NICHD-SECCYD system (NICHD-3) are reliable and valid tools for identifying parent-infant interaction qualities that predict later toddler mental health outcomes. These tools identified overlapping and different forms of vulnerability in mother-infant dyads. Choice of which measure clinicians might use will depend on the likely goals for intervention, the age of the infant and time constraints on assessment within busy clinical practice. This paper may serve as a valuable resource for researchers and clinicians in both perinatal and infant mental health fields, particularly those involved in the observational assessment of mother–infant relationship quality.

The datasets presented in this article are not readily available due to ethical constraints. Supporting data are available to bona fide researchers on approval of an application for access. Further information about the data and conditions for access are available at the University of Liverpool Research Data Catalogue: doi: 10.17638/datacat.liverpool.ac.uk/564. Requests to access the datasets should be directed to HS, hmsharp@liverpool.ac.uk.

The study, which involved human participants, was approved by the Cheshire North and West Research Ethics Committee on the 27 June 2006 and on the 7th June 2010 (reference numbers 05/Q1506/107 and 10/H1010/4 respectively). The study was conducted in accordance with the local legislation and institutional requirements. Written informed consent for participation in this study was provided by the participants and by the participants’ legal guardians/next of kin.

HS: Conceptualization, Funding acquisition, Investigation, Methodology, Supervision, Writing – original draft, Writing – review & editing, Project administration. SV: Formal analysis, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing, Conceptualization, Investigation, Supervision. HO’M: Conceptualization, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing. LB: Data curation, Investigation, Methodology, Project administration, Writing – original draft, Writing – review & editing. MR: Investigation, Methodology, Project administration, Writing – review & editing. CH: Formal analysis, Investigation, Project administration, Writing – original draft, Writing – review & editing. JG: Investigation, Methodology, Project administration, Writing – review & editing. AP: Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Supervision, Writing – original draft, Writing – review & editing.