Automated Progress-Monitoring for Literate Language Use in Narrative Assessment (LLUNA) - Fox, Carly; Jones, Sharad; Gillam, Sandra Laing; Israelsen-Augenstein, Megan; Schwartz, Sarah; Gillam, Ronald Bradley (Frontiers in Psychology, 2022)
Language sample analysis (LSA) is an important practice for providing a culturally sensitive and3 accurate assessment of a child’s language abilities. A child’s usage of literate language devices4 in narrative samples has been shown to be a critical target for evaluation. While automated5 scoring systems have begun to appear in the field, no such system exists for conducting progress-6 monitoring on literate language usage within narratives. The current study aimed to develop a7 hard-coded scoring system called the Literate Language Use in Narrative Assessment (LLUNA),8 to automatically evaluate six aspects of literate language in non-coded narrative transcripts.9 LLUNA was designed to individually score six literate language elements (e.g., coordinating and10 subordinating conjunctions, linguistic and mental verbs, adverbs, and elaborated noun phrases).11 The interrater reliability of LLUNA with an expert scorer, as well as its’ reliability compared to12 certified undergraduate scorers was calculated using a quadratic weighted kappa (Kqw). Results13 indicated that LLUNA met strong levels of interrater reliability with an expert scorer on all six14 elements. LLUNA also surpassed the reliability levels of certified, but non-expert scorers on four15 of the six elements and came close to matching reliability levels on the remaining two. LLUNA16 shows promise as means for automating the scoring of literate language in LSA and narrative17 samples for the purpose of assessment and progress-monitoring.