Child and adolescent psychiatric services are increasingly confronted with demands for evidence of the effectiveness of their interventions in terms of outcome. For German-speaking countries, Mattejat and Remschmidt [e.g., Ref. (1)] provide various concepts for systematic collection of outcome data. For English-speaking countries, Bickman et al. (2) looked at various approaches and developed recommendations for a modular system of outcome measurement. Here they found the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA), developed by Gowers et al. (3) in England, particularly important. HoNOSCA is based on the Health of the Nation Outcome Scales (HoNOS) (4, 5), which were developed in the United Kingdom in the early 1990s as a clinician-rated assessment instrument for routine use in adult mental health services. A major aim was the construction of a practicable instrument for differentiated assessment of the severity of mental illness that can be used as a routine clinical outcome measure.

Based on a review of the literature by Hunter et al. (6), HoNOSCA was developed taking account of specific needs in child and adolescent mental health, which resulted in an increase in the number of items (scales) from 12 to 13. Each of these biopsychosocial indicators flags mental health problems and can determine whether the young person will be referred to psychiatric care. Like HoNOS, HoNOSCA is a clinician-rated instrument for assessing the severity of mental illness. Hunter et al. (6) emphasized that this kind of graduated rating of severity complements a psychiatric diagnosis and is of central importance with regard to outcome and measurement of the course of illness. The 13 items, or scales, assess different areas of symptoms and functioning: Behavior, Impairments, Symptoms, and Social. Two additional items concern parents’ lack of understanding or lack of information about difficulties and about therapy, services, and other help and arrangements offered in connection with the patient’s mental health problems. Ratings on these two items are not a part of the total score calculated for items 1–13.

Regarding interrater reliability, previous studies found that the total score for the 13 items shows good-to-very good agreement, with intraclass correlations from 0.63 to 0.98 (3) [see also Ref. (7)]. Noteworthy is a study by Hanssen-Bauer et al. (8), which reported an intraclass correlation of raters in five countries of 0.84. For retest reliability, Harnett et al. (9) found, with a time interval of 2–3 weeks, a correlation between the total scores of r = 0.80. But there are also less favorable findings: Brann et al. (10) found a very low intraclass correlation (ICC = 0.06) for one item and only moderate intraclass correlations (ICC = 0.43) for three items. This discrepancy underlines the importance of careful rater training.

To assess the construct validity of HoNOSCA various methods have been used. Comparing HoNOSCA to other structured clinician ratings, Bilenberg (11) found associations between the HoNOSCA total score and the Global Assessment of Psychosocial Disability (GPAD) (12) of r = 0.62 (p < 0.001, N = 173). Yates et al. (13) reported an association of r = −0.64 (p < 0.01, N = 230) with the Children’s Global Assessment Scale (CGAS) (14) and a correlation of r = 0.62 (p < 0.01, N = 248) with the Paddington Complexity Scale (PCS) (13). For the PCS, Harnett et al. (9) found a somewhat lower correlation of r = 0.46 (p < 0.01, N = 51) and r = −0.47 (p < 0.01, N = 51) with HoNOSCA rates of change over a 3-month period. Looking at parent ratings, Yates et al. (13) found a correlation of r = 0.44 (p < 0.01, N = 37) with the Behavior Checklist (15, 16) and a correlation of r = 0.40 (p < 0.001, N = 208) with the Strengths and Difficulties Questionnaire (SDQ) (17), which we also used in this study.

Factor analysis of the four areas of functioning suggested by Gowers et al. (3, 9) yielded less unambiguous results, but no convincing evidence has since been found for these results.

Regarding criterion-related validity of HoNOSCA, Gowers et al. (3) tested whether HoNOSCA could differentiate between different patient groups. They found evidence of this insofar as they found that for clients in inpatient psychiatric treatment, who made up approximately 12% of the sample of N = 1,143, the HoNOSCA total score was M = 15.51 (SD = 7.19), but for clients in outpatient treatment, the total score was M = 11.18 (SD = 5.30). This difference was statistically significant [t(1 141) = −8.03, p < 0.001], and the effect size was nearly medium. As to age differences in HoNOSCA total scores, in a child and adolescent mental health service sample from the Melbourne area, Brann et al. (10) found significant differences among three age groups (n_<5 years = 34, n_{5–12 years} = 142, n_>12 years = 129) with a medium effect size (η² = 0.095). Post hoc, this significance with a medium effect size (d = 0.70) was ascribable to the difference between the 13- to 20-year age range (M = 15.21, SD = 6.66, n = 113) and the 5- to 12-year age range (M = 11.11, SD = 5.15, n = 125). In contrast, Harnett et al. (9) found no age differences.

At the level of individual HoNOSCA items, Kisely et al. (18) found that children younger than age 12 years had significantly lower scores than older children, with medium effect size, on the items non-accidental self injury and alcohol, substance/solvent misuse.

For children and adolescents aged 0–20 years (N = 173), Bilenberg (11) found that in patients with comorbidity (n₁ = 77), both the HoNOSCA total score baseline and the follow-up measurement 3 months later or at the end of therapy were significantly higher than in comparable patients without comorbidity (n₂ = 96).

In studies with longitudinal designs, Kisely et al. (18) found significant changes over time in HoNOSCA total scores for both outpatients and inpatients. For inpatients (n₁ = 30) the effect size was very large (d = 1.34; with an average measurement interval of 24.3 days); for outpatients (n₂ = 123) the effect size was small (d = 0.32; with an average measurement interval of 45.5 days). Harnett et al. (9) reported similar results. Quantified in raw scores, Gowers et al. (3) found a 38% reduction in the total score; this was after an average treatment duration of 3 months and in a sample of several hundred persons. Where clinicians reported definite improvement of the mental health problem, an average change in the HoNOSCA total score of M_d = 7.70 (SD_d = 5.40) was found. Where clinicians reported no change, the average change was only M_d = 1.60 (SD_d = 4.99), and where clinicians diagnosed a worsening of the mental health problem, the average change in the HoNOSCA total score was found to be M_d = −1.00 (SD_d = 4.53).

Bilenberg (11) reported on the usefulness of HoNOSCA in the context of a Danish field trial; about 80% of the clinicians surveyed found HoNoSCA to be clinically feasible and easy and fast to use. Gowers et al. (3) reported a mean time to complete the scale of 8.5 min.

The aim of this study was to report initial results on item characteristics and validity of the German-language version of HoNOSCA, the HoNOSCA-D, based on the following analyses: –

Item and scale analyses, homogeneity, internal consistency.

The first three analyses above concern the internal consistency and construct validity of HoNOSCA-D, and the last two pertain to criterion-related validity.

Health of the Nation Outcome Scales for Children and Adolescents, consisting in a score sheet and user guide, was translated by the first author of this article using the World Health Organization translation guidelines. HoNOSCA-D was then practice-tested at two offices of the child and adolescent psychiatric services of the Canton of St. Gallen (KJPD SG) in Switzerland for a period of 6 months. Clinicians’ ideas and suggestions for changes were recorded and then reviewed by a group of clinical experts, and some of them were incorporated in a revised version. The revised version of HoNOSCA-D was then back-translated by a native speaker of English and sent to the authors of HoNOSCA (Simon Gowers) for authorization.

The study was approved by the Eidgenössische Expertenkommission für das Berufsgeheimnis in der medizinischen Forschung (Swiss federal expert committee for professional confidentiality in medical research), which was also responsible for ethical approval. The data were collected from May 2010 and April 2012 in the context of a naturalistic field study at the KJPD SG. The KJPD SG is responsible for basic psychiatric services in a region with a population of approximately 500,000 persons, about one-fifth of whom are children and adolescents. The scales were part of a systematic outcome measurement and were used for both outpatients and inpatients aged 4;0 to 17;11 years. For all new patients, the case manager (psychiatrist, psychologist, or social worker) had to fill out HoNOSCA-D and decide on the diagnosis within the first two in depth consultations. The ICD-10 diagnosis was given then in agreement with the senior physician. The parents of all new patients received the SDQ questionnaire, which they usually filled out on-site and immediately before the first meeting. This interrogation is part of the usual intake procedure and is related to the diagnostic process and quality assurance. Parents usually were informed that if they don’t want to fill out an anonymized SDQ and therefore contribute to the quality assurance one actively has to pull out from the survey.

To evaluate the item and scale characteristics of HoNOSCA-D, after analysis of missing values, descriptive characteristic values were calculated and item and scale analyses carried out. The average interitem correlation (homogeneity) was calculated using Fisher’s z transformation. For construct validity, we computed Pearson correlations with the parent SDQ. We also carried out a principal axis factor analysis with varimax rotation. For criterion-related validity, we used a 2 × 3 ANOVA. To minimize statistical inference type 1 errors due to alpha inflation, we used Bonferroni correction of the level of significance. As a measure of effect sizes (23), for the correlations, we used their r measure (small: r = 0.1; medium: r = 0.3; large: r = 0.5) and for the ANOVA η² (small: η² = 0.01; medium: η ² = 0.06; large: η² = 0.14).

Table 1 shows the composition of the sample.

The HoNOSCA-D total sample of N = 1,553 contains, as a paired sample with n = 1,408, the sample of the parent SDQ. The difference is missing data.

Concerning difficulty index P (Table 2), there was a wide range, starting with items with low difficulty index values: hallucinations, delusions, or abnormal perceptions; alcohol, substance/solvent misuse; and non-accidental self injury. It is noteworthy that for every item the entire range of five points was used; there were no missing values.

The HoNOSCA-D total score was skewed to the left (skewness = 0.82, kurtosis = 0.84); difficulty index value was low, at 0.24. Discrimination of 7 of the 13 items was below 0.30. Internal consistency (Cronbach’s α) of the total score was 0.63.

In all, only 6 of 78 intercorrelations had at least a medium effect size of more than 0.3; the other effect sizes were small. After Fischer’s z transformation, the homogeneity of the scale was 0.11.

We correlated HoNOSCA-D with the parent SDQ (17, 19–21) (Table 3); the significance values for the 84 individual comparison were Bonferroni adjusted.

Of the correlations with at least medium effect size, noteworthy is the HoNOSCA-D item peer relationships, which was correlated with SDQ peer relationship problems, followed by disruptive, antisocial or aggressive behavior, which was correlated with SDQ conduct problems and hyperactivity and the SDQ total difficulties score. The HoNOSCA-D item overactivity, attention or concentration was correlated with SDQ hyperactivity, and emotional and related symptoms was correlated with SDQ emotional problems. Regarding assessment of the HoNOSCA-D total score, it was found to be moderately correlated with the SDQ total difficulties score.

Principal axis factoring and varimax rotation (Table 2)—with a mediocre Kaiser–Meyer–Olkin measure of sampling adequacy of 0.684, an eigenvalue of 1, and explained total variance of 30.62%—produced a four-factor solution.

The psychiatric diagnoses were combined in diagnostic groups, which do not consistently follow the single-digit mental disorders groups of the ICD-10, however (Table 5). The aim was to form meaningful evaluation categories in view of the HoNOSCA-D items.

Table 4 shows that for each disorder there was at least one HoNOSCA-D correspondence with a severity rating of at least 2 (2 = mild problem but definitely present). Emotional and related symptoms played a leading role here, whereby they were particularly severe for ICD-10 F50 Eating disorders and F3 Affective disorders. In second place were problems in family life and relationships. On the basis of the HoNOSCA-D total score, ICD-10 F2 Schizophrenia, schizotypal and delusional disorders, followed by F92 Mixed disorders of conduct and emotions and F3 Affective disorders, were the most severe disorders.

As mentioned in the note, the significance level in Table 5 is Bonferroni adjusted. Regarding age differences, the item score increased, if significant, steadily across the three age groups, whereby adolescents had a significantly highest score, unless overactivity, attention, or concentration, which had a significant lower score than the other two groups. The effect sizes of these main effects are negligible to small, with the exception of alcohol, substance/solvent misuse which shows a medium effect size. Concerning gender differences there show up only two significant main effects which are disruptive, antisocial or aggressive behavior [F(1, 1,547) = 24.51***, part. η² = 0.02] and overactivity, attention or concentration [F(1, 1,547) = 29.15***, part. η² = 0.02]. In both cases, boys reach significant higher values than girls. The HoNOSCA-D total score is equal between girls and boys [F(1, 1,547) = 8.17]. Supplementary there appears one significant interaction within non-accidental self injury [F(2, 1,547) = 10.08***, part. η² = 0.01], though the effect is small.

The item and scale analyses (see Table 2) showed that only 6 of 13 items had high enough discrimination for a priori justification of scale construction. The interitem correlations and homogeneity were also very low. This resulted in a relatively low internal consistency of 0.63 (Cronbach’s α). Harnett et al. (9) reported an even lower internal consistency of α = 0.45. This situation is actually not sufficient to ascribe satisfactory scale characteristics to HoNOSCA-D for scale construction in the classical sense. However, the finding that the interitem correlations and the homogeneity are so low allows the construction of an index, since a high share of the items are stochastically independent. The result of this is a HoNOSCA total score in which the contributions of the individual items come into effect independently of one another. Based on 13 dimensions, various clinical features in specific domains of functioning are described. As mentioned above, Harnett et al. (9) found it diagnostically useful to capture these various psychosocial and psychiatric features in the items. Gowers et al. (3) also pointed to the low interitem correlations and thus to the independence of the main scales (r = 0.13, range = 0.41–0.01). These findings are in line with Hunter et al. (6), who emphasized that HoNOSCA is an instrument that complements psychiatric diagnosis by means of an index of severity, meaning the extent of a disorder.

The results on intraclass correlations of HoNOSCA at the level of the individual items reported in the introduction above (3) made it clear that high interrater agreement is achievable in principle. It should be borne in mind that the frequent changes in resident and intern physicians and psychologists reduce the reliability of clinical observer ratings. Therefore, training must be provided frequently, and appropriate manuals must be made available.