This feasibility study was conducted in accordance with the Declaration of Helsinki. We received ethical approval from the institutional review boards of all three participating hospitals, namely the Second Xiangya Hospital of Central South University (Changsha, China), Hospital Universitario Germans Trías i Pujol (Barcelona, Spain) and Hospital Universitario Virgen del Rocío (Sevilla, Spain). Each study participant—both clinicians and patients—provided oral consent.

We employed a mixed-methods approach to evaluate the feasibility of using the OxMIS and OxMIV risk assessment tools in new clinical settings. Clinicians from three psychiatric hospitals, based in Changsha (China), Barcelona and Sevilla (Spain), were recruited using a convenience sampling method. We selected this sampling approach as data collection using other sampling methods, e.g., randomization, is often met with practical barriers and ethical issues in psychiatric hospitals (33–35). These clinicians were general psychiatrists (specialists), general practitioners (primary care physicians) and clinical psychologists with experience in conducting risk assessments of patients with severe mental illness. First, clinicians were asked to estimate the risk of suicide (within the next 12 months) for each patient on a continuum ranging from 0.5% (marked as being the average) to 5%. Second, each clinician was also asked to calculate the probability risk scores for both tools (using an online risk calculator at http://oxrisk.com), in addition to their routinely conducted risk assessments. Consenting patients were included if they were aged 18 years and older, and had a formal diagnosis of schizophrenia-spectrum disorders or bipolar disorders (as defined by the Diagnostic and Statistical Manual of Mental Disorders [DSM-5] or the International Classification of Diseases 10th Revision [ICD-10]).

Following this, clinicians were asked to complete semi-structured interviews by means of two questionnaires (one for each tool) to evaluate the utility and acceptability of the tools. These standardized questionnaires were developed and previously used to evaluate the feasibility of implementing OxMIS and another risk assessment tool FoVOx (Forensic Psychiatry and Violence Oxford) in clinical settings (31, 36). The OxMIV semi-structured interview questionnaire for clinicians was designed prior to the OxMIS one, and thus some items differ between the two questionnaires. Questions were based on the impact, accuracy, practicability and simplicity of the tools via multiple-choice (e.g., rating various characteristics of the tools), short answers (e.g., identifying facilitators and barriers to using the tools), and Likert scale formats (e.g., not at all, slightly, somewhat, very or extremely similar). Their answers were subsequently translated into English by members of the research team who were fluent in English and either Mandarin or Spanish. Any confidential information about patients was anonymised.

Data collection took place from 2019 to 2021 across all three research sites. Specifically, quantitative and qualitative data were collected between May and December 2019 (Changsha), January and February 2020 (Barcelona), and June and July 2021 (Sevilla).

There are currently no explicit numerical recommendations for sample size in feasibility studies, or even qualitative health research (37), in contrast to external validation studies (38–40). We followed a sample saturation principle to identify and select participants related to study aims. This approach is that past a certain point, further data collection is unnecessary, as it only leads to data redundancy (41). Guidelines for feasibility studies recommend using a minimum sample size of ten (42), and thus our sample size of 38 clinicians was sufficient.

Predictors in OxMIS and OxMIV can be broadly grouped into three categories: sociodemographic, clinical and familial factors (see Supplementary Material). We employed the same definitions as those used in the original derivation studies (27, 28). Data on risk factors were obtained from existing medical records, and supplemented by verbal information collected during clinical consultations.

OxMIS and OxMIV were first developed to estimate the risk of suicide and new violent offending within 1 year, respectively. In the derivation studies, suicide included death by intentional self-harm and undetermined deaths (i.e., ICD codes X60–84 and Y10–Y34) consistent with previous research (27, 43), and violent offending was defined as any act of physical and sexual violence against others, including illegal threats and harassment, that resulted in a conviction (28). Defining violent crime as conviction, rather than arrest, reduces the possibility of systemic biases in the ascertainment of violence amongst people with severe mental illness. However, violence being defined as conviction in the derivation study does not mean that it cannot be used to predict other forms of violent behavior, such as interpersonal violence based on clinical records and self-report. In fact, the tool has been validated for these non-crime outcomes (29). Thus, our primary outcomes were patient information on the relevant risk factors, OxMIS and OxMIV risk scores for each patient, suicide risk estimation of clinicians (in relation to OxMIS), and their responses to the semi-structured interviews.

The qualitative data was analyzed by two researchers (GB and JM) scoring independently. Using a thematic analysis approach, themes and subthemes were generated, and then compared between researchers to identify common findings. Any discrepancies were resolved by discussion with a senior researcher (RY). We used NVivo V.12 Software for qualitative data analysis (44).

In the quantitative analyses, the frequency or the median and interquartile range were calculated for each risk factor, as appropriate. Missing information was recorded as ‘unknown,’ which allowed for the calculation of the estimated risk score range. We reported the proportion of missing data for each predictor. Risk scores were calculated for each patient to estimate the probability of experiencing the two outcomes. Primary analysis was conducted to evaluate the proportion of patients in each risk category for suicide and violence, and compare them between tools and countries using Spearman's rank order correlation coefficient (Spearman's rho). Suicide risk, as measured by OxMIS, was grouped into three numerical categories (i.e., low [ <0.5%], moderate [where the range of risk scores for an individual included 0.5%], and increased [>0.5%]). This is a lower threshold than in the original OxMIS study and based on a pilot work estimating mean risk scores (31). For OxMIV, the original risk categorization was employed, and subsequently translated into violence risk levels (i.e., low [ ≤5%] or increased [>5%]). This threshold was selected based on the incidence of violence (following initial diagnosis) based on longitudinal research (6). We calculated the concordance rates for suicide by comparing clinical judgement with the risk categories based on OxMIS. To supplement this, we also calculated Spearman's rho between the clinician's own predicted risk value with OxMIS in the Chinese sample (which was not possible in the Spanish sample due to no individual level identifiers) (45). Such analysis could not be undertaken for violence risk due to no linkage information at patient level.

The funders of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report. All authors had full access to the data in the study and had final responsibility for the decision to submit for publication.

The characteristics of the Chinese and Spanish samples are reported in Table 1. In total, data from 147 patients were analyzed. More specifically, risk factors were extracted for 60 patients from China and 87 patients from Spain. There were some differences in the Chinese and Spanish samples in sex distribution and mean age. For patients with missing information on predictors, minimum and maximum values of coefficients were adopted to provide an estimated risk probability score (range of values). The mean of this range of values was used in correlational analyses for individuals with missing information.

We recruited a total of 38 clinicians (26 from China and 12 from Spain), the majority of which were general psychiatrists (86.8%). Most Chinese clinicians worked in intensive care units (61.6%) and psychiatric inpatient settings (general adult wards [53.3%]). Spanish clinicians were also based in inpatient settings (58.3%) and others were in outpatient settings (41.6%). The number of assessed patients per clinician also varied between countries, with most clinicians in China evaluating 1–4 patients (96.2%), and those in Spain assessing 5–9 (41.6%) and 10–20 patients (58.3%). Some Chinese clinicians reported using another tool or process for suicide (36 [60.0%] out of 60 cases) and violence risk assessment (23 [38.3%] out of 60 cases). Similar figures were found in Spain for suicide risk assessment (5 [41.6%] out of 12). However, none of the Spanish clinicians previously used a tool to evaluate violence risk (see Supplementary Table 1 for list of tools used).

OxMIS and OxMIV probability risk scores were calculated with information from medical records and clinical consultations. As estimated by OxMIS, the median probability of suicide within 1 year was 1.9% (range = 0.6 to 10.3%) in the Chinese sample and 1.0% (range = 0.2 to 12.5%) in the Spanish sample. Based on a pre-specified threshold of 0.5% to determine risk categories (i.e., low, moderate, and increased), most Chinese patients were at increased risk (n = 59, 98.3%). None were low risk, and one moderate. In the Spanish sample, 21 (24.1%) were categorized as low risk, two at moderate risk (2.3%), and 64 at increased risk (73.6%).

Using OxMIV, the median probability of violence within 1 year was 0.7% (range = 0.2 to 14.4%) in the Chinese sample and 0.8% (range = 0.0 to 30.2%) in the Spanish sample. Most patients in China (55 [91.6%]) were categorized as low risk according to a pre-specified cut-off (<5.0%). One individual was at moderate risk (range of estimated values included 5.0%) and 4 others were at increased risk (>5.0%). As for the Spanish sample, 83 patients (95.4%) were categorized as low risk and 4 (4.6%) at increased risk for violence.

We found a medium effect size for the correlation of the risk of suicide and violence in the Chinese sample, r_s (46) = 0.45, p < 0.001. Amongst the patients from Spain, a small effect size was found for the correlation between the risk of suicide and violence, r_s (85) = 0.23, p = 0.03.

In the semi-structured interviews, clinicians from Spain completed one questionnaire each (n = 12), whereas those from China completed one questionnaire per patient (n = 60). Half of clinicians described the OxMIS score as an accurate representation of risk (50.0%; 36/72), and almost all found that OxMIV compared similarly to their clinical judgement of risk (94.4%; 68/72). Overall, 41.7% of clinicians (30/72) reported that OxMIV had an impact on their clinical practice. A summary of the clinicians' detailed viewpoints in regards of OxMIS and OxMIV is provided in Tables 2, 3. Results from the multiple-choice items are presented in the Supplementary Material (Tables 2, 3). In addition, there was a significant correlation between the OxMIS risk score and clinical judgement, r_s (58) = 0.51, p < 0.001.

We identified common themes from detailed qualitative analysis of clinician semi-structured interviews about OxMIS and OxMIV (see Tables 2, 3 for summarized qualitative results). Their overall views on the practicality and future use of the tools were grouped into two broad themes: clinical benefits and potential improvements.

Clinicians underscored improved risk assessment, management, interpretation, and prevention as clinical benefits of incorporating these tools in their practice. They also described potential additional predictors and some overall issues. For OxMIS, these included general limitations, relevance of existing risk factors, others not currently included, a lack of repeated measures over time, and accuracy. Sub-themes of potential improvements for OxMIV also included general limitations and risk factors not currently identified.

In terms of practicality, all Spanish clinicians and most Chinese ones (for 49 [81.6%] out of 60 cases) stated that the OxMIS web-based calculator would be practical to use as part of a suicide risk assessment or treatment plan. A majority of clinicians indicated that they would consider using OxMIS in the future (88.8% overall; 90.0% in China and 83.3% in Spain). For OxMIV, very few clinicians reported having encountered practical barriers in their local clinical setting that limited its use (6.9%; 6.6% in China and 8.3% in Spain). Only one clinician from Spain said that it was “not at all likely” for them to use OxMIV in the future. The remaining responses ranged from slightly to extremely likely.

Limitations include selection of patients and clinicians, which reflected a convenience sample. While this is a valid approach, particularly when conducting research with individuals that can otherwise be hard-to-reach, some studies caution against generalizing findings due to the potential risk of selection bias (35, 52). We did not conduct an external validation of OxMIS and OxMIV as this was not the aim. Future research could consider this, and updating of the predictor coefficients may be required (53). Risk categories may also need to be adjusted. However, our findings are consistent with that of previous studies of OxMIS and OxMIV in which most participants were categorized at increased risk of suicide, but low risk of violence (29–31). Using an inter-rater design to assess the reliability of the tools should also be considered for future studies.

Further work is also necessary to establish whether OxMIS and OxMIV improve clinically relevant outcomes, such as rates of future adverse events and cost effectiveness of implementing these tools. Impact analysis is one recommended method to evaluate the real effect of a prediction rule on a given clinical setting. This could take the form of a multicenter randomized controlled trial (RCT)—with hospitals or clinics being assigned randomly either to the use of the prediction rule or conventional risk assessment practice. Patients would then be followed up to determine the impact of the tool on the outcome of interest (e.g., suicide or violence) (54). The feasibility of such studies will need to be determined.

Qualitative feedback regarding country-specific aspects highlighted one or two predictors that might not translate well to China. One such example is benefit recipiency in China, of which the prevalence was nill for OxMIS. This result was corroborated in the interviews with clinicians who suggested that this predictor was not applicable. Despite recent progress in welfare system reforms, social safety nets in China remain relatively weak when compared to other economically developed countries (55). As benefit recipiency is one of the weakest predictors in OxMIS, this is unlikely to have any material impact on the tool's accuracy, and could be removed altogether for Chinese populations. Other research could be undertaken to establish if there is a proxy measure that is associated with risk.

Further, the prevalence of both previous drug and alcohol misuse, and parental drug or alcohol misuse was particularly low in the sample of patients from China compared with Spain. Taken together, stigmatization against people with substance use disorder (56), and punitive measures associated with drug use (57), might preclude their reporting. Previous research also suggests that alcohol use is not associated with suicide in China (adjusted hazard ratio, 95% CI: 1.0, 0.8–1.3) (10). However, it is likely that substance misuse is associated with violence risk based on systematic reviews of the evidence, although this needs more work in middle income countries (58), including those with lower prevalences of substance misuse in patient populations. Considering that OxMIS and OxMIV were developed in Sweden and substance use is an important risk factor in Western countries (27, 28), further work can examine the effect of these risk factors in the Chinese population.

Both OxMIS and OxMIV include some mental health–related modifiable risk factors for managing risk of suicide and violence in people with severe mental illness, including recent antipsychotic, antidepressant or dependence treatment, substance misuse, and depression comorbidity. These risk factors can be justified as they provide key targets for clinical intervention, from which people at increased risk could benefit (46). Additional research on novel approaches to risk assessment is required to incorporate more detailed monitoring of fluctuations in symptoms and dynamic risk factors, although this will likely improve needs management rather than risk prediction (59).

Future investigations should consider differential clinical and legal definitions in the context of China or Spain, as these are relevant variables when conducting cross–cultural implementation research. For instance, while the concept of dangerousness toward others is universal, what constitutes a clinically or legally relevant threshold will likely differ between jurisdictions.