Usability can be defined as “the quality of a user's experience when interacting with a product, encompassing effectiveness, efficiency, and overall satisfaction of the user” (1). Usability is crucial to the success of interactive healthcare applications (2) beyond pilot stages, for efficiency, avoiding staff burnout (3), patient safety, and quality of care.

The usability of digital health interventions can be measured quantitatively and qualitatively. Quantitative measures include the System Usability Scale (SUS) created by John Brooke in 1986 (4), the industry standard for the quantitative assessment of usability (3), technologies (5), and products (6). SUS is a 10-question assessment proven reliable (7) and valid in high resource settings (8, 9). Scores of more than 68 are considered acceptable or above average. Although widely used and recognised, SUS does not explore human idiosyncrasies or delineate why particular usability issues occur during a user's experience (10).

User experience is termed as “UX” and Neilson's landscape of UX research methodologies (11) for understanding these issues better, includes scripted “think-aloud” usability studies, where an early prototype is tested in theoretical conditions before being released to the real world. Participants are asked to vocalise their thoughts and experiences while user-testing an intervention (12) generating qualitative feedback regarding the “what” but also the “why” of usability issues. This in combination with SUS has been shown to be more effective at explaining usability compared with SUS alone (2, 12). User Centred Design (UCD) and Agile are both established UX philosophies involving the user-focused iterative cycles of testing and refinement. While UCD is designer-led, centred on the end-user achieving their goals, Agile is more the remit of developers, focusing on the early and continuous delivery of working software to bring commercial value as early as possible (13). An Agile “product backlog” can structure the prioritisation of the smallest functional increments of work (“User-Stories”) within larger bodies of work (“Epics”), which can then be efficiently worked on simultaneously by different “scrum-teams.” Agile-UXD approaches are now being applied more frequently to the development of health apps for the patients' own self-management of diabetes (14), paediatric concussion (15), weight management (16), and smoking cessation (17). Reports of their use for health apps in low-resource hospital settings, where usability may be particularly important, are less common. Preliminary examples of Agile development in low resource settings include the development of a smart phone app for perinatal monitoring by Guatemalan midwives (18).

The NeoTree (Figure 1) is a recently developed, neonatal digital-health system, centred around an app which supports Health Care Professionals (HCP) working in low-resource neonatal units (20, 21). NeoTree aims to improve neonatal care and reduce newborn mortality by bringing evidence-based guidelines, algorithms, and digital data capture to the bedside. Within the NeoTree app, nurses complete digital forms, documenting admission, and discharge details for each sick newborn admitted to a neonatal unit. The app asks them to complete one field/question per page of the app. Figure 2 maps out the sections of the digital admission form that users must complete as they progress through the app, such as the embedded emergency clinical algorithm. Figure 3 shows a screen flow of the first nine pages of the app, demonstrating how a user is prompted to resuscitate a newborn who is not breathing and how to manage various newborn emergencies should they occur during admission. In this way, as the user completes the app, they receive decision support, prompts, and management support on how to manage a sick neonate according to best practise and guidance. Table 1 summarises the functions, aim, and goal of NeoTree to date.

Prior to being implemented in Malawi and Zimbabwe central hospitals (19, 22), the NeoTree app was piloted in Zomba Central Hospital (ZCH), Malawi (20). This 1-month pilot included some preliminary usability testing which resulted in early adjustments to the app and produced the beta-app, “Minimum Viable Product 1” (MVP1), the most basic working version that can be safely launched and tested (23). Although mean SUS score for this version was 86.1 (24) (relatively high), this was in the context of a small preliminary phased pilot in a district general hospital (ZCH). Since usability varies with context (4) and NeoTree was now to be implemented permanently, in a new tertiary referral centre (Kamuzu Central Hospital (KCH), Malawi), the app now required further user-testing and refinement.

The overall aim of this study was therefore to investigate the usability of the NeoTree β-app (MVP1) from both qualitative and quantitative perspectives. Objectives were to first conduct qualitative (UXD/Agile) usability testing of MVP1 and apply themes to the usability-focused development of MVP2. Second, to conduct a quantitative usability testing of MVP1 using SUS. Third, to evaluate the usage of the NeoTree β-app (MVP2) during the first 6 months of deployment in a Malawian tertiary neonatal unit.

Six participants attended usability interviews, eight completed SUS1, and eight completed SUS2 (participant details are included in Table 3). Those attending usability interviews included four women and two men, aged 28–43 years. Exact numbers of participants who gave ad-hoc feedback in the real-world pilot were not recorded.

In this study, we have demonstrated usability-focused optimisation of an app for neonatal HCPs in theoretical and real-world conditions. Rapid Agile analysis allowed abundant, detailed, context-specific, user-led refinements to be applied rapidly and iteratively, expediting deployment, and optimising usability. A new MVP2 was produced, and this was used widely on a low-resource neonatal ward, achieving high usability scores and coverage with short completion times. Multiple usability priorities relating to NeoTree as an EMR, and a clinical aid were uncovered and acted on. Similar bedside apps for low-resource settings may also benefit from this approach to prevent later implementation problems.

Results of this 2019 study can be directly compared with the previous smaller 1-month pilot in 2017 (Table 8). This larger 6-month study included more users (93 vs. 46), achieved higher SUS (89.4 vs. 86.1), higher coverage (100 vs. 70%), and shorter completion times (16 vs. 37 min for an admission). Together these might suggest that the MVP2 produced in this study was more usable than the previous MVP1. However, it is difficult to know the contribution of other factors in this new facility setting, such as the policy to use NeoTree on every baby, and the employment of NeoTree champions to provide technical support. Some of the usability themes and features from the pilot were similar to those of this study, however, this analysis was more sophisticated organising and sorting themes more clearly. Occasionally the same features were added in this study as were previously added (e.g., adding unknown options and making questions non-compulsory to ease progression through the app), but this time they were added to all parts of a now more complex app. Optimising usability in this way without compromising data integrity was more of a challenge here, as we approached the minimum dataset required for each baby. Complete data were now also crucial for quality improvement via new features of the system (diagnostic algorithm and connected dashboard). This trade-off between usability and data integrity could be a relevant consideration for other apps that combine data capture with quality improvement.

Expected usability themes produced in this study relating to NeoTree's data capture function, included “Exhaustiveness of data schema.” An exhaustive digital form that includes all relevant data fields and all possible list options is likely to reduce user frustration and forced errors, while providing a structured, standardised, and systematic user experience. Similar themes from other app studies include that the mPneumonia app allowed users to systematically capture all signs and symptoms of pneumonia (34), a Dutch digital intake tool produced a better overall picture of patients (35) and an Australian EMR prevented nurses from missing out important data inputs (36). The “relevancy of content” theme was also expected because this is consistent with lean product development which recommends that efficient systems should minimise waste (37). This in turn aligns with data protection requirements that data systems should only collect the minimum patient data needed to successfully accomplish a given task (38). For these reasons and to minimise possible harm from the burden of data collection (39), our study supports that adding new fields to clinical apps for low resource settings, should be kept to a minimum and recommends the creation of national minimum datasets already underway in Malawi.

Expected “clinical usability' improvements included adding extra pages to digitally replicate the original paper forms” (theme “cohesion with usual ward process”) aiming to reduce deviation from norms and increase buy-in. This theme corroborates a review of EMR-powered clinical decision support systems, which concluded that integrating new technologies into clinical work practices is essential for successful implementation (40). Moreover, adding educational images to improve clinical usability, was helpful for an app for illiterate midwives in Guatemala, substantiating that picture content may improve usability in low resource settings where there may be reduced literacy, or multiple languages spoken. In addition, “locally coherent clinical language” was true for apps in high resource settings where usability themes included “clarity of wording” (41) and “clear and unambiguous text feedback” (42), highlighting the importance of clear, context-specific written communication for health apps across settings.

Although SUS scores for NeoTree ß were expected to be high (following relatively high scores in the pilot), the lack of statistically significant rise in SUS after adjustments might be explained by four of the same staff completing both SUS1 and SUS2, or a possible inflation of baseline scores and therefore less room for improvement. High baseline SUS scores could have been due to a more competent group of staff using the app this time, or initial excitement and enthusiasm for a new intervention. Indeed, the perceived newness and modernity of the mPneumonia app in Ghana improved acceptance and was associated with the perception of improved care (34). Overall, NeoTree's SUS scores have been consistently higher than those of UK Emergency Department (ED) data systems, which recently scored below average (median SUS 53) (3) suggesting NeoTree may be easier to use than these. However, the UK study compared a much larger array of digital systems, while in Malawi NeoTree is the first such system, hence with no comparisons available subjective SUS scores might be inflated.

While think aloud feedback and SUS scores are both the subjective measures of usability, our objective usage statistics were encouragingly high after refinements were made, adding weight to the qualitative aspects of the study, and demonstrating that NeoTree MVP2 was used widely. The proficiency or efficiency of use, may be indicated by shorter completion times compared with the Zomba pilot (20), but these may also be attributable to more practice time over a 6-month period. In this study similar to the pilot, a significant proportion of admissions (one-fifth) were completed by nursing students which may reflect the apps educational advantages (20) or an ongoing reliance on students for service provision, in a resource-limited setting. The problem of staff shortages endures across African healthcare settings (43–46), and needs more consideration by Governments and NGOs as new technologies are introduced.

A limitation of the study might be the reliance on the subjective verbal expressions of a small number of users to guide and shape the application as per Agile. Objective log data of user errors (such as, completing a field and then deleting it or navigating to a page and then navigating back) might strengthen the study. However, log data analytics were not feasible within team capacity in a preliminary phase of the project. Despite the small sample size, qualitative data were rich, prompting abundant app adjustments. Log data analytics could potentially be made available in the future to strengthen follow on usability studies, with perhaps a more structured task analysis, but these fail to capture human idiosyncrasies and require large samples for statistical relevance. Agile methods allow lean iterative development at accelerated velocity, focusing on usability and outcomes with only a few participants, and are mainstream in industrial organisations. Increasingly academic publications are employing Agile methods. For example, a Kenyan mixed-method study combined qualitative interviews with human centred design workshops to engage users in designing and tailoring an intervention to fit the low-resource context (47).

Other limitations include that the interviews were not transcribed verbatim, and this might have limited the qualitative analysis by allowing bias during summarisation or interpretation of participant responses. However, the interview dialogue was relatively sparse and usability notes often related to observed interactions with the UI (such as clicking the wrong button), hence it was decided that transcribing the videos would not add significantly to the richness of data. Completion times were recorded manually from the app home page that may have been affected by human error. These data are amenable to automation in future studies.

This study show-cases usability-focused development of a digital app for neonatal HCPs in a low-resource neonatal unit, combining Agile methods with thematic analysis to expedite deployment and allow iterative user centric refinement during real-world use. System usability remained high after refinements were made and usage was high with 93 neonatal nurses using the app on 1,323 babies. Coverage of admissions and discharges exceeded numbers recorded on paper and completion times were shorter than a previous pilots. Results suggest that NeoTree was now tailored enough for successful use on a permanent basis and the large number of qualitative usability insights and subsequent refinements, promotes the importance of usability for digital intervention development alongside acceptability and feasibility. This study could inform the optimisation, uptake, and implementation of similar apps for other low-resource hospitals as they digitalise over the coming decades.

The data analysed in this study are part of an MDRes dissertation at UCL. For more information please contact the corresponding author.

Ethical approval for this study was confirmed with the University of Malawi, College of Medicine Research Council (P.02/19/2613) and the UCL Research Ethics Board (Ref. 6681/001).

CC conceptualised and conducted the study, analysed the data, and wrote the manuscript. MC is Malawi principle investigator, advised on study design, assisted with ethical approval, supported app implementation, and revised the manuscript. YM was chief NeoTree Ambassador overseeing KCH NeoTree activities and edited the manuscript. PD was also NeoTree Ambassador, facilitated usability interviews, contributed to analysis, and supervised the real-world pilot. TH-B was project manager, facilitated usability interviews, contributed to analysis, and oversaw NeoTree logistics. CN contributed to the conceptualisation and study design, analysis and edited the manuscript. YS advised on data analysis. DN contributed to data cleaning and quantitative analysis. KG contributed to analysis by reviewing usability themes. ML, FL, and MH supervised CC in her MD thesis, advising on study design and methodology, supervising CC in the field (remotely), and editing the manuscript. MH is principal investigator and contributed to the conceptualisation of the study. All authors contributed to the article and approved the submitted version.

This study was completed as part of an MDRes thesis for CC which was funded partly by a clinical fellow post at UCL, the Naughton Cliffe Matthews grant (PI MH) and partly by charity donations to the NeoTree charity. This research was funded in part by the Wellcome Trust [215742_Z_19_Z]. For the purpose of open access, author has applied a CC BY public copyright license to any Author Accepted manuscript version arising from this submission.

CN was employed by Spinspire Consultants. KG acquired her Agile certification from ICAgile but was not employed by ICAgile at the time of this work. YS was employed by Snowplow Analytics. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.