To optimise diagnosis, treatment, and ongoing care, healthcare providers (HCPs) and recipients of care (ROCs) need to exchange appropriate and accurate information (1, 2). Health-related information is often sensitive, and sharing necessitates empathy, compassion, and mutual trust between HCP and ROC (2–4). Yet, fostering these conversations can be difficult for multiple reasons, including high patient loads, limited time, and complex, sensitive subject matter (5, 6). In South Africa, all three reasons are relevant, given HCP shortages in the public sector and the high HIV prevalence in the population (3, 4, 7). Without supportive HCP-ROC interactions, ROC may feel afraid, ashamed, or unwilling to communicate necessary information about their health behaviours, needs, concerns, and preferences (8–11). Strategies for bridging this communication gap between ROCs and HCPs are critical to the success of patient-centred care.

Artificial intelligence (AI) has the potential to help bridge this gap (8, 9, 11). One approach is leveraging generative AI tools, specifically Large Language Models (LLMs), which have demonstrated a remarkable ability to answer questions, summarise, and provide information (12–15). Given their demonstrated capacity for empathy (16, 17) in recent research, LLMs could serve as conversational agents to enhance communication between HCP and ROC through personalised, empathetic conversations.

Research on attitudes and preferences regarding AI and its capabilities for empathetic communication has shown varied results (18, 19). While some studies show HCPs are preferred over AI conversational agents (20, 21), another study suggests AI conversational agents could offer more empathetic responses than human providers (22). Although the potential benefits of AI use in healthcare are promising, more evidence supporting its potential impact in resource-constrained settings is needed (23–25). To date, most studies have predominantly focused on higher resource settings, addressing conditions prevalent in these environments (26–29). Concerns have been raised about the ability of AI conversational agents to fully capture the nuances of healthcare systems, access barriers, and cultural norms in settings like South Africa (30) and the potential impact of these limitations on patient care requires careful examination. Before questions on scalability or health outcomes can be addressed, we must demonstrate proof of concept with a prototype showing that it works in principle, is safe, appropriate, and acceptable. In this research, we assessed the usability, acceptability, feasibility, and appropriateness of an LLM conversational agent for HIV vulnerability screening, information provision, and counselling.

South Africa, with its large HIV programme, constrained human resources, and high smartphone penetration, represents a context where AI counselling tools could both be impactful and feasibly implemented (31, 32).

While South Africa's HIV treatment program has substantially increased life expectancy, the epidemic persists (31); achieving control requires expanding access to effective HIV prevention (33, 34). ROCs that test HIV negative are candidates for various HIV prevention modalities. One of the most effective being oral daily PrEP, which is offered free at the point of care within the South African primary health care system (35). Increased PrEP uptake among vulnerable populations is important to meet HIV infection reduction targets (33, 34, 36). Productive discussions between HCP and ROC about HIV prevention are plagued by many barriers in the South African context. These barriers include the limited number of HCPs, stigma associated with sexual behaviour, and lack of knowledge, amongst others (37–39).

This study evaluates a generative AI intervention for HIV prevention in a low- and middle-income country. We co-designed and tested a prototype LLM-powered conversational agent, “Your Choice”, with ROCs and HCPs in a non-clinical setting in South Africa. The conversational agent was designed to facilitate conversations around HIV prevention, in particular to determine potential benefit from and readiness for HIV pre-exposure prophylaxis (PrEP) after a negative HIV test, and provide the HCP with summarised information to boost the performance of their in-person interaction with the ROC.

Between August and October 2023, we enrolled the initial 100 participants in our ROC study to test our initial text-only LLM-powered counselling prototype in a non-clinical setting. The final 30 ROC participants were enrolled between January and March 2024 to evaluate the second LLM-powered app prototype that included text-to-audio and audio-to-text features. HCP participants were enrolled between 20 September and 20 November 2023 and reviewed output based on the first prototype only. The LLM-powered app was assessed for usability, acceptability, and appropriateness among ROCs. In addition to these assessments, HCPs evaluated its feasibility.

A total of 150 participants (n = 130 ROC; n = 20 HCP) were enrolled and included in the study. The sample is described in Table 1. ROC participants were young, the majority (71.5%) between 18 and 34 years old, with more than half identifying as male (59.00%) and unemployed (60.0%).

Of 20 HCPs, there was a complete dataset for 16 - 4 completed the survey, but the survey data from REDcap did not sync and was lost. Of the 16 HCPs, 13 (81.3%) had over 3 years' experience in practice and 8 (50.0%) had over 3 years’ experience providing PrEP services.

Across most demographic characteristics examined, including age, sex, education, employment status and location, there were no statistically significant differences in AIM, IAM or SUS scores. Differences were observed only for sexual orientation and first-language English status. AIM and IAM scores did not differ significantly by sexual orientation (p = 0.47 and p = 0.45, respectively), while SUS scores differed across groups (p = 0.04). AIM and IAM scores did not differ by first-language English status (p = 0.79 and p = 0.24, respectively). SUS scores showed borderline evidence of a difference between groups (p = 0.05) Table 2.

The ROC participants were allocated by the research team to one of two LLM-powered conversational agents with the aim of achieving approximately equal numbers per LLM agent: 66 (51.0%) ChatGPT 3.5 Turbo agent and 64 (49.0%) Claude AI agent. Research assistants monitored enrolment numbers in real time and directed new participants to one or the other agent to maintain this balance; no formal randomisation procedure was used.

By following an HCD approach in a non-clinical, hypothetical context, it was possible to develop a prototype LLM-powered conversational agent that demonstrated preliminary proof of concept that LLMs could support specific aspects of ROC–HCP communication and HIV-prevention counselling in resource-constrained settings. Specifically, we found that both ROC and HCP were willing to engage with the LLM-powered conversational agent and summaries, respectively, and rated their experience of using them highly. Although both groups rated the intervention positively, we observed differences between ROC and HCP scores. ROCs gave exceptionally high scores for acceptability, appropriateness, and usability (AIM =4.95, IAM =4.95, SUS 93–97), indicating a very strong user experience. HCP scores were comparatively lower (AIM 4.5, IAM 4.4, SUS 78), but still in the “good” usability range and reflective of high acceptability and appropriateness. These differences likely reflect the distinct roles of ROCs as end-users and HCPs as service providers, who assess the tool in relation to workflow and clinical responsibilities. ROCs interacted directly with the conversational agent and appeared to derive immediate personal benefit, whereas HCP ratings may capture broader considerations such as workflow integration, feasibility, and the reality of using such tools in busy clinics. This difference underscores the importance of considering both end-user and provider viewpoints when assessing the implementation potential of LLM-powered tools.

AIM and IAM scores did not vary across demographic groups; the SUS findings, however, indicate small but notable differences in perceived usability by sexual orientation and first-language English status. These differences may reflect varying levels of familiarity with digital technologies, differing expectations, or variations in how easily participants interpreted the LLM's responses. Usability captures aspects such as confidence, clarity and smoothness of interaction, which may be more sensitive to individual experience than acceptability or appropriateness.

Further work is needed to understand the drivers of these differences and to ensure that future design iterations improve usability across all groups. This includes examining specific interaction challenges, qualitatively exploring the user experience, and developing multilingual and culturally responsive system features to support equitable engagement.

An analysis of the conversations between the LLM-powered conversational agent and ROCs showed a high level of openness, with ROCs engaging in personal and intimate discussions. In almost all situations, the LLM-powered conversational agent provided accurate and appropriate responses to the ROC. Most instances of inaccurate or misleading responses could be traced back to the LLM using outdated information or failing to understand colloquial language.

The high acceptability, appropriateness, and usability of the LLM-powered conversational agent has been shown in other studies that have evaluated similar conversational agents (46, 47), particularly those exploring stigmatised conditions. Our study was one of only a few studies (48) that focused on resource-limited settings and also incorporated the HCP's experience and viewpoint towards technology in the delivery of healthcare. While our work shows that participants were open to the use of technology in the delivery of healthcare, it does not necessarily translate directly into demand for this service. Further work to understand the user's preferences for these services will be critical.

Some research has shown that LLM-powered conversational agents can provide high-quality, empathetic responses to ROC and may be effective in providing counselling, particularly for stigmatised conditions (21, 46, 49). In addition to counselling, other studies have demonstrated their potential to provide health information, screen for disease vulnerability, and provide tailored care plans for individuals (46, 47). Much work has been done to understand and define empathy in the interactions between ROC and LLM-powered conversational agents, and results vary (17, 50, 51). Some studies have suggested that LLM's lack empathy, the ability to understand a ROC's perspective, and communicate this understanding but may demonstrate “cognitive” empathy (16). Recent studies on the use of LLM conversational agents as therapists found they may reinforce stigma and risk reinforcing harmful beliefs (52, 53).

The results from our work show support for the idea that an LLM-powered conversational agent might fill a critical gap in health care provision. We provide a unique focus on HIV prevention in settings where discussions about HIV, sex, and sexuality are often stigmatised and HCP time is limited. The ROC were able to have open and honest conversations and the LLM-powered conversational agent was able to provide empathetic responses. However, variation in the literature underscores the complexity of evaluating perceived empathy in AI systems. Differences in findings across studies may reflect differences in definitions and analysis, as well as individual preferences and contexts (50). For example, some studies assess “empathy” in purely functional terms, such as whether an LLM recognises emotions or generates supportive language that aligns with concepts of cognitive empathy (16). Other studies evaluate empathy as an inherently human emotional capacity, emphasizing that LLMs cannot genuinely experience emotions or feel with a patient (54, 55). These differing conceptual foundations lead to different conclusions.

Analysis may also differ. Some studies measure empathy through patient-reported perceptions, while others use behavioural coding frameworks or other evaluations of responses (54). These methodological differences capture distinct dimensions of “empathy”, resulting in differences in findings.

Different contexts also contribute to variability. Participants with positive expectations of AI may perceive LLM responses as more empathetic, whereas others may feel that LLMs cannot feel emotions (56, 57). Cultural norms, prior experience with HCPs further shape how empathy is judged. Those who have had negative experiences discussing sensitive information with HCPs may find LLMs to be more empathetic.

Differences in findings across studies may also reflect the rapid pace of technological improvement, since older systems quickly become outdated, and the fact that bespoke human-centred design approaches can produce more tailored and empathetic interactions. This highlights the importance of building healthcare systems and interventions that accommodate individual differences while keeping pace with technological advances. Future work should explore how contextual and individual differences influence the perceived empathy of LLMs, and investigate which healthcare use cases are most likely to be accepted and trusted by both ROCs and HCPs across diverse settings.

While this work highlights the enormous potential for improving in-person interactions with the healthcare system, it also has identified potential pain points. We did note instances where the LLM-powered conversational agent misinterpreted context or language, as has also been shown in other studies (48, 58). This raises the concern that a technology-based service may provide inappropriate or inaccurate information to a client. This argues for greater diversity (i.e., geography, language, context) in the training data, robust guardrails to guard against misinformation, and underscores the need for innovative ways to co-design and test LLM-powered conversational agents in the contexts they will be used, with the populations that will use them. There is also a need for ongoing monitoring to ensure information stays relevant and appropriate.

We did not include a comparison with traditional human counsellors, and therefore we cannot determine the relative effectiveness of an LLM-powered conversational agents compared to the current standard of care. While the prototype did exhibit occasional minor errors, it is important to note that there is substantial evidence that human counsellors may also make similar or even more serious mistakes. Without direct comparison, we cannot draw conclusions about relative safety or effectiveness. It is thus premature to interpret such errors as indicative of substandard performance; rather, they highlight the need for further empirical research to assess the comparative risks and benefits.

Stigma is a major barrier to the uptake of HIV services and can often be exacerbated by misinformation or unclear explanations (38, 59–61). Challenges regarding accuracy of the information provided by the LLM and its ability to respond in a way that reflects the nuances of counselling on highly sensitive, stigmatised matters have also been reported in other work (53, 62, 63). Systems that can navigate these complexities have the opportunity for greater impact on care, missing this could reduce safety and equity of such systems, putting people who are already vulnerable at risk. Prompt engineering and ensuring that reliable, current and contextually relevant data sources augment these services could improve efficacy of these tools.

Our study points to several important directions for future research and practice. There is a need to ensure a minimum standard regarding the accuracy and appropriateness of information provided by the LLM-powered conversational agent used in healthcare. Given the variability in how clients interpret counselling language, it is critical to monitor for unintended negative effects such as reinforcing stigma or overlooking emotional nuance when using an LLM-powered conversational agent. These responses may differ even when identical phrasing is used, depending on the individual's background and experience. To mitigate potential harm, ongoing quality assurance through both automated systems and human oversight (“human-in-the-loop” methods) is essential (53, 64, 65). Such mechanisms can help detect and address issues early, supporting the safe and context-sensitive integration of LLMs into healthcare in resource-limited settings. Where accuracy of out-of-the-box commercial LLMs are not up to standard, further work is needed on how to ensure information provided is consistently accurate and safe.

One of the main attractions for using LLM-powered conversational agent in healthcare is its potential to increase equity regarding access to healthcare services. Co-designing with target end-users to understand their concerns and barriers to care is important. Factors like digital literacy, ability to read and write are also important considerations. With this in mind, the second iteration of “Your Choice” had a voice to text addition. We also designed the app as a mediator between HCP and ROC, to be used within the health setting. This would ensure that ROC had access to the hardware required to use the app when visiting a clinic, and that those who had difficulty reading and writing could also use the audio to text functionality. However, a disadvantage of conversational agents for use in clinical is limited private spaces to use such an app. Future studies should explore the use of conversational agents for supporting self-care outside traditional clinics, while also addressing the challenges that currently limit their use in clinic spaces.

Last, at the pace AI technology is moving, and the access different people have to it, research and implementation need to move at a pace that matches these developments.

The first limitation is that this study was conducted in a controlled research environment, with participants recruited through the B-Hub and making use of research facility resources. While this may have produced more favourable findings than might occur under routine implementation, it was necessary to first establish proof of concept under stable conditions. There was no direct comparison with counselling delivered by human healthcare providers (HCPs), which limits our ability to benchmark the performance of the LLM-powered agents against standard-of-care interactions. Future research in real-world settings will be important to understand how contextual challenges may influence effectiveness and impact.

While the LLM conversational agents were not used to guide actual clinical care, the participants were actual end users interacting based on their lived experience resulting in accurate, contextually relevant data that can provide evidence to support the safe implementation of such technology in clinical care environment.

Another limitation is that we did not specifically include key populations, people with disabilities, or stratification to ensure variation in educational and literacy levels. Although we included different populations who could benefit from PrEP, the diversity of the sample was constrained by recruitment through the B-Hub and the relatively small sample size. This restricts the generalisability of our findings, as the perspectives captured may not fully reflect the experiences and needs of these important groups. Future work should prioritise inclusion of diverse populations to strengthen the relevance and applicability of results.

The last limitation is that, even though we included a diverse group of individuals, they were screened appropriately for language comprehension, so only people with some comfort communicating in English were included. This was necessary as the LLMs could not safely communicate in the other official South African languages. Despite this screening only a third of all participants were first language English speakers, despite being comfortable participating in English. This suggests that these results reflect participants with broader language diversity than English only. However, the ROC sample may have been biased towards those with higher literacy. This may have inflated the perceptions of usability and acceptability which limits generalisability of our findings. The ability to communicate in languages other than English will be critical for future versions to ensure equitable access; particularly in South Africa, where there are 11 official spoken languages.

This is one of the first studies to explore the use of LLM-powered conversational agents for HIV service delivery. We also found little research on LLM-powered conversational agents in LMICs, and even less that apply HCD involving both ROC and HCP. Another major strength was the multidisciplinary collaboration between a behavioural science research team with deep expertise in HIV and health systems in Southern Africa and a health-tech company with cutting-edge AI and software engineering experience. Our use of co-creation, integrating technology with behavioural science, and leveraging the B-Hub to rapidly and ethically embed local context was a key strength.

The HIV prevention use case in a resource-constrained setting has shown that LLM-powered applications hold immense potential to revolutionise healthcare. They may be able to improve outcomes by enhancing the ROC-HCP interaction, optimizing the use of limited human resources, creating private spaces for disclosure of sensitive information, and ensuring empathy in responding to ROC inquiries and disclosures.

However, there are areas of concern with using commercial LLMs largely “out of the box”. Rapid, human-centred design, testing, and scalable monitoring of LLM accuracy and empathy are needed, along with integrating local context and care guidelines to match AI's evolution. Likewise, more education about AI, both potential benefits and risks, research, and engagement with HCPs, policy makers, and local stakeholders is needed to ensure safe, equitable and effective use of AI, particularly in resource-limited settings.

In addition, lessons from this work highlight further considerations for scaling innovations like “Your Choice”. These include co-designing with potential users, embedding plans for scale and sustainability from the outset and throughout the design and research process. Leveraging existing platforms and creating integrated services rather than health-only or single-condition tools.