<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.3 20070202//EN" "journalpublishing.dtd">
<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="brief-report">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">Front. Public Health</journal-id>
<journal-title>Frontiers in Public Health</journal-title>
<abbrev-journal-title abbrev-type="pubmed">Front. Public Health</abbrev-journal-title>
<issn pub-type="epub">2296-2565</issn>
<publisher>
<publisher-name>Frontiers Media S.A.</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="doi">10.3389/fpubh.2020.00117</article-id>
<article-categories>
<subj-group subj-group-type="heading">
<subject>Public Health</subject>
<subj-group>
<subject>Perspective</subject>
</subj-group>
</subj-group>
</article-categories>
<title-group>
<article-title>A &#x0201C;Third Wheel&#x0201D; Effect in Health Decision Making Involving Artificial Entities: A Psychological Perspective</article-title>
</title-group>
<contrib-group>
<contrib contrib-type="author" corresp="yes">
<name><surname>Triberti</surname> <given-names>Stefano</given-names></name>
<xref ref-type="aff" rid="aff1"><sup>1</sup></xref>
<xref ref-type="aff" rid="aff2"><sup>2</sup></xref>
<xref ref-type="corresp" rid="c001"><sup>&#x0002A;</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/120273/overview"/>
</contrib>
<contrib contrib-type="author">
<name><surname>Durosini</surname> <given-names>Ilaria</given-names></name>
<xref ref-type="aff" rid="aff2"><sup>2</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/631932/overview"/>
</contrib>
<contrib contrib-type="author">
<name><surname>Pravettoni</surname> <given-names>Gabriella</given-names></name>
<xref ref-type="aff" rid="aff1"><sup>1</sup></xref>
<xref ref-type="aff" rid="aff2"><sup>2</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/177982/overview"/>
</contrib>
</contrib-group>
<aff id="aff1"><sup>1</sup><institution>Department of Oncology and Hemato-Oncology, University of Milan</institution>, <addr-line>Milan</addr-line>, <country>Italy</country></aff>
<aff id="aff2"><sup>2</sup><institution>Applied Research Division for Cognitive and Psychological Science, IEO, European Institute of Oncology IRCCS</institution>, <addr-line>Milan</addr-line>, <country>Italy</country></aff>
<author-notes>
<fn fn-type="edited-by"><p>Edited by: Koichi Fujiwara, Nagoya University, Japan</p></fn>
<fn fn-type="edited-by"><p>Reviewed by: Takanori Fujita, Keio University, Japan; Yoko Kiriyama, MICIN, Japan</p></fn>
<corresp id="c001">&#x0002A;Correspondence: Stefano Triberti <email>stefano.triberti&#x00040;unimi.it</email></corresp>
<fn fn-type="other" id="fn001"><p>This article was submitted to Digital Public Health, a section of the journal Frontiers in Public Health</p></fn></author-notes>
<pub-date pub-type="epub">
<day>28</day>
<month>04</month>
<year>2020</year>
</pub-date>
<pub-date pub-type="collection">
<year>2020</year>
</pub-date>
<volume>8</volume>
<elocation-id>117</elocation-id>
<history>
<date date-type="received">
<day>03</day>
<month>12</month>
<year>2019</year>
</date>
<date date-type="accepted">
<day>23</day>
<month>03</month>
<year>2020</year>
</date>
</history>
<permissions>
<copyright-statement>Copyright &#x000A9; 2020 Triberti, Durosini and Pravettoni.</copyright-statement>
<copyright-year>2020</copyright-year>
<copyright-holder>Triberti, Durosini and Pravettoni</copyright-holder>
<license xlink:href="http://creativecommons.org/licenses/by/4.0/"><p>This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.</p></license>
</permissions>
<abstract><p>In the near future, Artificial Intelligence (AI) is expected to participate more and more in decision making processes, in contexts ranging from healthcare to politics. For example, in the healthcare context, doctors will increasingly use AI and machine learning devices to improve precision in diagnosis and to identify therapy regimens. One hot topic regards the necessity for health professionals to adapt shared decision making with patients to include the contribution of AI into clinical practice, such as acting as mediators between the patient with his or her healthcare needs and the recommendations coming from artificial entities. In this scenario, a &#x0201C;third wheel&#x0201D; effect may intervene, potentially affecting the effectiveness of shared decision making in three different ways: first, clinical decisions could be delayed or paralyzed when AI recommendations are difficult to understand or to explain to patients; second, patients&#x00027; symptomatology and medical diagnosis could be misinterpreted when adapting them to AI classifications; third, there may be confusion about the roles and responsibilities of the protagonists in the healthcare process (e.g., Who <italic>really</italic> has authority?). This contribution delineates such effects and tries to identify the impact of AI technology on the healthcare process, with a focus on future medical practice.</p></abstract>
<kwd-group>
<kwd>decision making</kwd>
<kwd>artificial intelligence</kwd>
<kwd>ehealth</kwd>
<kwd>patient-doctor relationship</kwd>
<kwd>technology acceptance</kwd>
<kwd>healthcare process</kwd>
<kwd>patient-centered medicine</kwd>
</kwd-group>
<counts>
<fig-count count="1"/>
<table-count count="1"/>
<equation-count count="0"/>
<ref-count count="82"/>
<page-count count="9"/>
<word-count count="7156"/>
</counts>
</article-meta>
</front>
<body>
<sec sec-type="intro" id="s1">
<title>Introduction</title>
<p>In the last few years, Artificial Intelligence (AI) has been on the rise, and some think that this technology will define the contemporary era as automation and factory tools defined the industrial revolutions, or as computers and the web characterized recent decades (<xref ref-type="bibr" rid="B1">1</xref>&#x02013;<xref ref-type="bibr" rid="B3">3</xref>). These technologies, based on machine learning, promise to become more than simple &#x0201C;tools&#x0201D;; rather, they will be interlocutors of human operators that can help in complex tasks involving reasoning and decision making. The expression &#x0201C;machine learning&#x0201D; refers to a branch of computer science devoted to developing algorithms able to learn from experience and the external environment, improving performance over time (<xref ref-type="bibr" rid="B4">4</xref>&#x02013;<xref ref-type="bibr" rid="B6">6</xref>). More specifically, algorithms are able to detect associations, similarities, and patterns in data, allowing predictions to be made on the likelihood of uncertain outcomes.</p>
<p>AIs and machine learning are present in a number of commonly used technologies, such as email, social media, mobile software, and digital advertising. However, the near future of AI is not that it will continue to work outside of the end users&#x00027; awareness, as it mostly does nowadays; on the contrary, AI promises to become an active collaborator with human operators in a number of tasks and activities. AIs are able to analyze enormous quantities of data of various contents and formats, even where it is dynamically changing (Big Data). AIs identify associations and differences between data and provide human operators with outputs that are impossible to achieve by humans alone, at least in the same amount of time.</p>
<p>An example of such outputs are medical diagnoses and the identification of therapy regimens to be administered to patients. Health professionals (physicians especially) will increasingly interact with AIs to get information on their patients that will hopefully be more exact, specific, and based on objective data (<xref ref-type="bibr" rid="B7">7</xref>&#x02013;<xref ref-type="bibr" rid="B9">9</xref>). Diagnostic decision support could be considered the main application area for AI-based innovation in medical practice (<xref ref-type="bibr" rid="B10">10</xref>, <xref ref-type="bibr" rid="B11">11</xref>). Basically, machine learning devices are trained to classify stimuli based on initial examples. For instance, tumor types can be identified by the comparison of patient&#x00027;s TAC with information coming from scientific literature (<xref ref-type="bibr" rid="B12">12</xref>, <xref ref-type="bibr" rid="B13">13</xref>); the same can be done with pictures of skin lesions (<xref ref-type="bibr" rid="B14">14</xref>), optical coherence tomography in the case of sight diseases (<xref ref-type="bibr" rid="B15">15</xref>&#x02013;<xref ref-type="bibr" rid="B17">17</xref>), or the integration of clinical observations and medical tests for other diseases (<xref ref-type="bibr" rid="B18">18</xref>, <xref ref-type="bibr" rid="B19">19</xref>). While diagnosis is recognized by many as the main area for AI implementation in medicine and healthcare, others could be envisaged, as summarized in <xref ref-type="table" rid="T1">Table 1</xref>.</p>
<table-wrap position="float" id="T1">
<label>Table 1</label>
<caption><p>A resume of the main areas for AI implementation in healthcare and medicine.</p></caption>
<table frame="hsides" rules="groups">
<thead><tr>
<th valign="top" align="left"><bold>AI function in healthcare</bold></th>
<th valign="top" align="left"><bold>Description; AI is&#x02026;</bold></th>
<th valign="top" align="center"><bold>Examples</bold></th>
</tr>
</thead>
<tbody>
<tr>
<td valign="top" align="left">Diagnosis</td>
<td valign="top" align="left">Employed as a diagnostic support tool; it analyzes clinical/pathological data to identify the disease</td>
<td valign="top" align="center">(<xref ref-type="bibr" rid="B11">11</xref>, <xref ref-type="bibr" rid="B16">16</xref>, <xref ref-type="bibr" rid="B20">20</xref>, <xref ref-type="bibr" rid="B21">21</xref>)</td>
</tr>
<tr>
<td valign="top" align="left">Treatment (identification)</td>
<td valign="top" align="left">Involved in identification of treatment, often patient-specific solutions (genomics, precision medicine); it could participate in providing early interventions to delay the onset of chronic conditions (pre-emptive medicine)</td>
<td valign="top" align="center">(<xref ref-type="bibr" rid="B22">22</xref>&#x02013;<xref ref-type="bibr" rid="B25">25</xref>)</td>
</tr>
<tr>
<td valign="top" align="left">Health management/patient engagement</td>
<td valign="top" align="left">Featured in devices that collect data on patient health status and provide recommendations for everyday care (eHealth, Digital Therapeutics, Ambient Intelligence)</td>
<td valign="top" align="center">(<xref ref-type="bibr" rid="B26">26</xref>&#x02013;<xref ref-type="bibr" rid="B28">28</xref>)</td>
</tr>
<tr>
<td valign="top" align="left">Health Systems organization support/simulation</td>
<td valign="top" align="left">Used in agent-based simulations that model care coordination capabilities, providing insights for organizational improvements</td>
<td valign="top" align="center">(<xref ref-type="bibr" rid="B29">29</xref>, <xref ref-type="bibr" rid="B30">30</xref>)</td>
</tr>
</tbody>
</table>
</table-wrap>
<p>However, the study of AI in healthcare in its social-psychological aspects is still an underrepresented area. One important field is that of &#x0201C;Explainable Artificial Intelligence&#x0201D; (commonly abbreviated in XAI), namely the research on AI&#x00027;s transparency and ability to explain its own elaboration processes. The American Agency for Advanced Research Projects for Defense (DARPA) launched a program on XAI (<xref ref-type="bibr" rid="B31">31</xref>), and the European Parliament demands a &#x0201C;right to explanation&#x0201D; in automated decision making (<xref ref-type="bibr" rid="B32">32</xref>). Indeed, one issue with AI implementation in professional practice regards the fact that it is supposed to be used by non-professionals: doctors, marketers, or military personnel are not expected to become experts in informatics or AI development, yet they will have to interact with artificial entities to make important decisions in their fields. While one could easily agree with the analyses and outputs of AIs, <italic>trusting them</italic> and taking responsibility for decisions that will affect the &#x0201C;real world&#x0201D; is no easy task. For this reason, XAI is identified by many scholars as a priority for technological innovation. Miller and colleagues (<xref ref-type="bibr" rid="B33">33</xref>, <xref ref-type="bibr" rid="B34">34</xref>) maintain that AI developers and engineers should turn to social sciences in order to understand what is an explanation, and how it could be effectively implemented within AIs&#x00027; capacities. For example, Vellido (<xref ref-type="bibr" rid="B35">35</xref>) proposed that AIs learn to make their processes transparent via visual aids that help a human user to understand how a given conclusion has been reached; Pravettoni and Triberti (<xref ref-type="bibr" rid="B36">36</xref>) highlighted that explanation is rooted in interaction and conversation, so that a complete, sophisticated XAI would be reached when artificial entities were able to communicate with human users in a realistic manner (e.g., answering questions, learning basic forms of perspective-taking, etc.). In any case, besides working on AI-human interfaces, another field of great interest is that of AI&#x00027;s impact on professional practice or the prediction of possible organizational, practical, and social issues that will emerge in the context of implementation.</p>
<p>Though the contribution of AIs to medical practice is promising, their impact on the clinician-patient relationship is still an understudied topic. From a psychosocial point of view, it is possible that new technologies will influence the relationship between clinicians and patients in several ways. Indeed, the introduction of AI into the healthcare context is changing the ways in which care is offered to patients: the information given by AIs on diagnosis, treatment, and drugs will be used to make decisions in any phase of the healthcare journey (e.g., choices on treatment or lifestyle changes, deciding to inform relatives of one&#x00027;s health status, communication of bad news).</p>
<p>According to a patient-centered perspective, such care choices should be made by the patient and the doctor within a mutual collaboration, which points to the popular concept of <italic>shared decision making</italic>. This concept has become fundamental in the debate on patient-centered approaches to care, with the number of scientific publications on the subject rising more than 600% from 2000 to 2013 (<xref ref-type="bibr" rid="B37">37</xref>). Reviews show that the communication process and relationship quality among doctors and patients has a significant effect on patients&#x00027; well-being and quality of life, so that the proper communication style can alleviate the traumatic aspects of illness (<xref ref-type="bibr" rid="B38">38</xref>&#x02013;<xref ref-type="bibr" rid="B40">40</xref>). However, when prefiguring the adoption of AIs participating in diagnosis and therapy identification, it is possible that the same concept of &#x0201C;shared decision&#x0201D; should be updated, taking into consideration the contribution of artificial entities.</p>
</sec>
<sec id="s2">
<title>Who to Share Decision Making With?</title>
<p>Shared decision making has been proposed as an alternative paradigm to the &#x0201C;paternalistic&#x0201D; one (<xref ref-type="bibr" rid="B41">41</xref>, <xref ref-type="bibr" rid="B42">42</xref>). The latter model dominated disease-centered medicine, with the physician being authoritative and autonomous, giving recommendations to patients without taking into consideration their full understanding, personal needs, and feelings. While the &#x0201C;paternalistic&#x0201D; physician intended to act in the best interest of the patient, such an approach may be ineffective or counterproductive in the end, because the patient may not understand nor follow the recommendations (<xref ref-type="bibr" rid="B43">43</xref>). Shared decision making is a process by which patients and health professionals discuss and evaluate the options for a particular medical decision, in order to find the best available treatment that is based on knowledge that is accessible and comprehensible for both and satisfies both needs (<xref ref-type="bibr" rid="B44">44</xref>&#x02013;<xref ref-type="bibr" rid="B46">46</xref>). During this process, the patient is made aware of diagnostic and treatment pathways, as well as of related risks and benefits; also, the patient&#x00027;s point of view is taken into consideration in terms of preferences and personal concerns (<xref ref-type="bibr" rid="B47">47</xref>&#x02013;<xref ref-type="bibr" rid="B49">49</xref>).</p>
<p>In other words, shared decision making entails a process of communication and negotiation between the health professional and the patient in which both medical information (e.g., diagnosis, therapy, prognosis) and patient&#x00027;s concerns (e.g., doubts and request for clarification, lifestyle changes, worries for the future, etc.) are exchanged.</p>
<p>In the near future, where AI is expected to take a role in medical practice, it is important to understand its influence on shared decision making and on the patient-doctor relationship as a whole. In most of the health systems around the world, the patient has the right to be informed about which tools, resources, and approaches are being employed to treat his or her case; a patient will have to know that the diagnosis or even the medical prescriptions first came out of a machine, not through the human doctor&#x00027;s effort. Presumably, just the knowledge of the presence of a &#x0201C;machine&#x0201D; in the healthcare process could influence the attitudes of doctors and patients: from a psychological point of view, attitudes toward something may develop before any direct experience of it (e.g., as a product of hearsay, social norms, etc.), and influence subsequent conduct (<xref ref-type="bibr" rid="B50">50</xref>). Indeed, while medicine itself is inherently open to innovation and technology, some health professionals harbor negative attitudes toward technology for care (<xref ref-type="bibr" rid="B51">51</xref>, <xref ref-type="bibr" rid="B52">52</xref>), the main reasons being the risk they feel for patient de-humanization (<xref ref-type="bibr" rid="B53">53</xref>, <xref ref-type="bibr" rid="B54">54</xref>) or the fear that tools they are not confident in mastering may be used against them in medical controversies (<xref ref-type="bibr" rid="B55">55</xref>). Similarly, patients who do not feel confident in using technology (&#x0201C;computer self-efficacy&#x0201D;) benefit less from eHealth resources than other patients who do feel confident (<xref ref-type="bibr" rid="B56">56</xref>, <xref ref-type="bibr" rid="B57">57</xref>), and technological systems for healthcare are not expected to work as desired if development is not tailored to users&#x00027; actual needs and context of use (<xref ref-type="bibr" rid="B26">26</xref>). Though these data regard types of technologies different from AIs, they clearly show that technological innovation in the field of health is hardly a smooth process. While it is clear how the &#x0201C;technical&#x0201D; part of medicine (e.g., improving diagnosis correctness) would benefit from AIs and machine learning devices, their impact on the patient-doctor relationship is mostly unknown.</p>
<p>Furthermore, with the development of eHealth (<xref ref-type="bibr" rid="B58">58</xref>, <xref ref-type="bibr" rid="B59">59</xref>) and the diffusion of interactive AIs as commonly used tools (e.g., home assistants), it is possible that chronic patients (e.g., patients with obesity, arthritis, anorexia, heart disease, diabetes) will be assisted by AIs in their everyday health management. Indeed, for example, the American Food and Drugs Administration (FDA) has made &#x0201C;significant strides in developing policies that are appropriately tailored to ensure that safe and effective technology reaches users&#x0201D; [US FDA (<xref ref-type="bibr" rid="B60">60</xref>), p.2], promoting the development of Software as a Medical Device (SaMD), devices that play a role in diagnosis or treatment (not only health or wellness management). FDA-approved digital therapeutics include, for example, reSET developed by PEAR Therapeutics (<xref ref-type="bibr" rid="B61">61</xref>), which delivers cognitive-behavioral therapy to patients suffering from substance abuse (<xref ref-type="bibr" rid="B62">62</xref>). It is possible that similar future resources will include AIs that directly interact with patients based on natural language processing.</p>
<p>In other words, AIs will not be just a &#x0201C;new app&#x0201D; on doctors&#x00027; devices, but active interlocutors, able to deliver diagnosis, prognosis, and intervention materials to both the doctor and the patient. According to Topol (<xref ref-type="bibr" rid="B7">7</xref>), AI&#x00027;s implementation in care could potentially have positive effects, but this depends mainly on doctors&#x00027; attitudes: for example, if AI were to take on administrative and technical tasks in medicine, doctors would have the occasion to recover the &#x0201C;lost time&#x0201D; for consultation with and empathic listening to their patients, so to improve shared decision making. In this sense, AI would become an active go-between among care providers and patients.</p>
<p>This considered, it becomes fundamental to understand whether we should expect structural changes in the same context of shared decision making and medical consultation. Will patients interact with AIs directly? Will doctors encounter difficulties and obstacles in adapting their work practices to include technologies able to participate in diagnosis and treatment? Are patient-doctor decisions to be shared with artificial entities too?</p>
<p>At the present time, the scientific literature lacks research data to fully respond to these questions. However, by considering the literature on health providers&#x00027; reactions to technological innovation and the psychology of medicine, it is possible to prefigure some social-psychological phenomena that could occur in the forthcoming healthcare scenarios in order to prepare to manage undesirable side-effects.</p>
</sec>
<sec id="s3">
<title>A &#x0201C;Third Wheel&#x0201D; Effect</title>
<p>In common language, the expression &#x0201C;third wheel&#x0201D; refers to someone who is superfluous with respect to a couple. Typically, the focus of the expression is on this third person, who unintentionally finds him or herself in the company of a couple of lovers and feels excluded and out of place. On the other side of the relationship, the couple may be unaware of the stress caused to the third wheel, or they may feel awkward and uneasy because of the unwanted presence. In other words, a third wheel is someone who is perceived as an adjunct, something unnecessary, who may spoil the mood and negatively influence others&#x00027; experience. Despite just being a popular idiom, this expression is sometimes used in psychological research to describe relationship issues as experienced by research participants (<xref ref-type="bibr" rid="B63">63</xref>): new technology, specifically social media, has been called a third wheel as well because of its possible negative influence on relationship quality (<xref ref-type="bibr" rid="B64">64</xref>, <xref ref-type="bibr" rid="B65">65</xref>).</p>
<p>We propose to employ the expression &#x0201C;third wheel&#x0201D; to highlight an emergent phenomenon relating to the implementation of artificial entities in real-life contexts: while technologies become more and more autonomous, able to talk, to &#x0201C;think,&#x0201D; and to actively participate in decision making, their role within complex relationships may be unclear to the human interlocutors, and new obstacles to decision making could arise.</p>
<p>We identified three main ways a third wheel effect may appear in medical consultation aided by artificial intelligence: <italic>decision paralysis, or a risk of delay, &#x0201C;Confusion of the Tongues,&#x0201D; and role ambiguity</italic>. In the next sections, these will be described in detail.</p>
<sec>
<title>Decision Paralysis, or a Risk of Delay</title>
<p>As previously stated, current AIs are not transparent in their elaboration processes; that is, their interlocutors may have no clear representation of how AIs have reached a given conclusion: this could generate &#x0201C;trust issues,&#x0201D; especially when important decisions should be taken on the basis of these conclusions. According to Topol and his seminal book <italic>Deep Medicine</italic> (<xref ref-type="bibr" rid="B7">7</xref>), one positive consequence for AI implementation in medical practice that we could hope for is giving back time to doctors to reserve to empathic consultation and patient-centered medicine. Indeed, if AIs were to take on technical and administrative tasks in medicine, doctors could devote their attention to patients as individuals and improve the &#x0201C;human side&#x0201D; of their profession. However, we should take into account that doctors using AIs will need to contextualize and justify their role within practice and the relationship with the patient. It could be said that doctors will become &#x0201C;mediators&#x0201D; between their artificial allies and the patients: AI&#x00027;s conclusions and recommendations should be reviewed by the doctor, approved and refined, and explained to the patient, answering his or her questions. On the other side, future technologies could include opportunities for direct interaction between AIs and patients: for example, digital therapeutics, or eHealth applications providing assistance to patients and caregivers in the management and treatment of chronic diseases, could potentially include access to the AI providing diagnosis and therapy guidelines. However, we can foresee that a patient would still need approval and guidance from the human doctor for modifications to the treatment schedule, medication intake, specific changes to lifestyle, and everyday agenda.</p>
<p>Such a &#x0201C;mediation&#x0201D; role could be time-consuming and, at least at an organizational level, generate decision paralysis or delays. Imagine that a hospital tumor board has to make a decision on a patient&#x00027;s diagnosis and only half of the board members agree with the AIs&#x00027; recommendation; or, that a patient receives an important indication from the AI (e.g., stop taking a medication because wearable devices registered unwanted side effects), but he or she struggles to get in contact with his or her doctor to gain reassurance that this is the right thing to do.</p>
<p>These are examples of the implementation of AIs giving rise to a risk of delay. Though the technical processes could be accelerated, organizational and practical activities could be affected by the complex inclusion of an additional figure in the decision making process.</p>
</sec>
<sec>
<title>&#x0201C;Confusion of the Tongues&#x0201D;</title>
<p>The psychoanalyst Sandor Ferenczi used the expression &#x0201C;confusion of the tongues&#x0201D; to identify the obstacles inherent to communication between adults and children, who are inexorably heterogeneous in their mental representations of relationships and emotional experience. Since then, it has proved an effective expression to refer to interlocutors misinterpreting one another without knowledge.</p>
<p>The expression could be useful when we consider the utilization of AI in medical diagnosis, especially when the latter should be communicated to the patient. The physician is not a simple &#x0201C;translator&#x0201D; of information from the AI to the patient; on the contrary, he or she should play an active role during the process. Let us consider an example: an AI requires that the information on the patient&#x00027;s state is entered according to formats, categories, and languages that it is able to understand and analyze (e.g., data); however, it is possible that not all the relevant information for diagnosis could be transformed as such. How can doctors enter an undefined symptom, a general malaise, or a vague physical discomfort, if the patient himself or herself is hardly able to describe it? Even if trained in the understanding of natural language, the AI will not be able to integrate such information in its original form; this is not related to some sort of malfunction; rather, the AI does not have access to the complex and subtle emotional intelligence abilities that a human doctor can employ when managing a consultation with a patient. Specifically, one risk is that doctors would try to adapt symptoms to AIs&#x00027; language and capabilities, for example by forcing the information coming from the patient into predefined categories; this could be related to an exaggerated faith in the technology itself, which could lead human users to overestimate its abilities (<xref ref-type="bibr" rid="B66">66</xref>).</p>
<p>This could lead patients not being motivated to report doubts, feelings, and personal impressions; indeed, patients can feel when doctors are not really listening to them (<xref ref-type="bibr" rid="B67">67</xref>, <xref ref-type="bibr" rid="B68">68</xref>) and could experience a number of negative emotions ranging from anger to demoralization and a sense of abandonment (<xref ref-type="bibr" rid="B69">69</xref>&#x02013;<xref ref-type="bibr" rid="B71">71</xref>). Such experiences have a detrimental effect both on the success of shared decision making and on therapy effectiveness because the patient will not adhere to the recommendations (<xref ref-type="bibr" rid="B72">72</xref>, <xref ref-type="bibr" rid="B73">73</xref>). In other words, in this way, the new technology would become a source of patient reification, neglecting important elements that only humans&#x00027; emotional intelligence can grasp.</p>
</sec>
<sec>
<title>Role Ambiguity</title>
<p>When the press started to write about Artificial Intelligence in medicine, a number of authoritative medical sources expressed a firm belief: AI will not replace doctors; it will only help them to do their jobs better, especially by analyzing complex medical data. However, we still have no clear idea about the perception of AIs <italic>on the side of patients</italic>; though it is obvious that AIs will not take over doctors&#x00027; work, what will the patients think?</p>
<p>According to a recent survey by PwC on 11,000 patients from twelve different countries, 54% of the interviewees were amenable to the idea of being cured by artificial entities, 38% were against it, and rest were uncertain. The highest rates of acceptance can be traced to developing countries, which are open to any innovation in medicine, while countries used to high-level care systems were more critical.</p>
<p>This points to the need for AI innovation to be communicated and explained to patients in the right way, by justifying its added value but also by avoiding the risk that technology takes the place of human doctors <italic>in patients&#x00027; perception</italic>. For example, as shown by some of the first implementations of the AI system Watson for Oncology by IBM (<xref ref-type="bibr" rid="B74">74</xref>, <xref ref-type="bibr" rid="B75">75</xref>), it may happen that the diagnosis provided by AI does not mirror completely the ideas and assessments implemented by the doctor. It is possible that physicians, patients, and AIs will provide different narratives of diagnosis, prognosis, and treatment. This situation of ambiguity and disagreement could lead the patient to experience uncertainty, not knowing what opinion to follow, who really has authority, and who is actually working to help him or her.</p>
<p>When the doctor has to explain the role of AI in the consultation, he or she will have to reassure the patient that the recourse to such a technology is a desirable strategy to employ to provide the best possible consultation and treatment. However, in the perception of the patient, this communication may contain an implicit message, for example, that <italic>someone else</italic> is doing the doctor&#x00027;s work. A recent case was reported in the news worldwide where a patient and his family received a terminal diagnosis from the doctor on a moving robot interface: the family was shocked by the experience and perceived the use of the machine as an insensitive disservice (<xref ref-type="bibr" rid="B76">76</xref>).</p>
<p>This is an extreme example where a machine has been introduced in a delicate phase of the healthcare process: even if the machine was not acting autonomously, the effect was disastrous. Obviously, it is fundamental to build an empathic relationship with patients, especially with those dealing with the reality of death and grief (<xref ref-type="bibr" rid="B77">77</xref>, <xref ref-type="bibr" rid="B78">78</xref>); while speaking through a machine is &#x0201C;technically the same&#x0201D; as in person, a grieving patient or his caregiver could reasonably feel talking to a robot to be a tragically absurd situation. This example shows how the implementation of technology should be analyzed not only in terms of functionality and technical effectiveness but also from the point of view of patients (<xref ref-type="bibr" rid="B79">79</xref>), taking into account their reaction and its consequences for patient health engagement as well as their commitment to shared decision making.</p>
<p>But if the AI is good as my doctor, or maybe even better, whom should I trust? A similar issue exists in medicine already: when multidisciplinary care is offered, patients may experience anxiety and confusion because they have to schedule appointments with several doctors and are not sure who to refer to with specific questions or who to listen to when recommendations are (or appear to them) contradictory (<xref ref-type="bibr" rid="B80">80</xref>); patients may find it difficult to trust health providers when the recommendation received is unexpected or counterintuitive (<xref ref-type="bibr" rid="B81">81</xref>) and they sometimes consult multiple health professionals, searching for infinite alternative options, as if the cure were some goods to buy, a maladaptive conduct known as &#x0201C;doctor shopping&#x0201D; (<xref ref-type="bibr" rid="B82">82</xref>). In other words, when there is disagreement, doubt, or ambiguity in diagnosis and treatment, its effect on the patient&#x00027;s perception and behavior should be taken into consideration and adequately managed within the consultation.</p>
<p>On the side of the doctor, the description of symptoms, diagnosis, and prognosis given by the AI could be more clear and understandable than the patient&#x00027;s; indeed, AI uses medical language and adopts the perspective of a medical professional, relying on objective data and scientific literature. While a patient could often experience difficulties when trying to explain his or her experience, AI could provide a different &#x0201C;narrative&#x0201D; of the diagnosis that the doctor would perceive as more comprehensible and reassuring. In this case, it is possible that the patient&#x00027;s testimony would be undermined or partially ignored, this way losing trace of the nuances and peculiarity of the actual patient&#x00027;s situation, which only a fine-grained analysis of the subjective testimony could detect.</p>
<p>To sum up, AI could potentially take the role of doctors in patients&#x00027; perception or the opposite.</p>
</sec>
</sec>
<sec sec-type="discussion" id="s4">
<title>Discussion</title>
<p>In this contribution, we tried to identify possible dysfunctional effects of AI&#x00027;s inclusion in medical practice and consultation, conceptualizing them as multiple forms of a &#x0201C;third wheel&#x0201D; effect; besides prefiguring them, it is possible to sketch solutions to the issues to be explored by means of future research.</p>
<p>The three forms of the &#x0201C;third wheel&#x0201D; effect may affect three important areas of the medical consultation: <italic>organizational, communicational</italic>, and <italic>socio-relational</italic> aspects, respectively (see <xref ref-type="fig" rid="F1">Figure 1</xref> for a summary of the concept).</p>
<fig id="F1" position="float">
<label>Figure 1</label>
<caption><p>A summary of the &#x0201C;third wheel&#x0201D; effect with possible related solutions.</p></caption>
<graphic xlink:href="fpubh-08-00117-g0001.tif"/>
</fig>
<p>First, doctors and patients could experience a <italic>decision paralysis</italic>: decisions could be delayed when AIs&#x00027; recommendations are difficult to understand or to explain to patients. <italic>Decision paralysis</italic> may affect the o<italic>rganizational</italic> aspects of healthcare contexts. It refers to how AI technology will be implemented in healthcare systems that may struggle to adapt their timings, procedures, and organizational boundaries to innovation. Tackling these issues entails making plans for the management of AI implementation that take into consideration not only the benefits of AI for the &#x0201C;technical&#x0201D; aspects of medical activities but also the behavior of organizational units toward AI outcomes and how these outcomes fit among any care practice processes.</p>
<p>Second, the presence of AIs could lead to a &#x0201C;<italic>confusion of the tongues</italic>&#x0201D; between doctors and patients, because patients&#x00027; health information could be lost or transformed when adapted to AI&#x00027;s classifications. &#x0201C;<italic>Confusion of the tongues&#x0201D;</italic> affects <italic>communication between doctors and patients</italic>. It refers to the actual possibility for patients and doctors to understand each other and enact a desirable process of shared decision making. The solution to these possible issues involves the design of training resources for doctors that make them aware of how AI implementation could be perceived by patients; desirable practices within patient-doctor communication would include double-checking health-related information to address possible confusion arising from the delivery of relevant information mediated by AI.</p>
<p>Lastly, the involvement of AIs could cause confusion regarding roles in patient-doctor relationships when ambiguity or disagreement arises about treatment recommendations. <italic>Role ambiguity</italic> acts on <italic>socio-relational</italic> aspects in healthcare contexts. It refers to the unwanted effects on trust and quality of relationship related to the addition of an artificial interlocutor within the context. These relational aspects are important prerequisites for achieving a desirable healthcare collaboration. Therefore, solutions to role ambiguity issues would entail proper patient education on the usage of AI in their own healthcare journey, especially when intelligent technology resources interact with them directly, mediating treatment (e.g., eHealth). Future studies on the ethical implementation of AI in medical treatment should consider patients&#x00027; <italic>perception</italic> of these tools and forecast under which conditions patients may feel &#x0201C;put aside&#x0201D; by their doctor because health advice and treatment are delivered by autonomous technologies.</p>
<p>The identification of the psychosocial effects of AI on medical practice is speculative in nature: we should wait until these technologies become actual protagonists in a renovated approach to clinical practice in order to collect data about their effects on the scenario. As a limitation of the present study, we did not report research data; rather, we tried to sketch possible correlates of AI implementation in healthcare based on the literature in health psychology and the social science of technology implementation issues. We believe that consideration of such established phenomena may help pioneers of AI in healthcare to forecast (and possibly manage in advance) issues that will characterize AI implementation as well. Future studies may employ technology acceptance measures to explore health professionals&#x00027; and patients&#x00027; attitudes toward artificial intelligence. Moreover, qualitative research methods (e.g., ethnographic observation) could be employed within the pioneer contexts where AIs start to be used in medical consultation, in order to capture the possible obstacles to practice consistent with the third wheel effect prefigured here.</p>
</sec>
<sec id="s5">
<title>Author Contributions</title>
<p>ST conceptualized the ideas presented in the article and wrote a first draft. ID helped to refine the theoretical framework and edited the manuscript. GP contributed with important intellectual content and supervised the whole process. All authors contributed to revision, read and approved the submitted version.</p>
</sec>
<sec id="s6">
<title>Conflict of Interest</title>
<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.</p>
</sec>
</body>
<back>
<ref-list>
<title>References</title>
<ref id="B1">
<label>1.</label>
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Barrat</surname> <given-names>J</given-names></name></person-group>. <source>Our Final Invention: Artificial Intelligence and the End of the Human Era</source>. <publisher-loc>New York, NY</publisher-loc>: <publisher-name>St. Martin Press</publisher-name>. (<year>2013</year>).</citation></ref>
<ref id="B2">
<label>2.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Li</surname> <given-names>B</given-names></name> <name><surname>Hou</surname> <given-names>B</given-names></name> <name><surname>Yu</surname> <given-names>W</given-names></name> <name><surname>Lu</surname> <given-names>X</given-names></name> <name><surname>Yang</surname> <given-names>C</given-names></name></person-group>. <article-title>Applications of artificial intelligence in intelligent manufacturing: a review</article-title>. <source>Front Inform Technol Electron Eng.</source> (<year>2017</year>) <volume>18</volume>:<fpage>86</fpage>&#x02013;<lpage>96</lpage>. <pub-id pub-id-type="doi">10.1631/FITEE.1601885</pub-id></citation></ref>
<ref id="B3">
<label>3.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Makridakis</surname> <given-names>S</given-names></name></person-group>. <article-title>The forthcoming artificial intelligence (AI) revolution: its impact on society and firms</article-title>. <source>Futures.</source> (<year>2017</year>) <volume>90</volume>:<fpage>46</fpage>&#x02013;<lpage>60</lpage>. <pub-id pub-id-type="doi">10.1016/j.futures.2017.03.006</pub-id></citation></ref>
<ref id="B4">
<label>4.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Angermueller</surname> <given-names>C</given-names></name> <name><surname>P&#x000E4;rnamaa</surname> <given-names>T</given-names></name> <name><surname>Parts</surname> <given-names>L</given-names></name> <name><surname>Stegle</surname> <given-names>O</given-names></name></person-group>. <article-title>Deep learning for computational biology</article-title>. <source>Mol Systems Biol.</source> (<year>2016</year>) <volume>12</volume>:<fpage>878</fpage>. <pub-id pub-id-type="doi">10.15252/msb.20156651</pub-id><pub-id pub-id-type="pmid">27474269</pub-id></citation></ref>
<ref id="B5">
<label>5.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Ghahramani</surname> <given-names>Z</given-names></name></person-group>. <article-title>Probabilistic machine learning and artificial intelligence</article-title>. <source>Nature.</source> (<year>2015</year>) <volume>521</volume>:<fpage>452</fpage>&#x02013;<lpage>9</lpage>. <pub-id pub-id-type="doi">10.1038/nature14541</pub-id><pub-id pub-id-type="pmid">26017444</pub-id></citation></ref>
<ref id="B6">
<label>6.</label>
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Lawrynowicz</surname> <given-names>A</given-names></name> <name><surname>Tresp</surname> <given-names>V</given-names></name></person-group>. <article-title>Introducing Machine Learning</article-title>. In: <person-group person-group-type="editor"><name><surname>Lehmann</surname> <given-names>J</given-names></name> <name><surname>Volker</surname> <given-names>J</given-names></name></person-group>, editors. <source>Perspectives on Ontology Learning</source>, <publisher-loc>Berlin</publisher-loc>: <publisher-name>IOS Press</publisher-name>. (<year>2011</year>)</citation></ref>
<ref id="B7">
<label>7.</label>
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Topol</surname> <given-names>E</given-names></name></person-group>. <source>Deep Medicine</source>. <publisher-loc>New York, NY</publisher-loc>: <publisher-name>Basic Books</publisher-name>. (<year>2019</year>).</citation></ref>
<ref id="B8">
<label>8.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Wartman</surname> <given-names>SA</given-names></name> <name><surname>Combs</surname> <given-names>CD</given-names></name></person-group>. <article-title>Reimagining medical education in the age of AI</article-title>. <source>AMA J Ethics.</source> (<year>2019</year>) <volume>21</volume>:<fpage>146</fpage>&#x02013;<lpage>52</lpage>. <pub-id pub-id-type="doi">10.1001/amajethics.2019.146</pub-id><pub-id pub-id-type="pmid">30794124</pub-id></citation></ref>
<ref id="B9">
<label>9.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Wooster</surname> <given-names>E</given-names></name> <name><surname>Maniate</surname> <given-names>J</given-names></name></person-group>. <article-title>Reimagining our views on medical education: part 1</article-title>. <source>Arch Med Health Sci.</source> (<year>2018</year>) <volume>6</volume>:<fpage>267</fpage>&#x02013;<lpage>9</lpage>. <pub-id pub-id-type="doi">10.4103/amhs.amhs_142_18</pub-id></citation></ref>
<ref id="B10">
<label>10.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Amato</surname> <given-names>F</given-names></name> <name><surname>L&#x000F3;pez</surname> <given-names>A</given-names></name> <name><surname>Pe&#x000F1;a-M&#x000E9;ndez</surname> <given-names>EM</given-names></name> <name><surname>Vanhara</surname> <given-names>P</given-names></name> <name><surname>Hampl</surname> <given-names>A</given-names></name> <name><surname>Havel</surname> <given-names>J</given-names></name></person-group>. <article-title>Artificial neural networks in medical diagnosis</article-title>. <source>J Appl Biomed.</source> (<year>2013</year>) <volume>11</volume>:<fpage>47</fpage>&#x02013;<lpage>58</lpage>. <pub-id pub-id-type="doi">10.2478/v10136-012-0031-x</pub-id></citation></ref>
<ref id="B11">
<label>11.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Jiang</surname> <given-names>F</given-names></name> <name><surname>Jiang</surname> <given-names>Y</given-names></name> <name><surname>Zhi</surname> <given-names>H</given-names></name> <name><surname>Dong</surname> <given-names>Y</given-names></name> <name><surname>Li</surname> <given-names>H</given-names></name> <name><surname>Ma</surname> <given-names>S</given-names></name> <etal/></person-group>. <article-title>Artificial intelligence in healthcare: past, present and future</article-title>. <source>Stroke Vasc Neurol</source>. (<year>2017</year>) <volume>2</volume>:<fpage>230</fpage>&#x02013;<lpage>43</lpage>. <pub-id pub-id-type="doi">10.1136/svn-2017-000101</pub-id><pub-id pub-id-type="pmid">29507784</pub-id></citation></ref>
<ref id="B12">
<label>12.</label>
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Fakoor</surname> <given-names>R</given-names></name> <name><surname>Nazi</surname> <given-names>A</given-names></name> <name><surname>Huber</surname> <given-names>M</given-names></name></person-group>. <article-title>Using deep learning to enhance cancer diagnosis and classification</article-title>. In <source>Proceedings of the international conference on machine learning (Vol. 28)</source>. <publisher-loc>New York, NY</publisher-loc>: <publisher-name>ACM</publisher-name>. (<year>2013</year>).</citation></ref>
<ref id="B13">
<label>13.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hu</surname> <given-names>Z</given-names></name> <name><surname>Tang</surname> <given-names>J</given-names></name> <name><surname>Wang</surname> <given-names>Z</given-names></name> <name><surname>Zhang</surname> <given-names>K</given-names></name> <name><surname>Zhang</surname> <given-names>L</given-names></name> <name><surname>Sun</surname> <given-names>Q</given-names></name></person-group>. <article-title>Deep learning for image-based cancer detection and diagnosis &#x02013; A survey</article-title>. <source>Pattern Recognit</source>. (<year>2018</year>) <volume>83</volume>:<fpage>134</fpage>&#x02013;<lpage>49</lpage>. <pub-id pub-id-type="doi">10.1016/j.patcog.2018.05.014</pub-id></citation></ref>
<ref id="B14">
<label>14.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Esteva</surname> <given-names>A</given-names></name> <name><surname>Kuprel</surname> <given-names>B</given-names></name> <name><surname>Novoa</surname> <given-names>RA</given-names></name> <name><surname>Ko</surname> <given-names>J</given-names></name> <name><surname>Swetter</surname> <given-names>SM</given-names></name> <name><surname>Blau</surname> <given-names>HM</given-names></name> <etal/></person-group>. <article-title>Dermatologist-level classification of skin cancer with deep neural networks</article-title>. <source>Nature.</source> (<year>2017</year>) <volume>542</volume>:<fpage>115</fpage>&#x02013;<lpage>8</lpage>. <pub-id pub-id-type="doi">10.1038/nature21056</pub-id><pub-id pub-id-type="pmid">28117445</pub-id></citation></ref>
<ref id="B15">
<label>15.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>De Fauw</surname> <given-names>J</given-names></name> <name><surname>Ledsam</surname> <given-names>JR</given-names></name> <name><surname>Romera-Paredes</surname> <given-names>B</given-names></name> <name><surname>Nikolov</surname> <given-names>S</given-names></name> <name><surname>Tomasev</surname> <given-names>N</given-names></name> <name><surname>Blackwell</surname> <given-names>S</given-names></name> <etal/></person-group>. <article-title>Clinically applicable deep learning for diagnosis and referral in retinal disease</article-title>. <source>Nat Med</source>. (<year>2018</year>) <volume>24</volume>:<fpage>1342</fpage>&#x02013;<lpage>50</lpage>. <pub-id pub-id-type="doi">10.1038/s41591-018-0107-6</pub-id><pub-id pub-id-type="pmid">30104768</pub-id></citation></ref>
<ref id="B16">
<label>16.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Gargeya</surname> <given-names>R</given-names></name> <name><surname>Leng</surname> <given-names>T</given-names></name></person-group>. <article-title>Automated identification of diabetic retinopathy using deep learning</article-title>. <source>Ophthalmology.</source> (<year>2017</year>) <volume>124</volume>:<fpage>962</fpage>&#x02013;<lpage>9</lpage>. <pub-id pub-id-type="doi">10.1016/j.ophtha.2017.02.008</pub-id><pub-id pub-id-type="pmid">28359545</pub-id></citation></ref>
<ref id="B17">
<label>17.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Gulshan</surname> <given-names>V</given-names></name> <name><surname>Peng</surname> <given-names>L</given-names></name> <name><surname>Coram</surname> <given-names>M</given-names></name> <name><surname>Stumpe</surname> <given-names>MC</given-names></name> <name><surname>Wu</surname> <given-names>D</given-names></name> <name><surname>Narayanaswamy</surname> <given-names>A</given-names></name> <etal/></person-group>. (<year>2016</year>). <article-title>Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs</article-title>. <source>JAMA.</source> <volume>316</volume>:<fpage>2402</fpage>&#x02013;<lpage>10</lpage>. <pub-id pub-id-type="doi">10.1001/jama.2016.17216</pub-id><pub-id pub-id-type="pmid">27898976</pub-id></citation></ref>
<ref id="B18">
<label>18.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Abiyev</surname> <given-names>RH</given-names></name> <name><surname>Ma&#x00027;aitah</surname> <given-names>MKS</given-names></name></person-group>. <article-title>Deep convolutional neural networks for chest diseases detection</article-title>. <source>J Health Eng.</source> (<year>2018</year>) <volume>2018</volume>:<fpage>4168538</fpage>. <pub-id pub-id-type="doi">10.1155/2018/4168538</pub-id><pub-id pub-id-type="pmid">30154989</pub-id></citation></ref>
<ref id="B19">
<label>19.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Litjens</surname> <given-names>G</given-names></name> <name><surname>S&#x000E1;nchez</surname> <given-names>CI</given-names></name> <name><surname>Timofeeva</surname> <given-names>N</given-names></name> <name><surname>Hermsen</surname> <given-names>M</given-names></name> <name><surname>Nagtegaal</surname> <given-names>I</given-names></name> <name><surname>Kovacs</surname> <given-names>I</given-names></name> <etal/></person-group>. <article-title>Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis</article-title>. <source>Sci Rep.</source> (<year>2016</year>) <volume>6</volume>:<fpage>26286</fpage>. <pub-id pub-id-type="doi">10.1038/srep26286</pub-id><pub-id pub-id-type="pmid">27212078</pub-id></citation></ref>
<ref id="B20">
<label>20.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Bahaa</surname> <given-names>K</given-names></name> <name><surname>Noor</surname> <given-names>G</given-names></name> <name><surname>Yousif</surname> <given-names>Y</given-names></name></person-group>. <article-title>The Artificial Intelligence Approach for Diagnosis, Treatment Modelling in Orthodontic</article-title>. In <person-group person-group-type="editor"><name><surname>Naretto</surname> <given-names>S.</given-names></name></person-group>, editor, <source>Principles in Contemporary Orthodontics</source>, InTech (<year>2011</year>).</citation></ref>
<ref id="B21">
<label>21.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Ramesh</surname> <given-names>AN</given-names></name> <name><surname>Kambhampati</surname> <given-names>C</given-names></name> <name><surname>Monson</surname> <given-names>JR</given-names></name> <name><surname>Drew</surname> <given-names>PJ</given-names></name></person-group>. <article-title>Artificial intelligence in medicine</article-title>. <source>Ann R Coll Surg Engl</source>. (<year>2004</year>) <volume>86</volume>:<fpage>334</fpage>&#x02013;<lpage>8</lpage>. <pub-id pub-id-type="doi">10.1308/147870804290</pub-id><pub-id pub-id-type="pmid">15333167</pub-id></citation></ref>
<ref id="B22">
<label>22.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Gubbi</surname> <given-names>S</given-names></name> <name><surname>Hamet</surname> <given-names>P</given-names></name> <name><surname>Tremblay</surname> <given-names>J</given-names></name> <name><surname>Koch</surname> <given-names>C</given-names></name> <name><surname>Hannah-Shmouni</surname> <given-names>F</given-names></name></person-group>. <article-title>Artificial intelligence and machine learning in endocrinology and metabolism: the dawn of a new era</article-title>. <source>Front Endocrinol.</source> (<year>2019</year>) <volume>10</volume>:<fpage>185</fpage>. <pub-id pub-id-type="doi">10.3389/fendo.2019.00185</pub-id><pub-id pub-id-type="pmid">30984108</pub-id></citation></ref>
<ref id="B23">
<label>23.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Somashekhar</surname> <given-names>SP</given-names></name> <name><surname>Sep&#x000FA;lveda</surname> <given-names>MJ</given-names></name> <name><surname>Puglielli</surname> <given-names>S</given-names></name> <name><surname>Norden</surname> <given-names>AD</given-names></name> <name><surname>Shortliffe</surname> <given-names>EH</given-names></name> <name><surname>Rohit Kumar</surname> <given-names>C</given-names></name> <name><surname>Ramya</surname> <given-names>Y</given-names></name></person-group>. <article-title>Watson for oncology and breast cancer treatment recommendations: agreement with an expert multidisciplinary tumor board</article-title>. <source>Ann Oncology</source>. (<year>2018</year>) <volume>29</volume>:<fpage>418</fpage>&#x02013;<lpage>23</lpage>. <pub-id pub-id-type="doi">10.1093/annonc/mdx781</pub-id><pub-id pub-id-type="pmid">29324970</pub-id></citation></ref>
<ref id="B24">
<label>24.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Liu</surname> <given-names>C</given-names></name> <name><surname>Liu</surname> <given-names>X</given-names></name> <name><surname>Wu</surname> <given-names>F</given-names></name> <name><surname>Xie</surname> <given-names>M</given-names></name> <name><surname>Feng</surname> <given-names>Y</given-names></name> <name><surname>Hu</surname> <given-names>C</given-names></name></person-group>. <article-title>Using artificial intelligence (Watson for Oncology) for treatment recommendations amongst Chinese patients with lung cancer: feasibility study</article-title>. <source>J Med Internet Res.</source> (<year>2018</year>) <volume>20</volume>:<fpage>e11087</fpage>. <pub-id pub-id-type="doi">10.2196/1108</pub-id><pub-id pub-id-type="pmid">30257820</pub-id></citation></ref>
<ref id="B25">
<label>25.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Itoh</surname> <given-names>H</given-names></name> <name><surname>Hayashi</surname> <given-names>K</given-names></name> <name><surname>Miyashita</surname> <given-names>K</given-names></name></person-group>. <article-title>Pre-emptive medicine for hypertension and its prospects</article-title>. <source>Hypertens Res.</source> (<year>2019</year>) <volume>42</volume>:<fpage>301</fpage>&#x02013;<lpage>5</lpage>. <pub-id pub-id-type="doi">10.1038/s41440-018-0177-3</pub-id><pub-id pub-id-type="pmid">30560889</pub-id></citation></ref>
<ref id="B26">
<label>26.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Triberti</surname> <given-names>S</given-names></name> <name><surname>Barello</surname> <given-names>S</given-names></name></person-group>. <article-title>The quest for engaging AmI: patient engagement and experience design tools to promote effective assisted living</article-title>. <source>J Biomed Inform</source>. <volume>63</volume>:<fpage>150</fpage>&#x02013;<lpage>6</lpage>. <pub-id pub-id-type="doi">10.1016/j.jbi.2016.08.010</pub-id><pub-id pub-id-type="pmid">27515924</pub-id></citation></ref>
<ref id="B27">
<label>27.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Davenport</surname> <given-names>T</given-names></name> <name><surname>Kalakota</surname> <given-names>R</given-names></name></person-group>. <article-title>The potential for artificial intelligence in healthcare</article-title>. <source>Future Healthc J.</source> (<year>2019</year>) <volume>6</volume>:<fpage>94</fpage>. <pub-id pub-id-type="doi">10.7861/futurehosp.6-2-94</pub-id><pub-id pub-id-type="pmid">31363513</pub-id></citation></ref>
<ref id="B28">
<label>28.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Triberti</surname> <given-names>S</given-names></name> <name><surname>Durosini</surname> <given-names>I</given-names></name> <name><surname>Curigliano</surname> <given-names>G</given-names></name> <name><surname>Pravettoni</surname> <given-names>G</given-names></name></person-group>. <article-title>Is explanation a marketing problem? the quest for trust in artificial intelligence and two conflicting solutions</article-title>. <source>Public Health Genomics</source>. (<year>2020</year>). <pub-id pub-id-type="doi">10.1159/000506014</pub-id>. [Epub ahead of print].<pub-id pub-id-type="pmid">31991406</pub-id></citation></ref>
<ref id="B29">
<label>29.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kalton</surname> <given-names>A</given-names></name> <name><surname>Falconer</surname> <given-names>E</given-names></name> <name><surname>Docherty</surname> <given-names>J</given-names></name> <name><surname>Alevras</surname> <given-names>D</given-names></name> <name><surname>Brann</surname> <given-names>D</given-names></name> <name><surname>Johnson</surname> <given-names>K</given-names></name></person-group>. <article-title>Multi-agent-based simulation of a complex ecosystem of mental health care</article-title>. <source>J Med Syst.</source> (<year>2016</year>) <volume>40</volume>:<fpage>1</fpage>&#x02013;<lpage>8</lpage>. <pub-id pub-id-type="doi">10.1007/s10916-015-0374-4</pub-id><pub-id pub-id-type="pmid">26590977</pub-id></citation></ref>
<ref id="B30">
<label>30.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hamet</surname> <given-names>P</given-names></name> <name><surname>Tremblay</surname> <given-names>J</given-names></name></person-group>. <article-title>Artificial intelligence in medicine</article-title>. <source>Metabolism</source>. (<year>2017</year>) <volume>69</volume>:<fpage>S36</fpage>&#x02013;<lpage>40</lpage>. <pub-id pub-id-type="doi">10.1016/j.metabol.2017.01.011</pub-id></citation></ref>
<ref id="B31">
<label>31.</label>
<citation citation-type="web"><person-group person-group-type="author"><name><surname>Gunning</surname> <given-names>D</given-names></name></person-group>. <source>Explainable Artificial Intelligence (XAI): Program Update Novmeber 2017</source>. Defense Advanced Research Projects Agency (DARPA) (<year>2017</year>). Retrieved from: <ext-link ext-link-type="uri" xlink:href="https://www.darpa.mil/attachments/XAIProgramUpdate.pdf">https://www.darpa.mil/attachments/XAIProgramUpdate.pdf</ext-link></citation></ref>
<ref id="B32">
<label>32.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Edwards</surname> <given-names>L</given-names></name> <name><surname>Veale</surname> <given-names>M</given-names></name></person-group>. <article-title>Enslaving the algorithm: from a right to an explanation to a right to better decisions?</article-title>. <source>IEEE Secur Priv.</source> (<year>2018</year>) <volume>16</volume>:<fpage>43</fpage>&#x02013;<lpage>54</lpage>. <pub-id pub-id-type="doi">10.1109/MSP.2018.2701152</pub-id></citation></ref>
<ref id="B33">
<label>33.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Miller</surname> <given-names>T</given-names></name></person-group>. <article-title>Explanation in artificial intelligence: insights from the social sciences</article-title>. <source>Artif Intell.</source> (<year>2019</year>) <volume>267</volume>, <fpage>1</fpage>&#x02013;<lpage>38</lpage>. <pub-id pub-id-type="doi">10.1016/j.artint.2018.07.007</pub-id></citation></ref>
<ref id="B34">
<label>34.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Miller</surname> <given-names>T</given-names></name> <name><surname>Hower</surname> <given-names>P</given-names></name> <name><surname>Sonenberg</surname> <given-names>L</given-names></name></person-group>. <article-title>Explainable ai: beware of inmates running the asylum or: how i learnt to stop worrying and love the social and behavioural sciences</article-title>. In: <source>Proc. IJCAI Workshop Explainable AI (XAI), 2017</source>,. (<year>2017</year>) pp. <fpage>36</fpage>&#x02013;<lpage>42</lpage>. <pub-id pub-id-type="doi">10.1016/j.foodchem.2017.11.091</pub-id></citation></ref>
<ref id="B35">
<label>35.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Vellido</surname> <given-names>A</given-names></name></person-group>. <article-title>The importance of interpretability and visualization in machine learning for applications in medicine and health care</article-title>. <source>Neural Comput Appl.</source> (<year>2019</year>) <fpage>1</fpage>&#x02013;<lpage>15</lpage>, <pub-id pub-id-type="doi">10.1007/s00521-019-04051-w</pub-id></citation></ref>
<ref id="B36">
<label>36.</label>
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Pravettoni</surname> <given-names>G</given-names></name> <name><surname>Triberti</surname> <given-names>S</given-names></name></person-group>. <source>Il Medico 4.0</source>. <publisher-loc>Milan</publisher-loc>: <publisher-name>EDRA</publisher-name>. (<year>2019</year>).</citation></ref>
<ref id="B37">
<label>37.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>L&#x000E9;gar&#x000E9;</surname> <given-names>F</given-names></name> <name><surname>Thompson-Leduc</surname> <given-names>P</given-names></name></person-group>. <article-title>Twelve myths about shared decision making</article-title>. <source>Patient Educ Couns.</source> (<year>2014</year>) <volume>96</volume>:<fpage>281</fpage>&#x02013;<lpage>6</lpage>. <pub-id pub-id-type="doi">10.1016/j.pec.2014.06.014</pub-id><pub-id pub-id-type="pmid">25034637</pub-id></citation></ref>
<ref id="B38">
<label>38.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Barry</surname> <given-names>MJ</given-names></name> <name><surname>Edgman-Levitan</surname> <given-names>S</given-names></name></person-group>. <article-title>Shared decision making &#x02014; the pinnacle of patient-centered care</article-title>. <source>N Engl J Med.</source> (<year>2012</year>) <volume>366</volume>, <fpage>780</fpage>&#x02013;<lpage>1</lpage>. <pub-id pub-id-type="doi">10.1056/NEJMp1109283</pub-id><pub-id pub-id-type="pmid">22375967</pub-id></citation></ref>
<ref id="B39">
<label>39.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Simpson</surname> <given-names>M</given-names></name> <name><surname>Buckman</surname> <given-names>R</given-names></name> <name><surname>Stewart</surname> <given-names>M</given-names></name> <name><surname>Maguire</surname> <given-names>P</given-names></name> <name><surname>Lipkin</surname> <given-names>M</given-names></name> <name><surname>Novack</surname> <given-names>D</given-names></name> <name><surname>Till</surname> <given-names>J</given-names></name></person-group>. <article-title>Doctor-patient communication: the Toronto consensus statement</article-title>. <source>BMJ.</source> (<year>1991</year>) <volume>303</volume>:<fpage>1385</fpage>&#x02013;<lpage>7</lpage>. <pub-id pub-id-type="doi">10.1136/bmj.303.6814.1385</pub-id><pub-id pub-id-type="pmid">1760608</pub-id></citation></ref>
<ref id="B40">
<label>40.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Stewart</surname> <given-names>MA</given-names></name></person-group>. <article-title>Effective physician-patient communication and health outcomes: a review</article-title>. <source>CMAJ</source>. (<year>1995</year>) <volume>152</volume>, <fpage>1423</fpage>&#x02013;<lpage>1433</lpage><pub-id pub-id-type="pmid">7728691</pub-id></citation></ref>
<ref id="B41">
<label>41.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Bragazzi</surname> <given-names>NL</given-names></name></person-group>. <article-title>From P0 to P6 medicine, a model of highly participatory, narrative, interactive, and augmented medicine: Some considerations on Salvatore Iaconesi&#x00027;s clinical story</article-title>. <source>Patient Prefer Adherence</source>. (<year>2013</year>) <volume>7</volume>:<fpage>353</fpage>&#x02013;<lpage>9</lpage>. <pub-id pub-id-type="doi">10.2147/PPA.S38578</pub-id><pub-id pub-id-type="pmid">23650443</pub-id></citation></ref>
<ref id="B42">
<label>42.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>M&#x000FC;ller-Engelmann</surname> <given-names>M</given-names></name> <name><surname>Keller</surname> <given-names>H</given-names></name> <name><surname>Donner-Banzhoff</surname> <given-names>N</given-names></name> <name><surname>Krones</surname> <given-names>T</given-names></name></person-group>. <article-title>Shared decision making in medicine: the influence of situational treatment factors</article-title>. <source>Patient Educ Couns.</source> (<year>2011</year>) <volume>82</volume>:<fpage>240</fpage>&#x02013;<lpage>6</lpage>. <pub-id pub-id-type="doi">10.1016/j.pec.2010.04.028</pub-id><pub-id pub-id-type="pmid">20542403</pub-id></citation></ref>
<ref id="B43">
<label>43.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Spencer</surname> <given-names>KL</given-names></name></person-group>. <article-title>Transforming patient compliance research in an era of biomedicalization</article-title>. <source>J Health Soc Behav.</source> (<year>2018</year>) <volume>59</volume>. <fpage>170</fpage>&#x02013;<lpage>84</lpage>. <pub-id pub-id-type="doi">10.1177/0022146518756860</pub-id><pub-id pub-id-type="pmid">29443620</pub-id></citation></ref>
<ref id="B44">
<label>44.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Ford</surname> <given-names>S</given-names></name> <name><surname>Schofield</surname> <given-names>T</given-names></name> <name><surname>Hope</surname> <given-names>T</given-names></name></person-group>. <article-title>What are the ingredients for a successful evidence-based patient choice consultation?: a qualitative study</article-title>. <source>Soc Sci Med.</source> (<year>2003</year>) <volume>56</volume>:<fpage>589</fpage>&#x02013;<lpage>602</lpage>. <pub-id pub-id-type="doi">10.1016/S0277-9536(02)00056-4</pub-id><pub-id pub-id-type="pmid">12570976</pub-id></citation></ref>
<ref id="B45">
<label>45.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Marzorati</surname> <given-names>C</given-names></name> <name><surname>Pravettoni</surname> <given-names>G</given-names></name></person-group>. <article-title>Value as the key concept in the health care system: how it has influenced medical practice and clinical decision-making processes</article-title>. <source>J Multidiscip Healthc.</source> (<year>2017</year>) <volume>10</volume>:<fpage>101</fpage>&#x02013;<lpage>6</lpage>. <pub-id pub-id-type="doi">10.2147/JMDH.S122383</pub-id><pub-id pub-id-type="pmid">28356752</pub-id></citation></ref>
<ref id="B46">
<label>46.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Fioretti</surname> <given-names>C</given-names></name> <name><surname>Mazzocco</surname> <given-names>K</given-names></name> <name><surname>Riva</surname> <given-names>S</given-names></name> <name><surname>Oliveri</surname> <given-names>S</given-names></name> <name><surname>Masiero</surname> <given-names>M</given-names></name> <name><surname>Pravettoni</surname> <given-names>G</given-names></name></person-group>. (<year>2016</year>). <article-title>Research studies on patients&#x00027; illness experience using the narrative medicine approach: a systematic review</article-title>. <source>BMJ Open</source>. <fpage>6</fpage>:<lpage>e011220</lpage>. <pub-id pub-id-type="doi">10.1136/bmjopen-2016-011220</pub-id><pub-id pub-id-type="pmid">27417197</pub-id></citation></ref>
<ref id="B47">
<label>47.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Chawla</surname> <given-names>NV</given-names></name> <name><surname>Davis</surname> <given-names>DA</given-names></name></person-group>. <article-title>Bringing big data to personalized healthcare: A patient-centered framework</article-title>. <source>J Gen Intern Med.</source> (<year>2013</year>) <volume>28</volume>:<fpage>660</fpage>&#x02013;<lpage>5</lpage>. <pub-id pub-id-type="doi">10.1007/s11606-013-2455-8</pub-id><pub-id pub-id-type="pmid">23797912</pub-id></citation></ref>
<ref id="B48">
<label>48.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Corso</surname> <given-names>G</given-names></name> <name><surname>Magnoni</surname> <given-names>F</given-names></name> <name><surname>Provenzano</surname> <given-names>E</given-names></name> <name><surname>Girardi</surname> <given-names>A</given-names></name> <name><surname>Iorfida</surname> <given-names>M</given-names></name> <name><surname>De Scalzi</surname> <given-names>AM</given-names></name> <etal/></person-group>. <article-title>Multicentric breast cancer with heterogeneous histopathology: a multidisciplinary review</article-title>. <source>Future Oncol.</source> (<year>2020</year>) <volume>16</volume>:<fpage>395</fpage>&#x02013;<lpage>412</lpage>. <pub-id pub-id-type="doi">10.2217/fon-2019-0540</pub-id><pub-id pub-id-type="pmid">32026709</pub-id></citation></ref>
<ref id="B49">
<label>49.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Riva</surname> <given-names>S</given-names></name> <name><surname>Antonietti</surname> <given-names>A</given-names></name> <name><surname>Iannello</surname> <given-names>P</given-names></name> <name><surname>Pravettoni</surname> <given-names>G</given-names></name></person-group>. <article-title>What are judgment skills in health literacy? A psycho-cognitive perspective of judgment and decision-making research</article-title>. <source>Patient Prefer Adherence.</source> (<year>2015</year>) <volume>9</volume>:<fpage>1677</fpage>&#x02013;<lpage>86</lpage>. <pub-id pub-id-type="doi">10.2147/PPA.S90207</pub-id><pub-id pub-id-type="pmid">26648700</pub-id></citation></ref>
<ref id="B50">
<label>50.</label>
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Terry</surname> <given-names>DJ</given-names></name> <name><surname>Hogg</surname> <given-names>MA</given-names></name></person-group>. <source>Attitudes, Behavior, and Social Context: The Role of Norms and Group Membership</source>. <publisher-loc>Hove, UK</publisher-loc>: <publisher-name>Psychology Press</publisher-name> (<year>1999</year>).</citation></ref>
<ref id="B51">
<label>51.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Huryk</surname> <given-names>LA</given-names></name></person-group>. <article-title>Factors influencing nurses&#x00027; attitudes towards healthcare information technology</article-title>. <source>J Nurs Manag.</source> (<year>2010</year>) <volume>18</volume>:<fpage>606</fpage>&#x02013;<lpage>12</lpage>. <pub-id pub-id-type="doi">10.1111/j.1365-2834.2010.01084.x</pub-id><pub-id pub-id-type="pmid">20636510</pub-id></citation></ref>
<ref id="B52">
<label>52.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Jacobs</surname> <given-names>RJ</given-names></name> <name><surname>Iqbal</surname> <given-names>H</given-names></name> <name><surname>Rana</surname> <given-names>AM</given-names></name> <name><surname>Rana</surname> <given-names>Z</given-names></name> <name><surname>Kane</surname> <given-names>MN</given-names></name></person-group>. (<year>2017</year>). <article-title>Predictors of osteopathic medical students&#x00027; readiness to use health information technology</article-title>. <source>J Am Osteopath Assoc.</source> <fpage>117</fpage>:<lpage>773</lpage>. <pub-id pub-id-type="doi">10.7556/jaoa.2017.149</pub-id><pub-id pub-id-type="pmid">29181520</pub-id></citation></ref>
<ref id="B53">
<label>53.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kossman</surname> <given-names>SP</given-names></name> <name><surname>Scheidenhelm</surname> <given-names>SL</given-names></name></person-group>. <article-title>Nurses&#x00027; perceptions of the impact of electronic health records on work and patient outcomes</article-title>. <source>CIN &#x02013; Comput Inform Nurs.</source> (<year>2008</year>) <volume>26</volume>:<fpage>69</fpage>&#x02013;<lpage>77</lpage>. <pub-id pub-id-type="doi">10.1097/01.NCN.0000304775.40531.67</pub-id><pub-id pub-id-type="pmid">18317257</pub-id></citation></ref>
<ref id="B54">
<label>54.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Weber</surname> <given-names>S</given-names></name></person-group>. <article-title>A qualitative analysis of how advanced practice nurses use clinical decision support systems</article-title>. <source>J Am Acad Nurse Pract.</source> (<year>2007</year>) <volume>19</volume>:<fpage>652</fpage>&#x02013;<lpage>67</lpage>. <pub-id pub-id-type="doi">10.1111/j.1745-7599.2007.00266.x</pub-id><pub-id pub-id-type="pmid">18042131</pub-id></citation></ref>
<ref id="B55">
<label>55.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Liberati</surname> <given-names>EG</given-names></name> <name><surname>Ruggiero</surname> <given-names>F</given-names></name> <name><surname>Galuppo</surname> <given-names>L</given-names></name> <name><surname>Gorli</surname> <given-names>M</given-names></name> <name><surname>Gonz&#x000E1;lez-Lorenzo</surname> <given-names>M</given-names></name> <name><surname>Maraldi</surname> <given-names>M</given-names></name> <etal/></person-group>. <article-title>What hinders the uptake of computerized decision support systems in hospitals? a qualitative study and framework for implementation</article-title>. <source>Implemen Sci.</source> (<year>2017</year>) <volume>12</volume>:<fpage>113</fpage>. <pub-id pub-id-type="doi">10.1186/s13012-017-0644-2</pub-id><pub-id pub-id-type="pmid">28915822</pub-id></citation></ref>
<ref id="B56">
<label>56.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Cho</surname> <given-names>J</given-names></name> <name><surname>Park</surname> <given-names>D</given-names></name> <name><surname>Lee</surname> <given-names>HE</given-names></name></person-group>. <article-title>Cognitive factors of using health apps: systematic analysis of relationships among health consciousness, health information orientation, eHealth literacy, and health app use efficacy</article-title>. <source>J Med Internet Res.</source> (<year>2014</year>) <volume>16</volume>:<fpage>e125</fpage>. <pub-id pub-id-type="doi">10.2196/jmir.3283</pub-id><pub-id pub-id-type="pmid">24824062</pub-id></citation></ref>
<ref id="B57">
<label>57.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kelley</surname> <given-names>H</given-names></name> <name><surname>Chiasson</surname> <given-names>M</given-names></name> <name><surname>Downey</surname> <given-names>A</given-names></name> <name><surname>Pacaud</surname> <given-names>D</given-names></name> <name><surname>Payton</surname> <given-names>FC</given-names></name> <name><surname>Par&#x000E9;</surname> <given-names>G</given-names></name> <etal/></person-group>. <article-title>The clinical impact of ehealth on the self-management of diabetes: a double adoption perspective</article-title>. <source>J Assoc Inf Syst</source>. (<year>2011</year>) <volume>12</volume>:<fpage>208</fpage>&#x02013;<lpage>34</lpage>. <pub-id pub-id-type="doi">10.17705/1jais.00263</pub-id></citation></ref>
<ref id="B58">
<label>58.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Afra</surname> <given-names>P</given-names></name> <name><surname>Bruggers</surname> <given-names>CS</given-names></name> <name><surname>Sweney</surname> <given-names>M</given-names></name> <name><surname>Fagatele</surname> <given-names>L</given-names></name> <name><surname>Alavi</surname> <given-names>F</given-names></name> <name><surname>Greenwald</surname> <given-names>M</given-names></name> <etal/></person-group>. <article-title>Mobile software as a medical device (SaMD) for the treatment of epilepsy: development of digital therapeutics comprising behavioral and music-based interventions for neurological disorders</article-title>. <source>Front Human Neurosci.</source> <volume>12</volume>:<fpage>171</fpage>. <pub-id pub-id-type="doi">10.3389/fnhum.2018.00171</pub-id><pub-id pub-id-type="pmid">29780310</pub-id></citation></ref>
<ref id="B59">
<label>59.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Sverdlov</surname> <given-names>O</given-names></name> <name><surname>van Dam</surname> <given-names>J</given-names></name> <name><surname>Hannesdottir</surname> <given-names>K</given-names></name> <name><surname>Thornton-Wells</surname> <given-names>T</given-names></name></person-group>. <article-title>Digital therapeutics: an integral component of digital innovation in drug development</article-title>. <source>Clin Pharmacol Ther</source>. (<year>2018</year>) <volume>104</volume>:<fpage>72</fpage>&#x02013;<lpage>80</lpage>, <pub-id pub-id-type="doi">10.1002/cpt.1036</pub-id><pub-id pub-id-type="pmid">29377057</pub-id></citation></ref>
<ref id="B60">
<label>60.</label>
<citation citation-type="web"><person-group person-group-type="author"><collab>US Food and Drug Administration</collab></person-group>. <source>Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML) Based Software as a Medical Device (SAMD)&#x02014;Discussion Paper and Request for Feedback</source>. (<year>2019</year>). Available online at: <ext-link ext-link-type="uri" xlink:href="https://www.fda.gov/files/medical%20devices/published/US-FDA-Artificial-Intelligence-and-Machine-Learning-Discussion-Paper.pdf">https://www.fda.gov/files/medical%20devices/published/US-FDA-Artificial-Intelligence-and-Machine-Learning-Discussion-Paper.pdf</ext-link></citation></ref>
<ref id="B61">
<label>61.</label>
<citation citation-type="web"><person-group person-group-type="author"><collab>Pear Therapeutics Inc</collab></person-group>. <source>reSET for Substance Use Disorder</source>. (<year>2018</year>). Retrieved from: <ext-link ext-link-type="uri" xlink:href="https://peartherapeutics.com/reset/">https://peartherapeutics.com/reset/</ext-link> (accessed April 16, 2020).</citation></ref>
<ref id="B62">
<label>62.</label>
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Budney</surname> <given-names>AJ</given-names></name> <name><surname>Borodovsky</surname> <given-names>JT</given-names></name> <name><surname>Marsch</surname> <given-names>LA</given-names></name> <name><surname>Lord</surname> <given-names>SE</given-names></name></person-group>. <article-title>Technological innovations in addiction treatment</article-title>. In: <person-group person-group-type="editor"><name><surname>Danovitch</surname> <given-names>I</given-names></name> <name><surname>Mooney</surname> <given-names>L</given-names></name></person-group>, editors. <source>The Assessment and Treatment of Addiction</source>. <publisher-loc>St. Louis, MO</publisher-loc>: <publisher-name>Elsevier</publisher-name> (<year>2019</year>). p. <fpage>75</fpage>&#x02013;<lpage>90</lpage>.</citation></ref>
<ref id="B63">
<label>63.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Cosson</surname> <given-names>B</given-names></name> <name><surname>Graham</surname> <given-names>E</given-names></name></person-group>. <article-title>I felt like a third wheel&#x00027;: Fathers&#x00027; stories of exclusion from the &#x02018;parenting team</article-title>. <source>J Family Studi.</source> (<year>2012</year>) <volume>18</volume>:<fpage>121</fpage>&#x02013;<lpage>9</lpage>. <pub-id pub-id-type="doi">10.5172/jfs.2012.18.2-3.121</pub-id></citation></ref>
<ref id="B64">
<label>64.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Clayton</surname> <given-names>RB</given-names></name></person-group>. <article-title>The third wheel: The impact of Twitter use on relationship infidelity and divorce</article-title>. <source>Cyberpsychol Behav Soc Netw.</source> (<year>2014</year>) <volume>17</volume>:<fpage>425</fpage>&#x02013;<lpage>30</lpage>. <pub-id pub-id-type="doi">10.1089/cyber.2013.0570</pub-id><pub-id pub-id-type="pmid">24690067</pub-id></citation></ref>
<ref id="B65">
<label>65.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Halpern</surname> <given-names>D</given-names></name> <name><surname>Katz</surname> <given-names>JE</given-names></name> <name><surname>Carril</surname> <given-names>C</given-names></name></person-group>. <article-title>The online ideal persona vs. the jealousy effect: two explanations of why selfies are associated with lower-quality romantic relationships</article-title>. <source>Telemat Inform.</source> (<year>2017</year>) <volume>34</volume>:<fpage>114</fpage>&#x02013;<lpage>23</lpage>. <pub-id pub-id-type="doi">10.1016/j.tele.2016.04.014</pub-id></citation></ref>
<ref id="B66">
<label>66.</label>
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Jochemsen</surname> <given-names>H</given-names></name></person-group>. (<year>2008</year>). <article-title>Medical practice as the primary context for medical ethics</article-title>. In <person-group person-group-type="editor"><name><surname>Weisstub</surname> <given-names>D.</given-names></name> <name><surname>Diaz Pintos</surname> <given-names>G.</given-names></name></person-group> Editors, <source>Autonomy and human rights in health care. An international perspective</source>. <publisher-loc>Dordrecht</publisher-loc>: <publisher-name>Springer</publisher-name>.</citation></ref>
<ref id="B67">
<label>67.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Charon</surname> <given-names>R</given-names></name></person-group>. <article-title>Narrative medicine as witness for the self-telling body</article-title>. <source>J Appl Commun Res</source>. <volume>37</volume>:<fpage>118</fpage>&#x02013;<lpage>31</lpage>. <pub-id pub-id-type="doi">10.1080/00909880902792248</pub-id></citation></ref>
<ref id="B68">
<label>68.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Smith</surname> <given-names>SK</given-names></name> <name><surname>Dixon</surname> <given-names>A</given-names></name> <name><surname>Trevena</surname> <given-names>L</given-names></name> <name><surname>Nutbeam</surname> <given-names>D</given-names></name> <name><surname>McCaffery</surname> <given-names>KJ</given-names></name></person-group>. <article-title>Exploring patient involvement in healthcare decision making across different education and functional health literacy groups</article-title>. <source>Soc Sci Med.</source> (<year>2009</year>) <volume>69</volume>:<fpage>1805</fpage>&#x02013;<lpage>12</lpage>. <pub-id pub-id-type="doi">10.1016/j.socscimed.2009.09.056</pub-id><pub-id pub-id-type="pmid">19846245</pub-id></citation></ref>
<ref id="B69">
<label>69.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Harris</surname> <given-names>CR</given-names></name> <name><surname>Darby</surname> <given-names>RS</given-names></name></person-group>. <article-title>Shame in physician-patient interactions: patient perspectives</article-title>. <source>Basic App Soc Psychol</source>. (<year>2009</year>) <volume>31</volume>:<fpage>325</fpage>&#x02013;<lpage>34</lpage>. <pub-id pub-id-type="doi">10.1080/01973530903316922</pub-id></citation></ref>
<ref id="B70">
<label>70.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kee</surname> <given-names>JWY</given-names></name> <name><surname>Khoo</surname> <given-names>HS</given-names></name> <name><surname>Lim</surname> <given-names>I</given-names></name> <name><surname>Koh</surname> <given-names>MYH</given-names></name></person-group>. (<year>2017</year>). <article-title>Communication skills in patient-doctor interactions: learning from patient complaints</article-title>. <source>Health Prof Educ</source>. <volume>4</volume>:<fpage>97</fpage>&#x02013;<lpage>106</lpage>. <pub-id pub-id-type="doi">10.1016/j.hpe.2017.03.006</pub-id></citation></ref>
<ref id="B71">
<label>71.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Miaoulis</surname> <given-names>G</given-names></name> <name><surname>Gutman</surname> <given-names>J</given-names></name> <name><surname>Snow</surname> <given-names>MM</given-names></name></person-group>. <article-title>Closing the gap: the patient-physician disconnect</article-title>. <source>Health Mark Q</source>. (<year>2009</year>) <volume>26</volume>:<fpage>56</fpage>&#x02013;<lpage>8</lpage>. <pub-id pub-id-type="doi">10.1080/07359680802473547</pub-id><pub-id pub-id-type="pmid">19197588</pub-id></citation></ref>
<ref id="B72">
<label>72.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Ozawa</surname> <given-names>S</given-names></name> <name><surname>Sripad</surname> <given-names>P</given-names></name></person-group>. <article-title>How do you measure trust in the health system? A systematic review of the literature</article-title>. <source>Soc Sci Med.</source> <volume>91</volume>:<fpage>10</fpage>&#x02013;<lpage>4</lpage>. <pub-id pub-id-type="doi">10.1016/j.socscimed.2013.05.005</pub-id><pub-id pub-id-type="pmid">23849233</pub-id></citation></ref>
<ref id="B73">
<label>73.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Taber</surname> <given-names>JM</given-names></name> <name><surname>Leyva</surname> <given-names>B</given-names></name> <name><surname>Persoskie</surname> <given-names>A</given-names></name></person-group>. <article-title>Why do people avoid medical care? a qualitative study using national data</article-title>. <source>J Gen Intern Med.</source> <volume>30</volume>:<fpage>290</fpage>&#x02013;<lpage>7</lpage>. <pub-id pub-id-type="doi">10.1007/s11606-014-3089-1</pub-id><pub-id pub-id-type="pmid">25387439</pub-id></citation></ref>
<ref id="B74">
<label>74.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Rajkomar</surname> <given-names>A</given-names></name> <name><surname>Dean</surname> <given-names>J</given-names></name> <name><surname>Kohane</surname> <given-names>I</given-names></name></person-group>. <article-title>Machine learning in medicine</article-title>. <source>New Engl J Med.</source> (<year>2019</year>) <volume>380</volume>:<fpage>1347</fpage>&#x02013;<lpage>58</lpage>. <pub-id pub-id-type="doi">10.1161/CIRCULATIONAHA.115.001593</pub-id></citation></ref>
<ref id="B75">
<label>75.</label>
<citation citation-type="web"><person-group person-group-type="author"><name><surname>Ross</surname> <given-names>C</given-names></name> <name><surname>Swetlitz</surname> <given-names>I</given-names></name> <name><surname>Thielking</surname> <given-names>M</given-names></name></person-group>. <source>IBM Pitched Watson as a Revolution in Cancer Care: it&#x00027;s Nowhere Close</source>. <publisher-loc>Boston</publisher-loc>. (<year>2017</year>) Retrieved from <ext-link ext-link-type="uri" xlink:href="https://www.statnews.com/2017/09/05/watson-ibm-cancer/">https://www.statnews.com/2017/09/05/watson-ibm-cancer/</ext-link> (accessed April 16, 2020)</citation></ref>
<ref id="B76">
<label>76.</label>
<citation citation-type="web"><person-group person-group-type="author"><name><surname>Nichols</surname> <given-names>G</given-names></name></person-group>. <article-title>Terminal patient learns he&#x00027;s going to die from a robot doctor</article-title>. (<year>2019</year>) Retrieved at: <ext-link ext-link-type="uri" xlink:href="https://www.zdnet.com/article/terminal-patient-learns-hes-going-to-die-from-a-robot-doctor/">https://www.zdnet.com/article/terminal-patient-learns-hes-going-to-die-from-a-robot-doctor/</ext-link> (accessed Mar. 26, 2019).</citation></ref>
<ref id="B77">
<label>77.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Durosini</surname> <given-names>I</given-names></name> <name><surname>Tarocchi</surname> <given-names>A</given-names></name> <name><surname>Aschieri</surname> <given-names>F</given-names></name></person-group>. <article-title>Therapeutic assessment with a client with persistent complex bereavement disorder: a single-case time-series design</article-title>. <source>Clin Case Stud</source>. (<year>2017</year>) <volume>16</volume>:<fpage>295</fpage>&#x02013;<lpage>312</lpage>. <pub-id pub-id-type="doi">10.1177/1534650117693942</pub-id></citation></ref>
<ref id="B78">
<label>78.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Rosner</surname> <given-names>R</given-names></name> <name><surname>Pfoh</surname> <given-names>G</given-names></name> <name><surname>Kotou&#x0010D;ov&#x000E1;</surname> <given-names>M</given-names></name></person-group>. <article-title>Treatment of complicated grief</article-title>. <source>Euro J Psychotraumatol</source>. (<year>2011</year>) <volume>2</volume>:<fpage>7995</fpage>. <pub-id pub-id-type="doi">10.3402/ejpt.v2i0.7995</pub-id><pub-id pub-id-type="pmid">22893810</pub-id></citation></ref>
<ref id="B79">
<label>79.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Wiederhold</surname> <given-names>BK</given-names></name></person-group>. <article-title>Can artificial intelligence predict the end of life&#x02026;And do we really want to know? <italic>Cyberpsychol</italic></article-title>. <source>Behav. Soc. Netw</source>. <volume>22</volume>:<fpage>297</fpage>. <pub-id pub-id-type="doi">10.1089/cyber.2019.29149.bkw</pub-id></citation></ref>
<ref id="B80">
<label>80.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Kedia</surname> <given-names>SK</given-names></name> <name><surname>Ward</surname> <given-names>KD</given-names></name> <name><surname>Digney</surname> <given-names>SA</given-names></name> <name><surname>Jackson</surname> <given-names>BM</given-names></name> <name><surname>Nellum</surname> <given-names>AL</given-names></name> <name><surname>McHugh</surname> <given-names>L</given-names></name> <etal/></person-group>. <article-title>One-stop shop: lung cancer patients&#x00027; and caregivers&#x00027; perceptions of multidisciplinary care in a community healthcare setting</article-title>. <source>Transl Lung Cancer Res</source>. (<year>2015</year>) <volume>4</volume>:<fpage>456</fpage>&#x02013;<lpage>64</lpage>. <pub-id pub-id-type="doi">10.3978/j.issn.2218-6751.2015.07.10</pub-id></citation></ref>
<ref id="B81">
<label>81.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Briet</surname> <given-names>JP</given-names></name> <name><surname>Hageman</surname> <given-names>MG</given-names></name> <name><surname>Blok</surname> <given-names>R</given-names></name> <name><surname>Ring</surname> <given-names>D</given-names></name></person-group>. <article-title>When do patients with hand illness seek online health consultations and what do they ask?</article-title> <source>Clinical Orthopaedics and Related Research</source> (<year>2014</year>) <volume>472</volume>:<fpage>1246</fpage>&#x02013;<lpage>50</lpage>. <pub-id pub-id-type="doi">10.1007/s11999-014-3461-9</pub-id><pub-id pub-id-type="pmid">24442841</pub-id></citation></ref>
<ref id="B82">
<label>82.</label>
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Yeung</surname> <given-names>RY</given-names></name> <name><surname>Leung</surname> <given-names>GM</given-names></name> <name><surname>McGhee</surname> <given-names>SM</given-names></name> <name><surname>Johnston</surname> <given-names>JM</given-names></name></person-group>. <article-title>Waiting time and doctor shopping in a mixed medical economy</article-title>. <source>Health Econ</source>. (<year>2004</year>) <volume>13</volume>:<fpage>1137</fpage>&#x02013;<lpage>44</lpage>. <pub-id pub-id-type="doi">10.1002/hec.871</pub-id><pub-id pub-id-type="pmid">15386684</pub-id></citation></ref>
</ref-list>
<fn-group>
<fn fn-type="financial-disclosure"><p><bold>Funding.</bold> ST and GP were supported by MIUR - Italian Ministry of University and Research (Departments of Excellence Italian Law n.232, 11th December 2016) for University of Milan. ID was supported by Fondazione Umberto Veronesi.</p>
</fn>
</fn-group>
</back>
</article>