There are a number of challenges associated with the application of human ethical theories to AI. Many of these challenges are outlined by Pflanzer et al. (2023) and include factors such as: human preferences, challenges with applying ethical theories to machines, attribution of blame and severity of consequence, and trust repair challenges. Some researchers have called for AI to include state awareness functions that resemble human consciousness as a means to promote more ethical machines (Chella et al., 2019). From this perspective, AI that possesses state-level awareness and “experience” might be better suited for ethical reasoning. This is similar to the notion of an Artificial Moral Agent (Dignum, 2017).

However, many (such as Bigman and Gray, 2018) have expressed concern regarding such approaches, noting that human acceptance of such capabilities (even if technically-feasible—which is a position that is still unclear) would be quite low. Humans tend to prefer other humans as the decision maker when the decisions are moral in nature. Specifically, humans are preferred over machines in situations involving driving, medical decisions, and military situations. The rationale for this preference is the notion that humans have greater experience and agency to navigate moral complexities. Interestingly, humans can favor other humans over AI even when the AI has demonstrated the ability to outperform other humans in a process referred to as algorithm aversion (Dietvorst et al., 2015). Hence, human preferences for other humans over AI as moral decision makers is one challenge in applying human ethics to AI.

Secondly, it is challenging to apply virtue ethics, utilitarianism, or consequentialism to machines because expectations for ethical behavior of machines are higher and more difficult to achieve than expectations of ethical behavior of humans. When considering the ethicality of humans, it is quite common for one to evaluate the features and characteristics of the human and determine if those features are good or bad. Competence is a core consideration for trustworthiness of machines (Hancock et al., 2011). However, humans and machines do not start at the same level when considering competence. Due to human individual differences such as the Perfect Automation Schema (see Dzindolet et al., 2002), humans have higher starting levels of perceived capability with machines relative to humans. These differences between initial trust in humans compared to machines has been further elaborated by Madhavan and Wiegmann (2007) who state that humans tend to: (1) have higher expectations of technology relative to humans, (2) view technology as invariant and humans as variant, (3) tend to be less forgiving of machine-based errors relative to humans, and (4) tend to view machines in performance terms and humans in relational terms. Research has indeed confirmed that machines tend to pay a higher cost for errors relative to humans. A series of studies by Sundvall et al. (2023) found that a robot was blamed more harshly than humans when it applied utilitarian logic to a situation that invoked more folk-ethics (such as saving two boaters who caused an accident rather than saving the innocent victim). Research by Laakasuo et al. (2022) found that robotic nurses were evaluated more negatively (compared to humans) in an ethical dilemma wherein they forcibly applied medication to a patent. Shariff et al. (2017) conducted a large survey of drivers and found that even though autonomous cars may result in fewer accidents than humans, human drivers attributed greater severity to accidents that involved autonomous vehicles relative to accidents caused by humans. Thus, the application of virtue ethics, utilitarianism, and consequentialism to AI comes with limitations.

Third, the issue of intentionality is harder to decipher when the referent is an AI. Perceived benevolence, the beliefs that another agent has one's best interests in mind when making a decision and acting, is a core trustworthiness attribute (Mayer et al., 1995). People are more prone to trust others when they believe that those other agents have their best interests at heart. There are two challenges related to the attribution of intentionality toward AI to include the issue of agency, and the issue of transparency. It is highly probable that future AI systems deployed in a military context will need to evaluate multiple goals and select actions in accordance with those goals. There may be times when goals are somewhat competing—in the case of competing time demands vs. resource utilization, for instance. Yet unlike humans, wherein the notion of agency is understood and fully assumed, AI may vary in the degree to which they have been delegated bounded authority for a given situation. Thus, even if machines communicate goal alignment with a particular human partner, machines may not possess the delegated authority to act on behalf of that intent. Without agency to act on behalf of communicated intent, a human partner may not believe and attribute positive attitudes toward a machine agent.

Research by Lyons et al. (2022) confirm this effect. Their research demonstrates that the benefits of benevolently-framed intent are highest when a robot also possesses the highest degree of decision latitude to act on that intent. So for a future AI to convey benevolence the AI needs to actually be able to act on behalf of that intent. Imagine, for example, an AI-based rescue robot identifies a victim and communicates an interest to help the victim. However, the victim watches the robot pass her/him by because it was following a pre-programmed route and did not possess the delegated authority to deviate from that route. It is highly likely that conveying an interest to help would be met with negative responses if the AI could not actually provide help due to prior programming. This subtle, but critical issue is not one that humans are accustomed to dealing with in other humans, but it is an issue for AI systems that will need to be addressed.

AI systems suffer another limitation in that, even if they are delegated bounded authority for a given task, intent in relation to a human may be opaque. Lyons (2013) talks about the importance of transparency of intent within human-robot interactions. To date, it is unclear whether behaviors from an AI will be attributed to benevolent intentions or simply to the programming guiding the technology. For an AI to be perceived as helping and supportive of one's goals it must be clear that the AI is working toward goals that benefit the human, this will require design features that make such attributions clear. Research has shown that framing a robot's behavior in self-sacrificial terms can increase trust and trustworthiness of a robot (Lyons et al., 2021). However, great care must be taken to ensure that if goal alignment is conveyed to the human that the AI is actually working in support of said goals, lest the human perceive that the AI is trying to exploit the human. Thus, when trying to apply virtues such as competence and benevolence of an AI, one must consider the issues of opacity, dispositional biases, and design for transparency.

Some researchers have called for moral competence in technologies such as social robotics (Malle and Scheutz, 2014). There is certainly value in designing physically-embodied technologies with capabilities to better communicate and interact with humans. Due to limitations in contextual awareness and adaptability, which would preclude broad applications of moral competence in robots, this could be envisioned to occur on a limited scale for tasks. Specifically, the social norms and value-oriented language to communicate on behalf of those norms could be structured to be executable in task-specific ways. This would be consistent with Dignum (2017) Functional level of moral competence. Such methods might include the ability to communicate in relation to norms and to engage in trust repair and explanation when behaviors violate norms (Malle and Scheutz, 2014; Lyons et al., 2023). A recent study by Lyons et al. (2023) examined human reactions to a situation where a robot violated a behavioral expectation for a robot to follow a plan issued by a human operator. They found that trust and trustworthiness decreased following the behavioral violation. However, explanation strategies geared toward providing the rationale for a behavioral deviation (in this case by offering an observational rationale) were effective in thwarting the decreases in trust and trustworthiness. Notably, participants evidenced no decrease at all in ability perceptions when this (observational rationale) explanation was provided.

In the above sense, like the moral competence discussed by Malle and Scheutz (2014), the robot is simply recognizing a norm violation and responding accordingly. This could be useful in narrow contexts, but less feasible in broad contexts. However, recent research has found that trust repair strategies are typically ineffective for repairing trust violations by AI for ethical violations (Schelble et al., 2023). Schelble et al. (2023) exposed participants to an unethical AI (in this case a violation of virtue ethics wherein the AI team was told to avoid collateral damage in an air-to-ground strike, but the AI engaged in a strike that resulted in significant collateral damage) and tested two trust repair strategies (apology and denial) and found that neither repaired trust for an unethical AI. Thus, traditional strategies for repairing trust may be ineffective for trust repair associated with ethical violations from AI systems.

In summary, there are a number of challenges with using human-centric ethical models for evaluating AI systems. Humans have preferences for non-AI agents in moral decision-making. It is difficult to attribute virtue to a machine and to ascribe intentionality to its actions, it is hard to know who to blame when AI systems make an error and how severe that error really is, and repair trust of AI systems when they eventually do make an error (as all agents do—human or AI-based) is difficult when the violation is ethical in nature. For all of these reasons, one might move away from attributions of AI ethicality and toward responsible use of the AI—which pulls in the broader human-AI systems perspective.

Awareness begins with education regarding the ethical use of AI within the communities of AI researchers, designers, regulators, insurers, acquirers, leaders of team developing or using AI, and end users of AI technology. “Therefore, Responsible AI also requires informal participation of all stakeholders, which means that education plays an important role, …” (Dignum, 2019, p. 48). It is a known problem that many computer science curriculums do not have much of an emphasis on AI ethics (Reidy, 2017). The current manuscript advocates that AI developers and robotics engineers get exposure to ethics during their training. This education could include, at a minimum, (1) courses to introduce ethics, (2) courses to discuss examples of prior AI ethics issues, and (3) courses to discuss methods that promote responsible use of AI. Education is also needed beyond just the AI developers. Leaders in organizations need to be educated on the realistic capabilities and potential ethical issues surrounding the use of AI in their organizations. Generally, this could help to promote appropriate expectations of AI within the organization's strategy and vision. Additionally, it could help to anticipate and address potential ethical shortcomings regarding the use of AI in the military and across society more broadly. Finally, it is imperative to cultivate an AI-literate general workforce. As organizations develop, test, and field AI systems, it is important for the general population of workers to understand the basics of AI in order to be responsible users of AI.

One way to measure this understanding may be to evaluate whether they can interpret the contents of a model card (Mitchell et al., 2019) or datasheets for datasets (Gebru et al., 2021) sufficiently to appropriately bound their use of the AI model in question. While there are currently no standards for documenting ML datasets, the model cards and datasheets for datasets ideas suggest that AI developers use a structure to document a trained model or the data used to train it. Model cards are envisioned to be brief documents accompanying a trained ML model, sometimes referred to as a “nutrition label” for AI models, that provide the context where models are intended to be used, expected performance, and a description of what information the model was trained on. For example, in big data contexts such as supervised learning on populations of humans, a model card might describe how well the model performs across race, geographic locations, age, sex, cultures, or skin types. For another example, a model trained using reinforcement learning might describe the fidelity of the simulation or physical environment it was trained on, the sets of conditions it has been trained and evaluated on, conditions where behavior may not be reliable, and failure rates. Datasheets for datasets are envisioned to include the motivation for the AI tool, the intended purpose of the tool (including who created it, who funded its creation, and whether there was a specific application in mind), the composition of the dataset (what is in it, how large it is, are there subpopulations, how it is maintained), the collection process associated with the dataset (how was the dataset created, who was involved, how recent is it, has it been updated, if there is data related to people did the people know about the dataset, were they informed about possible uses?), and the recommended uses of the dataset (Gebru et al., 2021).

Medical doctors and fighter pilots have very specialized training to meet highly regulated standards that allows a person to make some assumptions about expertise. Even with unfamiliar doctors or fighter pilots, one can make inferences about the referent due to the training and experience that those individuals possess and have demonstrated through standardized evaluations. However, one will likely not have the same confidence that a medical doctor can repair a motor vehicle. Likewise, narrow AI systems that are based on ML algorithms are completely limited by the training data used to create them. This means that the training data itself can be used as a means to better calibrate one's understanding and expectations regarding an AI tool. The World Economic Forum suggested that all developers of ML datasets document the provenance, creation, and intended use of datasets to avoid bias in the application of ML models (World Economic Forum, 2018). Lyons et al. (2018) talk about this in terms of using the training datasets as a means to understand and communicate the learning affordances from which the algorithms have been trained. Learning affordances associated with datasets could involve the types of data (i.e., size, creation, intended use), types of content (specific attributes related to the content of this dataset), environmental constraints (i.e., what types of uncertainty has this training dataset been exposed to?), and stability (i.e., has the dataset been updated, what is the battle rhythm for updates and who provides updates?). Understanding these features of a dataset could help to inform whether one's intended use of an AI matches the tool's original intended purpose and whether or the dataset used to train the AI algorithm is appropriate for that targeted use.

Similarly, Gebru et al. (2021) discuss the concept of datasheets for datasets, and Mitchell et al. (2019) have proposed documentation of trained models using model cards. Acknowledging that training datasets are the key driver of ML models, datasheets for datasets would help to create meta-data that would allow a user of AI to understand how well that AI tool (and its associated training data) matches the intended use of the AI. Obviously, if the targeted context does not match the intended target context, one might think twice before using that AI for the task. This methodology could also help to reduce bias associated with the use of AI by preventing the use of restricted datasets for applications to populations not well-represented within the dataset. Through labeling the generalizability of a dataset, users of AI can better understand who (i.e., what groups) AI tools might be most effective for—and potentially what groups to avoid using the AI tool for. Herein, the overall objective of providing this type of information about the dataset is to calibrate users' expectations with regard to the appropriateness for a particular AI tool for a given desired use.

Human-centered approaches have emerged as an influential approach in human-machine interaction. Human-centered AI (HCAI) is an approach pioneered by Shneiderman (2020) which encourages privacy, security, environmental protections, social justice and human rights. This is consistent with value sensitive design wherein the design method considers human values as a key feature of the design process (Friedman et al., 2006). The key element in the human-centered approach is to augment humans rather than focus on replacing them. To do so, one must first understand the tasks being performed, the value structures that exist within these task contexts, the human actors that are present, and any contextual constraints that influence what is deemed acceptable (by humans) in these task situations. In addition to understanding human value structures and their contextual nuances, AI must be designed to augment humans and to maximize human control over the AI (Shneiderman, 2020). Designing with maximal human control does not mean that machines should avoid higher levels of automation. In contrast, HCAI approaches emphasize that both high human control and high machine control is possible simultaneously (Shneiderman et al., 2016). HCAI principles can support more effective human-AI systems and should be used when developing novel AI systems (Xu et al., 2023). Human-centered design helped to inform the design of a novel supervisory control system within the U.S. Air Force Research Laboratory. Driven by a thorough understanding of operator needs and preferences, an intuitive interface design was created to maximize operator directability and to facilitate a flexible full spectrum-of-control wherein the gamut of manual control, playcalling approaches, and fully autonomous responses were shown to augment operations (Calhoun et al., 2021).

Parallel to the HCAI movement, has been the emergence of agent transparency research. Transparency is one of three pillars for responsible AI along with fairness and accountability (Dignum, 2019). Agent transparency can be referred to as the agent's ability to communicate information to the human operator in a clear and efficient manner, which allows the operator to develop an accurate mental model of the system and its behavior (Chen et al., 2020). Research on transparency has increased in recent years keeping pace with emergent AI systems that are often opaque (e.g., ML systems). There are two dominant models within the transparency literature: the Situation Awareness-based Agent Transparency (SAT) model and the Lyons transparency model for human-robot interaction (Bhaskara et al., 2020).

The SAT model leverages Endsley (1995) Situation Awareness model to create methods for perception (level I), awareness (level II), and projection (level III) of agent rationale and behavior (Chen et al., 2018). Research has shown that interfaces that invoke level III transparency can improve performance and increase trust compared to lower levels of transparency (Mercado et al., 2016). Interestingly, these benefits did not come at the cost of higher workload—which is consistent with the HCAI principles noted above. The Lyons model of HRI transparency emphasizes robot-to-human (r-TO-h) transparency which promotes signaling information about the robot's purpose, task understanding and status, analytic methods, and environmental understanding (Lyons, 2013). Additionally, this model discusses the need for robot-of-human (r-OF-h) transparency which emphasizes information about the team roles and division of labor between the human and robot, as well as bidirectional information about the robot's understanding of the human state (e.g., cognitive workload; Lyons, 2013). The SAT model has been applied to autonomous squad member technologies, RoboLeader applications, and command and control (C2) applications (see Chen et al., 2018). The Lyons model has been applied to robotic scenarios (Lyons et al., 2021) and commercial aviation automation (see Lyons et al., 2016b). While many transparency approaches have focused on the human-machine interface part of the human-AI system, transparency related information can be acquired across the lifecycle of the human-AI interaction to include design, test, use, and debriefing opportunities (Miller, 2021).

Run time assurance (RTA) is the process of augmenting a complex system, such as a trained neural network, to ensure that its output meets some desired properties (safety, ethics, performance, etc.). RTA theory is sourced from the control theory community, where it is often implemented as a wrapper that monitors the output of a complex controller (e.g., neural network control system) and modifies its output when necessary to assure safety (Hobbs et al., 2023). An example RTA system is the Auto-GCAS system mentioned earlier. This system monitors the state (position, velocity, orientation, etc.) of the aircraft relative to surrounding terrain for imminent collisions and intervenes by switching from the complex control system (in this case a human pilot) to a backup controller (roll-to-wings-level and 5-G pull up maneuver) to avoid the collision, as depicted in Figure 1. While control theory definitions of RTA focus on the safe output of a control system, the authors generalize the definition here to be applicable to the acceptable outputs of a neural network.

The final element for moving toward responsible use of AI is to design and implement robust joint human-AI training opportunities. Development of mental models of one's collaborative partners is vital to human-AI teaming success (Musick et al., 2021). Shared mental models allow team members to adapt to change and helps teams to interpret events in the environment in similar ways—promoting common responses, strategies, and expectations within teams (Salas et al., 2008). A key pathway toward mental model development is through team cognition—which is typically manifest through communication (Musick et al., 2021). Traditionally, mental models refer to one's awareness (and a team's shared awareness) of equipment used within the team, team tasks (i.e., strategies, procedures, and contingencies for accomplishing tasks), the team composition (i.e., skillsets, preferences, abilities, knowledge within the team), and team interaction features (i.e., communication patterns, roles, dependencies) (Cannon-Bowers et al., 1993). Robust mental models of AI systems can be developed through education, considering the learning affordances of ML training datasets, developing effective human-machine interfaces, and by understanding the capabilities and limits of design features such as run time assurance methods—as noted above. However, one thing is still needed for the development of robust mental models—rich experience with the AI across contexts.

Scenario-based training is one means to test and validate that the AI is working as intended. Joint human-AI training is a concept discussed in Lyons et al. (2017). “The scenarios used during the human-machine training should test the envelope both in terms of performance expectations but also uncertainty. Testers will want scenarios that create morally contentious situations for the autonomy to see how it will react to ambiguous stimuli” (Lyons et al., 2017, p. 44). As noted by Lyons et al. (2017), instructional scaffolding could be used to progressively increase the level of difficulty (or the level of AI uncertainty) for a human to observe how the AI handles the additional complexity. The key issues associated with joint human-AI training are to expose the human to the AI across a gamut of contexts that vary in complexity and uncertainty. The experience garnered from such observations will facilitate rich mental models of the AI that can be applied to future contexts. The end goal of joint human-AI training is to enable predictability for how the AI handles tasks in a variety of task contexts. The more challenging and wider variety of contexts, the greater the benefit of the joint human-AI training in terms of establishing the right expectations of the AI.