When framing the research questions and search terms, the Population, Intervention, Comparison, Outcome, and Context (PICOC) method was used Carrera-Rivera et al. (2022) and Kitchenham and Charters (2007). Table 1 presents the PICOC terms for this research.

This research addresses the following three Research Questions (RQs), formulated from the PICOC terms.

Articles for the SLR were selected from IEEE Xplore, ACM Digital Library, Science Direct by Elsevier, and Springer. These digital libraries were chosen for their relevance to the RQs and their reputability for publishing peer reviewed articles.

Using the PICOC criteria, the following search string was synthesized: (“Machine Learning” OR “Artificial Intelligence” OR “Neural Network” OR “Reinforcement Learning” OR “Large Language Model” OR “Transformer”) AND (“Formal Methods” OR “Formal Verification” OR “Temporal Logic” OR “Model Checking” OR “Theorem Proving” OR “Runtime Verification” OR “Reactive Synthesis” OR “Automated Reasoning”) AND (“Testing” OR “Traditional Verification”) AND (“Safety” OR “Quality” OR “Safe”) AND (“Safety-Critical” OR “Critical System” OR “Life-Critical” OR “Mission-Critical” OR “High-Reliability”).

An additional title search term was used for Springer to narrow down results further: “Formal” OR “Temporal Logic” OR “Verification” OR “Model Checking” OR “Runtime.” Because Science Direct only allows eight Boolean connectors per query, the search string was split into the following two search strings, for Science Direct only:

The inclusion and exclusion criteria are presented in Table 2. Criteria are adapted from Carrera-Rivera et al. (2022) with several additions. Due to the significant development in ML in the last five years, articles were chosen to be no earlier than 2020. Despite their lower quality as opposed to conference and journal articles, workshop articles were included, as these articles frequently present novel ideas for future exploration.

In addition to the criteria presented in Table 2, studies that do not directly apply formal methods to ML systems were excluded. This exclusion applies to studies that formally verify controllers or code synthesized by an ML model, or to studies focusing on autonomous systems broadly. Articles that do not mention “formal verification” or “formal methods” explicitly were removed. Articles related to cybersecurity were also excluded, as this SLR focuses on safe ML as opposed to secure ML.

After the initial articles were selected using the inclusion and exclusion criteria on the full list, the quality assessment checklist, shown in Table 3, was used primarily to characterize the quality of the articles and help determine future research directions. Articles were not excluded based on the quality assessment criteria, as this work is primarily interested in the ideas and research directions within safe ML through formal methods. Additionally, due to the limited nature of work in this area, it was counterproductive to further exclude articles outside of the inclusion and exclusion criteria. The questions are adapted from Zhou et al. and utilize the Reporting, Rigor, Credibility, and Relevance criteria (Zhou et al., 2015). For each article, each question received a score of either 0 – “No,” 0.5 – “Partially,” or 1 – “Yes.”

After obtaining the final set of research articles, data was systematically extracted for analysis using the data extraction form presented in Supplementary Table 1. The data extraction form was generated using guidelines from Kitchenham and Charters (2007).

Note that fields related to experimental design and results were not directly used to answer the RQs. Extracting information about each experiment was used to validate the significance of the authors’ findings and therefore the relevance of the proposed method. As a result, these fields primarily assisted with the quality assessment of each study. Experimental data was also used to help determine the application area or domain when answering RQ2.

After extracting data using the data extraction form, the data was analyzed and synthesized through two methods. First, a bibliometric analysis was performed on the data to determine bibliometric information such as which conferences or journals produce the most amount of research in this area, the top three countries where research in this area is being conducted, and the citation counts for the different articles.

Next, the technical content from the articles was analyzed from the extracted data to answer the RQs. This was accomplished by comparing the responses to each data item across articles to determine trends, application areas, challenges, and future research directions. The findings are then synthesized to answer the RQs in Sections 4 and 5. Data is presented both quantitatively and qualitatively. Separately, an analysis of the quality of the articles is presented after executing the quality assessment checklist on each article.

Reachability analysis for neural networks computes the set of all outputs the network can produce when its inputs—and, if modeled, its parameters (weights and biases)—vary within specified bounds. This technique propagates sets (e.g., represented by intervals or star sets) through layers or time steps; some solutions compute a sound over-approximation that provably contains every true output, while other architectures admit exact set propagation (Meyer, 2023; Choi et al., 2025).

S22 extends the star-set reachability framework to RNNs to aid in formally verifying the robustness of RNNs (Choi et al., 2025). The authors develop both exact and over-approximate reachability algorithms that incrementally unroll recurrent layers, encode dependencies between current and past hidden states via Minkowski sum of star sets, and offer soundness and completeness guarantees. Unrolling refers to the technique of expanding the recurrent network across a chosen number of time steps by creating a separate copy of the recurrent layer for each time step. This transformation converts the RNN’s cyclic, time-dependent structure into a deep, acyclic Feed-Forward Neural Network (FFNN) at the cost of producing a much larger network (Choi et al., 2025).

S23 proposes a mixed-monotonicity-based reachability method for computing interval over-approximations of a NN’s output when both its inputs and its parameters lie within known bounds (Meyer, 2023). In this approach, the network’s partial networks are treated as static functions whose derivatives are known to lie within a certain bound. For each output neuron, a worst-case input corner is chosen based on whether each slope is positive or negative. The input corners are then used to compute tight upper and lower output bounds. This approach handles uncertainties by treating both input values and all weights as uncertain within given intervals and uses interval arithmetic on the network’s Jacobian to propagate these uncertainties layer by layer. Rather than completing this step once, the method is applied to every contiguous block of layers (i.e., all partial networks). These interval bounds are then intersected to result in a tighter overall output range. This approach is applicable to any Lipschitz-continuous activation function (Meyer, 2023).

On the other hand, S28 presents a high-parallelization framework for formally verifying feed-forward DNNs under bounded input uncertainty by computing interval over-approximations of layer outputs (Hafaiedh et al., 2025). Interval over-approximation is a static analysis technique that computes, for each neuron in the network, a conservative lower-upper bound on its activation by propagating input uncertainty through the network. The framework checks if, for all inputs satisfying some precondition, the DNN’s outputs satisfy a specified post-condition. The method runs several incomplete verifiers in parallel, and for each neuron takes the intersection of their interval bounds to refine the layer’s over-approximation before feeding it forward (Hafaiedh et al., 2025).

S12, S29, S31, and S34 apply reachability and over-approximation techniques to NNs for image-based applications (Parameshwaran and Wang, 2025; Zhong et al., 2023; Tang et al., 2023; Ashok et al., 2020). S12 aims to improve the scalability and efficiency of the verification problem for image-based NNs by performing formal verification, specifically linear bound propagation using 𝛼 − 𝛽 – 𝐶𝑅𝑂𝑊𝑁, on structured input spaces within the latent space. This technique thereby reduces input dimensionality and computational complexity. Specifically, the authors propose the Scalable and Interpretable Verification of Image-Based Neural Network Controllers (SEVIN). The method learns a structured latent representation of the controller’s input space, thereby decreasing the computational complexity of the verification task and making formal verification more scalable (Parameshwaran and Wang, 2025).

Similarly, S29 introduces the Abstract Refinement Enhancer for Neural network verifAcation (ARENA), which constructs a linear constraint encoding of a NN’s behavior using abstract interpretation bounds, then iteratively refines that encoding. ARENA targets potential violations of robustness and proves whether the regions are spurious (Zhong et al., 2023). S31 introduces the tool Fast Grouping for Multi-neuron Relaxation (FaGMR), a verifier that encodes the network and an input-perturbation region into linear constraints. The authors also address verifying robustness properties. Nonlinear activations are over-approximated by convex constraints, then a Linear Programming (LP) solver checks whether the predicted class can change within the perturbation region (Tang et al., 2023). S34 proposes DeepAbstract, an abstraction-based method that clusters neurons within each hidden layer based on empirical behavior (Ashok et al., 2020). Each neuron is represented by its vector of activation values over a chosen input set, then k-means groups neurons with similar activation vectors. Neurons in each cluster are merged, yielding a smaller abstract network. Verification is then run on the abstract network.

On the other hand, S30 and S43 focus specifically on Semantic Segmentation NNs (Pal et al., 2023; Tran et al., 2021). S30 performs set-based reachability analysis for neural networks, specifically star-set reachability using the “approx-star” method (Pal et al., 2023). The work presents a standardized benchmark intended to support fair and repeatable comparisons of verification approaches for semantic segmentation. S43 performs reachability analysis on Semantic Segmentation NNs, using ImageStar set representations and LP optimization to compute bounds needed for over-approximation (Tran et al., 2021). The authors introduce a relaxed reachability variant for Rectified Linear Unit (ReLU) layers and pixel-classification reasoning, which is controlled by a relaxation factor that reduces how many LP problems are solved.

S37 focus specifically on reachability analysis and abstract interpretation for CNNs (Kirov et al., 2023). The CNN’s safety-relevant requirements are formalized as mathematical properties over bounded input perturbation sets and bounded trajectory modification sets. For each property, the verifier over-approximates the CNN’s reachable output set over the constrained input set and checks whether the output set violates the output bound. When over-approximation yields “unknown,” randomized simulation searches for an explicit violating example.

S41 instead explores safety assurance when quantizing DNNs for use on resource constrained devices (Zhang et al., 2023). The authors use both Differential Reachability Analysis (DRA) and MILP. Given a DNN, its corresponding fully quantized QNN, an input region, and an allowed error bound, the method verifies whether the maximum output deviation between the DNN and QNN is always less than the bound for all inputs in the region. DRA attempts to prove the bound quickly; if it cannot, the MILP encoding is used to decide the property exactly.

Lastly, S42 applies star-based reachability analysis to DNNs used for time-series regression tasks (Pal et al., 2023). The framework encodes bounded sensor noise as a set of possible inputs and then propagates that set through the network layer-by-layer to obtain an output reachable set. The reachable output bounds are then compared to permissible bounds to determine robustness at each time step and across a sequence.

SMT-based verification for NNs encodes both the neural network’s computation and the safety or robustness property as logical formulas in rich theories (e.g., linear real arithmetic or bit-vectors) and then utilizes optimized SMT solvers to check for violations or to prove the absence of counterexamples (Das et al., 2025; Bachiri et al., 2025). Abstraction/Refinement (AR) verifies a DNN by first constructing a smaller over-approximating model whose correctness implies correctness of the original. If the abstract query is UNSAT, the original is UNSAT; if it is SAT, the produced counterexample is checked on the original network and refinement is applied only for spurious counterexamples or otherwise inconclusive abstract results. By iteratively refining the abstraction, AR trades precision for solver performance and can be substantially faster than verifying the full network directly (Elboher et al., 2024).

S3, S4, S27, S32, and S35 specifically apply SMT-based verification to DNNs (Nuhu et al., 2022; Samadi et al., 2024; Elboher et al., 2024; Zhao et al., 2022; Paterson et al., 2021). After determining candidate unsafe input subregions (i.e., candidate unsafe sub-requirements), S3 utilizes the Marabou framework to validate the unsafe input space on a DNN (Nuhu et al., 2022). Marabou, an SMT-based framework for verifying DNNs, validates whether the input subregion violates the safety property. The Negative Selection Algorithm, a meta-heuristic algorithm, is used to search for candidate unsafe input subregions. Next, the subregions are fed into Marabou to formally verify that the corresponding safety property is violated, thereby validating the unsafe subregion and unsafe sub-requirement.

Similarly, S4 uses Marabou to systematically insert perturbations into the DNN under test and determine which safety properties are violated as a result (Samadi et al., 2024). The authors propose utilizing Marabou to analyze the resilience of a DNN to input perturbations. Specifically, the study aims to determine the threshold weights that result in property violations and focuses on Single Event Upsets (SEUs) and Multi-Bit Upsets (MBUs). An SEU is a single bit flip within the memory’s stored data, which impacts the weights of the DNN parameters. SEUs are performed by randomly selecting a weight within the DNN and flipping a bit in its binary representation. An MBU occurs when multiple bits are flipped in the system memory or processor of the DNN due to multiple SEUs in the network parameters. The proposed framework simulates both SEUs and MBUs within DNN network parameters. The authors then analyze the resulting effect on DNN performance and adherence to properties. The output of the DNN is compared with the anticipated output from the training data.

S27 focuses on AR for SMT-solving by introducing residual reasoning, a scheme that captures and reuses information (e.g., already-proven safe regions of the search tree) across successive AR iterations, thereby pruning redundant work and accelerating verification of DNNs (Elboher et al., 2024). The authors contribute a formal, sound, and complete residual-reasoning framework for DNN verification as well as a detailed design for extending the Marabou verifier to support residual reasoning. Figure 6 displays the verification loop described by S27, where Γ is a context formula encoding scenarios already shown safe.

S32 applies Counterexample-Guided Abstraction Refinement (CEGAR) to DNN verification to determine whether a DNN satisfies specified input/output safety properties formulated as verification queries (Zhao et al., 2022). The framework involves building an over-approximated abstract network of the DNN and iterating AR using counterexamples. Backend formal verification is accomplished using Marabou. S35 introduces DeepCert, which performs constraint-based formal verification of DNN robustness using Marabou (Paterson et al., 2021). DeepCert encodes a perturbation to an image as constraints linking original pixels, perturbed inputs, and a perturbation bound, then determines whether a misclassification is possible within that bound.

On the other hand, S19 focuses on verifying the robustness of SDNNs, in safety-critical applications, from adversarial attacks using formal verification (Das et al., 2025). The authors specifically explore modeling and verification using SMT constraints. The framework receives two inputs: the SDNN and a property for verification. The formal verification framework then rigorously analyzes the SDNN using SMT-solving and either declares property adherence or presents a counterexample. The authors also present an interval bound derivation to improve the performance of the SMT solution further.

S20 presents an SMT-based verification of QNNs by approximating QNN values with rational-number versions (Bachiri et al., 2025). The authors combine rational approximations using set theory with SMT-based verification over rational arithmetic programs. The authors’ approach consists of four steps:

On a separate note, S33 models a NN controller as a transition predicate and utilizes an inductive invariant method for proving time-unbounded safety (Zhou and Tripakis, 2024). The framework decomposes the verification step so that the NN-related implication is handled using an NN verification engine and the environment-related implication is handled using an SMT solver.

S39 formally verifies an RL-trained controller for vertical collision avoidance (Genin et al., 2021). The authors compute conservative safe command ranges for states in a critical region, discretize the state space into cubes and over-approximate the safe bounds per cube, then use an SMT solver to prove whether the NN’s command acceleration is always within safe bounds for each critical cube. The result is a classification of cubes as verified safe versus potentially unsafe.

Focusing on a different area of ML research, S45 applies SMT solving to Spiking NNs, which are NNs that process information via discrete “spikes” over time (similar to human brains) (Banerjee et al., 2023). The framework translates the step-by-step execution of a Spiking NN into SMT constraints, where Boolean variables represent whether each neuron spikes at each timestep, and real variables represent each neuron’s instant potential and stored potential. Properties are then verified by checking satisfiability of a combined constraint system representing the Spiking NN behavior, input constraints, and violation of the desired output condition. If satisfiable, the solver provides a counterexample input spike train. If unsatisfiable, the property holds for all allowed inputs.

Lastly, S46 focuses specifically on SMT solving for Long Short-Term Memory (LSTM) networks, which are a type of RNN designed for sequential and time-series data (Moradkhani et al., 2023). The authors train an LSTM model, extract learned parameters, and encode the LSTM’s step-to-step update equations as an SMT constraint system. They then use an SMT solver with Bounded Model Checking capabilities to determine whether the LSTM can violate the intended safety specification.

MILP/ILP verification for NNs encodes a NN and a safety specification over a bounded input set into a set of linear and integer constraints/inequalities. Integer (binary) variables capture discrete choices introduced by activations and quantization; the solver asks a feasibility question: “Is there an input in this set that violates the property?” If the solver finds one, that assignment is a concrete counterexample; if none exists, the property is certified for that region. MILP mixes continuous and integer variables, whereas ILP uses only integers (Zhang et al., 2022; Mistry et al., 2022).

S5 presents an approach in which fixed-point primitives are encoded as operations in MILP to allow the authors to apply MILP solving to QNNs (Mistry et al., 2022). Otherwise, MILP cannot be used directly on QNNs. The verification problem is converted from a validity problem to a satisfiability problem, which allows the MILP solver to determine satisfaction (with a produced counterexample) or unsatisfaction (indicating a successful verification or valid input formula). Rather than being used for optimization, the MILP solver is used to check the feasibility of the equations. Similarly, S14 presents an encoding method for QNNs that reduces the verification problem to ILP (Zhang et al., 2022). The authors introduce piecewise constant functions for the encoding of QNN activation functions, which are then further encoded as integer linear constraints using additional Boolean variables. Both S5 and S14 use Gurobi as the backend MILP/ILP solver.

Separately, S36 applies MILP to ReLU NNs to verify monotonicity properties for a safety-critical avionics system (Vidot et al., 2022). A model is monotone if an input change in a specified direction results in an output moving in only one direction. The authors encode the ReLU NN and the monotonicity requirement into MILP constraints. They then check monotonicity over many paired input sub-spaces. By running MILP feasibility checks on each sub-space, they classify regions where monotonicity holds, fails everywhere, or partially fails.

Lastly, S6 combines polynomial inclusion computation with barrier certificate generation to formally verify synthesized DNNs from RL using linear programming solvers (Zhao et al., 2023). The authors abstract a DNN controller as a polynomial using Bernstein polynomials to facilitate verification by polynomial inclusion. Safety adherence is determined by the existence of barrier certificates under the abstract controller, and Sum-of-Squares optimization is used to search for these barrier certificates. If a barrier certificate is found, adherence to the corresponding safety property is determined for all executions of the original DNN controller.

Within formal methods broadly, Model Checking provides an automated framework to verify that system designs meet desired specifications. An abstract model M of the system under verification is constructed using finite state automata. Programs are modeled as transition systems, represented using nodes, variables, and transitions. A set of correctness and safety formulas F is defined using a logic formalism. The formulas describe the required correctness properties of the system. Next, the state space of M is systematically explored to check if F is consistently satisfied. If a property is violated, the model checker provides a counterexample demonstrating where the violation occurs (Paul et al., 2023; Jhala and Majumdar, 2009).

S2 applies Symbolic Model Checking to NNs by formally modeling a trained NN and verifying the model to estimate robustness to noise (Naseer et al., 2020). The study introduces the Formal Analysis of Neural Network (FANNet), which is composed of three main procedures. The first procedure is behavior extraction, in which the formal model of the NN is built using the weights and activations of the network on known test samples. The temporal properties are translated into the logical language of the model checker. The second procedure is noise tolerance analysis, in which the specific noise tolerance for the NN is algorithmically determined. The last procedure is the adversarial noise vector extraction, in which a unique array of noise patterns that the NN is sensitive to is built (Naseer et al., 2020). S9 applies Bounded Model Checking to a simplified version of a DNN modeling adaptive cruise control behavior (Nenchev, 2025). However, the model checked may not describe all behaviors of the actual system, therefore introducing potential gaps in the evaluation. The proposed approach should therefore be used alongside other verification techniques.

Both S1 and S44 apply Model Checking to RL-based systems (Tao et al., 2025; Adelt et al., 2023). S1 proposes a framework for formally specifying requirements, constructing an abstract model, and model checking an RL-based system (Tao et al., 2025). The study provides specification templates for formally specifying requirements of RL-based systems. A model construction process for generating the abstract model is then presented. Finally, the authors present Reinforcement Learning Verification-as-a-Service (ReLVaaS), a framework allowing users to specify formal requirements, construct the abstract models, and perform model checking using PRISM and Storm. On the other hand, S44 combines shielding with Statistical Model Checking-based learning for an RL agent (Adelt et al., 2023). The RL agent is constrained by a verified shield, and the authors prove, in Differential Dynamic Logic (dL) and KeYmaera X, that actions satisfying the agent’s contracts preserve safety and resilience. The framework then enforces, at runtime, that the agent can only choose those safe actions. Statistical Model Checking is used both during training and for statistical evaluation to compute confidence intervals of property satisfaction.

S26 introduces Deep Statistical Model Checking (DSMC), a framework that treats a trained neural network as an oracle to resolve nondeterminism in a Markov Decision Process (MDP), yielding a Markov chain whose behavior can be assessed via Statistical Model Checking (Gros et al., 2022). DSMC treats a trained neural network as a black-box oracle that resolves nondeterministic choices in a formally specified MDP. Whenever the MDP reaches a decision point, it queries the NN, using the current state, for the next action. The result is a fully probabilistic Markov Chain that can then be used for analysis.

On a separate note, S25 presents a unified Model Checking framework for ReLU RNNs that leverages a polyhedron abstraction domain and bidirectional propagation to symbolically verify both qualitative and quantitative temporal and robustness properties (Liang et al., 2024). The contribution is a systematic verification framework that combines polyhedron forward propagation, dimension preserving abstraction, and Monte Carlo sampling. Polyhedron forward propagation is used to track sets of possible RNN states. Dimension-preserving abstraction is used to curb combinatorial explosion of polyhedral vertices. Lastly, Monte Carlo sampling and backward propagation are used for quantitative estimation of satisfaction probabilities and to refine approximations.

Model Checking of MARL frameworks is explored by S15, which applies Model Checking to an abstract representation of a Markov game and joint policies (El Mqirmi et al., 2021). The study presents Assured Multi-Agent Reinforcement Learning (AMARL), a formal verification framework for MARL systems. AMARL formally guarantees the safety of agents acting in an unknown environment. The authors introduce the Abstract Markov Game (AMG) and present a procedure for automatically generating AMGs to conduct verification over.

S21 applies Model Checking to a framework combining Federated Learning, Genetic Algorithms (GA), and Meta-learning into a multi-layer Industrial Cyber Physical Systems (ICPS) framework named FedGA-Meta (Guendouzi et al., 2025). The authors first model the ICPS. Key elements of the ICPS, such as sensors, actuators, edge/fog/cloud servers, the network links, the data flows, and the Federated Learning process itself, are formalized as a Labeled Transition System (LTS). The behavioral models are then represented as timed automata in UPPAAL, and each property is posed as a query to the UPPAAL model checker. UPPAAL then exhaustively explores every possible execution within specified timing bounds to verify that each property holds, producing a counterexample otherwise. This provides a machine-checked guarantee that the FedGA-Meta system satisfies the stated properties (Guendouzi et al., 2025).

Lastly, S40 first utilizes active automata learning using Angluin’s L* algorithm to learn a minimal Deterministic Finite Automata (DFA) abstracting an RNN’s behavior (Khmelnitsky et al., 2021; Angluin, 1987). Statistical Model Checking is then used to determine whether the DFA abstraction violates a specification. When violations are found, the method checks whether the RNN truly violates the property or whether the DFA abstraction inaccurately represents the RNN, refining the learned DFA accordingly (Khmelnitsky et al., 2021).

Broadly, Runtime Verification is a lightweight formal method applied at runtime to provide rigorous guarantees of property adherence for a system. Runtime Verification provides precise information on the runtime behavior of the monitored system at the cost of limited execution coverage. Within this method, safety properties are specified in a formal language, a monitor is generated from the formal specification, and the monitor is instrumented to extract information from the system to verify property adherence against the system trace (Bartocci et al., 2018).

S8 combines runtime assurance with Theorem Proving and SMT-solving to formally verify an airborne collision avoidance system that utilizes an NN for decision making (Cofer et al., 2022). The runtime monitor evaluates the flight plan generated by the collision avoidance system and contains a safe backup planner. The results of the evaluation are then fed into a decision logic component which selects a backup flight plan that ensures safe flight. The decision component decides on a safe flight plan based on a tabular specification of safety rules. The decision logic code is synthesized from a formal specification, and formal proofs of correctness are produced after each step during synthesis. Additionally, the runtime monitor is modeled using the Architecture Analysis and Design Language (AADL) and the study formally analyzes the monitor, using SMT-solving, against safety properties. The Assume Guarantee Reasoning Environment (AGREE) is used to analyze the runtime assurance architecture (Cofer et al., 2022).

Similarly, S16 utilizes both Theorem Proving and runtime monitoring within a modified formally constrained RL framework (Hunt et al., 2021). The study introduces Verifiably Safe Reinforcement Learning (VSRL). Rather than assuming a perfect simulator state, VSRL first trains a lightweight object detector, using only a handful of labeled examples, to extract the positions of safety-critical objects from raw Red Green Blue frames. Instead of embedding constraints directly into the RL algorithm, VSRL wraps the original environment with a safety guard. Whenever the agent proposes an unsafe action, that action is replaced at runtime by a uniformly sampled safe one. The authors show that this method produces a refined MDP in which all actions are safe and whose optimal policies exactly correspond to the best safe policies in the original problem. The authors also use dL to model the environment and controller as hybrid programs. The safety properties are then specified and proven in the dL proof calculus, thus providing a formal certificate of safety from the starting safe state (Hunt et al., 2021).

S38 also focuses on RL-based systems and formally verifies an RL agent implemented via MATLAB Simulink’s RL toolbox (Adelt et al., 2021). The RL component’s acceptable behavior is specified as a hybrid contract in dL, constraining which actions are permitted in which observed states. The Simulink model is transformed into a dL hybrid program where the RL agent is represented as a monitored nondeterministic choice over safe actions consistent with the contract. Safety properties are proven deductively for the entire closed-loop system, and runtime monitors ensure the trained agent’s executed actions satisfy the contract during simulation and training.

S10 performs runtime monitoring of multiple parallel ML models for Over-The-Air (OTA) updates in CPSs (Guissouma et al., 2023). An initial ML model, such as an Artificial Neural Network (ANN), is first deployed after training and the first validation phase. While operating, new data may be added to the original training base, requiring an updated ML model to be sent OTA to the vehicle. As a result, a new version of the ML model is retrained. Further, multiple models may be retrained with different configurations under the new training data. These newly trained versions are then deployed to the vehicle as Shadow Versions (SVs), where each model runs in parallel to the Active Version (AV). While all versions are running, online monitoring to ensure safety is simultaneously performed on all models, and safety properties are expressed in STL. The data during monitoring is collected and analyzed, and when one SV is found to be more robust or safe than the AV, the AV is replaced with that SV. The AV serves as the baseline for future updates (Guissouma et al., 2023).

On the other hand, S11 incorporates formal symbolic reasoning into the construction of runtime monitors for DNNs (Cheng, 2021). The monitors are based on building an abstraction out of neuron activation patterns from the training data. In order to accomplish this, a set of neurons to be monitored is selected. Then, feature vectors are formed by taking the values of all monitored neurons for each input into the DNN. The monitor is constructed algorithmically by building a compact set representation that contains all feature vectors. The result is a sound guarantee, in that a warning over an input means that there is no close input within the training dataset. However, one issue with this approach is that in real-world deployment, the method can yield many false alarms. As a result, the study proposes a different monitor construction algorithm with robustness guarantees (Cheng, 2021).

On a separate note, S13 applies Runtime Verification to verify ML-driven chatbots (Ferrando et al., 2023). A runtime monitor is placed outside of a chatbot environment to ensure that the interaction between the user and chatbot follows a specified interaction protocol. The study presents Runtime Verification for Rasa (RV4Rasa), a framework for the runtime verification of chatbots developed in Rasa. These chatbots can be run locally on a computer rather than solely on the cloud. However, the framework can be extended to other chatbot environments. RV4Rasa checks, based on structured data generated during the Natural Language Understanding (NLU) step, whether the user and chatbot follow a specified interaction protocol (Ferrando et al., 2023).

Lastly, S18 develops runtime monitors analyzing the input and output channels of black-box neural networks, especially CNNs and ANNs (Tripuramallu et al., 2024). Safety properties are formalized as Valued Discrete Timed Automata (VDTA), and the runtime monitors are synthesized from the VDTAs. The runtime monitor operates over the inputs into the CNN and the outputs of the CNN.

A shield is a reactive system implemented alongside the learning agent that enforces safety properties specified in a formal language. The learning agent observes the environment and selects an appropriate action, which is then checked by the shield and corrected if the chosen action is deemed unsafe (Newcomb et al., 2024). Note that shielding is also utilized in S15 and S44, however, shielding was not the primary formal technique presented in these studies and they are therefore both discussed in Section 4.4.

S17 presents a dynamic shielding mechanism for MARL where shields dynamically split and merge depending on agent behavior, promoting collaboration between agents to ensure safety (Xiao et al., 2023). When agents are at risk of conservative behavior, such as when their shields block them from taking action due to lack of coordination, the independent shields can merge into one shield that maintains the safe behavior of the agents. When the agents move apart from each other, the shields split apart into multiple shields.

The authors present an algorithm for shield synthesis called K-Step Look Ahead Shields, which is a variant of traditional shield synthesis. The result is potentially improved computational efficiency when shields dynamically centralize and decentralize, as well as improved coordination among agents. The authors use Linear Temporal Logic to formalize safety specifications. They also theoretically prove that their shield synthesis technique guarantees safety (Xiao et al., 2023).

Control Barrier Function (CBF) methods assign a scalar safety score to each state. The safe set is then the super-level set containing all states with scores above a safety threshold, e.g., zero. A function is a CBF if, for every safe state, there exists at least one action that guarantees the next state remains in the safe set. Controllers can then use this condition as a safety filter to choose actions that keep trajectories inside the safe set. S7 reduces a learned Value Function (VF) used within RL into a CBF and verifies that the CBF is valid. If the CBF is valid, the CBF may be used as a formal safety certificate for the RL policy. Experiments validate that learned CBFs as safety filters or certificates are feasible through RL (Tan et al., 2024).

Risk verification is a data-driven formal verification approach that, from system execution traces, computes high-confidence bounds on statistical risk metrics over the distribution of robustness values for a given formal specification. The method therefore quantifies the tail risk—the risk from the extreme end of the robustness distribution—and the severity of potential violations (Cleaveland et al., 2022).

S24 applies risk estimation to NN controllers (Cleaveland et al., 2022). Safety requirements are formalized as either simple state constraints or STL formulas. The study estimates, from trajectory data, the risk that an NN-controlled stochastic system will violate a formal specification. Then, the study characterizes how that risk changes when the system is perturbed (e.g., due to environmental changes or modeling errors), by deriving bounds based on measures of system closeness. The steps of the verification framework are as follows:

This section answers the second half of RQ1 by detailing how the described formal methods improve upon traditional verification approaches. S1-2, S4, S8-10, S12, S14, S16, S23, S26, S28, S31-32, S36-37, S41 and S43 briefly discuss limitations to traditional verification that warrants the investigation of formal methods for providing correctness guarantees on ML systems. The evidence from these studies supports deeper studies into formal methods as opposed to traditional methods for enforcing safe ML.

S1 briefly discusses how traditional verification techniques on RL-based systems only provide statistical guarantees at most, and more rigorous techniques, such as formal methods, are therefore necessary to ensure the trustworthiness of RL-based systems (Tao et al., 2025). Also related to RL-based systems, S16 mentions that providing a purely statistical guarantee that an RL agent remains within safe states requires an infeasible amount of training data (Hunt et al., 2021).

Regarding NNs and DNNs, S23 also discusses statistical testing and explains how statistical testing of autonomous systems utilizing NNs is insufficient when verifying the safety of these systems (Meyer, 2023). S2 explains that testing an NN on a complete input set is impossible as the input set is often infinite. Therefore, testing is insufficient when verifying an NN (Naseer et al., 2020). Similarly, S4 mentions that traditional verification techniques on DNNs, such as simulations or random noise analysis, are insufficient for providing formal guarantees against errors. Therefore, traditional methods cannot capture the complete set of potential faults, which is necessary for safety critical applications (Samadi et al., 2024). S8 explains that because the behavior of NNs is largely attributed to its training data, it is generally impossible to determine the correctness of a NN through white-box methods or design analysis (Cofer et al., 2022). S9 describes that traditional verification, such as simulation testing, is non-exhaustive by nature. Therefore, incorporating formal methods into the verification process will help obtain additional completeness guarantees of autonomous software (Nenchev, 2025).

On a similar note, S10 mentions that simulation-based verification methods are insufficient for verifying real-time systems, and as a result, some software errors only become apparent after deployment (Guissouma et al., 2023). S14, S28, and S41 explain that traditional verification techniques, i.e., testing, for DNNs are useful for finding input samples that lead to incorrect behavior. However, these techniques cannot prove the absence of input samples leading to incorrect behavior (Zhang et al., 2022; Hafaiedh et al., 2025; Zhang et al., 2023). S26 discusses how human inspection is not a viable verification technique to use for NNs as it is for traditional programming. The complex function representation of a NN makes it challenging to verify through mechanical analysis of important properties (Gros et al., 2022).

Lastly, S12 argues that existing methods for verifying image-based NNs are limited by the high dimensionality and complexity of image inputs. Therefore, these methods face scalability challenges and computational inefficiencies. Additionally, traditional verification techniques often treat NNs as black-boxes, which makes it challenging to understand the decision making within each controller (Parameshwaran and Wang, 2025).

RQ2 asks, “in which domains or applications have these formal methods been successfully employed for safe ML?” Table 4 presents the application areas or domains identified, the number of articles that correspond to that application/domain, and the source IDs. The largest number of articles discuss the formal verification of NN and DNN controllers, followed by RL policies and NNs/DNNs for perception.

RQ3 asks, “What gaps, limitations, and challenges exist within current work combining formal methods and ML safety, and what future research directions can be identified from these gaps?”