Uninformed search strategies refer to a group of search techniques wherein the algorithm has no access to any information regarding how far the goal state is, though the algorithm can check if the current state is a goal state or not. It is also known as blind search. Table 1 summarizes the current uninformed search techniques (Russell and Norvig, 2003), including breadth-first search, depth-first search, depth-limited search, iterative deepening search, uniform cost search, bidirectional search, Dijkstra (Dijkstra, 1959), and their performance criteria. Among uninformed search methods, the Dijkstra algorithm is considered as benchmark both in terms of time and space complexity since its complexity does not depend either on the branching factor or the depth of the solution, but only depends on the number of nodes in the network.

The primary benefit of heuristic-based strategies is that the search space can be narrowed down. Many shortest path algorithms with a reduced search space have been proposed, such as best-first search (Pearl, 1984), greedy best-first search, hill climbing search, and A* (Hart et al., 1968) as shown in Table 1. These types of search methods attempt to narrow down the search space by utilizing various sources of additional information. Also, the A* algorithm is complete and optimal only if its heuristic function is admissible and monotonic, and many modern algorithms such as D* lite (Koenig and Likhachev, 2002) and LPA* (Koenig and Likhachev, 2001) are based on A*. Therefore, the A* algorithm stands out in the class of informed search because of its completeness and optimality.

Incremental search strategies are applicable for networks in which only a small number of link costs are likely to change at a time. Since these changes affect only a part of the graph, recomputing shortest paths for the entire graph is not necessary. Instead, it is possible to update paths corresponding to only a subset of the graph. As a result, such methods are used to solve dynamic shortest path problems, which require determining shortest paths repeatedly as the topology of a graph or its link costs only change partially. A representative incremental search algorithm is the Ramalingam and Reps’ algorithm (Ramalingam and Reps, 1996b) (RR), also known as the DynamicSWSF-FP algorithm. However, if all the link costs in the graph change, incremental search algorithms are unable to capitalize on previous search results.

The field of RL has grown significantly over the past decades, with applications ranging from robotics (Polydoros and Nalpantidis, 2017), communications (Luong et al., 2019) to gaming AI (Vinyals et al., 2019). Among RL algorithms, Q-routing is a method that was proposed primarily for communication networks but can also be used in transportation systems (Boyan and Littman, 1994). Q-routing is based on Q-learning (Sutton and Barto, 2018) and does not need to know the link costs to start off, and can learn optimal paths over time through experience. In particular, for every node x in the graph, a 2D Q-table, Q x ( d , y ) , is maintained that contains the cost of transitioning from x to node y (where y is one of the neighbors of node x , i.e., y ∈ N ( x ) , if the final destination is node d . During a training episode, at each step, the next move is chosen based on the current values in the Q-table, the move is executed, and the actual time experienced while executing the move is stored. Based on this experience, a Bellman update of the Q-values is performed as follows: Q x d , y * = Q x d , y * + η m i n z ∈ N y *   Q y * d , z + t − Q x d , y * (1) where y * is the next move chosen based on current Q-values, i.e., y * = argmin y ∈ N x   Q x d , y (2)

In Equation 1, z belong to the list of the adjacent nodes of y * , η is the step-size, and the t is the time experienced estimated to go from x to y * which serves as a surrogate for the link cost (reward). In other words, if Q-values are consistent, then Q x ( d , y * ) , where y* is computed using Equation 1, should be equal to the time estimated from x to y * plus the cost associated with the best move thereafter, and the update is proportional to the difference from this desired value.

As shown in Table 2, Q-routing requires less run-time complexity as it is executed more often. During training mode or offline mode, it takes O ( T ∗ n b ) complexity to run the Q-routing algorithm, where T is the number of training loops, n is the number of nodes and b is branching factor. But, time complexity is reduced to O ( 1 ) during online mode.

We first test the Dijkstra and A* algorithms on our simulated transportation network when there is no vehicle. The purpose of this test is to check the performance of the heuristic function in A * . When there is no traffic in the network, the link costs are the link lengths and the network is static. Figure 9 shows the identical performance of A * and the Dijkstra algorithm in terms of the quantile-quantile (Q-Q) plot between the optimal route costs of the two algorithms for the following list of 10 OD pairs: (1,25), (5,21), (10,16), (6,24), (22,4), (4,16), (18,5), (12,20), (17,5) and (22,9).

When traffic is introduced into the simulated network, the network becomes dynamic and link costs are estimated using methods described in Section 3.2. Dijkstra, A * and Q-routing are tested in this dynamic network. Q-routing parameters are found to be optimal with a learning rate of 0.29 and 200 iterations. Figure 10 shows a graph with route cost for OD =(1,20) for each learning rate. Route cost starts to be minimal for a learning rate of 29 × 1 0 − 2 .

In the dynamic simulated network, A * , Dijkstra, and Q-routing are invoked every 10 s of simulation with all OD pairs, and a few OD pairs are selected for illustrating and comparing route costs. Figure 11,13,12,14 show the cost in terms of sum of the travel times for minimum cost routes for OD = ( 1,25 ) , ( 5,21 ) , ( 4,16 ) and (6,14). It can be seen that the data points for Q-routing overlap with those for Dijkstra. Moreover, Supplementary Appendix A illustrates the Q-Q plots between route costs between A * , Q-routing, and Dijkstra for 10 OD pairs: (1,25), (5,21), (10,16), (6,24), (22,4), (4,16), (18,5), (12,20), (17,5), (22,9).

Based on Figures 11–14, it is clear that the A * algorithm produces non-optimal shortest routes, resulting in larger route costs as compared to those achieved by Dijkstra and Q-routing. This is because the heuristic function for A * , used in this simulated network no longer remains admissible as the link costs become dynamic. At any node during the A * run-time, the heuristic function always overestimates the route cost to the destination.

On the other hand, Q-routing delivers comparable route costs to Dijkstra during the entire simulation of the network. Following points should be considered while interpreting the plots in Supplementary Appendix Figures A1–A10 of Supplementary Appendix A. In a perfect information setting wherein the true link costs are accessible to the algorithm, it is well known that the Dijkstra algorithm gives the optimal shortest path between any two nodes of a network (Cormen et al., 2022). That is, Dijkstra gives the best-case cost when correct link costs are known. The costs corresponding to the Dijkstra algorithm shown in the plots of Supplementary Appendix Figures A1–A10 are those when Dijkstra was run as if the true costs were known. Although the true costs are not known in a realistic setting, the x-axis coordinates in the aforementioned Q-Q plots represent the theoretical upper bound of the performance for respective OD pairs. Along the y-axis we plot the costs produced by the Q-learning algorithm, wherein Q-learning is performing routing without knowing the costs. Thus, if in the Q-Q plot, the scatter is around the 45° line, it goes on to showing that the Q-learning algorithm, under an imperfect information setting, tends to perform close to an algorithm which gives the theoretical upper bound under perfect information setting. This is our main result, i.e., Q-learning tends to give near optimal result. Further, Supplementary Appendix B visualizes the agreement and differences in shortest routes estimated by Q-routing, Dijkstra and A * for OD =(1,25) pair.