Before a ship berths at the destination port, it needs to report the ship type, expected arrival time, amount of cargo to be loaded and unloaded, and expected departure time. Based on this information, the port assigns berths and quay cranes to ships to minimize the total port cost. The berth-quay crane allocation process can be mapped to a two-dimensional space-time diagram. As shown in Figure 1, the time-variant quay crane allocation allows the quay crane to leave to serve other vessels at any time during the execution of loading and unloading activities but not to cross other quay cranes while working. The solid line in the figure indicates a feasible allocation of quay cranes, and the dashed line indicates an infeasible allocation of quay cranes. The core of time-variant quay crane assignment is its flexibility, which allows the quay cranes to switch service between different vessels while ensuring safety and efficiency. For example, the No. 1 quay crane can provide service for the No. 4 vessel when the No. 3 vessel has not yet left the harbor; the No. 2 quay crane can connect to the unloading and loading operation of the No. 2 vessel after the completing its tasks at the No. 5 vessel. Such a time-variant scheduling program enables more flexible scheduling of quay crane resources and improves the loading and unloading efficiency of the terminal.

To achieve time-variant scheduling of quay cranes, we must determine which quay cranes serve each ship in each time period. We construct an integer programming model with the objective of minimizing the total cost of ship delay cost, quay crane service cost, and quay crane movement cost by considering the quay crane working range and constraints on quay crane crossing. In the model, we take the ship berthing time, berthing position, and the serial number of the ship served by each time period of the quay crane as decision variables. In order to better study this problem, the following assumptions are made in the next discussion: 1) Ship berthing is not limited by tide, water depth, mechanical failure, etc., 2) Ship unloading and loading services are continuous and cannot be stopped in the middle, and each ship has a range of the number of quay cranes that can be allocated; 3) Quay cranes cannot cross each other; 4) The movement time of the quay cranes is negligible.

For better understanding, we provide an explanation of the notation used in the model in Table 1.

We construct an integer programming model with the objective of minimizing the total ship delay cost, quay crane service costs, and quay crane movement costs by taking into account the quay crane working range, constraint limitations of quay crane crossings, and other elements as follows: min   C = ∑ i ∈ S ∑ t ∈ T c t service ∑ q ∈ Q z i q t + c i delay e i − d i + c move ∑ t ∈ T ∑ q ∈ Q v i q t . (1)

Equation 1 addresses the objective function, that the total cost of the port is the lowest, and has three components: the first term is the cost of the quay cranes service, the length of time the quay cranes service the ship, and the cost of the quay cranes at different moments of the service. The second is the cost of the ship’s delays; this item will only be established if the ship’s departure time is later than the expected time of departure; otherwise, it will be zero. The third is the movement cost of the quay cranes, which comes from the personnel dispatch and resource consumption caused by the stopping and restarting of the quay cranes and is established when the quay cranes change the service ship or serve the ship for the first time. s . t .   b i ≥ a i , ∀ i ∈ S . (2)

Equation 2 ensures that each ship berths later than its arrival time, avoiding impractical scheduling arrangements. e i = b i + f i , ∀ i ∈ S . (3)

Equation 3 calculates the departure time of a ship based on the loading and unloading time of the ship. p i + l i + r ≤ p j + M ⋅ 1 − y i j , ∀ i , j ∈ S , (4) y i j + y j i ≤ 1 , ∀ i , j ∈ S , (5) e i ≤ b j + M ⋅ 1 − x i j , ∀ i , j ∈ S , (6) x i j + x j i ≤ 1 , ∀ i , j ∈ S , (7) x i j + x j i + y i j + y j i ≥ 1 , ∀ i , j ∈ S . (8)

A series of constraints from Equations 4–8 ensures that only one vessel can berth at the same location at any given time, avoiding scheduling conflicts in time or space. p i + l i ≤ L , ∀ i ∈ S . (9)

Equation 9 requires that the berthing position of the ship be on the shoreline of the port and that the feasibility of the berthing position be ensured. e i ≤ T max , ∀ i ∈ S . (10)

Equation 10 requires all ships to leave port during the berth planning period. ∑ i ∑ q z i q t ≤ Q max , ∀ i ∈ S , ∀ t ∈ T . (11)

Equation 11 ensures that the total number of quay cranes serving ships at any one time is less than the total number owned by the terminal. ∑ q z i q t ≥ q i min , ∀ i ∈ S , ∀ t ∈ T , ∀ q ∈ Q , (12) ∑ q z i q t ≤ q i max , ∀ i ∈ S , ∀ t ∈ T . (13)

Equations 12, 13 require that the number of quay cranes on the service ship at each point in time be greater than the minimum required number of quay cranes and less than the maximum number of quay cranes that the ship can accommodate. ∑ t ∑ q η i t z i q t ≥ w i , ∀ i ∈ S , ∀ t ∈ T . (14)

Equation 14 ensures that the loading and unloading tasks of the ship can be accomplished. M ⋅ m i t ≥ ∑ q z i q t , ∀ i ∈ S , ∀ t ∈ T , (15) m i t ≤ ∑ q z i q t , ∀ i ∈ S , ∀ t ∈ T . (16)

Equations 15, 16 indicate that if ship i is serviced in time slot t , then m i t takes the value of 1 and otherwise 0. f i = ∑ t m i t , ∀ i ∈ S . (17)

Equation 17 calculates the total loading and unloading time for each ship. ∑ t = 1 t max − 1 | m i , t − m i , t + 1 ≤ 2 , ∀ i ∈ N . (18)

Equation 18 ensures that the quay crane is available to serve the ship for each time period after it berths until the ship departs. z i a t q a l − z j b t q b l + z i a t + z j b t − 2 M ≤ 1 − y i j M , ∀ i ∈ S , ∀ j ∈ S , ∀ t ∈ T , ∀ a ∈ Q , ∀ b ∈ Q . (19)

Equation 19 ensures that the quay cranes do not cross. If ship i is to the right of ship j and y i j is 0, the equation is constant; if ship i is to the left of ship j , the right-hand side of the equation is 0; and if both ships require quay crane service at the same time, the quay crane that serves ship i must be all the way to the left of the quay crane that serves ship j . The equation also ensures that the quay crane that serves ship j will not cross. q k l − p i − l i ≤ M 1 − z i k t , ∀ i ∈ S , ∀ t ∈ T , ∀ k ∈ Q , (20) p i − q k r ≤ M 1 − z i k t , ∀ i ∈ S , ∀ t ∈ T , ∀ k ∈ Q . (21)

Equations 20, 21 provide that a ship can only be serviced within the working limits of the quay crane. v i q t ≤ z i q t , ∀ i ∈ S , ∀ t ∈ T , ∀ q ∈ Q , (22) v i q t ≤ 1 − z i q t − 1 , ∀ i ∈ S , ∀ t ∈ T , ∀ q ∈ Q , (23) v i q t ≥ z i q t − z i q t − 1 , ∀ i ∈ S , ∀ t ∈ T , ∀ q ∈ Q . (24)

Equation 22 through Equation 24 ensure that v i q t records the first time at time t that a quay crane q moves to a ship i to service it. Equation 22 ensures that the quay crane only moves to a ship if it must service that ship. Equation 23 states that when the quay crane q has already serviced ship i at the previous time period t − 1 , the move is not recorded. Equation 24 indicates that when quay crane q did not serve ship i in the previous time period t − 1 , and quay crane q serves ship i in this time period t , then a movement of the quay crane is recorded. p i ≥ 0 , b i ≥ 0 , ∀ i ∈ S , (25) x i j , y i j ∈ 0 , 1 , (26) m i t ∈ 0 , 1 , ∀ i ∈ S , ∀ t ∈ T , (27) z i q t ∈ 0 , 1 , ∀ i ∈ S , ∀ t ∈ T , ∀ q ∈ Q . (28)

Equation 25 defines the non-negativity of berthing position and berthing time. Equations 26–28 define the 0–1 variables.

We utilize the Q-learning algorithm to determine the genetic operators during the population iteration process and set the agent of the Q-learning as populations. The two-dimensional states and actions form a three-dimensional Q-table. The agent calculates the current state of the population and selects the appropriate action (i.e., genetic operator) to enhance the genetic algorithm’s convergence and optimization capabilities. The algorithm framework of the Q-learning part is shown in Figure 4.

We define the state of the environment in which the agent is located to be determined by the diversity of individuals in the population and the number of repetitions of the optimal individual fitness in the population that have not been updated. The diversity of individuals in the population is defined by the information entropy (as shown in Equation 30), where x i is the individual fitness in the population, p x i is the probability of occurrence of the individual fitness, − ∑ i = 1 n P x i log 2 ⁡ P x i is the information entropy of the population, which is log 2 ⁡ n when the population is completely disordered, and H x is defined as the complexity of the population. The number of repetitions without updating represents the number of times the current individual repeats the optimal state keeping. H X = − ∑ i = 1 n P x i log 2 ⁡ P x i log 2 ⁡ n . (30)

We define the state space as in Table 2, where i t e r is the number of repeated non-updates, and G is the total number of iterations.

Traditional genetic algorithms often employ a single crossover and mutation method, which can easily lead to local optima. Moreover, these operations are applied to the entire chromosome, making local adjustments challenging for relatively fit individuals. We implement diverse crossover and mutation strategies at three coding levels: ship berthing order, ship berthing position, and the number of quay cranes required for ships during different time periods. The actions of the Q-learning algorithm in this study are defined in Table 3.

We set two strategies in selecting the action: the first one is to use the Q-table for selection, and the other one is to select by greedy strategy. The greedy value is defined in Equation 31, where ε max is the hyperparameter represents the maximum greedy rate, f is the current iteration number, and G is the total iteration number. We use random selection of actions when r a n d o m < ε ; otherwise, one is randomly selected in the Q-table from the actions corresponding to the first three Q-values of the current state. ε = ε max 1 + e 10 × f − 0.6 × G G . (31)

After performing an action, the agent obtains a reward for that action, and we set the reward value as in Equation 32. The Q-table is updated as shown in Equation 33, where Q s t , a t denotes the Q-value for each generation of selecting action a t based on the state s t . α ∈ 0 , 1 , and γ ∈ 0 , 1 denote the learning rate and discount factor. max ⁡ Q s t + 1 , a t + 1 denotes the maximum Q-value at the next state s t + 1 when taking the next action a t + 1 . r e w a r d = + 1 if  f gbest new − f gbest old > 0 , 0 otherwise . (32) Q s t , a t = Q s t , a t + α reward t + 1 + β ⁡ max ⁡ Q s t + 1 , a t + 1 − Q s t , a t . (33)

We take the data from a port on 4 July 2023 as an example for our analysis. The quay shoreline extends for a total length of 800 m and is equipped with eight quay cranes, each with a working distance of 300 m. Fifteen ships arrive at the port successively, and their details, including the captain’s information, arrival time, expected departure time, and the amount of cargo to be loaded and unloaded, are shown in Table 4. We wrote the code in Python 3.9 and executed it on a computer featuring a Core i9 2.50 GHz CPU, 16.0 GB of RAM, and a 64-bit Windows 11 operating system. The parameter settings are shown in Table 5.

Based on the above relevant parameters, we calculate the results as follows: the cost of quay crane service is 235,320 CNY, the cost of quay crane movement is 70,670 CNY, the cost of ship delay is 35,000 CNY, and the final total cost is 340,990 CNY. The final allocation plan of the ship is shown in Figure 5.

The quay crane scheduling plan is shown in Figure 6. We illustrate with the example of Quay Crane 1. Quay Crane 1 initially remains idle and then commences service for Ship 2 mid-way through its stay; subsequently, it transitions to serve Ship 4. After completing the service, the quay crane stops working. It resumes service at 00:00 the following day to assist Ship 11 and ceases operation upon completion of this service. Quay Cranes 5, 6, and 7 also implement time-variant allocation. To ensure that Ship 12 departs on schedule, they cease service after a designated period of unloading and loading for Ship 12, thereby reducing the operational costs associated with the quay cranes.

Section 4 describes our use of the Q-learning algorithm to select genetic operators, which enhances the search speed of the algorithm. However, it is also possible to optimize the algorithm without using the Q-learning algorithm. To demonstrate the effectiveness of the Q-learning algorithm, we have designed two genetic operator selection methods for comparative analysis: random selection and selection using the Q-learning algorithm. We conducted five sets of experiments, with the number of quay cranes set at eight and a maximum of 3,000 iterations for the algorithm. Each operator’s results were run five times. The experimental results are shown in Table 6.

The comparison results indicate that the performance gap between the two algorithms is minimal when the number of ships is low. However, as the number of ships increases, the optimization capability of the Q-learning-assisted algorithm is significantly superior to that of the random selection method. This is attributed to the fact that with a smaller number of ships, the algorithm’s complexity is manageable. In contrast, as the number of ships grows, the random selection method introduces uncertainty and a higher likelihood of becoming trapped in local optima. The algorithm optimized with Q-learning also exhibits a reduced running time, demonstrating its superior search capabilities.

To verify the effectiveness of the algorithm presented in this article, we compare it with the CPLEX solver and genetic algorithm. The genetic algorithm adopts the traditional crossover and mutation logic and simultaneously performs crossover and mutation operations on the three layers of codes. We calculate the average of five runs for each algorithm under various scenarios, with the final results presented in Table 7. Given the difficulty of solving large-scale problems with CPLEX, we impose a 3-hour time limit on the solver, terminating it after this duration. The gap calculationformula is defined as follows, taking the genetic algorithm shown in Equation 34 as an example: G A P = O b j cplex − O b j G A O b j G A × 100 % . (34)

From Table 7, it can be found that when the number of quay cranes is constant, as the number of ships increases, the overall cost and solution time of the ships will increase accordingly. This is because an increase in the number of ships leads to an increase in the complexity and workload of loading and unloading operations, which increases the overall cost and solution time. In terms of cost, our proposed algorithm demonstrates lower costs across all examples, with its optimal costs significantly lower than those of the genetic algorithm. In comparison with CPLEX, our algorithm has a significantly lower GAP value than the genetic algorithm, indicating that its cost is much closer to the optimal cost. In certain large-scale scenarios (for example, 30 × 5 and 30 × 10), our proposed algorithm’s GAP is negative, indicating that its performance surpasses that of CPLEX, which is limited by time constraints. This further confirms the superior search capability of our algorithm in handling large-scale cases.

In Table 7, “Running time/s” refers to the computational time required to complete the optimization process, measured in seconds. This metric is used to evaluate the efficiency of the proposed method. The running time of our proposed algorithm is generally lower than that of the genetic algorithm and much lower than that of CPLEX, which reflects the high computational efficiency. Q-learning effectively reduces the running time by introducing a dynamic mechanism for operator selection. Q-learning leverages the performance of past iterations to dynamically adapt the selection of genetic operators. This ensures that only the most effective operators are utilized at each stage of the optimization process, thereby avoiding low-efficiency operations and reducing unnecessary computational effort. In summary, the algorithm proposed in this article offers significant advantages in terms of solution speed and accuracy.