Carbon emission is produced directly or indirectly by various manufacturing links, such as raw materials consumption, machine operation, transportation, and metal debris treatment (Gutowski et al., 2005). In this paper, the electrical energy consumption of machine operation and the energy consumption of coolant treatment are identified as the main carbon emission sources in CEA-FJSP.

The CEA-FJSP is a multi-objective optimization problem, considering both economic and environmental benefits. The scheduling objectives are to simultaneously minimize C max = max ⁡ ⁡ { C i n i | i = 1,2 , … , n } and T C E . Therefore, the mathematical model of CEA-FJSP is formulated as: min ⁡ f = min { w 1 C max + w 2 T C E } (7) s . t . { C max ≥ C i j , ∀ i , j ( a ) C i j = S i j + t i j k , S i j ≥ 0 , ∀ i , j , k ( b ) ∑ M k ∈ M i j x i j k = 1 , ∀ i , j ( c ) S i , j + 1 ≥ C i j , ∀ i , j ( d ) C i ′ j ′ − C i j ≥ t i ′ j ′ k ,   y i j i ′ j ′ , k = 1 ( e ) (8) Eq. 7 shows that the objective function minimizes the weighted sum of C max and T C E , converting the multi-objective optimization problem into a single-objective optimization problem, where w 1 and w 2 are the weights corresponding to C max and T C E , respectively. Eq. 8 shows the five constraints. Constraint (a) in Eq. 8 describes the relationship between makespan and the operation completion time. Constraint (b) ensures that the operation completion time is equal to the sum of the start time and the processing time. Constraint (c) specifies that an operation can be assigned to and processed by only one machine. Constraint (d) guarantees the precedence constraint between the operations of the same job. Constraint (e) shows that a machine can process only one operation at a time.

An MDP mainly includes three components: state, action, and reward. A complete decision-making process of MDP is called an episode, consisting of T = ∑ i = 1 n n i decision steps in CEA-FJSP, where one decision step corresponds to one interaction. At the decision step t , the scheduling agent perceives the state s t of the scheduling environment. Then, the state features are fed into the scheduling policy that in turn selects an action a t . After the execution of the action a t , an unscheduled operation is selected and assigned to a candidate machine. Hence, the selected operation becomes scheduled. After that, the scheduling environment releases a reward r t to reflect the change of the scheduling objectives, as well as updates to a new state s t + 1 ready for the next interaction.

The goal of the scheduling policy is to determine the best-matched action for a given state. In this paper, a DNN with parameter θ consisting of six fully connected layers is employed to parameterize the scheduling policy denoted as π θ ( a t | s t ) . The input layer has ten neurons equal to the number of the state features, and the output layer outputs the probability distribution over the nine actions. Each of the first three hidden layers has sixty-four neurons while the fourth hidden layer has thirty-two neurons, and the Tanh activation function is used for all hidden neurons.

The PPO algorithm is adopted to train the policy network, where the state-value function V ( s t ) is approximated by another DNN with parameter ∅ , denoted as V ∅ ( s t ) . V ∅ ( s t ) has the same structure as π θ ( a t | s t ) except that the output layer consists of only one neuron, and shares the first three hidden layers with π θ ( a t | s t ) to utilize the learned abstract features.

DRL establishes an interaction framework between the agent and the environment using the MDP components: state, action, and reward. The agent learns to optimize its decision-making policy through the interaction, i.e., tunning its policy network π θ ( a t | s t ) . Figure 3 illustrates the PPO-based DRL training process for the CEA-FJSP, where a training cycle includes a sampling phase and an update phase. Two same policy networks π θ o l d and π θ are set-up in the beginning of training to facilitate the training process. During a training cycle, π θ o l d remains unchanged throughout the sampling and update phases, while π θ is updated multiple times during the update phase.

In the sampling phase, π θ o l d interacts with the scheduling environment to collect sufficient state-action-reward tuples, ( s t , a t ,   r t ) , and store into a memory buffer. In the update phase, π θ is updated for several epochs with the collected data. After that, π θ o l d copies π θ and then starts the next training cycle. The surrogate objective loss function of the policy network is defined as: L t C L I P = E t [ min ( π θ ( a t | s t ) π θ o l d ( a t | s t ) A ^ t ,   c l i p ( π θ ( a t | s t ) π θ o l d ( a t | s t ) , 1 − ε , 1 + ε ) A ^ t ) ] (16) where E t [ ⋅ ] denotes the empirical average, π θ ( a t | s t ) π θ o l d ( a t | s t ) is the importance sampling weight, and c l i p ( · ) is the constraint function with hyperparameter ε to ensure the similarity between π θ and π θ o l d . A ^ t is the generalized advantage estimation (GAE) function.

The value network is updated through the mean squared error (MSE) loss function: L t V F = E t [ ( V ∅ ( s t ) − V t t a r g ) 2 ] = E t [ ( V ∅ ( s t ) − ∑ i = t T r i ) 2 ] (17)

Due to parameter sharing, the entire network model is trained with the loss function: L t C L I P + V F + S = E t [ L t C L I P − c 1 L t V F + c 2 S [ π θ ] ( s t ) ] (18) where S [ π θ ] ( s t ) is the entropy bonus to encourage exploration, while c 1 and c 2 are the coefficients.

The pseudo-code of the training process is presented in Algorithm 1. Here, N training instances are initialized at the beginning of a training cycle to prevent the DRL scheduling model from overfitting a specific instance. The data collected from the sampling phase is used to calculate the cumulative gradients to update parameters θ and ∅ for K epochs.

Algorithm 1

Training process for CEA-FJSP using PPO

Input: training cycles L; memory buffer M; update epochs K: number of training instances N

Five Mk03Ex instances were generated in each training cycle based on the Mk03 configuration in Table 4 with the parameters in Table 5. These instances were used to train the proposed DRL scheduling model to produce the DRL-Mk03Ex scheduling solver. Table 6 lists the values of hyperparameters of Algorithm 1. Both weights w 1 and w 2 were set to 0.5 to equally evaluate the contribution of makespan and carbon emissions to the reward. Furthermore, reward scaling (Engstrom et al., 2020) was adopted to stabilize the training process. The hardware for training was a PC with a single Intel Xeon E5-2678 V3 @ 2.50 GHz CPU and a single NVIDIA RTX A2000 GPU. Algorithm 1 was implemented using Python 3.7, with PyTorch to deploy the network model.

Figure 4A–C show the training histories of the reward, C max and T C E , respectively. It can be seen from Figure 4A that the reward gradually increases with the advance of the training process. Figure 4C shows that T C E continuously contributes positively to the reward, as it decreases monotonously along the timeline. It can be seen from Figure 4B that C max increases till about the 2200th cycle and then decreases until the end. The results show that the contribution of T C E to the reward surpasses C max in the early stage of optimization, and finally, both T C E and C max are optimized by the DRL scheduling model. All three curves in Figure 4 begin to converge around the 7000th cycle and all of them oscillate slightly thereafter. Therefore, the training process had better stop around the 7000th cycle or the performance could get worse, exhibiting a kind of overfitting behavior.

One hundred additional Mk03Ex instances different from those used in the training stage were generated to test the DRL-Mk03Ex against the proposed scheduling rules S R 1 to S R 9 and GA (Yin et al., 2017) respectively.

Figure 5 shows the performance of DRL-Mk03Ex over the Mk03Ex instances. It can be seen from the figure that DRL-Mk03Ex outperforms all the scheduling rules and GA on the testing instances, i.e., it achieves the lowest average makespan and the lowest average total carbon emission. Although GA and some scheduling rules ( S R 5 , S R 6 , S R 8 ) perform well in reducing makespan or total carbon emission, none of the scheduling rules and GA can simultaneously minimize the two objectives. Furthermore, DRL-Mk03Ex is also significantly better than the scheduling rules and GA in terms of N P . The results confirm the superiority and optimization ability of DRL-Mk03Ex.

The DRL scheduling solver built on the Mk03Ex instances (DRL-Mk03Ex) was tested on the Mk01Ex, Mk02Ex, and Mk04Ex to Mk10Ex instances. That is, to say, the instances used for testing were different from the ones used for training, and the difference was significant in the sense that the testing and the training instances were sampled from different configurations. To compare the performance, Table 7 shows the average results of three metrics over 100 instances for nine different instance configurations. The proposed scheduling rules S R 1 to S R 9 and GA are used as the baselines and the best values of each metric are highlighted in bold font in Table 7.

Table 7 shows that DRL-Mk03Ex achieves the best solutions in most instances compared with the scheduling rules and GA. Furthermore, Mk01Ex and Mk02Ex instances have a simpler configuration than the Mk03Ex instances, while Mk04Ex to Mk10Ex instances have a more complex configuration. This means the DRL-Mk03Ex can be bidirectionally generalized. Besides, DRL-Mk03Ex can achieve comparable performance with GA in the simple scheduling instances, while surpassing GA in the complex instances. Moreover, DRL-Mk03Ex is also more robust than the scheduling rules. For example, A C changes from 52.86 to 595.41 s for DRL-Mk03Ex, while from 100.5 to 1380.95 s for S R 1 . The performance fluctuates even more wildly for the complex instances among the scheduling rules. For example, S R 5 and S R 3 achieve 560.22 and 1421.84 s as A C values for the Mk10Ex instances, respectively.

The Mk03Ex instances were used to train the DRL scheduling model under various weight combinations ( w 1 ,   w 2 ) : W C 1 = ( 0.0 ,   1.0 ) , W C 2 = ( 0.25 ,   0.75 ) , W C 3 = ( 0.5 ,   0.5 ) , W C 4 = ( 0.75 ,   0.25 ) , and W C 5 = ( 1.0 ,   0.0 ) . Consequently, five DRL scheduling solvers were built. These solvers had the same model structure distinguished by different parameter values. Figure 6 illustrates the resultant C max and T C E as well as carbon emission components, C E p , C E r and C E f for processing state, idle state, and coolant treatment, respectively, of the five solvers.

The results demonstrate the nonlinearity of the DRL scheduling solvers. The C max does not vary monotonously with the weight w 1 ; nor does the carbon emission with the weight w 2 , i.e., either the makespan or the carbon emission is affected jointly by w 1 and w 2 . It also implies that the DRL scheduling solvers cannot directly control the sub-optimization objectives. Instead, the weights should be treated as optimization parameters in the sense that the weighted optimization objective can be figured out for a given instance by adjusting w 1 and w 2 . For example, W C 2 is the best among the five weight combinations for the Mk03Ex instances.

For each weight combination, the three carbon emission components, C E p , C E f , and C E r , contribute roughly 56%, 34%, and 10% to T C E , respectively. Specifically, the machines produce the most carbon emission when processing an operation, and the least carbon emission when in the idle state. The carbon emission caused by the coolant treatment is also nonnegligible. Furthermore, it can be observed that as C E p , C E f , and C E r maintain a similar tendency with the T C E along the weight change. The results stimulate the possibility to simplify the representation of carbon emission by replacing T C E with C E p or with the sum of C E p and C E f .