There are two types of cryomodules, namely, type-A and type-B. The main difference between them lies in the fact that type-A contains one resonating cavity, whereas type-B contains two of them. For more details concerning the cryomodules design and performance, please refer to [5,6].

In terms of thermodynamics, both types of cryomodules undergo different heat loads. First is the static heat load induced by the heat transfer between the cold and their surrounding parts. Second is the dynamic heat load due to the RF resistive losses in the resonating cavity. Heat load amplitudes are different for the two types of cryomodules. For the 4.4°K bath, those characteristics as well as the volumes are given in Table 1.

The cryogenic system is in charge of keeping the superconducting RF cavities under their superconducting critical temperature at all times. This requirement is achieved by keeping the cavities submerged in a liquid helium bath. As the cavities are fed with RF power, a liquid helium bath ensures the resistive heat loads extraction at the cavity's surface walls. As a consequence, the temperatures of the cavities are kept uniform and stable at 4.4°K far beyond their superconducting critical temperature of 9.2°K. Would the cavity be partially exposed out of the liquid helium bath, it would undergo a quench ³ . Three main elements, shown in Figure 1, insure this constraint: a phase separator filled with liquid helium at 4.4°K and 1,200 mbar, a thermal shield that surrounds the phase separator and is kept at 60°K, and finally a valves box containing all the valves used to control cryogenics operation. As the phase separator is the most critical element of the cryogenic system, we will only focus on that element and its associated valves. The Figure 1 presents a simplified scheme with the subsystems of interest.

The phase separator is fed with liquid helium through the input valve, which is used to regulate the level of liquid. Due to thermal heat load, liquid helium evaporates and is returned to the cold box. In that process, gas goes through the output valve, which is used to regulate the pressure within the phase separator.

Both the valves and the phase separator have been modeled. The equations governing the operation of the valve are the ones given in the standard ISA [7], whereas the phase separator dynamics are described through energy and mass balance. The equations have been implemented in the Simcryogenics library [8] of MATLAB^®, which is a modeling tool used to simulate and optimize cryogenic systems. Helium properties are extracted from tabulated data using the HEPAK^® package. As those equations have been extensively described in [9–11], they will not be discussed in this article. Rather, the comparison between experimental and simulation results will be emphasized.

The simulation results for both cryomodule types have been compared to experimental data. For each of the cryomodules, an operating scenario has been performed starting from stable operating conditions ⁴ . This scenario is a series of steps applied to the input and output valve opening command. The same values have been applied to the model and to the real process in an open-loop manner. The comparison obtained for cryomodule 1 (the first one on the line considering the beam direction) is shown in Figure 2. The comparison shows a good agreement between experimental and simulated data for both level and pressure dynamics. It is worth mentioning that the uncertainty of the modeled liquid helium level increases with time as the level is an integrator system. Furthermore, the high uncertainty on the pressure at time t = 1,500 s is mostly due to the valve position uncertainty: an error of ±1% on valve position could lead to a pressure uncertainty up to 10 mbar. Finally, the pressure peak occurring at t = 1,200 s is due to a pressure oscillation in the cryodistribution (i.e. the inlet boundary of the model).

Similar results have been observed on all other cryomodules. The following criteria have been calculated for each comparison: C r = 100 ⋅ ∫ t i n i t t f i n a l V m e s t − V s i m t 2 d t V m o y i n i t ⋅ t t o t a l , (1) where Cr is the criteria representing the integral of the error between measured and simulated data. t _init , t _final , and t _total are, respectively, the initial time, the final time and the overall duration of the scenario, whereas V _mes (t) and V _sim (t) are, respectively, the measured and simulated values. Finally, V m o y i n i t designates the mean value at the beginning of the scenario. Normalizing by the overall duration and the mean value makes it possible to compare multiple scenarios with different durations and operating conditions. The criteria values obtained for the scenario shown in Figure 2 and applied to all cryomodules are plotted in Figure 3. The latter plot gives an important insight into the usability of such a model in a generic way for all cryomodules across the LINAC. In fact, obtained Cr values give a deviation sufficiently small to be considered for applications such as control and fault detection. The next sections will investigate this feasibility.

Cryogenic system control loops are critical items that can affect the overall accelerator. Two requirements are to be met in order to allow the nominal operation of the RF cavity. The first is to ensure that the temperature of the cavity remains below its critical value. Otherwise, the cavity could quench ⁵ . To do so, the cavity is submerged in a liquid helium bath, and the level of liquid helium is regulated through a PID (Proportional–Integrator Derivative) controller acting on the input valve (see Figure 1). The goal is to maintain a level at 90% ± 5% which is high enough to maintain the overall cavity fully submerged with a comfortable operating margin. The second is to ensure that the shape of the cavity does not change as the performances of the resonator are intrinsically linked to the cavity shape. This could be seen in the expression of its unloaded quality factor: Q f 0 = G R s , (2) where Q _f0 is the unloaded quality factor, Rs is the surface resistance, and G is the geometric factor that depends on the surface and the volume of the cavity. As the cavity is submerged in liquid a helium bath, any pressure variation (ΔP) in the separator will induce a mechanical force on the cavity walls that slightly deforms the cavity. This results in a variation of the geometric factor that can lead to a drop in the cavity quality factor, hence significantly reducing the nominal cavity accelerating gradient.

Considering the bandwidth of the cavity and its associated RF system, a pressure variation limit of ΔP = ±5 mbar has been set up for the SPIRAL2 cryomodules. It is worth mentioning that the nominal pressure of the helium bath is 1,200 mbar, which means that a ΔP of ±5 mbar represents a tolerance of ±0.41%. Both the level and the pressure are regulated through PID controllers. Although the PID performance is enough to achieve the level requirement, it is not the case for the pressure requirement. Even using a state-of-the-art [12] PID (Proportional, Integral Derivative) tuning tool, we were not able to maintain the pressure variation within a range of ±5 mbar for long periods of time (i.e., more than a few hours) without having significant overshoots. This is probably due to the fact that the two regulation loops are coupled: an action on the input valve influences the level and the pressure. A similar statement is also true for the output valve: an action on the output valve has an impact on both on the pressure and the level. As PID controllers are more suitable in the case of linear SISO ⁶ system, another control algorithm is necessary to achieve the project requirement.

Few parameters have to be considered while choosing the most suitable solution for the cryomodules control loops. First, the cryomodule cryogenic system is a multiple inputs multiple outputs (MIMO) system with two valves as inputs and the level and pressure as outputs. As there is internal coupling between all inputs and outputs, a controller that can handle this coupling is mandatory. Second, as the accelerator will be used for many years, it must be a solution proven on multiple systems with full documentation. Third, the controller has to be implemented in a dedicated PLC (Programmable Logical Controller) with a limited amount of calculation capacity. Considering those parameters, an LQ (Linear Quadratic) controller seems to be a good candidate. The block diagram of such controller applied to our system is given in Figure 4. The mathematical development of this controller has already been described in [9,11]. In this section, we will only recall the main equations of the discrete LQ controller and focus on experimental results.

The principle of a LQ controller is to synthesize a state feedback gain such that the command input is given by: u k = − K ⋅ x k , (3) where K is the state feedback gain and x the state of the system. K is calculated so that it minimizes the following quadratic cost: J = ∑ i = k ∞ x i T ⋅ Q ⋅ x i + u i T ⋅ R ⋅ u i , (4) where J is the cost, and Q and R are respectively state and input weights. As for gain and integral time for a PI controller, the goal is to tune the matrices Q and R to fulfill the process specifications. Details about the way to tune those gains are given in [9].

The calculation of the state feedback gain K requires the state-space model of the system which could be directly generated with the previously described model and a linearization algorithm such that the one described in [13]. To allow a comparison between the existing PID and the proposed regulation law, the LQ controller has been implemented on the existing PLC of each cryomodule. Even if they have a limited calculation capacity (a work memory of 192 Ko), it is more than enough for the proposed LQ controller which only requires around 30 multiplications/additions per sampling time. This is due to the fact that only the control law described in Eq. 3 and its associated Luenberger observer [14] have been implemented. The calculation of the state feedback gain K that minimizes the cost Eq. 4 could be carried out offline using dedicated optimizers.

Classical tests like set point variation and output disturbance test (using electrical heaters as disturbance sources) have been performed on both PID and LQ controllers. The results of those tests show that the LQ regulator is slightly better in terms of robustness and speed than the PI controller. Nevertheless, our main interest is to see which controller is able to respect the level and pressure requirements during a long period of operation. To check this, the following experiment has been realized for two nights ⁷ . During the first night (12 h), all cryomodules are regulated by PID controllers. During the second night, they are regulated by LQ controllers. For both cases, the level requirement has been respected, but not the pressure requirement. To illustrate this, the number of times the pressure overshoots the threshold of ±5 mbar has been used as a metric to compare the two controllers. The result of the comparison is given in Figure 5. The LQ controller shows no pressure overshoot at all, whereas the PID controller shows multiple ones. However, the LQ controller could not perfectly dump pressure oscillations. To illustrate this, a counter of overshoot with a tightened pressure threshold of ±3 mbar (in comparison with the specification of ±5 mbar) has also been plotted in Figure 5. Nevertheless, the results obtained with the LQ regulator are satisfying considering process requirements. As the algorithm has been deployed in the PLCs for the purpose of the test, it is already available for the current operation. This new control strategy is an important improvement that could reduce the accelerator downtime as one pressure overshoot may arise safety chains that shuts down the accelerator beam. One drawback of such method is the knowledge of the thermal set point of operation. This setpoint depends on static load, RF losses, and other effects such as beam loading. A deviation from the setpoint due to isolation vacuum leaks or field emitters in the cavity might make the LQ control worse than a simple PID ⁸ . In this matter, having state observers able to monitor the thermal behavior of a cryomodule is vital. Such an observer could drive the change of the thermal operation set point and LQ inputs to automatically adapt to the real state of the system. The synthesis of precision state observers using supervised learning will be the subject of future studies. The next section is the first study of a thermal load observer based on a twin model synthesis as a starting point for future planned studies.

As the RF signal injected in the resonator is sinusoidal, it generates energy dissipation in the cavity walls called AC losses [15]. Those losses are considered an indicator of the cavity state: an abrupt raise of those losses can indicate that a part of the cavity is no more in a superconducting state. This could be the premise of a global quench of the cavity. On another timescale, a slow increase of the dissipated AC losses can indicate a pollution of the cavity with non-superconducting elements. In the case of SPIRAL2, there is no continuous measurement of these AC losses. Measurements can only be performed when the cavity is not in operation as the measurement method is intrusive [16]. There is no operating solution in the case of SPIRAL2 that would allow us to perform such measurements online and without disturbing the process.

To solve this problem, we proposed a method to estimate these losses based on the phase separator model and an extended Kalman filter (EKF) [17].

From the phase separator point of view, the AC losses represent an external thermal heat load. The more AC losses, the more heat has to be extracted through the evaporation of liquid helium. Equations that link the AC losses to the thermal heat load are given in [18]. Knowing this, measuring the AC losses is equivalent to measuring the heat load extracted by the liquid helium bath. Nevertheless, as for the AC losses, there is no continuous measurement of the heat load dissipated in each cavity in the SPIRAL2 cryogenic system. Discontinuous measurement can be made by measuring the evaporating rate of the liquid helium [19], but once again it is an intrusive method that could not be realized during operation. This is where the cryomodule model becomes very useful: using the model and process measurements such as phase separator level and pressure as well as valve opening, it is possible to predict the current heat loads. Therefore, the idea is to synthesize an observer (also called a virtual sensor in that case) of the heat load.

An extended Kalman filter seems to be the best choice as it is designed to work with nonlinear processes and has been successfully used in many applications [20,21]. The process diagram of such an observer applied to our process is described in Figure 6 where it is decomposed into elementary steps represented as a number in green circles:

• 1: calculate phase separator internal energy (e) and density (ρ) through property interpolation using bath pressure and liquid level.

In a nutshell, we use the difference between estimated values based on the model (i.e., e _est and ρ _est ) and values (e and ρ) directly calculated from measurement (tabulated data in HEPACK), to correct the estimated heat load based on the model equations. The complete algorithm is being deployed in cryomodule PLCs. The following results were obtained using an external computer connected to the data acquisition system of the process. In that way, it was possible to directly get sensor process values but with a delay of few seconds.

To evaluate the estimation capacity of the extended Kalman filter, a reference was needed. A controllable resistive heater thermally linked to the liquid helium phase separator was used for this purpose. Actual power dissipated in the helium bath ⁹ showed good agreement with the electrical power of the heater (within 1 Watt). Stepped variations of the heater power were used to test the synthesized virtual sensor. The results of Figure 7 showed an exceptionally good prediction of the heater power. The actual dynamics precision was better than 2 Watts for heat loads lower than 20 Watts. The absolute average estimation error was equal to 0.7 W which represents about 3% of the maximal tested heat load. This means that the estimation precision could reach 1 W if the estimation is averaged on a sufficiently long timescale. Above 20 Watts, the method used to measure the actual heat load dissipation of the heater introduces a bias higher than 2 Watts which limits the interpretability of the results. However, the presented virtual sensor ¹⁰ shows its capability to predict heat load in real-time with a precision of few Watts. This is enough to detect anomalies during operation.

“Anomaly detection” is used to designate algorithms capable of identifying events or items differing from the majority of the events/items. For the case of plant monitoring these algorithms could be used to address the problem of continuous fault detection on process actuators or transmitters. These kinds of algorithms are particularly suitable for large processes which contain thousands of actuators and transmitters because it is almost impossible for a single operator to continuously check the functioning of each element within the process. For example, in the SPIRAL2 cryogenic system, there are more than 70 control valves and 300 transmitters.

In such a case, we demonstrate the possibility to use the cryomodule model to perform actuator malfunction detection. To be more specific, we try to predict if the output value of one cryomodule (see Figure 1) is undergoing a deadband ¹¹ problem. We use machine learning (ML) algorithms to predict the malfunction.

This section is decomposed into two parts. First, the generation of the dataset used to train ML algorithms is explained. Then, the ML algorithms themselves are described as well as their performances.

Nowadays, the main concern when working with ML algorithms is the generation of a clean dataset rather than the algorithm itself. Why so? Because it exists many libraries that already contain codes for all the commonly used ML algorithms. In Python, some of the most popular libraries are TensorFlow [22], PyTorch [23], and Skitlearn [24].

In the present work, the MATLAB statistics and ML toolbox [25] are used. All the data used for the anomaly detection problem are simulated data. Nevertheless, white noise has been added to each input and output of the model. The amplitude of this noise has been defined such that the simulated data look like the real measurements. Furthermore, slow fluctuations have been added to the input and output boundaries pressures in order to mimic the real operating conditions. To a naked eye, it is almost impossible to differentiate simulated data from measured data.

Before generating a dataset for valve anomaly detection, it is required to model the deadband problem on the output valve. In our case, the deadband has been set to random values between 1% and 4% to generate different test cases. The following signals have been recorded:

• phase separator pressure

Only the valve command (and not the real position) is considered. It mimics the case where valves are not equipped with a position indicator. In total, 500 time series of 60 s have been simulated. The dataset has been perfectly balanced: in half of the cases, the valve was subject to deadband, and in the other half it was not. For the two ML algorithms described in the next sub-sections, we used a standard cross-validation method. So, the overall dataset has been decomposed into a training set (60% of the data), a validation set (20% of the data), and a testing set (20% of the data). Thus we are able to perform hyper-parameters ¹² tuning for each tested ML algorithm.

The first solution is to use a classification learner to determine if a valve is faulty or not. This kind of algorithms require features as input and not time series. So, features were extracted from each time series of the dataset. As we do not know which features would be most suitable to identify a deadband problem, we calculate all the most common ones (i.e., variance, peak to peak, skewness, and kurtosis, etc.). In our case, we define 36 features which are few enough not to be concerned with limitations due to computer performances. But if it was the case, it would still be possible to use the same brute force approach and apply a principal component analysis [26,27] to reduce the number of features. Consequently, for all the time series of the dataset, each of the 4 measured signals has been transformed into a list of 9 features that could be used as input for a classification learner ¹³ .

Once again, as there is no methodology to choose the best classification algorithm, we trained multiple ones and selected the one with the highest accuracy. Thanks to parallel computing it takes less than a few minutes to train multiple algorithms including decision trees, support vector machine (SVM), logistic regression, and nearest neighbors.

It appears that SVM with Gaussian kernel [28,29] gets the best performance among the other algorithms. SVM with Gaussian kernel is particularly suitable for our problem as we have a small number of features (less than 1,000) and not too much data to get concerned with the computation time issue.

The final results obtained with the SVM are given in Table 2. They are compared to the results obtained with another method: a deep neural network presented in the next section.

The second idea while developing the valve anomaly detection consists of a Long Short Term Memory (LSTM) network [30]. The main advantage of this deep learning algorithm is the fact that time series signals could be directly used as network inputs. It means that there is no need to calculate features in that case. Nevertheless, it generally requires more data to train this kind of network than for an SVM.

The architecture of the LSTM network is given in Figure 8. As one can see, the network is decomposed into five layers:

• The sequence input layer used to sequence data to the network.

In total, it took 200 training epochs with a constant learning rate of 0.001 to train the network. This took less than 1 min of computation time.

In this section, we compare the performances of the synthesized SVM and the LSTM algorithm. The comparison is based upon usual ML metrics: accuracy, precision, recall, and F1 score. More details about those metrics are available in [31]. Comparison is performed on a test set of 100 time series used only for this purpose (and not for training). Results are given in Table 2.

As one can see, both SVM and LSTM algorithms show good results in terms of error predictions. Nevertheless, the SVM results are slightly better. As the implementation complexity of those two algorithms is quite similar, the best option would be to deploy an SVM algorithm on the system to get an online anomaly detector. It is worth mentioning that anomaly detection has been tested on the cryomodules only to remain consistent with the rest of the article. Nevertheless, it would be more interesting to generate an anomaly detector for process-critical elements such as rotating machines of the cryogenic system: the turbines and the compressors.