Anomaly Detection for the Centralised Elasticsearch Service at CERN - Andersson, Jennifer R.; Moya, Jose Alonso; Schwickerath, Ulrich (Frontiers in Big Data, 2021)
For several years CERN has been offering a centralised service for Elasticsearch. This dynamic infrastructure currently consists of about 30 independent Elasticsearch clusters, covering more than 180 different use cases. Due to its size and complexity, the installation produces a huge amount of internal monitoring data which can be difficult to process in real time with limited available person power. Early on, an idea was therefore born to process this data automatically, aiming to extract anomalies and possible issues building up in real time, allowing the experts to address them before they start to cause an issue for the users of the service. Both deep learning and traditional methods have been applied to analyse the data in order to achieve this goal. This resulted in the current deployment of an anomaly detection system based on a one layer multi dimensional LSTM neural network, coupled with applying a simple moving average to the data to validate the results. This paper will describe which methods were investigated and give an overview of the current system, including data retrieval, data pre-processing and analysis. In addition, reports on experiences gained when applying the system to actual data will be provided. Finally, weaknesses of the current system will be briefly discussed, and ideas for future system improvements will be sketched out.