AUTHOR=Andersson Jennifer R. , Moya Jose Alonso , Schwickerath Ulrich 
  
TITLE=Anomaly Detection for the Centralised Elasticsearch Service at CERN
  
JOURNAL=Frontiers in Big Data
  
VOLUME=Volume 4 - 2021
  
YEAR=2021
  
URL=https://www.frontiersin.org/journals/big-data/articles/10.3389/fdata.2021.718879
  
DOI=10.3389/fdata.2021.718879
  
ISSN=2624-909X
  
ABSTRACT=For several years CERN has been offering a centralised service for Elasticsearch. This dynamic infrastructure currently consists of about 30 independent
Elasticsearch clusters, covering more than 180 different use cases. Due to its size and complexity, the installation produces a huge amount of internal monitoring data which can be difficult to
process in real time with limited available person power. Early on, an idea was therefore born to process this data
automatically, aiming to extract anomalies and possible issues building up in real time, allowing the experts to address them before they start to cause an issue for the users of the service.
Both deep learning and traditional methods have been applied to analyse the data in order to achieve this goal.
This resulted in the current deployment of an anomaly detection system based on a one layer multi
dimensional LSTM neural network, coupled with applying a simple moving average to the data to validate the results. This paper will describe which methods were investigated and give an overview of the current
system, including data retrieval, data pre-processing and analysis.
In addition, reports on experiences gained when applying the system to actual data will be provided.
Finally, weaknesses of the current system will be briefly discussed, and
ideas for future system improvements will be sketched out.