Enabling real-time adaptation of machine learning models at x-ray Free Electron Laser facilities with high-speed training optimized computational hardware - Milan, Petro Junior; Rong, Hongqian; Michaud, Craig; Layad, Naoufal; Liu, Zhengchun; Coffee, Ryan (Frontiers in Physics, 2022)
The emergence of novel computational hardware is enabling a new paradigm for rapid machine learning model training. For the Department of Energy's major research facilities, this developing technology will enable a highly adaptive approach to experimental sciences. In this manuscript, we present the per-epoch and end-to-end training times for an example of a streaming diagnostic that is planned for the upcoming high-repetition-rate x-ray Free Electron Laser, the Linac Coherent Light Source-II. We explore the parameter space of batch size and data parallel training across graphics processing units and Reconfigurable DataFlow Units across multiple interconnected host nodes. We show the landscape of training times with a goal of full model retraining in under 15 minutes. Although a full from scratch retraining of a model may not be required in all cases, we nevertheless present an example of the application of emerging computational hardware for adapting machine learning models to changing environments in real-time, during streaming data acquisition, at the rates expected for the data fire hoses of accelerator-based user facilities.