Using open-science workflow tools to produce SCEC CyberShake physics-based probabilistic seismic hazard models - Callaghan, Scott; Maechling, Philip J.; Silva, Fabio; Su, Mei-Hui; Milner, Kevin R.; Graves, Robert W.; Olsen, Kim B.; Cui, Yifeng; Vahi, Karan; Kottke, Albert; Goulet, Christine A.; Deelman, Ewa; et al. (Frontiers in High Performance Computing, 2024)
The Statewide (formerly Southern) California Earthquake Center (SCEC) conducts multidisciplinary earthquake system science research that aims to develop predictive models of earthquake processes, and to produce accurate seismic hazard information that can improve societal preparedness and resiliency to earthquake hazards. As part of this program, SCEC has developed the CyberShake platform, which calculates physics-based probabilistic seismic hazard analysis (PSHA) models for regions with high-quality seismic velocity and fault models. PSHA hazard models provide estimates of possible future peak ground motions for sites of interest that are useful for building engineering, insurance rates, and disaster planning. The CyberShake platform implements a sophisticated computational pipeline that includes over 15 individual codes written by 6 developers. These codes are heterogeneous, ranging from short-running high-throughput serial CPU codes to large, longrunning, parallel GPU codes. Additionally, CyberShake simulation campaigns are computationally extensive, typically producing tens of terabytes of meaningful scientific data and metadata over several months of around-the-clock execution on leadership-class supercomputers. We present the workflow software stack required to support CyberShake campaigns, including open-source workflow tools and custom solutions. We identify how the CyberShake platform and supporting tools enable us to meet a variety of challenges that come with large-scale simulations, such as automated remote job submission, data management, and verification and validation. This platform enabled us to perform our most recent simulation campaign, CyberShake Study 22.12, from December 2022 to April 2023. During this time, our workflow tools executed approximately 32,000 jobs, and used about 770,000 node-hours on the Summit system at Oak Ridge Leadership Computing Facility. At peak, we utilized 73% of the system. Our workflow tools managed about 2.5 PB of total data, and automatically staged 19 million output files totaling 74 TB back to archival storage on the University of Southern California's Center for Advanced Research Computing systems. CyberShake extreme-scale workflows have generated simulation-based probabilistic seismic hazard models that are being used by seismological, engineering, and governmental communities.