High-Performance Computing (HPC) is entering an era where energy efficiency, even considered a secondary design objective, has become an important factor for system design and operations. Across the last couple of decades, this has meant: (i) the design of computing architecture and systems with higher FLOPS/Watt ratio¹, (ii) the development of hardware extensions to support energy-proportionality, like the introduction of the performance states, dynamic-voltage and frequency scaling, low-power idle states (Kodama et al., 2020; Jadhav et al., 2022), (iii) the development of integrated hardware extensions for monitoring power and energy consumption at runtime (Vysocky et al., 2018; Hackenberg et al., 2014), and (iv) the development of system-level and user-level power management runtime systems to control and optimize the energy and power consumption during HPC production (Cesarini et al., 2021; Corbalán and Brochard, 2018; Madella et al., 2025).

In the pursuit of higher peak performance amid the stagnation of lithographic technology scaling gains, HPC systems have increasingly prioritized efficiency by favoring designs with simpler cores and SIMD acceleration. Over the years, this trend has led to heterogeneous node architectures and the introduction of RISC ISAs into the HPC race. The Fujitsu A64FX (Yoshida, 2018) was the first ARM-based HPC processor to be deployed at a large scale, powering the Fugaku supercomputer. Fugaku ranked #1 in the Top500 list from June 2020 to June 2022 and is currently ranked 7th as of June 2025.². Its architecture is based on the A64FX processor, built on the ARMV8.2-A SVE (scalable vector extension) architectures with 48 compute cores, high-speed high bandwidth memory generation 2 (HBM2) memory, and vector processing units. The Fujitsu A64FX processor, designed for the Fugaku supercomputer, offers decent power efficiency for its workloads, providing a promising basis for continued development of energy-efficient HPC architectures (Sreepathi and Taylor, 2021).

In HPC, performance and capability are attained by distributing applications across multiple compute nodes interconnected by low-latency networks. The Message Passing Interface (MPI) is the standard application runtime to exchange data and synchronize workload (Hsu and Feng, 2005; Walker, 1994). In recent years, several power management runtime systems have been proposed for HPC systems (Cesarini et al., 2021; Corbalán and Brochard, 2018). Following the HPC PowerStack initiative architecture, these runtime systems can be categorized into three layers: job-manager, system-manager, and node-manager (Madella et al., 2025). The node manager controls and monitors hardware at the node level and applies power management policies for the processor and node. The system manager is responsible for the system-wide power budget and constraints. It monitors performance and power telemetry in real-time and implements power shifting and reallocation algorithms to track the power demand and redistribute capacity to maximize job performance under a power cap. The job manager focuses on lowering the energy consumption of each job by instrumenting parallel applications to isolate their computational phases and applying application-aware power-management policies.

The primary goal of this layered approach is to convert application inefficiencies into energy savings while minimizing performance loss. Indeed, recent works have studied the sustainability of HPC production from the perspective of carbon footprint emission, showing that both Scope2 and Scope3 emissions are equally important during an HPC system's lifetime (Li et al., 2023; Chadha et al., 2023). Scope2 relates to the carbon emission from energy consumption, while Scope3 relates to carbon emission from the fabrication and installation of the computing system. Given the similar weight of Scope2 and Scope3 carbon emission in today's supercomputer systems and electricity mix composition, reducing energy consumption at a cost of longer execution time leads to a higher Scope3 accounted cost and may lead to a loss in sustainability. It is thus essential to monitor the actual energy consumption (energy-to-solution) together with the actual carbon intensity and minimize the overhead of power management policies in terms of the application's performance (time-to-solution).

To minimize this overhead, these power-management runtime systems employ policies that identify inefficiencies during application execution phases and selectively reduce the hardware performance/resources when workloads are constrained by communication, memory, or I/O. At such phases, scaling back hardware resources does not impair application performance. COUNTDOWN (Cesarini et al., 2021, 2020) is one such runtime system that focuses on detecting and isolating communication-bound phases in MPI applications and reducing the performance of the computing unit in these moments. The MERIC (Cesarini et al., 2020) runtime system and energy measurement library, which optimizes energy efficiency by adjusting hardware configurations, focuses on identifying application phases and reporting energy consumption, enabling searches for optimal operating points that minimize energy consumption for each phase.

Energy consumption can be attained by reading vendor-specific hardware interfaces such as RAPL (for Intel and AMD CPUs), NVML (for NVIDIA GPUs), or Fujitsu-provided APIs for A64FX (ARM). It is important to note that the accuracy and reliability of these monitoring interfaces are not uniform and can vary between CPU/GPU architectures and vendors. This variability presents a challenge for porting runtime systems developed for one architecture, like Intel-specific interfaces, to newer architectures like the ARM-based A64FX processor. For instance, MERIC already supports the A64FX, while COUNTDOWN only targets the Intel processors and NVIDIA GPUs.

To address this limitation, in this paper, we introduce CNTD_MERIC, a framework that integrates the MERIC runtime system and energy measurement library (Vysocky et al., 2018) with the COUNTDOWN runtime system. CNTD_MERIC brings architecture-independent power and energy monitoring capabilities to MPI applications running on heterogeneous systems.

The following are the main contributions of this paper:

The paper is organized as follows. Section 2 provides the background and limitations of existing power management libraries. Section 3 provides the design details and implementation of CNTD_MERIC integration. Section 4 presents our experimental setup and results on A64FX and Intel Xeon Max 9468 systems using NAS parallel applications.

Power and energy efficiency have become primary design goals in high-performance computing (HPC), particularly as we progress toward exascale systems. Many runtime solutions have been proposed to reduce the energy consumption while maintaining acceptable performance levels. Key threads of research include runtime energy management and energy-aware frameworks. Most of them rely on exploiting communication slack, workload imbalance, or application phases using Dynamic voltage and frequency scaling (DVFS) and low-power state transitions.

Early software-based energy optimization techniques, such as Adagio (Rountree et al., 2009) and EAR (Corbalan et al., 2020), leverage DVFS to reduce energy consumption in MPI applications by exploiting communication slack and load imbalance. Adagio relies on offline profiling and manual phase annotation to model frequency scaling, limiting its adaptability to dynamic workloads. While EAR introduces a transparent, runtime approach through its (DynAIS) algorithm, which automatically detects iterative application regions (e.g., loops) and computes dynamic application signatures to drive DVFS decisions without requiring prior application knowledge or user intervention. However, both systems primarily target long-running iterative phases, leaving fine-grained power management opportunities unexplored.

To overcome these limitations, Cesarini et al. (2021) introduced COUNTDOWN, a tool for fine-grained, performance-neutral energy saving. COUNTDOWN specifically targets MPI communication and synchronization phases. It addresses the problem of overheads in transitioning to low-power states by using a timeout strategy that distinguishes between short and long MPI phases. It dynamically intercepts MPI calls and injects DVFS and other power-state transitions (e.g., P-, C-, and T-states) only when beneficial. A notable strength of COUNTDOWN is its transparency. It requires no modifications to the application source code and works seamlessly through dynamic linking. Evaluation on NAS benchmarks and Quantum ESPRESSO showed up to 50% energy savings with less than 5% performance overhead. However, its energy-saving capabilities are currently focused on systems with RAPL support and Intel-based platforms, limiting its portability to emerging architectures such as ARM-based processors (e.g., A64FX). In their other work, COUNTDOWN Slack (Cesarini et al., 2020), the authors specifically focus on exploiting slack time. While the original COUNTDOWN applied power saving to all MPI communication phases, COUNTDOWN Slack introduces artificial barriers to isolate slack time and only reduces power during non-critical waiting periods. This enhancement avoids slowing down data transfer and ensures persistent performance with less than 3% overhead and is still capable of achieving significant energy savings (up to 22% in large-scale runs).

READEX (Schuchart et al., 2017) does not focus on a single specific type of workload, but splits a complete application execution into phases of a similar workload type (similar hardware resource requirements) at the granularity of tens up to hundreds of milliseconds. In a complex application, we may expect this dynamic boundness change to occur frequently. The MERIC runtime system is exploiting the energy savings from hardware and system-software parameters configuration changes to fit each application phase, bringing power savings while not extending application runtime.

Another library, EVeREST-C (Yue et al., 2025), is a runtime tool that helps save CPU energy in HPC applications by adjusting core and uncore frequencies using DVFS. It relies on a simple metric, Instructions Per Second (IPS), to detect whether code is compute or memory-bound, and applies energy-saving changes without modifying the application. It achieved up to 11% energy savings on SPEC benchmarks and 8% improvement over other MPI-based methods, with less than 3% performance overhead. While the IPS-based design is intended to be multi-architecture, the current results are limited to x86 platforms. However, uncore frequency control is only available for Intel processors. It may miss very short phases, require users to specify acceptable performance degradation, and be less effective with irregular communication patterns in MPI applications.

In the literature, various DVFS-aware runtime frameworks have been proposed. Each targets different trade-offs between energy efficiency and application performance. For instance, CPU MISER (Ge et al., 2007) is a runtime system that leverages user-defined performance thresholds and adjusts CPU frequency based on observed performance counters. While it achieves significant energy savings, especially in memory-bound applications, its effectiveness is highly dependent on platform-specific counter configuration and static thresholds, making it less adaptable across diverse architectures or dynamic workloads. The Uncore Power Scavenger (UPS) (Gholkar et al., 2019) targets uncore frequency scalings, adjusting the power domain to save the energy in compute-bound phases, but it relies on hardware-specific monitoring like DRAM power counters and limited to Intel processors. Venkatesh et al. (2015) propose EAM (Energy-Aware MPI), an application-oblivious runtime that reduces power during MPI slack time using DVFS and idling, integrated into the MVAPICH2 library. It achieves up to 41% energy savings under 4% performance loss across many applications. However, its dependence on accurate slack prediction limits performance with irregular or fine-grained communication.

Recently, there has been increased focus on optimizing performance and energy efficiency on ARM-based HPC systems such as Fugaku, which is powered by the Fujitsu A64FX processors. Kodama et al. (2020) performed the foundational analysis of supercomputer Fugaku's A64FX processor, by exploring three key power control modes: eco, boost, and core retention using micro-benchmarking (STREAM and DGEMM). Their study showed that combining eco mode with core retention can significantly reduce power without performance loss in memory-bound workloads, while boost mode improves compute-bound performance. However, eco mode only saves power for memory-bound tasks, and improvement in the performance of compute task result in more energy cost. The analysis was limited to synthetic workloads. Also, power use varies a lot between different parts of the supercomputer, making it hard to optimize the whole system evenly. Expanding this work, the authors evaluated Fugaku's A64FX processor using the SPEC CPU and SPEC OMP benchmark suites to compare its performance and power behaviors with Intel Xeon Processors. The comparison results confirmed A64FX's advantages in memory-intensive workloads and validated the earlier findings on power-saving modes. In this study, the authors also highlighted the higher performance of the Fujitsu compiler over GCC. Despite this, Power-saving modes reduce energy but often sacrifice performance (Kodama et al., 2021).

Kodama et al. (2022) further explored this work on real-world HPC workloads from the SPEC suite. They demonstrate that a combination of eco and boost mode could reduce energy consumption by 17% along with slight performance gains 2%, validating earlier works in practical contexts. However, it excelled in scalable, memory-intensive HPC tasks with strong power efficiency, but its reliance on manual tuning highlighted challenges for broader adoption. Alappat et al. (2021) proposed the Execution-Cache-Memory (ECM) model for A64FX to predict and analyze energy-performance tradeoffs. It shows offline tuning of frequency and memory bandwidth parameters can help to save energy from 18 to 31%. However, their models are static and do not support dynamic workloads.

A review of the studies presented above demonstrates significant progress in understanding and controlling energy usage on A64FX systems. However, they often rely on offline profiling, static policies, or coarse-grained runtime controls. They don't provide MPI communication awareness, automated DVFS control, or direct energy monitoring on ARM-based supercomputers. To address these limitations, CNTD_MERIC leverages vendor-specific interfaces (e.g., Fujitsu A64FX APIs) through MERIC to provide hardware-level energy measurement capabilities. This unified framework combines the runtime adaptive DVFS and timeout-based logic of COUNTDOWN with the MERIC's monitoring support, targeting x86 and ARM-based architectures. It addresses a critical gap in MPI runtime energy/power management by enabling dynamic, system-level power and energy monitoring specifically tailored for ARM-based HPC architectures such as A64FX.

This section outlines the methodology we propose to develop a unified framework called CNTD_MERIC, which integrates the MERIC energy monitoring library with the COUNTDOWN power-aware runtime system. This integration allows platform-independent energy measurement and profiling of HPC applications. To describe the integration process, we first introduced the internal structure and key functions of MERIC and COUNTDOWN that are relevant for this work. Then we present the steps followed to create the CNTD_MERIC framework.

This section presents the experimental setup and tests that we conducted.

In this work, we present CNTD_MERIC, a unified and multi-architecture (x86 and ARM) framework for profiling MPI-based HPC applications. CNTD_MERIC extends the capabilities of the original COUNTDOWN by integrating it with the MERIC library through a C-compatible abstraction layer. It enables seamless support for heterogeneous energy monitoring systems, such as Intel's RAPL and Fujitsu's A64FX. We performed a series of experiments and evaluated the framework on Intel Xeon Max 9468 and Fujitsu A64FX compute nodes provided by the IT4Innovations supercomputing center using a diverse set of workloads, including NAS Parallel Benchmarks, STREAM, deep learning inference, and Quantum ESPRESSO. Our tool successfully captured the distinct energy-performance profiles for each of them. This ability to provide a unified energy-performance perspective across diverse workloads demonstrates the suitability of CNTD_MERIC for large-scale HPC and ML workloads.

The key finding of our results shows that CNTD_MERIC preserves the benefits of both libraries and introduces negligible overhead. Both configuration modes (Analysis and Enable) show nearly identical execution times across all benchmarks, with a minimum variation between −0.25% and +0.45%. Similarly, energy consumption and average power are closely aligned, confirming an overhead within −5% to +3%, with respect to the original COUNTDOWN configurations. On Intel nodes, the framework also demonstrated optimization capability, achieving energy and power savings of approximately 13%–21% across the NAS benchmarks, proving that the integrated library retains the full energy-efficiency potential of the original COUNTDOWN implementation. This validated that CNTD_MERIC is also capable of delivering energy-efficient execution while preserving application performance. In addition to single-node validation, we conducted multi-node experiments on the Fujitsu A64FX system with the Wide ResNet workload, NAS Parallel Benchmark (BT Class D), and Quantum ESPRESSO on up to eight nodes. For Quantum ESPRESSO, the workload scaled from 2 to 8 nodes using a hybrid MPI+OpenMP configuration. In all workloads, we observed substantial reductions in execution time. While Wide ResNet and BT exhibited a non-monotonic energy trend, Quantum ESPRESSO demonstrated a monotonic increase in energy consumption, rising moderately from 139.7 kJ to 201.829 kJ. Across all workloads, average power increased almost linearly with the number of nodes, consistent with proportional resource utilization across the cluster. This highlights CNTD_MERIC's capability to provide stable, accurate runtime, energy, and power monitoring on multi-node ARM-based HPC systems.

We acknowledge that the current validation is limited to CPU-based systems specifically Intel x86 and ARM A64FX architectures. While we describe CNTD_MERIC as multi-architecture, the portability and applicability to GPUs or other accelerators remain to be explored. In future work, we plan to extend CNTD_MERIC framework toward co-tuning strategies. Our goal is to coordinate COUNTDOWN's tuning of MPI communication phases with MERIC's tuning of computation, memory, and I/O-bound phases so that both components work together at runtime. We also intend to explore integration with emerging system-wide frameworks such as PowerStack, and to assess the possibility of adding GPU power monitoring and control on systems that support it. These steps will allow us to develop a more adaptive and multi-architecture co-tuning approach.