We recruited young men and women for our study among medical students through advertisements on the internal Charité student’s platform and social media. Those interested were eligible for inclusion if they were between the ages of 18 and 30 and had no history of competitive training. On the other hand, interested were excluded if they had any form of cardiovascular, metabolic, and neurological diseases, or any physical impairments that would prevent participation in an incremental test on a treadmill. Following explanations of the study aim and protocol, including experimental procedures and known risks, participants provided informed written consent prior to commencing study participation. Based on a sample size calculation using the results of a comparative study between commercial trackers and a portable ECG (Godino et al., 2020), the study sample was planned with 20 participants. To ensure conclusive statistical results at the end of the study, we recruited a total of 22 participants (11 women, 11 men). Their anthropometric data were as follows: age, mean 24.0 (SD 2.4) years; body weight, mean 70.2 (SD 7.7) kg; height, mean 176 (SD 9.1) cm; body mass index, mean 22.6 (SD 1.6) kg/m². The study was approved by the Ethics Committee of Charité–Universitätsmedizin Berlin (Date: 9 April 2021, EA 4/050/21).

The research and consumer-grade wearables considered for evaluation in this study were selected from a study protocol designed for the purpose of providing scientific information on their reliability for the use in the setting of population monitoring in SSA (Barteit et al., 2021). Table 1 provides details of the consumer- and research-grade wearables in the present study.

The measurements were conducted in the laboratories of the Institute of Physiology, Charité–Universitätsmedizin Berlin on weekdays between 9:00 and 14:30 in September 2021. Study participants followed a structured, laboratory-based protocol that included two different resting phases followed by different locomotion phases on a motorized treadmill (Figure 1). In particular, we wanted to simulate intensities typical of the daily routines of subsistence farmers in SSA regions. For example, the metabolic equivalent of task (MET) for the classic Bruce treadmill protocol is estimated to be 4.2 MET for the first stage and 8.3 MET for the third stage according to the FRIEND equation (Kokkinos et al., 2017). MET values of 4.5 and 7.8 correspond to routine chores with small animals and shovel or pitchfork work, respectively (Pickett et al., 2015).

Participants were first equipped with various devices. The Tucky device was placed under the right armpit using Tucky double-sided adhesive Adh21. Due to an initial detachment on the first study participant, we have since positioned the device closer to the chest and additionally secured it with medical adhesive tape. The Tcore sensor and headband was fastened on the participants’ forehead. The Faros was positioned on the chest and secured with medical adhesive tape to ensure signal quality. GENEActiv and Withings were placed on the wrist of the non-dominant arm, with GENEActiv positioned directly above Withings.

After these initial setups, participants sat for 10 min and then rested for an additional 10 min while standing. Following this period of rest, measurements continued on a motorized treadmill (h/p/cosmos quasar med 4.0, Nussdorf-Traunstein, Germany). Similar comparative studies also utilized treadmills as test environments (Thiebaud et al., 2018; Thomson et al., 2019). On the treadmill, participants were fitted with a mask over their mouth and nose which was connected to the Cortex Metalyzer, and were also fitted with a harness system to prevent falls on the treadmill. After a 3-min rest on the treadmill, the study participants started the Bruce protocol continuing until complete exhaustion. The classic Bruce treadmill test consists of 3-min stages, with speed and slope increasing every 3 minutes without breaks (Fletcher et al., 2013). The first four stages are as follows: Stage1: 2.7 km/h (1.7 mph), 10%; Stage2: 4.0 km/h (2.5 mph), 12%; Stage3: 5.4 km/h (3.4 mph), 14%; Stage4: 6.7 km/h (4.2 mph), 16%.

Following the treadmill test, the collected data were retrieved and stored on a study computer. Data from Withings and Tucky were downloaded from their respective platforms and spiroergometric data were exported via MetaSoft software, while Tcore, Faros, and GENEActiv data were transferred directly from their internal storage. To ensure synchronization among all considered data logs for later data analysis, timestamps were documented during the experiments. First, the times on the computers associated with the different monitors were recorded at the beginning of each measurement day to account for potential time offsets. This was necessary because GENEActiv, Faros and Tcore were initialized with the study computer, while Withings and Tucky were initialized with the same mobile device, and spiroergometry was conducted using a separate computer. Secondly, the time (on the study computer) at which the rest and activity measurements began was noted.

In both resting states, the heart rate was similar for both methods. With increasing physical activity, HR_Withings did not increase to the same extent as the HR_Faros (Figure 3A). At the 4th stage, the mean difference between the methods was −12 bpm, the largest and statistically significant (Table 2). Correlations were strong and positive for Sit and Stand (r ≥ 0.82, LCCC ≥ 0.76). However, the agreement between HR_Withings and HR_Faros decreased with increasing physical activity (Table 3). For example, MAPE was more than twice as high from Stage2 (≥10%) as during both resting phases (≤4%).

CBT_Tucky was consistently lower than CBT_Tcore in all phases (Figure 2B), which was confirmed by statistical analysis (Table 2). The difference between the methods was smallest at rest (Sit, −0.8°C, t ₂₀ = −5.44, P < 0.001) and largest in the fourth stage of the Bruce test (−1.8°C, t ₂₀ = −10.35, P < 0.001). CBT_Tucky remained unchanged across different situations (ranged between 36.3°C and 36.5°C), while CBT_Tcore increased with physical effort (ranging between 37.2°C and 38.1°C). Similar to HR, the correlations between the temperature monitors declined with physical activity. In addition, LoA became wider and the MAPE increased (Table 3).

At a treadmill speed of 2.7 km/h (Stage1), step counts were similar between SC_Withings and SC_GENEActiv (72.2 vs. 71.5 steps/min, z = 0.54, P = 0.61). However, the difference between the methods increased with increasing speed (Figure 3C), while SC_Withings increasingly exceeding SC_GENEActiv (Table 2). For example, at a treadmill speed of 6.7 km/h (Stage4), SC_Withings exceeded SC_GENEActiv by about 17 steps/min. (152.2 vs. 134.9 steps/min, t ₂₀ = 8.07, P < 0.001). On the other hand, LoA at Stage4 was only half as wide as at Stage1 (Table 3). MAPE was highest in Stage1 (38%), but was only around 10% in the following three stages.

EE_IC increased with each subsequent intensity level of the Bruce test (3.1, 4.6, 6.8 and 9.7 MET for Stage1 to Stage4). Reference EE_IC was significantly underestimated by both alternative methods, EE_Withings and EE_GENEActiv, in each of the four treadmill stages (Figures 3D,E). The bias to IC increased for both methods during the first three stages (up to −2.9 MET). At Stage4, the bias was only −1.9 MET (EE_Withings) and −1.4 MET (EE_GENEActiv), but the LoA was widest at this stage. Although the agreement between EE_GENEActiv and EE_IC appeared to be better than between EE_Withings and EE_IC, the agreement between the methods for EE was generally low (Table 3).

In a previous validation study of wearables for HR measurement, a LCCC>0.80 was presented as an acceptable accuracy (Gillinov et al., 2017). Accordingly, our results showed that the Withings device demonstrated acceptable agreement with Faros for low physical activities (Sit: LCCC = 0.76, Stand: LCCC = 0.91, Stage1: LCCC = 0.81). The same applies if MAPE threshold is less than 10% (Boudreaux et al., 2018). In our study, MAPE was ≤4% during both resting states and ranged between 7 and 12% during the treadmill locomotion. However, in another study, the device under test was only considered valid if several criteria were met, e.g., LCCC>0.90 and MAPE<5% (Navalta et al., 2020). Furthermore, agreement in HR with the criterion measure during physical activity seems to be lower than during rest, which is in line with previous findings (Thomson et al., 2019; Bent et al., 2020). Devices that use photoplethysmography to monitor HR tend to be inaccurate at higher intensities of physical activity due to artifacts caused by intense hand movements (Castaneda et al., 2018; Bent et al., 2020; Navalta et al., 2020). In addition to motion artefacts from physical activity, ambient light, misalignment between the skin surface, and poor tissue perfusion can also be a source of error (Alzahrani et al., 2015). Skin tone is apparently not a source of errors (Bent et al., 2020), which is an important observation for studies involving African populations, for example. Interestingly, Stahl et al. (2016) observed a decrease in MAPE at treadmill speeds of >3.2 km/h, attributing this to improved perfusion due to increased intensity. In the present study, a small decline in MAPE was observed at treadmill speeds of >4.0 km/h. Nevertheless, not only the user of wrist-worn HR monitor or the ambient conditions seem to affect measurement accuracy, but also the device itself. Müller et al. (2019) investigated the validity of HR measures of a high-cost consumer-based tracker and a low-cost tracker in a laboratory setting, showing the high-cost tracker had smaller errors and a higher agreement with the criterion measure than the low-cost tracker.

For a step counter to be considered accurate, the MAPE should be less than 1% compared to the criterion measure when walking on a treadmill at a speed of 4.8 km/h (Tudor-Locke et al., 2006). In a recent review of the validation of treadmill step-counting technologies, median MAPE values for wrist-worn monitors ranged from 6.6% to 10.7% at speeds between 3.2 and 6.4 km/h (Moore et al., 2020). In our study, the MAPE ranged from 8% to 38% at speeds between 2.7 and 6.7 km/h (Stage1 to Stage4). In addition, the bias was lowest for Stage1 at 0.6 steps/min and highest for Stage4 at 17.3 steps/min, indicating an increasing overestimation in steps by the Withings Pulse HR with increasing treadmill speed. On the other hand, one could argue that estimating steps using a step counting algorithm with tri-axial acceleration data is not a gold standard. Therefore, we compared the estimates in our study with published hand-counted steps from treadmill experiments of Ducharme et al. (2021) and Tudor-Locke et al. (2019) (Supplementary Table S1). It was shown that both the SC_Withings and the SC_GENEActiv estimated about 17 steps/min less at speed of 2.7 km/h, which was the largest difference compared to published data. Low accuracy of step counting at slow walking speeds is a common issue with wrist-worn wearables (Moore et al., 2020). At treadmill speeds of 5.4 and 6.7 km/h, differences between hand-count SC_Withings were about −12 and −18 steps/min, while differences between hand-count and SC_GENEActiv were only about 2 and -1 steps/min. These observations suggest a paradox: bias was best at slow walking speed of 2.7 km/h because both wearables were equally inaccurate. Since the use of raw acceleration data provides a flexibility in processing, selecting a better performing step count function should be considered. For example, Ducharme et al. (2021) recently published a transparent algorithm for step detection, and the open-source Verisense step count algorithm has been optimized (Maylor et al., 2022; Rowlands et al., 2022). While Withings Pulse HR utilizes changes in the acceleration caused by foot impact during walking, the exact algorithm is not disclosed.

The EE_Withings showed low overall agreement with EE_IC during the treadmill test. The same applies to EE_GENEActiv, where acceleration data from GENEActiv was used to estimate EE using a prediction formula for physical activity energy expenditure (White et al., 2016). In both comparisons, the MAPE value was very high at Stage1 (≥200%), but decreased with increasing treadmill locomotion levels and was lowest in Stage4 (Withings: 60%, GENEActiv: 26%). However, Passler et al. (2019) considered a tested device valid if MAPE is less than 10%. The decrease in MAPE with increasing treadmill speed (and grade) indicates better agreement with higher physical workload. In fact, estimated HR by wrist-worn photoplethysmography devices in combination with physiological modeling tended to have lower MAPE for EE estimation during activities above the aerobic threshold (Parak et al., 2017). Moreover, in the present study both wrist-worn devices for EE estimation clearly underestimated the EE for the criterion measure (indirect calorimetry). Wearable trackers for EE estimation predominantly underestimate EE even in a controlled environment (Evenson et al., 2015; Wahl et al., 2017; Fuller et al., 2020). Wearables were typically examined while worn on the wrist (Fuller et al., 2020), though a greater accuracy can be achieved when placed on the hip or shirt collar (Woodman et al., 2017). EE estimates from devices worn on the wrist or hip generally vary in accuracy depending on physical intensity and type of activity (Howe et al., 2009; O’Driscoll et al., 2020). Recently, Ogata et al. presented an equation to improve EE estimation using accelerometer-based MET value and individual HR and showed that estimated total energy expenditure in rescue workers was one-third higher with the combined approach than with the accelerometer-based method alone (Ogata et al., 2024).

Most wearable thermometers were developed to continuously monitor skin temperature, few in order to reflect changes in CBT (Tamura et al., 2018). In the present study, we compared two sensors attached to the skin: the Tucky thermometer under the right armpit and the Tcore sensor on the forehead. Although adding 0.7°C to the measured values of Tucky improved agreement with rectal temperature, correlations between Tucky’s rectal measurements and Tcore’s CBT estimate decreased with increased physical activity (highest during Sit and the lowest during Stage4 of the Bruce treadmill test). In addition, the bias in each of the six activity phases was at least −0.8°C, indicating that Tucky’s rectal measurements underestimated traditional rectal temperature measurement. For example, Gunga et al. (2008) validated the Tcore precursor with rectal temperature measurement during treadmill activities (25%–55% maximum work intensity) at different ambient temperatures, demonstrating a good agreement during resting (Bias: 0.01°C, LoA: 0.74 to 0.72) and working periods (Bias: 0.08°C, LoA: 0.77 to 0.61) at ambient temperature of 25°C. Wearable thermometers are considered in agreement if they comply with the clinically meaningful recommendations of bias of ±0.5°C and LoA of ±1.0°C (Tamura et al., 2018). In our study, however, the bias was at least −0.8°C and the LoA were −2 to 0°C at Stand (narrowest) and −3.5°C to 0.2°C at Stage4 (widest). Our results suggest that the higher the intensity of physical activity, the lower the accuracy of Tucky’s measurements. This inaccuracy could be attributed to the thermoregulatory processes of the skin. Increased physical activity can lead to increased perspiration, which aims to cool the skin and CBT through evaporation. In the context of varying and intensive physical activity, Tucky under the armpit did not achieve sufficient accuracy with CBT. Similar observation was reported for another adesive axillary thermomenter patch. Temperatures of adesive axillary thermomenter showed good agreement with those from the conventional axillary method (Bias: 0.15°C, LoA: 1.13 to 0.99), but failed to those of the bladder as the CBT (Bias: 1.11°C, LoA: 3.19 to 0.98) (Boyer et al., 2021).

This study has several strengths. Firstly, we investigated two devices, Withings Pulse HR and Tucky thermometer, that had not been validated in an independent lab study previously, focusing on their utility for in-field assessment of physiological variables in different situations of varying physical activity. Therefore, a structured protocol consisting of successively changing intensities of activity was implemented. A structured procedure and laboratory-based setting enabled a high precision of comparison and reproducibility of results.

This study was limited to healthy, fair-skinned adults aged 20–29 years. Future research should include a more diverse cohort and a comparison of multiple skin tones, especially when using optical heart rate monitors. Motion that largely affects positioning of wearables, such as treadmill running for a wrist-worn tracker, may impact accuracy and the significance of validation research. Potential interference between devices worn simultaneously on the same wrist might also represent a possible limitation of the study. Additionally, although treadmill-based incremental testing can represent the cardiovascular strain of physical activity during agricultural work, it does not correspond to the actual biomechanics and motions of such physical activity.