Although CORBA is actually an open standard defined by the Object Management Group (OMG, 2021), it contains several design flaws, which are consequently inherited by all implementations, such as TAO (Schmidt et al., 1998), TAOX11 (Remedy IT, 2019), or omniORB (Grisby, 2022). Due to its design by committee, it suffers from several issues regarding complexity, redundancy, and missing features (Henning, 2006). Because real-time CORBA is a modification of the original specification (OMG, 2005), it incorporates the same issues, rendering it a suboptimal solution. Especially when targeting resource-constrained platforms such as MCUs, the high complexity of CORBA inevitably results in high resource requirements, which such devices can rarely (and if so, just barely) satisfy. As a result, all implementations of newer versions of the CORBA specifications only support sophisticated operating systems, such as Linux or Windows, but are not designed to be deployed on MCUs.

Originally developed by Bruyninckx (2001), the Orocos project is a collection of libraries and tools for the efficient development of robotics software, which is portable and has high runtime performance, with real-time support being one of its core aspects. Moreover, it can be combined with other middlewares if desired. For certain communication schemes, Orocos employs CORBA (cf. section 1.1.1; optional for local, non-distributed setups), and it can also be integrated with ROS (Quigley et al., 2009) and ROS 2 (Macenski et al., 2022). However, Orocos is not designed to be deployed on MCUs and does not feature validation of timing constraints at runtime.

R2P has been developed by Migliavacca (2013) as an alternative to ROS (Quigley et al., 2009), LCM (Huang et al., 2010), and FAMOUSO (Schulze and Zug, 2008; Schulze, 2009) with improved support for hard real-time systems. It is focused to be used on MCUs, which communicate via a controller area network (CAN), specifically the RTCAN protocol (Migliavacca et al., 2013). As such, it seemed appropriate for the AMiRo platform at first glance, as it is also based on ChibiOS (di Sirio, 2020, 2022), just like AMiRo-OS (Schöpping et al., 2016), the real-time operating system (RTOS) of AMiRo. Unfortunately, R2P relies on dynamic memory allocation as it employs memory pools. Therefore, complete determinism cannot be guaranteed. Because the development of the project ceased in 2016, there was no support to implement the according modifications, and there is little detailed documentation about the project to be found.

The arguably most popular solution in robotics is ROS, which introduced real-time support with ROS 2 (Quigley, 2015; Macenski et al., 2022) and its extensions, micro-ROS (OSRF, 2022a) and RT-ROS (Wei et al., 2016; Faust, 2022), respectively. However, the overall performance of ROS 2 strongly depends on the utilized data distribution service (DDS; OMG, 2015) implementation (Maruyama et al., 2016). Regarding MCUs and micro-ROS, DDS for eXtremely Resource Constrained Environments (DDS-XRCE; OMG, 2019) must be employed, which specifies a centralized communication topology with multiple clients interacting with a single server—an undesired architecture for distributed systems composed of equal participants. Because its popularity makes it the de facto standard solution in robotics, ROS 2 has been evaluated to a particular extent.

Although the ROS ecosystem is generally very powerful, there are indications that the real-time capabilities of ROS 2 are still not optimal. First, Kay and Tsouroukdissian (2015) presented empirical results of the official ROS 2 demo application for real-time use cases “pendulum demo,” which were promising overall. However, they showed occasional latency spikes, especially when the CPU was put under load. Because these benchmarks are rather dated and real-time capabilities might have been optimized since, another set of benchmarks was conducted, ¹ which still confirms the limited suitability of ROS 2 for hard real-time applications. The results of these benchmarks are depicted in Figure 1 and reveal two important findings. The histograms show the results of the same benchmarks as presented by Kay and Tsouroukdissian (2015), and the original result data is resembled very well. However, no “outliers” were visible in the presentation due to the linear scaling of the frequency axis. When the CPU is put under load, these “outliers” become even more frequent and resemble a Gaussian distribution. This is problematic regarding real-time systems because latency is not limited by an upper bound. The issue becomes even more evident when considering the goal of a jitter of fewer than 30 µs (3% of 1 ms period), as defined by Kay and Tsouroukdissian (2015). For the benchmark with additional CPU load, almost 25% of all data points violate that constraint. Although the cause for the worse performance, despite process priorities configured in favor of the “pendulum demo” (real-time vs. nice), may be rooted in the OS rather than ROS 2, there are obviously no mechanisms in place to limit execution times or at least notify about high latencies.

Further evidence of the limited real-time capabilities of ROS 2 can be found in the performance analyses by Maruyama et al. (2016). The data reveal that ROS 2 scales worse than linear in many situations. Hence, strict timing constraints become exponentially more difficult to meet when system complexity increases. The data also show strong latency variations, even exceeding 10% of the median for data sizes of 1 MB and more. Overall, the real-time capabilities of ROS 2 remain insufficient for scenarios where timing constraints are critical and must be respected and its high resource requirements render it unsuitable for MCUs without sacrificing decentralization.

One of the most important characteristics of any distributed communication framework is decoupling. Participants can be coupled in three domains, all of which should be omitted:

• space: If participants must know each other in order to exchange information, they are coupled in space.

Furthermore, Eugster et al. (2003) described six fundamental concepts of interaction and how communication between participants is realized. The publish–subscribe paradigm eventually achieves decoupling in all three domains and thus is the most powerful technique in this regard. Another very popular concept is remote procedure calls (RPCs), tightly coupled according to Eugster et al. (2003), but allowing for “pulling” communication, whereas information can only be “pushed” with publish–subscribe. Fortunately, the coupling can be relaxed by an extension of RPC, called “future” (Ananda et al., 1992) or “wait-by-necessity” (Caromel, 1993; cf. Eugster et al., 2003).

In order to establish communication between producers and consumers in a decoupled manner, addressing information needs to be abstracted. There are several approaches to how this can be realized (Eugster et al., 2003).

• topic/service-based: Producers provide information via a certain topic or service, usually identified by a name. Consumers can declare interest in specific information by such identifiers and will eventually receive all data provided via the according topic/service.

Hybrid techniques are also possible, such as the scope-based approach of Robotics Service Bus (RSB) (Wienke and Wrede, 2011). It employs a URI format for scope names, introducing a hierarchy and filtering capabilities without defining any requirements on the actual information payload.

Quality of service (QoS) is typically used to track information such as latencies and the number of delivered and discarded messages. However, timing constraints are of major importance to guarantee system stability and safety regarding real-time systems. Therefore, the according mechanisms can prioritize important communication and preempt others. Furthermore, the system can be monitored to detect critical failures (i.e., timing violations) as soon as possible and initiate an appropriate reaction. Hence, when timing constraints are not met, temporal behavior tracking and execution of defined routines are also part of QoS in the nomenclature of this work.

Often referred to as “deadline,” a maximum expected latency τ for information propagation can be specified. It is defined as a function l(τ, Δt) with Δt the duration of an operation so far: l τ , Δ t = 1 , if  Δ t ≤ τ 0 , otherwise (1)

For periodic tasks, it is often required that all iterations take similarly long and execution time varies as little as possible. For real-time systems, this “jitter” must be limited by an upper bound δ. By tracking the minimum and maximum iteration times (t _min and t _max) during system operation, the validity of the synchronicity constraint s(δ, t _min, t _max) can be calculated by s δ , t min , t max = 1 , if  t max − t min ≤ δ 0 , otherwise (2)

Although this constraint is not considered at all by most authors, it is crucial to detect the complete failure of individual components in an event-driven system at runtime. If a data source does not provide any further information, no data processing pipeline will be triggered, and neither latency nor synchronicity constraints will ever be violated. A possible solution to this challenge is the implementation of a dead man’s switch, which is monitored and checked for regular activation by another component. Alternatively, rate constraints can be validated without the need for a dedicated monitor by introducing a maximum period ϵ between subsequent data points and comparison of the current time t and the time of the latest data point t _i−1: r ϵ , t i − 1 , t = 1 , if  t − t i − 1 ≤ ϵ 0 , otherwise (3)

This trivial class has no real-time constraints at all. As a result, u = 1 always holds.

As the most general class, the mapping Δt↦u can be of any form for SRT. Each component may define an individual mapping function to calculate u in order to assess the quality of each data point during operation. While monotonic decreasing functions are most common, partial functions can be used to model desired temporal frames (i.e., to represent jitter constraints). Most notably, the NRT and FRT classes are, in fact, special cases of SRT.

This class further restricts u to be either 1 or 0 (“valid” or “invalid”). Such a distinction makes sense from an implementation perspective for two reasons. Calculations and representations do not require “sophisticated” data types, such as float. Because many MCUs do not feature an FPU (floating-point unit), emulation of such types is computationally expensive and may result in temporal inconsistency. Furthermore, this allows for a general calculation of u, depending on constraints for latency, synchronicity, and rate: u = l τ , Δ t ⋅ s δ , t min , t max ⋅ r ϵ , t i − 1 , t (4) Note that each factor can be “deactivated” by setting its parameter (τ, δ, or ϵ, respectively) to ∞.

Similar to FRT, the usefulness u is interpreted more strictly for this class. Violations of hard real-time constraints (u = 0) are considered severe incidents. The system is assumed to be in an undefined state. It may even be dangerous to itself and its environment. Such events must be detected and handled as quickly as possible to restore a stable situation and prevent any harm. This detection of violations is a key feature of µRT and is implemented using timers, which will trigger either a recovery attempt (can be defined for each component individually) or a system panic (system-wide default) exactly when a violation occurs.

Furthermore, µRT forbids HRT transmissions to be “overwritten,” which might result in the loss of a vital data point. As a result, further communication may be blocked as long as one or more HRT consumers have not processed previous messages yet (cf. section 2.6).

µRT fully embraces the paradigm of event-driven architectures and avoids any periodic polling. This ensures that all events are handled as soon as possible with minimal latency and task prioritization and preemption are left to the scheduler of the underlying RTOS. Due to the absence of periodic “synchronization points,” jitter may increase, though, if the runtime complexity of some components varies greatly during operation. However, real-time software, in general, should be designed to exhibit consistent processing time in the first place. As a result, systems using µRT are more sensitive to bad implementations (concerning real-time characteristics), so developers will be encouraged to optimize such a code. Nevertheless, periodic execution of tasks with a given frequency remains desired in many situations, such as reading sensor data. With µRT, such behavior can be realized by means of periodic timers, which regularly fire with a specified frequency and emit events that eventually trigger task execution. This method is more elaborate as it involves an additional component (the timer) to achieve the time-triggered operation of the event-driven system, but that is actually intended. Because event-triggered task execution should be preferred in most situations, µRT deliberately encourages developers to follow this software design paradigm.

Another important aspect of µRT is its approach to tracking latencies and validation of timing constraints at runtime, which are not defined per data point (e.g., sensor data have to be processed within a certain time frame), but by individual participants in the system (e.g., information must be received before it is older than τ). This approach effectively results in usability values per data point and consumer, so each participant can define and validate its individual timing constraints independently. Especially if additional stages are added to a data processing pipeline, making it more expensive regarding computation time, constraints of later stages remain valid and require no adjustment.

A crucial detail for this approach to work is correct tracking of the origin times of information, which is not the same as when the data are being transmitted within a system. While that data contains information, the latter emerges as soon as an event or state is observed and not only when it is encoded into some form of data structure, such that t info ≤ t data (5) holds. This definition is particularly important with regard to data processing pipelines, in which multiple components are arranged in a chain. After each stage in that pipeline, data is transmitted to the following component, and t _data,i increases continuously, whereas t _info,i remains unchanged. ³ The benefits of this approach become obvious when analyzing the opposite case. If each component k ∈ 2 , … , n in a data processing pipeline of length n would define its own relative deadline τ _k > 0 regarding its preceding stage, the overall deadline of the entire pipeline τ′ would be defined by τ ′ ≤ ∑ k = 2 n τ k (6)

The edge case of an equilibrium would only occur if all components fully exhaust their time budget: Δt _k = τ _k ∀k. As soon as any component requires less time (Δt _k < τ _k ), the overall time budget τ′ is also reduced by that difference: τ′ ← τ′ − (τ _k − Δt _k ). As a result, pipelines might miss τ′ because individual components are too fast, or all τ _k need to be increased to compensate for this effect, resulting in an overly optimistic initial value of τ′, which is no longer related to the actual use case. By referencing all timing constraints to the absolute origin time of information t _info, the constraints of the entire pipeline are defined exactly by the last component, and each previous component defines the timing constraints of the pipeline up to that stage.

µRT specifies an information time t _info,i per data point i and validation of all timing constraints always refer to this value. For latency constraints, the point in time t _l,i at which the deadline τ is missed is hence defined by t l , i = t info , i + τ (7) and µRT can thus arm a timer to trigger as soon as t _l,i has elapsed, indicating a timing violation. The two critical times for synchronicity constraints, t s min , i and t s max , i , are likewise defined by t s min , i = t max − δ and (8) t s max , i = t min + δ (9) While t s max , i is validated by means of a timer as well (actually only a single timer is required for latency and synchronicity validation; cf. Equation 12; sections 2.6 and 2.7), t s min , i is checked whenever data is retrieved by the consumer. If it was fetched too early, a timing violation has occurred. Regarding the rate, only the most critical constraint ϵ′ among all n consumers of a data source needs to be considered for the definition of the critical time t _r,i+1: t r , i + 1 = t info , i + ϵ ′ with (10) ϵ ′ = min ϵ 1 , … , ϵ n (11)

Hence, validation of ϵ′ is not performed every time data is fetched by a consumer, but only when it is provided by the producer. Therefore, rate validation does not require another timer for each consumer but only one per producer (at most; cf. section 2.6).

Due to these mechanics, some possible side effects should be kept in mind when working with µRT. When data is provided by a producer, the contained information might already violate the latency or synchronicity constraints of consumers: t data , i > min t l , i , t s max , i (12) A similar effect occurs whenever the difference between t _info,i and t _data,i exceeds a rate constraint: t data , i − t info , i > ϵ ′ ⇔ t data , i > t info , i + ϵ ′ ⇔ t data , i > t r , i + 1 (13) As a result, a timing violation is detected as soon as the data is committed. Strictly speaking, this is already too late because, in both cases, the critical point in time has already elapsed. For µRT’s validation mechanisms, there is obviously no way of knowing about such data before it exists. Although detection of timing violations might be delayed in such situations, µRT still acts as quickly as possible.

There is yet another possible edge case that should be considered when specifying the rate constraints of consumers. Because of Equation 5, µRT might detect a rate violation, although future data would provide valid information: t info , i + 1 < t r , i + 1 < t data , i + 1 (14) Therefore, rate checks in µRT are rather conservative, and constraints ϵ should be specified with this in mind.

Finally, a last particularity of µRT’s timing validation mechanics is worth pointing out. As already mentioned, all constraints are defined by consumers, so in a sequential data processing pipeline, there are two steps for which no constraints can be defined: the observation, which initiates the pipeline, and the ultimate action at its end. In the former case, latency and jitter do not apply, and the rate can be validated by the subsequent component. However, for the latter case, the timing of the final action can only be validated by yet another consumer. The component that executes the action needs to provide information to confirm that the action has been conducted, and the additional component—also called monitor—consumes this information and validates its real-time behavior. While the use of such monitors seems complicated and inefficient at first glance, a single component can monitor all pipelines in a system, resulting in only minimal overhead.

For control of concurrent access to shared data structures, µRT relies on mutex locks and condition variables. Although lock-free methods are generally to be preferred for real-time systems (Anderson et al., 1997), they are difficult to realize for many aspects of µRT, in particular without the C concurrency support library (stdatomic.h), which was only introduced with C11 (ISO, 2011) and would make µRT unusable for projects that do not support this version of the C standard. Especially in the context of MCUs, more conservative standards are often preferred (even Linux was only recently lifted to C11 from C89; Torvalds et al., 2022). Conversely, most MCUs feature only a single core anyway, such that the benefits of a lock-free implementation are rather limited.

In order to adapt µRT to the specific requirements of individual use cases, multiple feature flags and settings are provided to configure the implementation at compile time. First and foremost, the three subsystems—synchronization, publish–subscribe, and RPC (cf. Figure 2)—can each be enabled or disabled as required, and another global flag selects between debug and release builds. These settings are particularly useful if code size needs to be reduced to reduce ROM utilization. Selecting a release build also disables many sanity checks and improves performance tremendously.

There are several more settings regarding the two communication subsystems. On the one hand, tracking of profiling information can be enabled or disabled for each of the two subsystems. While such information can help track down bottlenecks, the logic obviously requires additional resources in ROM, RAM, and CPU time. On the other hand, the validation of timing constraints can be enabled or disabled as required via a total of five flags; for publish–subscribe validation of latency, synchronicity and rate constraints can be selected individually, and the same applies for latency and synchronicity constraints for RPC interaction. Even with all validation logic disabled, µRT still distinguishes the four real-time classes (cf. section 2.2), which is a perfectly legitimate use case. Once again, disabling these components saves resources in all three domains.

Further settings allow fine-tuning µRT even further by setting the sizes (i.e., number of bits) for several frequently used data types, such as temporal delays ⁴ and identifiers for topics and services (cf. sections 2.6 and 2.7). There are also flags to select alternative algorithms for selected components of µRT, although these are not recommended for most scenarios and therefore are not discussed in this work. Finally, more settings allow configuring and interfacing OSAL, the abstraction layer for interaction with the operating system (cf. section 2.4).

The memory footprint of µRT is of major importance as the middleware is targeted to be used on (32-bit) MCUs, for which available resources are very limited. The sizes of the integrated flash memory of such devices typically range from 16 to 1,024 kB, so compiled images need to be rather compact. System memory is even more scarce, with entry-level products featuring no more than 8 kB of RAM and only the most powerful devices exceeding 256 kB. Therefore, it is a crucial requirement for µRT to exhibit a small footprint in both regards. Therefore, it can be deployed on a wide range of MCUs.

Table 2 shows the sizes of all major components of µRT in system memory and compares them to R2P, as presented by Migliavacca (2013). Most noticeably, many rows in the table contain no values for R2P at all due to the lack of such components because R2P does feature neither synchronization mechanisms nor RPC interaction. Compared to µRT, it also omits a core component. Because there is only a single core per instance, these 37 bytes should be negligible in most scenarios. Other than that, nodes are about 50% larger for µRT, which is a significant increase, but with an absolute value of 56  B, the footprint is still considered reasonable. Both components of the synchronization subsystem are acceptably small, with no more than 32 B. Publishers are much smaller for µRT and require only 4 B (25% compared to R2P), as they essentially hold just a pointer to the topic. Topics are also larger for µRT than they are with R2P. Especially when QoS is enabled, the memory footprint is 157% larger, but even without QoS, the increase is still 57%. It should be noted that each topic in µRT already holds a message, so those 32 B need to be subtracted for an apples-to-apples comparison, resulting in a somewhat smaller increase of 100% with QoS enabled and even the same 56 B as for R2P with QoS disabled. The size of subscribers in µRT depends on their real-time class (cf. section 2.2) but is estimated in Table 2 with the upper bounds. While they are slightly smaller when QoS is disabled (92%), size increases by a factor of 2 when enabled. Finally, the footprints of RPC components are similar to those of the publish–subscribe subsystem. Requests are again much larger when QoS is enabled, with an even starker difference of 88 B (157%). Conversely, services are of constant size, with 48 B acceptably small. Overall, the memory footprints of µRT’s components are about 50%–150% larger than their pendants in R2P, but absolute values remain small enough to be reasonable even for entry-level MCUs.

When considering the required amount of flash memory, µRT can scale from a modest 2.2 kB to 18.2 kB due to its high configurability (cf. section 2.3.4). Unfortunately, a direct comparison with R2P is difficult to achieve because Migliavacca (2013) only stated values, including the RTOS; hence, such was not conducted for this work. Absolute footprint sizes of µRT’s core, the three subsystems, and a complete configuration are given in Table 3 and represent the worst-case scenario among all evaluated MCUs.¹⁰ When comparing memory footprints in the binary image, the publish–subscribe subsystem is the most expensive component, with RPC close behind. Conversely, synchronization functionality has a minor impact of no more than 1 kB, even for debug builds. When comparing release and debug scenarios, footprints increase between 49% and 95% for the latter (67% on average). Enabling QoS is less expensive, with only 13%–50% larger footprints (29% on average). In both cases, impacts are most pronounced for the publish–subscribe subsystem. Considering the required flash memory exceeding 16 kB with everything enabled and considering that a sophisticated RTOS is required, µRT is hardly a viable option for MCUs with 32 kB of flash or even less. Based on our experience, a minimum of 128 kB is recommended for development (debug builds) and 64 kB for deployment (release builds), even with optimizations (i.e., garbage collection and link time optimization) enabled.

In order to evaluate the runtime performance of µRT, an extensive set of benchmarks has been conducted. ¹¹ These benchmarks have been designed in a way that scaling effects for each middleware component can be assessed, as well as differences between real-time classes. Therefore, scenarios have been implemented carefully to represent worst-case situations. Thread priorities were set in a way that nodes would constantly block each other, and the MCUs were continuously stressed by an additional low-priority thread. Moreover, each benchmark was repeated 1,000 times, so meaningful minimum and maximum values could be obtained to assess synchronicity performance.

The graphs in this section depict only the results of the worst-performing component (e.g., the node with the highest latency). Note that each graph consists of a line and a shaded area “hanging” below. Although the former represents the largest measured values, the latter depicts the range between the lowest and highest values in the result data. However, for most graphs, the shaded area is barely visible. Except for the data shown in Figure 6A, all results were obtained using an STM32L476RG (Cortex-M4 @ 80 MHz).

First, performance scales linearly with the number of nodes in a system, as shown in Figure 6A. For this benchmark, a single timer triggered all nodes, which just measured the latency until the event was eventually processed. As expected, absolute performance varies significantly between different MCUs. Although most MCUs demonstrate consistent performance, the STM32G0 exhibits significant temporal variance, which is most probably caused by its Cortex-M0+ core.

Figure 6B depicts performance scaling for publish–subscribe and RPC interaction with increasing payload sizes. Again, performance scales linearly, as expected. More interestingly, the first communication of an RPC interaction (until the service dispatched the request) is significantly faster than publish–subscribe, although the return communication (until the answered request is retrieved again) takes longer overall. This finding indicates that fire-and-forget requests should be used whenever possible. The graphs also show the potential performance gains if zero-copy communication was possible with µRT. Especially when comparing the time scales with Figures 7, 8, copy operations have only a minor impact on the overall performance.

For benchmark results of the publish–subscribe subsystem, as depicted in Figure 7, two striking features catch the eye. First, for an increasing number of communication hops, topics and (HRT) subscribers, synchronicity characteristics deteriorate. A closer look at the data reveals that only the very first interaction is significantly faster (by 13.2% for topics), and excluding this data point results in a jitter of 1.2% instead of 12.2% regarding topics. Second, performance graphs of communication hops and topics do not scale linearly but slightly exponentially. Again, this is only the case for HRT interaction, whereas performance scales linearly for all other real-time classes. When recalling the estimated complexity of publish–subscribe interaction from Equation 15, the reason becomes obvious. Neither the number of hops nor the number of topics can be increased without increasing the number of subscribers s _HRT. Doing so effectively results in a complexity multiplied by s _HRT and, therefore, a quadratic increase in latency because validation timers of all HRT subscribers need to be updated with every interaction: s HRT ⋅ O α ⋅ m + β ⋅ 2 p + γ ⋅ s HRT = O s HRT ⋅ α ⋅ m + β ⋅ 2 p + γ ⋅ s HRT = O α ⋅ m ⋅ s HRT + β ⋅ 2 p ⋅ s HRT + γ ⋅ s HRT 2 (17)

Nonlinear performance scaling for HRT communication is an accepted trade-off made by µRT, though, in order to validate timing constraints for such critical components. Moreover, the rather low expression of exponential scaling in the data suggests that the corresponding weight γ is rather small compared to α and β. Figure 7A shows constant performance for an increasing number of messages. This benchmark does not represent the worst-case scenario, as t _info increased with every interaction, so messages did not have to be enqueued (general case; linear complexity) but could just be appended (constant complexity). With this in mind, the presented graph must be interpreted as typical performance rather than worst-case performance, which scales linearly.

A similar picture emerges for the RPC subsystem. Performance evaluation, as presented in Figure 8, apparently shows the same effects as the publish–subscribe, but more pronounced. Again, recalling the complexity estimation from Equation 16 reveals the cause. As with HRT subscribers for publish–subscribe interaction, increasing the number of HRT requests r _HRT effectively results in quadratic scaling regarding this factor: r HRT ⋅ O α ⋅ q + β ⋅ 4 p + γ ⋅ r HRT − 1 = O r HRT ⋅ α ⋅ q + β ⋅ 4 p + γ ⋅ r HRT − 1 = O α ⋅ q ⋅ r HRT + β ⋅ 4 p ⋅ r HRT + γ ⋅ r HRT 2 − r HRT (18) Notably, performance does not scale exponentially when increasing number of communication hops this time because there is only a single request per hop (r _HRT = 1), so the hindmost term has no effect in this case. Although exponential effects are much more pronounced for RPC than was the case for publish–subscribe, the weighting factor γ remains relatively small.

Overall, performance figures confirm the overall linear scaling of µRT, although some scenarios exhibit exponential latency increase. However, this is only the case for HRT communication, and even then, nonlinearity is little pronounced for scales reasonable for MCUs. Especially when considering the findings concerning RPC interaction, request queues with lots of HRT requests should be avoided in the first place for the sake of a responsive system.

First, a within-person design was chosen for the study, and it was subdivided into four parts, conducted in weekly sessions of 2 h each.

1. ex ante: Participants were asked to fill in a questionnaire about demographic information and prior experience in various areas of computer science. For the latter, six-level Likert scales were used with the available options “none,” “novice,” “advanced beginner,” “competent,” “proficient,” and “expert.”

Questionnaires for the three tasks were identical, each consisting primarily of a USE questionnaire as defined by Lund (2001), but supplemented by three further questions—level of completion (self-estimation), required time, and the number of requests for assistance—as well as corresponding free-text fields to account for qualitative responses. Obviously, the results of those questionnaires are the dependent variables in this study.

The independent variable of interest was the employed middleware—ROS versus µRT—so the two most important parts of the study are the line-following tasks. The decision was made against a between-person design because the scenarios could not match each other exactly. As ROS cannot be run on the MCUs of AMiRo, that task had to be conducted in a simulation environment rather than an actual robot. Conversely, no simulator integration for µRT exists so far, so these scenario differences were inevitable. Furthermore, the acquisition of participants for a study of this scale is difficult, especially during the COVID-19 pandemic, making those a very scarce “resource.” The information gained per participant hence needed to be maximized. Moreover, the in-person design with this order—ROS first and µRT thereafter—does not compromise the validity of this study’s results because many software developers in robotics are already experienced with ROS and would use µRT as supplemental middleware on the real-time level of a system. Hence, the study design represents real use case scenarios very well.

Each task of this study was designed so it could be completed in about 90 min, but participants were permitted up to 4 h. For optimal comparability, both line-following tasks were designed identically: participants had to create a new node, add a subscriber to receive sensor information from a prepared topic, write a simple logic to calculate a two-dimensional motion vector, and interface an existing service to make AMiRo move. The quality of the implemented line-following algorithm was of no concern for this study, as the focus was on understanding middleware concepts and the ability to apply these to actual implementation.

In total, 21 students in computer science participated in the study. Ages ranged 21–28 (median 23.5) years. Of the 21 students, thirteen were male and six were female participants, and two were not specified. All students had a bachelor’s degree except one who had no academic degree yet. Thirteen (62%) participants had already worked with ROS before, and seven (33%) had experience in robotics, although self-assessed skill levels ranged no higher than “competent.” The same applies to experience in the two relevant programming languages, C++ and C, whereby the mean expertise for the former was about 1/2 skill level higher. Notably, experience in Python was much higher, with the median at “competent.” This result confirms the decision to restrict the ROS task to C++ because the strongly differing programming skills might have influenced the study results. Finally, although 11 (52%) participants had worked with MCUs before, none stated any expertise with RTOSes, indicating that the previous experiences were made in bare-metal programming of MCUs instead of using sophisticated RTOS software environments.

Unfortunately, although all students did work on all tasks, not all questionnaires were completed. Of the 21 participants, only 12 (57%) assessed the ROS line-following task, 10 (48%) submitted valid questionnaires for the µRT tutorial, and 6 (29%) rated the µRT line-following task. Moreover, complete data about all three tasks are available for just five students (24%). For this reason, results will be distinguished hereafter by whether they were obtained from the entire cohort or only from the five participants who provided complete information. Those two subsets of the cohort will be referred to as group A and group B.

Before examining the results of the USE questionnaires, the evaluation of the three additional questions provides insights into the correlation between participants who only filled in the questionnaires about ROS (group A) and who also provided information about their experience with µRT (group B). Figure 9 shows that the ROS task completion level varies much more for group A, whereas group B performed consistently well. In fact, the level of completion of the ROS task and whether or not a participant filled in the questionnaire about the µRT task are highly correlated: ρ = 0.678. This finding suggests that many students had issues understanding the concepts of publish–subscribe and RPC interaction in general and could not apply those with either middleware. Although discrepancies between the two groups are not as pronounced for the time required and the number of requests for assistance, where average and median values are similar, variances of the answers to these questions are also much higher for group A.

Participants, on average, required 1/2 to one more hour to complete the line-following task with µRT than using ROS. Participants also asked for assistance five times more often when working on the µRT task. These two findings are again correlated with ρ = 0.746 for ROS and ρ = 0.809 for µRT. This indicates that participants could have completed the tasks in less time if they had been able to solve them completely on their own. While the overall higher number of requests for assistance for µRT shows that tutorial and documentation need to be improved, a participant stated for the µRT tutorial task that two of five requests for assistance were due to hardware issues with the robot rather than about µRT. This comment indicates that difficulties of understanding are not five times higher for µRT. However, as such differentiation was not considered in the study design, no definitive statement can be made based on the obtained data.

Results of the USE questionnaires are depicted in Figure 10 and show strong differences between groups A and B. Although µRT seems to outperform ROS in all four aspects when considering group A, results are more heterogeneous for group B. Due to the small size of the latter, the exact values depicted in Figure 10B cannot be considered significant. However, the overall positive trend for all aspects of USE and both middlewares is evident.

When investigating the correlations between previous experience in various areas of computer science and the information given in the USE questionnaires depicted in Figure 11, several further observations were made. First, the results of the USE questionnaire regarding the ROS task are correlated to prior experience in robotics but are uncorrelated to MCUs, both of which meet expectations. When investigating the correlation matrix regarding µRT, the first thing that stands out is the consistently inverse correlations. In particular, the strong (inverse) correlations with prior experience with MCUs and ROS are noteworthy but difficult to interpret. Either developers without previous knowledge in these topics rate µRT positively, or those who already have such experience rate µRT negatively, or both. Again, no clear conclusion can be drawn due to the small sample sizes (12 and 6 participants, respectively), so these findings should be considered and further investigated in future studies.

Finally, the USE questionnaires also contain qualitative questions, and some answers given by participants are worth mentioning. When asked about the most negative aspects of ROS and µRT, respectively, many complaints were made about the high complexity of ROS and the insufficient documentation of µRT. There were also some comments about µRT not supporting object-oriented programming languages such as C++ and Python. On the contrary, many praised ROS that it supports exactly those languages and that it is an open-source project with a large community supporting it. However, positive mentions about µRT were its simplicity and that it is “not overloaded with features.”