Opportunity++ is a precisely annotated dataset designed to support AI and machine learning research focused on the multimodal perception and learning of human activities (e.g., short actions, gestures, modes of locomotion, higher-level behavior).

This is an important area of research in wearable, mobile and ubiquitous computing Kim et al. (2010); Bulling et al. (2014); San-Segundo et al. (2018); Plotz and Guan (2018) systems able to model human behaviour and automatically recognise specific activities and situations enable new forms of implicit activity-driven interactions Lukowicz et al. (2010), which can be used to provide industrial assistance Stiefmeier et al. (2008), to support independent living or assist in sports and healthcare Avci et al. (2010); Patel et al. (2012); Feuz et al. (2015); Lee and Eskofier (2018) — see Demrozi et al. (2020) and Chen et al. (2021) for recent reviews.

Opportunity++ is a significant multimodal extension of a previous dataset called OPPORTUNITY—including previously unreleased videos and video-based skeleton tracking.

The former OPPORTUNITY dataset Roggen et al. (2010) was used in a machine learning challenge in 2010 Sagha et al. (2011), which led to a meta-analysis of competing approaches and the establishment of baseline performance measures Chavarriaga et al. (2013), and was then publicly released in 2012 ¹ .

Over the years, OPPORTUNITY became a well established dataset. For instance, it was used in one of the seminal work on deep learning for human activity recognition from wearable sensors Ordóñez and Roggen (2016); Morales and Roggen (2016), and it has been used recently in fields as varied as machine learning model compression for embedded systems Thakker et al. (2021), transfer learning research Kalabakov et al. (2021), ontology-based activity recognition Noor et al. (2018), unsupervised domain adaptation Chang et al. (2020), zero-shot learning Wu et al. (2020), convolutional feature optimisation Hiremath and Ploetz (2021) and deep network architecture optimisation Bock et al. (2021); Pellatt and Roggen (2021).

In recent years, there has been a growing interest in the combination of data obtained from video-based systems together with the time-series data originating from on-body sensors Rey et al. (2019); Kwon et al. (2020); Fortes Rey et al. (2021). Several reasons underpin this type of research: video-based activity recognition is naturally complementary to other sensing modalities Zhang et al. (2019); there is a wide availability of public videos that can be used as additional training source (e.g., YouTube) Kwon et al. (2020); and it makes sense to opportunistically use whichever sensing modalities are available around the user at any point in time Roggen et al. (2013). The original OPPORTUNITY dataset, however, did not contain any video camera data, nor data derived from Supplementary Video data. A similar limitation is seen with video-focused datasets, which generally do not include other sensor modalities Chaquet et al. (2013).

Opportunity++ addresses these limitations by enhancing the OPPORTUNITY dataset with previously unreleased video footage and video-based skeleton tracking. This will allow researchers to further build on this well-recognized dataset, while at the same time opening new research opportunities in multimodal sensor fusion.

Opportunity++ contains the recordings of 4 people engaging in a early-morning routine in a naturalistic home environment.

We designed the activity recognition environment and scenario to generate many activity primitives, yet in a realistic manner. Participants were instructed to prepare a coffee and sandwich as part of their breakfast, have the breakfast, and finally clean the kitchen putting utensils and foodstuffs back in place or in the dishwasher. This scenario allows to generates a large number of elementary actions (interaction between the hands and objects or hands and the environment) in a natural manner.

The environment, shown in Figure 1C, was set up to resemble a common kitchen fitted with all the required appliances (i.e., dish washer, fridge, etc.). The appliances as well as the object used to complete the activities were instrumented with a wide variety of motion and switch sensors. An inertial motion capture system and an UWB localization system was embedded in a jacket worn by the users performing the activities.

The final release of OPPORTUNITY++ includes the data of 4 users, performing 5 ADL runs and 1 Drill run each. The dataset includes a total of 19.75 h of sensor data annotated with multiple tracks: 1.88 h of actions performed with any of the two hands (ML Both Hands), 6.01 h of locomotion status (Locomotion), 3.02 h of annotated data for action performed with a specific hand and 4.89 h of high level activities. Moreover, the sensors placed on the objects produced a total of 3.92 h of annotated data.

Overall, the dataset comprise of more than 24,000 unique annotations, divided in 2,551 activity instances for ML both arm, 3,653 activity instances of locomotion, 12,242 action instances performed with a single specific hand, 122 instances of high level activities and 6,103 instances of interaction with the object in the kitchen.

Due to the common high-level scenario, the data is quite balanced amongst the 4 users, as visible from Figure 1.

Opportunity++ can be used in to address a wide range of research questions. We indicate here a few potential uses of the dataset:

• Multimodal fusion: Fusion between multiple distinct sensor modalities can be explored, notably between skeleton data obtained from the videos and the wearable sensors to evaluate the potential improvements of a multimodal activity recognition approach (Chen et al. (2016)). This can be interesting when designing activity recognition which aim at obtaining the highest performance. For instance, in assisted-living scenarios occupants may wear a few wearable devices but also be observed by cameras.