Excluded are applications in sports (45) and hand gesture recognition (46, 47). Hand gesture recognition is excluded because it is a form of non-verbal communication, as opposed to being used for assessing functional UE movement.

Job-related assessments of skillful UE functional motion using kinematics are included. These assessments are similarly motivated by the need for more objective measures of performance (10, 11) and follows closely with the health-oriented kinematic analysis workflow. The methods developed for job-related assessment applications can also be applied to health applications involving the UE.

Winter (23) describes the scientific approach to biomechanics, which this paper uses to represent the kinematic analysis workflow associated with UEFAs (23). We make an addition to the kinematic analysis process to include movement segmentation, which has previously been identified as necessary for a variety of kinematic analyses (40, 48). The resulting process (see Figure 1) consists of movement measurement (Section 3), segmentation (Section 4), description and analysis (Section 5), and assessment and interpretation (Section 6). Definitions for each component are in Table 1.

Kinematics is concerned with quantifying the details of movement itself (e.g., position displacement, velocity, acceleration, and jerk) and not the forces that cause the movement, where the goal is to use kinematics to provide actionable information for the domain expert. Kinematic data are collected by either direct measurement or optical systems (23).

Measurement tools are needed that minimize the impact of encumbrance on natural movement, provide near real-time kinematic data with minimal noise and inaccuracies, and are relatively inexpensive to own and operate. A 2019 systematic review of low-cost optical motion capture for clinical rehabilitation indicated the need for better measurement methods and validation studies, although most papers reviewed were not specific to UE functional motion (56).

There has recently been substantial progress on 2D and 3D human pose estimation using low-cost sensors, where the goal is to infer a representation of the body from images, video, or inertial sensor data. Kinematics can then be derived from the output representation. Table 2 represents a taxonomy of these methods, inspired by previous taxonomies (30, 57). Included in Table 2 are recent reviews and research papers for each methodological approach, along with a non-exhaustive list of works evaluating the utility of these methods for measuring UE kinematics. A comprehensive review of human pose estimation algorithms is beyond the scope of this paper, where instead we include a brief description of recent measurement approaches categorized by one of three input data types—RGB, RGB-D, and inertial data—and synthesize recent results from studies evaluating their utility for use cases involving UE functional motion. Radio frequency devices (e.g., WiFi) that do not require transmitters placed on the body have also been used for pose estimation (30). However, these methods currently have low spatial resolution, and we are not aware of their usage for UEFAs.

Additionally, we consider four representations used in human pose estimation—planar, kinematic, keypoint, and volumetric—along with their respective input data types (30) (see Table 2 for associations between representations and input data): ∙

Planar: This representation models the shape and appearance of the human body, which is usually represented as rectangles approximating the contours of the body.

While using optical motion capture and IMUs to measure complex functional UE movement kinematics are popular, there are other ways to measure functional UE movement that tend to be application-specific. For example, haptic virtual environments record precise kinematic information via encoders (87) and provide a customizable workspace to assess functional UE movement, which has uses in rehabilitation (26) and surgical skill assessment (88). Kinematic measurements have also been recorded from real laparoscopic box trainers, which has been used to evaluate surgical skill (51, 52, 89–93). UE kinematic measurements have been recorded by the objects people manipulate, as is the case with ultrasound probes that have been used to assess the skill of obstetric sonographers (11). Handwriting on digitizing tablets are used for assessing neurodegenerative diseases (42, 94) and dysgraphia (95) using kinematic information of the pen tip and pen pressure on the writing surface.

Domain experts need to understand how well measurement systems work and whether they can be adopted for their applications. The computationally-oriented literature tends to focus on evaluating the accuracy and run-time of new measurement methods, such as assessing keypoint localization error for 2D pose estimation using the keypoint representation (30). For adoption in healthcare applications involving UEFAs, test-retest reliability (50, 68, 70, 96) and validity (24, 50, 59, 66, 68) need to be assessed. Furthermore, accuracy assessments (25, 63, 64, 67, 73, 74, 78) that do not rely solely on healthy participants (17, 38) are needed. Although the reviewed pose estimation methods are finding utility in health applications related to measuring UE movement, more widespread adoption of these tools require further assessments of measurement accuracy, validity, and reliability (9) on quantities that are important to movement scientists (38), e.g., joint angle (17–21, 50, 62).

Table 4 includes publicly available data sets with labeled activity and functional UE movements, although most of these data sets also include non-UE motion. These data sets have sequences with potentially multiple segment classes, requiring temporal segmentation. We are not aware of publicly available data sets with labeled motions found in clinically validated UEFAs, which are necessary for reducing the burden associated with the UE kinematic analysis workflow. Lin (108) identified rehabilitation-focused data sets for the UE and lower extremity. However, the only rehabilitation-focused UE functional motion data set (109) could not be found online. Zhang et al. (99) and Hu et al. (100) include state-of-the-art action detection performance measures for a variety of data sets, and most of the referenced data sets in Table 4 include benchmark performance results using supervised and unsupervised approaches.

Activities are the highest level in the functional motion hierarchy (see Table 3). Activity recognition (35, 36, 102) is useful for assessments where individuals are being evaluated in their natural environments throughout the day, where it may be useful to automatically identify activities an individual is doing. At-home health monitoring is especially important to clinicians because improvement in clinical measures does not necessarily mean UE performance improvement in free-living and unstructured environments (49), where improvements in the latter is the goal.

Human activity recognition has received attention from the computational research community; however, many of the methods need labeled data, which are not rehabilitation specific (see Table 4). Additionally, activity recognition as part of UEFAs is a relatively undeveloped area. Inaccurate commodity measurement systems (e.g., wearable sensors), non-validated outcome measures, and human factors challenges are current barriers to use of data capture and analysis (86). Furthermore, current measures used for at-home UEFAs are not activity-specific and instead summarize different aspects of UE usage throughout the day (49, 103). Activity recognition methods are useful for at-home UEFAs (84, 103), but their utility is not as well defined compared to the automated segmentation of functional movements and primitives (see Table 3). This paper does not thoroughly review human activity recognition methods due these aforementioned issues. Additionally, the activities performed during UEFAs, which are most commonly done in clinics, research labs, or as part of job-related assessments and training, are pre-defined to include only the activities of interest.

Functional movement segmentation approaches, also known as human action detection in the computational literature (98–100), typically use supervised or unsupervised learning. Progress in functional movement segmentation algorithms for UE functional motions has benefited from the availability of labeled data sets. Algorithmic development has therefore been largely focused on these well-annotated data sets because it is easier to compare and evaluate algorithms.

Lin et al. (48) provides an organizing framework for functional primitive segmentation, which includes online and offline methods. A variety of approaches are reviewed in (48) for general primitive segmentation (i.e., includes full-body primitives and gesture recognition) that apply to a variety of tasks (i.e., many UE functional motions require reaching, grasping, etc.). This section focuses specifically on methods for UE functional motion, either of the UE or an end effector.

Feature vector thresholds and zero-crossings (48) work well for simple actions and small data sets that allow researchers to visually verify the movement segments. Engdahl and Gates (97) segmented functional UE movement during pre-defined activities of daily living (ADLs) into reaching and object manipulation phases using a fixed-velocity magnitude threshold. Cowley et al. (12) and Engdahl and Gates (15) evaluated UE prosthesis users compared to able-bodied individuals on a set of standardized ADLs and segmented the movement primitives using pre-defined velocity magnitude thresholds. Li et al. (132) accounted for differences in participant kinematics while transporting objects by selecting 50% of movement time as when the hand reached a target position.

Approaches that use thresholds and zero-crossing tend not to perform well with complex functional movements (48), particularly for reaching motions (133). Some measurement systems allow for the collection of events (i.e., additional context about what the individual is doing, such as making contact with objects) in addition to kinematics, such as in haptic virtual environments. These events can be used to indicate action segments, e.g., person grasped object, person released object (26). Jackson et al. (26) showed that primitive segmentation, such as reaching and grasping, using a fixed-velocity magnitude threshold can result in incorrect primitive segments, requiring more robust computational approaches. To remedy this, (26) proposed a movement primitive segmentation approach that uses distance from the object and event recordings to segment reaching from object manipulation. This method has since been used to segment the reach and dwell primitives of pen point trajectories during the Trail Making Test to assess cognitive function (106).

Additional approaches to segmenting UE movement primitives include using 2D hand trajectories for identifying different hand-drawn shapes by segmenting the trajectory into strokes based on large changes in the angle between line segments and the horizontal axis (134). Motivated by robotic imitation learning, visual information, specifically kinematics derived from the Kinect pose estimation software, has been used to segment functional UE movements into reaching, manipulation, and release (135).

Given labeled data sets for both functional movements and primitives, segmentation evaluation measures include accuracy, precision, recall, overlap between ground truth and predicted segment classes, and the ordering of predicted segments (48, 98, 125). Unsupervised and supervised functional primitive segmentation algorithms can use the same data sets for evaluation (e.g., as has been done with JIGSAWS (91)). However, due to challenges associated with creating ground truth labels for functional primitives, verification of temporal segmentation results is limited (48). One of the major challenges with acquiring data sets of motion primitives is that it is still unclear what separates the different primitive phases using kinematics alone, especially given variations in pathologies, impairments, and movement strategies. Similarly, functional movement labels are not reliably identified across raters (98). Additionally, many of the available data sets focus on healthy, able-bodied populations, which may not properly indicate whether a segmentation approach will generalize to populations of interest to domain experts (48).

Given that many existing clinical measures are already validated and well-known by rehabilitation professionals (41), automating these measures may offer additional benefit because clinicians are already familiar with them and would benefit from potential resource or time savings. For example, (137) used machine learning to infer clinically validated scores of UE motor impairment and movement quality in stroke and traumatic brain injury survivors using wearable sensor data. Barth et al. (138) evaluated a method for predicting the UE functional capacity, as defined by the Action Research Arm Test score, of individuals with first-ever stroke using early clinical measures and participant age.

Development, evaluation, and automation of currently non-validated measures, such as some that use kinematic data, should also continue in parallel to automating the output of validated measures. For instance, clinically relevant gait parameters (e.g., walking speed, cadence) and validated gait measures (e.g., the Gait Deviation Index and the Gross Motor Function Classification System score) have been inferred from 2D keypoint human pose estimates using a single RGB camera (79); a methodology which could apply to UE functional motion. An UE-specific example is the development of a kinematic-based quantitative measure of UE movement quality post-stroke from motions performed during two widely used qualitative assessments, where the quantitative measure was found to be strongly correlated with the qualitative assessment results (83). Similarly, a measure of movement quality from UE kinematics of individuals with chronic stroke symptoms captured during a rehabilitation game was evaluated against established UEFAs (84).

Note that kinematic descriptions and analyses tend to be explainable and expert-derived, e.g., in contrast with representation learning. This is largely due to intervention decisions being the responsibility of a human that must be able to interpret the data. However, this does not preclude the use of methods such as deep learning to help with analysis, as is the case with (79).

Time series data mining techniques have been successfully used for segmenting motions and analyzing skill in the robotic surgery setting (127, 131, 139–141). Some of these methods have also been used for the analysis phase, such as converting trajectories to string representations (e.g., symbolic aggregate approximations (SAX) (127)) and comparing time series using a method called dynamic time warping (DTW) (131, 139, 140). DTW is useful because it allows the measurement of similarity between two time series with varying speeds. The motivation for these works is that surgical motion classes (e.g., grab needle, pull needle, rotate suture once (141)) and surgical skill levels follow distinctive patterns. While movement segmentation are often a focus of these works, the use of DTW represents a direction where kinematic measures are computed based on comparisons, as opposed to computing a measure from an individual’s kinematics only. For example, (139) computed a score based on how the trajectories of the robotic instrument tips compared to “optimal” trajectories during a simulated surgical task. While it is unclear what an optimal trajectory would be in a clinical setting, the time series data mining techniques these surgical motion segmentation and skill evaluation methods use could be relevant for identifying patterns in functional UE motion on standardized tasks.

As more measures are developed and validated, it is possible that for a particular functional motion there could be many measures used to describe it (49). Another research direction is to use computational methods for visualizing high dimensional data, such as using t-SNE (142), UMAP (143), or principal component analysis (49, 83). Using dimensionality reduction techniques to project high dimensional data to two or three dimensions for plotting could be useful for seeing how the evaluated individual compares to others.

The lack of validated and standardized kinematic-based outcome measures are a substantial barrier to more widespread usage of kinematics by domain experts (16). Although domain-specific researchers are likely better positioned to address this problem, computer-assisted tools that make descriptions and analyses easier to acquire will enable a wider group of domain-specific researchers to develop and evaluate kinematic-based outcome measures.

Although adoption of kinematic analyses is currently limited, researchers have recently used machine learning and artificial intelligence to automate aspects of the interpretation and assessment process (41, 42, 52, 94, 95, 107, 144–146). Whereas machine learning in the previous section is used to output outcome measures that a domain expert would interpret as part of their decision-making process, the methods considered here automatically output an assessment (e.g., the presence of a disease) based on input kinematic-based outcome measures (see Section 5). Pereira et al. (144) provides a systematic review of machine learning approaches and data sets for inferring the diagnosis of Parkinson’s Disease using kinematic measurements, among other data sources. Classification models trained on kinematic features have been used to predict the skill level of laparoscopic surgeons (52, 107). Handwriting on consumer tablets has been used for automated diagnosis of dysgraphia (95) and neurological disease (42, 94, 145, 146).

How domain experts physically interface with kinematic-based outcome measures, either from UEFAs or during free living, has also been studied from the perspective of human-centered design (147, 148). These outcome measures are just one of a variety of inputs domain experts use in their assessments, necessitating consideration of how to integrate these various inputs into a system easily used by domain experts. For example, a 2020 survey on requirements for a post-stroke UE rehabilitation mobile application showed that rehabilitation clinicians in the United States and Ethiopia valued the ability to record video of UE function, automatically update performance measures, graphically display patient performance in a number of factors, and see current quality of life and pain levels, among other desired features (147). Similarly, in (148), rehabilitation clinicians qualitatively evaluated a prototype dashboard that visualized UE movement information in stroke patients. The dashboard was then revised based on their feedback and presented in (148). User studies like these will be essential to successfully integrating kinematics analyses into domain expert workflows.

A variety of tools are available for measuring kinematics, from low-cost optical and wearable sensors to high-cost optical motion capture systems. Although not offering the same accuracy as optical motion capture, low-cost sensors and pose estimation algorithms provide an opportunity for wider usage of kinematics for specific applications, primarily for measuring gross movements (71, 83, 84). Integration of these more flexible systems will depend on developing more accurate human pose estimation methods that generalize to populations of interest to domain experts (17), measure relevant quantities for the particular domain, e.g., 3D pose and joint angles (38), and are shown to be reliable, responsive, and valid (9, 86). Wider adoption of low-cost measurement sensors depends on whether the data from these systems combined with specific kinematic UEFAs are demonstrated to be valid and reliable, as is done in (83, 84). Alternatively, kinematic UEFAs that indicate an acceptable measurement error range could help identify what movement measurement approaches to use in practice. Ease-of-use is another barrier to using markerless pose estimation methods more widely. Some works have developed software packages that are more accessible (e.g., two or more smartphones can be used, user-friendly application) outside of the laboratory, while also incorporating methodological modifications to improve kinematic quantities (e.g., body models) (62). Ease-of-use may also be why the movement science community has frequently used the Kinect for markerless motion capture (see section 3.2.2.1), where the Kinect has a built in pose estimation capability accessible via a relatively simple application programming interface (API).

A variety of methods exist for movement segmentation, which could help automate the processing of data before analysis and interpretation by a domain expert. Movement segmentation, along with measurement, represent the most costly and burdensome parts of the kinematic analysis workflow for UEFAs that computer-assisted methods could help address. Current segmentation workflows used by researchers will not scale to the volume of data expected as kinematic measurements and analyses become more prevalent outside the laboratory. Automated segmentation approaches, along with improved measurement approaches, will enable more widespread kinematic data capture and processing, especially in unconstrained natural environments, e.g., at home. More accessible kinematic data capture and segmentation would give a wider range of domain experts access to kinematics analyses to support the further validation and standardization (16) of kinematics-based outcome measures and the administration of measures that have been validated.

In addition to the outstanding problems of algorithm generalizability and the general lack of algorithm verification due to difficulties with acquiring labeled data (48), motion hierarchies (see Table 3) tend to be inconsistently defined. Although there appears to be agreement across the computational and health literature that there are at least three levels of motion (2, 102), different names for these levels could be confusing to domain experts and limit their application. Consensus on a functional motion hierarchy amongst computational researchers and domain experts will be necessary for segmentation algorithm development. Standardization of a functional motion hierarchy will help researchers curate more relevant data sets, where those data sets will be essential to further development of segmentation algorithms for evaluation and learning-based algorithms. The lack of relevant, rehabilitation-focused data sets that follow a standardized motion hierarchy needs considerable attention by the research community, where the requirements for those data sets will require expertise from both computational researchers and domain experts.

The validation of kinematic measures is essential to more widespread usage. However, a valid kinematic measure computed using a specific measurement and segmentation approach may not be valid using another type of measurement and segmentation approach, making it difficult to generalize kinematic measures that have not been validated with a particular set of measurement and segmentation methods (9). Furthermore, we believe that computing the kinematic descriptions and analyses themselves is not a burdensome aspect of the kinematic analysis process if the kinematic data are accurate (section 3) and properly segmented (section 4).

Developing methods to more easily measure and output existing validated clinical measures appears to be a valuable direction to pursue because of familiarity and existing use by clinicians (41). In addition to existing measures, developing methods to better measure kinematics, compute kinematic outcome measures, and validate them should be pursued in parallel. An approach to developing and evaluating new quantitative measures from kinematics is to compare the kinematics-based measure to currently used assessments that have been demonstrated to be valid and reliable (83, 84). In healthcare, improving the quality of outcome measures and making the assessments easier to administer are important for patient outcomes and documentation. Outcomes research is used to understand the effectiveness of health services and interventions, or outcomes, necessitating outcome measures that are both valid and reliable (149). Furthermore, the need for repeated assessments to inform interventions throughout the rehabilitation cycle (150) necessitates easily acquired and sensitive movement quality measures (9).

There is currently no consensus on how domain experts should use kinematic measures (9). Additionally, it is currently not known how computer-aided assessments or diagnosis (42) would best integrate into the kinematic analysis workflow used by a domain expert beyond use as a screening tool because of potential biases in the data (e.g., cultural and impairment variations), small reference data sets, and limited data on whether automated systems actually improve health outcomes (39, 41). Although there has been success in automating aspects of robot-assisted surgical skill assessment and handwriting analysis, it is unlikely that domain experts, especially clinicians, will be replaced with fully autonomous systems responsible for deciding on interventions (42). Instead, a potentially more tractable approach is for computer-assisted methods to be designed to assist domain-experts in making decisions by providing more objective information (138).

Integrating artificial intelligence and autonomous systems (41) into domain expert processes is challenging and raises questions about reliability, trust, generalizability, and how domain experts and individuals can interface with the autonomous system. McDermott et al. (151) provides a framework for interviewing domain experts and establishing requirements that can enable an effective human-autonomy partnership. System-level user requirement studies can also inform the integration process (147, 148). Additionally, cognitive systems engineering research could be an area that provides valuable quantitative evaluations on how computational tools integrate into domain expert workflows, such as the recently proposed joint activity testing framework (152). The need for more user-friendly kinematics measurement, segmentation, and analysis methods, as well as investigating how to integrate kinematic analyses into domain expert workflows (i.e., human factors), underscores the multidisciplinary approach required to meaningfully improve the quality and administration of UEFAs.