Frailty is a state of increased vulnerability to stress, which may lead to a diminished homeostatic capacity across multiple physiological systems (Fried et al., 2009). Frail older adults are at an increased risk of premature death and various adverse health outcomes, including falls, fractures, disability, and dementia, all of which could result in a poor quality of life and an increased cost of healthcare resources, such as emergency department visits, hospitalisation, and institutionalisation (Kojima et al., 2019). The comprehensive geriatric assessment (CGA), which serves as the basis for geriatric medicine and research, is primarily aimed at identifying and quantifying frailty by examining various risk-prone domains and body functions (Lee et al., 2020). Fried’s frailty phenotype (FFP), the most acceptable face to face evaluation for frailty, includes five components, namely weakness, slowness, exhaustion, low physical activity, and unintentional weight loss; trained personnel takes up 20 min for one case (Fried et al., 2001). The Rockwood frailty index (RFI), involving 70 clinical deficits, which is usually generated from comprehensive health records, was less available for seniors with limited medical resources (Rockwood et al., 2005).

Human locomotion is a common daily activity and is also an acquired yet complex behaviour. It requires the involvement of the nervous system, many parts of the musculoskeletal apparatus, and the cardiorespiratory system (Adolph and Franchak, 2017). Individual gait patterns are influenced by age, personality, mood, and sociocultural factors. Some age-related neurological cases, such as sensory ataxia and Parkinson’s disorders, lead to unique gait disorders (Pirker and Katzenschlager, 2017). Furthermore, the preferred walking speed in older adults is a sensitive marker of general health and survival (Pirker and Katzenschlager, 2017).

However, recent researchers have focused on understanding the impact of the frailty state on various gait parameters beyond speed, because only the gait speed might not be sufficient to classify the frailty state of an individual. An improved classification can be achieved by referring to parameters such as the signal root mean square and total harmonic distortion instead of simply relying on the gait speed (Martinez-Ramirez et al., 2015). Previous studies have suggested that transitionally frail individuals exhibit a reduced locomotive speed, cadence, stride length, increased stride time, double support (as a percentage of the gait cycle), and stride time variability as compared to healthy individuals (Schwenk et al., 2014). Artificial neural networks might also help to further investigate the frailty of gait. Dawoon recently analysed the gait statistics gathered from gyroscopes placed on the feet using a long short-term memory network-based classifier (Jung et al., 2021). Akbari performed a Kinect-sensor machine learning methodology as a frailty classifier via functional assessment exercises including a walking test (Akbari et al., 2021).

Different approaches have been implemented to determine the gait characteristics in clinical research. Numerous studies were based on signals from floor sensors or wearable sensors, which can relatively provide precise time and space information. However, these accessory devices have limited their applicability, specifically in developing districts (Muro-de-la-Herran et al., 2014). Human gait recognition and behaviour understanding (GRBU), mostly without the use of contact sensors, has become a major research branch of machine vision using artificial neural network tools and has a wide range of applications in the field of anti-terrorism, intelligent monitoring, access control, criminal investigation, pedestrian behaviour analysis, reality mining, and medical care (Luo and Tjahjadi, 2020).

Gait recognition and behaviour understanding are primarily divided into data-driven (model-free) and knowledge-based (model-based) methods, based on the requirement of any relevant human pose parameters for feature extraction. The idea underlying model-based methods is the application of mathematical constructs to analyse walking movement as a representation of gait appearance using several ellipses or segments (Yoo et al., 2002). The main advantages of the model-based approach are that it can reliably handle occlusion (particularly human body self-occlusion), noise, scale, and rotation, as well as overcome poor robustness and its dependence on precise modelling of the human body. A model-free GRBU method extracts the statistical information of gait contours in a gait cycle and matches the contours reflecting the same shape and motion characteristics. A gait energy image (GEI) (Li et al., 2018), which is a classical representation of gait features, derives many energy images of related features, such as the frame difference energy images (Chen et al., 2009), gait optical flow images (Lam et al., 2011), and pose energy images (Roy et al., 2012). The advantages of these approaches are that: (1) they can obtain more comprehensive spatial information, focusing on each silhouette; and (2) they can gather more temporal information because specialised structures are utilised to extract sequential information. However, the GEI-like method requires a high computational power.

A huge demand for telemedical care emerged owing to the increasing spread of the corona virus disease 2019 (COVID-19), particularly among the older adults. The application of deep learning algorithms introduces real-time health analysis via video recording devices, such as a monitoring camera, web camera, or smartphones (Banik et al., 2021). A machine vision-based self-reporting method has the potential to significantly enhance accessibility and reduce costs in frailty evaluation. A deep learning gait assessment, conducted in a recent trial, based on a wearable sensor or force plates, is promising for disease screening and in-home monitoring (Zhang et al., 2020). However, older adults’ resistance to wearable devices and the additional cost of equipment might limit the scenarios of these technologies.

This study is aimed at developing a machine vision driven geriatric disease gait identification method without using a contact sensor or index and at exploring its potential as a frailty screening tool possessing the characteristics of convenience, objectivity, rapidity, and non-contact (Figure 1). In this study, a multidimensionally labelled gait video set of an older adult was established. Then, a series of hyperparameters in machine vision networks were optimised and evaluated for gait feature extraction and identification. The predictive power of the frailty and pre-frailty (patients at risk for frailty who fulfil some, but not all, criteria for frailty) evaluation was measured using the area under the curve (AUC) of the receiver operating characteristic (ROC). These methods may be generalised to any gait-related disease identification processes. The current approach provides unreported frailty and pre-frailty screening tools, which may potentially be generalised to gait-related diseases or in-home health monitoring, with the characteristics of convenience, objectivity, rapidity, and non-contact.

In the current study, a machine vision method without using a contact sensor was implemented to identify frailty and pre-frailty among older adults based on their walking behaviour. First, an FFP-state-labelled senior walking video dataset consisting of 222 participants was created. All images of their gait sequences were treated using the body key point information extraction application AlphaPose. DPose2Seg, with a fully convolutional DenseNets segmentation module, also trained by an open-source human body image set, was the silhouette segmentation measure for the previous gait set. Gait body key points and silhouette information were used in a trained recognition network, DGaitset. The sequence-level feature extraction part in the trained DGaitset generated a customised CAE to compress the gait feature in a 64 × 64 matrix using the key points and silhouette sequences. We found that both machine vision methods (AlexNet and VGG16) equipped with better predictive performance globally than age and grip strength, as well as than 4-m-walking-time in healthy and pre-frailty classifying task.

In the pre-treatment stage, opensource AlphaPose and optimised Pose2Seg were used as tools for body key point extraction and silhouette segmentation. The first step of body key point extraction was the location of the human body, i.e., bounding the human within boxes. The inevitable small errors in body localisation can cause failures in a single-person body key point extraction. AlphaPose can handle inaccurate bounding boxes and redundant detections (Fang et al., 2017). The key point information output from the pre-trained AlphaPose not only provides mathematically constructed pose information to three silhouette segmentation methods but also provides part of the input for DGaitset and customisation of CAE. The replacement of layers of the residual unit with a fully convolutional DenseNets structure increased the precision of segmentation, which introduces the candidacy of DPose2Seg along with AlphaPose to function as a silhouette-generated method for other gait research.

As gait motion was a one-circle period, all silhouette clouds were represented in a single period. Our fundamental network, Gaitset, a model-free GRBU method, directly learns the representation of every frame silhouette independently via a CNN and set pooling instead of measuring the similarity between a pair of silhouette templates or sequences (Chao et al., 2019). Furthermore, a structure called HPM was used to map the set-level feature into a more discriminative space. Thereafter, the recognition was completed by calculating the distance between the representations of different samples. Gaitset exhibited a faster and more effective performance in individual re-identification tests in comparison with the previous model-free methods. The three major optimisations of DGaitset were as follows: replacement of max function with the attention module (in frame and sequence level), replacement triplet loss to weighted sum of softmax loss and hard triplet loss (in sequence level), and dual-channel input manner network (in global structure). DGaitset, a hybrid of model-based and model-free GRBU manner, performed better than the Gaitset and LGaitset approaches in the recognition task, and also suggests that DGaitset is a better candidate for the gait feature extraction and compression using the original video.

AlexNet used the residual unit activation function after the convolutional layers and softmax for the output layer, as well as applied max pooling instead of average pooling (Gu et al., 2018). VGG uses very small convolutional filters and very deep (16 and 19 layers) models (Bajić et al., 2019). The design decisions in the VGG models have become the starting point for the simple and direct use of CNNs in general.

The limitation of the current programme was the gait feature labelled by the FFP assessment excluding the prognosis events such as death, major cardiovascular events, or re-hospital, which directly point to a state of frailty because follow-up data of the WCHAT study is currently unavailable. The scale of the current walking video database and the unbalanced physical frailty state prevalence in the community-based cohort also limited the performance of the machine vision frailty classifier. As the potential clinical-gait-machine vision applications based on the current research may focus more on disease screening than accurate diagnosis, discarding random samples from the healthy group in the data compilation stage, or increasing the cost of the frailty group in the algorithm modification stage could increase the sensitivity to frailty state in the future development (Van Hulse et al., 2007; Krawczyk, 2016).

Although AlphaPose is a reliable method for building body key point images, its 17 key points (nose, left and right eyes, ears, shoulders, wrists, hips, knees, and ankles) did not include any points on the feet (Task Force Members et al., 2013). This deficit might cause an increase in noise around the feet compared with other body parts in silhouette segmentation. The latest body point extraction algorithm could label ankle, heel, and foot index, such as BlazePose and Zou’s method, which might provide a better choice for the machine vision body reconstruction model in this field (Bazarevsky et al., 2020; Zou et al., 2020). As most non-linear machine learning methods (Horst et al., 2019), part of the analysis processes in the current program were not straightforward, understandable, and interpretable.

Our methodology performs unique advantages in identifying the pre-frailty state, which might provide a clue for developing a novel biomarker. Pre-frailty is usually not as typical as frailty, which limits the proper preventive treatment, such as physical exercise, nutritional interventions, and implements (Serra-Prat et al., 2017). Furthermore, the current camera-based identification methods might extend their potential applications from frailty to other geriatric syndromes, such as cognitive impairment (Amboni et al., 2013).

A contact-free self-reported frailty assessment tool, based on this method, might help healthcare personnel (HCP) minimise their exposure to SARS-CoV-2-contaminated environment and equipment. It is well known that frailty status is a better predictor of prognosis than age in the COVID-19 therapy process (Hewitt et al., 2020). The evaluation of FFP depended on the face-to-face evaluation of HCPs, and HCP might not be able to enough detailed information within 30 min to make a comprehensive RFI evaluation for elderly patients diagnosed with hearing, visual or cognitive impairment. However, the cumulative exposure time of HCP to SARS-CoV-2 would increase the risk of transmission [National Center for Immunization and Respiratory Diseases (NCIRD), 2021].

After further mobile optimisation, our methodology might also expand in-home application scenarios with the rapid growth of smart device owners, globally (Silver and Taylor, 2019). With the rapid and large-scale growth of the elderly population in the world, there is a huge gap between the supply and demand of health monitoring and disease screening. Solutions aimed at reducing the strain on elderly care facilities and promoting independence, such as technology-enabled home-care services, will become the major part of the elderly care model in the near future (Mesko et al., 2018). Machine vision with artificial neural network tools has produced opportunities for convenient at-home screening of geriatric diseases such as frailty (Ahmed et al., 2020).

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

The studies involving human participants were reviewed and approved by the Biomedical Ethics Committee of West China Hospital, Sichuan University. The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

YL contributed to the design and carried out the current study. RW and YD contributed to modification of the experimental protocol. RH and QT contributed to optimisation of the silhouette segmentation module. XH and LQ contributed to key point extraction. BY and ZW contributed to the feature extraction module. YM, XLi, HW, XLiu, LZ, LD, ZX, and CX contributed to the film and edited the gait video. MG, XS, JJ, JC, XLin, and LX were involved in baseline information gathering. HG and HY analysed the experimental data. BD drafted the manuscript. XHH supervised the current research. All authors contributed to the article and approved the submitted version.

XLi is employed by Aviation Industry Corporation of China 363 Hospital. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.