Recent efforts in the field of computer vision have applied deep learning techniques to build models for automated physical activity and posture recognition. Different camera types have been used to detect patient mobility, including Red-Green-Blue (RGB) cameras, depth cameras, and cameras that capture both color and depth images, such as the Microsoft Kinect device. Multi-view settings using multiple cameras installed at different positions also have been used for capturing a more encompassing view of the patient room (28, 31, 41, 42).

Generally, automated detection of patient mobility using computer vision first requires patient identification in the scene. After patient recognition, manually annotated datasets are used to train the model to classify patient pose and mobility. Such systems have been able to accurately classify patients' high-level activities such as nothing in bed (doing nothing, lying in bed), in-bed activity, out-of-bed activity, and walking (30), and postures such as lying in bed, sitting on the bed, sitting on the chair, and standing (22) in the ICU. Depth camera-based systems have been able to classify four provider activities: “moving the patient into and out of bed” and “moving the patient into and out of a chair” without incorporating the challenging step of patient recognition (48, 49).

While computer vision techniques can identify a patient's posture, accelerometer devices can quantify the movement intensity. Wearable devices allow for convenient data collection and analysis since they provide continuous and patient-specific data streams. Wearable accelerometers have been used for examining physical activity patterns in different cohorts in various ICU settings, including delirium patients, sepsis patients, and patients with unilaterally motor impairment (10–13, 43, 50).

Building on activity intensity detection, wearable accelerometer devices have been used for determining energy expenditure. Previously energy expenditure estimation of accelerometer devices has been validated in the healthy adult population (14). However, it has been shown that energy expenditure is overestimated in ICU patients by comparing mechanical ventilation with indirect calorimetry. The current methodology of estimation of energy expenditure relies on the detection of physical activity. It does not incorporate physiological conditions such as fever with shivering that may alter the energy expenditure (51). Accurate estimation of energy expenditure in ICU patients enables optimizing enteral feeding details to prevent overfeeding and underfeeding, both of which increase the risk of infection and prolonged weaning from mechanical ventilation (52).

Accelerometer devices have also been used to detect frailty (23), a geriatric syndrome defined as “a clinically recognizable state of increased vulnerability, resulting from the aging-associated decline in reserve and function across multiple physiologic systems such that the ability to cope with every day or acute stressors is compromised” (53). There is increasing evidence of frailty being an indicator for decreased reserve and increased vulnerability in critical care patients (54, 55). Frailty has been shown to increase the risk of both adverse events such as death and discharge to skilled nursing homes and prolonged hospitalization and loss of independence after hospital discharge (56, 57).

ICU patients experience decreased functional status and more muscle atrophy (58) exacerbated by minimal physical activity. They also suffer from impaired consciousness and attention (59), further compounded by disrupted circadian rhythm and sedation. While previous research has demonstrated the importance of physical activity in ICU patients, the increased risk of falls is brought on by confusion and agitation (60). Computer vision can detect falls in hospital settings (26, 27, 44, 45), as well as accelerometer-based monitoring systems (20, 21). Still, these approaches have not been adequately investigated in the ICU.

Despite the importance of accurate assessment and monitoring of physical activity, energy expenditure, fall risk, and frailty in ICU patients, few studies have investigated the use of pervasive sensing to facilitate assessment and monitoring.

Behavioral indices such as pain facial expressions are different from physical activity indices. They do not elucidate gross variations in patients and thus could elude the nurses' observation. Facial expressions can also be influenced by sedative-hypnotics and analgesics commonly used in the ICU. While there is no validated facial expressions score in critical care, a few preliminary studies have examined anxiety-related facial expressions. Anxiety is highly prevalent in critical care patients (61, 62). However, it is rarely screened in routine care settings in the ICU (62). Computer vision approaches have been used for anxiety detection (63, 64), focusing on features such as head movement, mouth, and eye movement, and heart activity as indicators of anxiety. Patient head pose angle and variability have also been studied using computer vision and associated with pain and agitation indicators (22, 65–67).

Continuous and objective monitoring of patient pain in the ICU, including for non-communicative patient populations, can pave the way for real-time adjustments to analgesics for optimal patient care, patient experience, and better health outcomes.

While wearable accelerometers have previously been used for studying pain (68), no study has investigated the relationship between pain and physical activity in the ICU settings, leaving unanswered the question of the complicated relationship between mobility, mental agitation, stress, and pain. The issue of pain in the ICU has many aspects. In addition to the potential effect of pain on a patient's physical activity, facial expressions and physiological signals may also be affected by pain. Previous work has investigated the feasibility of pain detection using vital signs (69, 70). Although this approach uses data routinely collected in the ICU, it has not shown strong specificity for pain detection. Formalizing facial expression of pain using facial action units (71) and advances in deep learning and computational power available have made it more plausible to move toward automated detection of facial expression of pain in the ICU. Facial expressions are assessed manually using several behavioral pain scales such as Non-verbal Pain Scale (NVP), Behavioral Pain Scale (BPS), and Critical Care Pain Observation Tool (CPOT), particularly for non-communicative patients (72–74). Researchers have used deep learning approaches to detect facial expressions of pain and to recognize individual facial action units associated with pain. Still, robust automated detection of pain in the ICU scene based on facial information requires more research and validation (75–79).

Agitation is prevalent in the ICU and is a large factor in conditions such as delirium (80). Current methods for assessing delirium rely on transient rather than continuous assessment, which is an important limitation given the waxing and waning characteristics which help define delirium. Over-sedation has been shown to lengthen ICU duration and put a patient at higher risk for delirium (81). In comparison, under-sedation has been linked with increased agitation and a higher risk of self-extubation (82, 83). Optimizing sedation to better control patients' agitation may lower the patient's risk of removing endotracheal tubes (84). Similar to pain, accurate detection of a patient's agitation and sedation levels will improve the administration of sedative interventions to optimize clinical decisions. Previous research has used sensors for monitoring agitation in critical care settings. Agitation detection methods have shown strong performance using accelerometers (16, 17, 85), image-based approaches (29), and pupillometric video devices (86). Using wearable devices to study anxiety, researchers used Google Glass to discover that heart rate, but not spontaneous blink rate, changes in anxious patients (87).

Previous studies have shown generally poor sleep quality in critical care settings (88). Sleep disruption in the ICU has been linked to various factors, including but not limited to the type and severity of the underlying medical condition, round the clock health care activities, enteral feeding, medication side effects, lack of natural light exposure and noise levels, and general disruptions to patients' circadian rhythm (88–90).

Sleep disturbance in ICU patients has been studied to determine its effect on patient outcomes and recovery (91) and has been shown to increase the risk of a longer stay in the ICU, worse discharge outcomes, impaired defense mechanism, and sleep disturbances that persist or develop after discharge (90). Determining a patient's sleep quality during their stay in the ICU allows for evaluating the effectiveness of the administered sleep hygiene interventions. Polysomnography, as the gold standard for studying sleep, has previously been investigated in ICU patients. However, polysomnography data require interpretation and might not be feasible for continuous data collection throughout the patient's stay in the ICU since it typically includes several EEG leads, electro-oculography, and chin electromyography (92, 93).

Accelerometer devices have been evaluated in quantifying sleep in healthy populations (15). However, previous studies have shown that they overestimate sleep in the ICU settings, possibly because the implemented sleep detection algorithms rely on a lack of physical activity in determining sleep events. ICU patients typically have low activity throughout the day, resulting in fractured sleep and shallower sleep stages (8). Other researchers have used computer vision to detect sleep pose, but these studies have been investigated based on data from healthy adults, limiting the generalizability of their performance to the ICU setting (28, 94).

To encourage sleep hygiene, hospitals generally implement official guidelines for regulating the presence of visitors in the ICU. However, visitations and interactions with the environment may be beneficial in improving patients' experience by reducing their anxiety, leading to a lowered risk of delirium and an overall more positive experience during their ICU stay (95, 96). Accurate detection of the number of visitors and healthcare personnel in the room and environmental factors such as a room's noise and light at all hours allows for quantifying the effects of such disruptions on patients' sleep quality and circadian rhythm integrity. Computer vision has been used to determine the number of people in ICU care rooms (31, 33, 97) to understand the association between visitation and clinical care disruptions to patients with patients' sleep hygiene and outcomes. Such information could assist in developing more accurate evidence-based visitation and sleep quality guidelines for ICU patients.

Light and noise intensity levels are primary contributors to sleep deprivation and fragmentation in the ICU (91). Previous studies have demonstrated the feasibility of using affordable light and noise intensity sensors in ICU rooms (22, 98). Several studies using noise intensity sensors in the ICU have determined that noise level frequently surpasses the levels recommended by the World Health Organization (WHO) (99, 100). Various sleep hygiene interventions in the ICU population incorporate light and noise exposure limits with mixed results in efficacy. Accurate, continuous light and noise intensity measures can enhance our understanding of the effect of environmental factors on patient sleep and the efficacy of sleep hygiene guidelines (90).

Deep learning methods discussed in many of the computer vision-based studies mentioned in this review require large, labeled datasets for training and validation. The developed models need to be validated in diverse populations in ICU settings and consider age, gender, primary diagnosis, and race. However, ethical and legal reasons rightly prevent the construction of public datasets to protect the privacy of patients, their visitors, and the healthcare team (112). Existing Privacy guidelines typically mandate deidentification and temporary recordings, while the current state of research still requires raw recordings of the ICU room for post-event annotations and analysis. While research in this direction could potentially facilitate the clinical workflow, researchers should be cognizant of the privacy implications and the tradeoff between privacy and technological benefits.

An essential intermediate step in several computer vision approaches is detecting a patient in the room of each video frame. Previous research (113–115) has shown promising results in facial recognition in each single frame or tracking the patient face during the recording, but most of these studies are validated on an ideal frontal full-face view of a patient's face or with standard lighting. Real-world ICU rooms may be crowded, with varying degrees of lighting and numerous objects that could partially obscure the patient's face, such as ventilators or oxygen masks. Patient face identification will be even more challenging due to variations in the face angle and obscuring elements such as facial hair or glasses. Improved patient recognition adapted to the ICU setting will make computer vision solutions more robust. Because of the variable location of the bed and patient, the ideal developed methodology should be agnostic to the bed/patient location and position.

Sleep detection methods still require further research for reliable use in ICU patients. Future approaches may focus on multimodal models and using wearable sensors to collect information on activity, heart rate, and body temperature. Current wearable activity sensors determine sleep based on a lack of activity and wake periods, but this does not account for the minimal activity of the ICU patients, thus resulting in low specificity.

Researchers also need to consider the effect of pain relief medications, nociceptive generators, and interindividual differences in pain processing in studying patients' pain. However, this is a challenging concept since medication effect changes over time depending on pharmacokinetics such as age, sex, weight, body surface area, renal and hepatic function, fluid shifts, medication dosage, time since administration, administration route, and drug-drug interactions. This complexity often increases for continuous infusions of medications. Additionally, model development relies on patient self-report and nurses' observation for non-verbal patients. The uncertainty regarding how to translate pain intensity assessments into objective, rational clinical decisions for analgesic therapies that reduce pain intensity and patient risk further complicates developing generalizable and reliable models. There is also a need to integrate outcomes pertaining to pain, analgesic requirement, and “patient function” to provide a more holistic perspective on recovery trajectories.

Another implementation challenge for using wearable devices in the ICU is the requirement of current devices to be tightly secured on the skin to better capture patients' subtle movements. This may prove to be an inconvenience for some patients, as continuous contact with the skin might cause irritation or even could pose risks for infection, tissue ischemia, compartment syndrome, and wound breakdown. Furthermore, for patients with medical equipment on their wrists, wrist-worn accelerometer devices might not be an option unless future medical equipment includes built-in accelerometers.

The inclusion of multiple data streams for different patient care tasks using pervasive sensing also requires generalizability, interoperability, scalability, and reliability of the systems. Interoperable and generalizable systems can operate together and be incorporated into different monitoring platforms, while scalable systems will accommodate collecting data from a larger number of patients. The accompanying analytical algorithms need to be adaptable for new hardware choices. Moreover, there must be a consensus on the measured variables better to evaluate the performance of proposed methods and devices. Ultimately, the positive impact of such systems in critical care needs to be rigorously assessed and validated.

To improve the adoption of pervasive sensing in routine care in the ICU, developed models and devices should be easily manageable by the care team with minimal or no required training. Any developed model should be validated to reduce the false alarm rate in the ICU -as is investigated with vital sign-based alarms (116)-and should optimize the visualization approach to prevent data fatigue. The presentation of new information should be determined by considering the preferences of the physicians and their team and could include facets such as a daily summary, continuous display or separate tabs, or simple alarms for specific, pre-determined events.

Ultimately, any developed detection and prediction model should report a patient's status in real-time to allow the ICU team to implement timely interventions, such as incorporating more active physical therapy regimens, improving sleep hygiene routines, and adapting administered medications. The necessary communications infrastructure and reporting medium should be optimized to avoid alarm fatigue to the already overburdened ICU nurses.