The human hand can have extensive movements and functions crucial for daily activities. Hand dysfunction can arise from various conditions, including various neurological disorders, hand injury, or due to ageing. The primary goals of hand rehabilitation are to restore movement, strength, and dexterity while enhancing the patient's ability to perform daily activities. Effective rehabilitation programs are designed to meet the unique needs of each individual, ensuring that recovery is both targeted and comprehensive. One of the primary goals of these programs is to improve the range of motion, which is achieved through stretching and mobility exercises that enhance joint flexibility and reduce stiffness. These exercises are essential for restoring natural movement patterns, especially after injury or surgery. In addition to mobility, rehabilitation often focuses on strengthening muscles through resistance training. This approach helps rebuild muscle strength and endurance, enabling patients to perform daily activities with greater ease and confidence. Furthermore, a crucial component of rehabilitation is the enhancement of coordination and fine motor skills, commonly referred to as dexterity. Through tasks and exercises that require precision, timing, and hand-eye coordination, such as picking up small objects, tracing shapes, or gesture-based interactions, patients can regain control over fine motor functions essential for independence in everyday tasks.

Evaluating hand function and the effectiveness of rehabilitation interventions involves various methods. Traditional methods include manual grip strength, range of motion, and functional ability tests, often conducted by occupational therapists. Traditional therapy often faces limitations such as low patient motivation and difficulty in home-based deployment. Motion capture systems and electromyography (EMG) are used to analyze hand movements and muscle activity, offering quantitative data on function and recovery. The advancement of Smart devices such as sensors, wearable sensors, Microcontrollers, and the Internet of Things (IoT) has revolutionized the collection of qualitative and objective information about patients' motor functions at a low cost, enhancing treatment efficiency (1). These platforms use common objects equipped with sensors and displays to monitor patients' activity during rehabilitation exercises based on Activities of Daily Living (ADL), such as drinking, cooking, and cleaning. Moreover, these solutions provide consistent tools for transparent monitoring of ADLs at home, facilitating better management of hand dysfunction. In recent years, game-based, AI-driven interaction in rehabilitation exercises has shown the ability to enhance patient engagement and adherence, with studies indicating they may be more effective than conventional rehabilitation methods (2, 3).

This paper aims to create a simple, easy-to-use web-based application for improving dexterity and hand-eye coordination for ADL tasks. We have reimplemented old arcade games in Python and integrated the controls via hand tracking, enabling a gamified cognitive and dexterity training experience. Simple diagnostic tools and a gamified experience provide an informative and engaging experience. The aim is to use existing deep learning methods to create camera-based applications instead of fantasy equipment, which is often impossible with financial constraints. This also enables low-cost hand-therapy solutions that do not rely on expensive hardware. A web-based interface will enable anyone with an internet connection to use the application, eliminating several constraints. The present work focuses on demonstrating the technical feasibility and usability of a camera-based gesture recognition system for rehabilitation.

Krizhevsky et al. (4) introduced Convolutional Neural Networks as a viable and scalable alternative for image classification and object recognition tasks instead of hard-coded and hand-designed features. He et al. (5) modified the CNN architecture proposed in (4) and its successor architecture by introducing the notion of a skip connection. The skip connection (a residual connection) connects convolutional blocks further down the neural net chain and its feed-forward chain. Howard et al. (6) introduced the notion of depth-wise separable convolution, which leads them to greatly improved efficiency gains, thus allowing them to deploy CNNs on smaller devices with constrained hardware resources like smartphones. Sandler et al. (2) improved the existing MobileNet architecture and introduced MobileNet v2. It presents the inverted residual structure, where the skip connections are between the bottleneck layers. This leads to more efficiency gains. The CNN models used in this research are slight variations of this architecture.

Dosovitskiy et al. (3) introduced the Vision Transformer (ViT) architecture, marking the first adaptation of the transformer model for image recognition tasks. However, Zhu et al. (7) demonstrated through detailed experimental analysis that ViTs tend to underperform in low-data scenarios, where Convolutional Neural Networks (CNNs) remain more effective—an insight that supports the use of CNN-based architectures in this research. In parallel, Bazarevsky et al. (8) proposed an optimized version of the MobileNetV2 architecture, which serves as the foundation for the object recognition models employed in this project via the MediaPipe library. MediaPipe Solutions, developed by Google, provides a comprehensive suite of AI and ML tools, including pre-trained models for efficient hand tracking and gesture recognition (9, 10).

Alfieri et al. (11) reviewed recent literature on gamification in musculoskeletal rehabilitation, highlighting its effectiveness compared to conventional therapies. Oña et al. (12) presented a novel system that enhances the assessment and rehabilitation of eye-hand coordination. The system suggested by the authors uses serious games in virtual environments to evaluate hand-eye coordination. Alarcón-Aldana et al. (1) presented a novel system that combines serious games with robotic devices to enhance the assessment and rehabilitation of eye-hand coordination. Panaite et al. (13) concluded that the game-based rehabilitation system has been found to motivate patients to continue their rehabilitation, as they feel rewarded when they win. Wang et al. (14) developed a system that combines AR technology, an air pressure-detecting device, and game-based learning to make the rehabilitation process more engaging and less tedious for patients.

Jha et al. (15) investigated the efficacy of a novel glove-based virtual rehabilitation system named Rehab Relive Glove, designed for patients with post-traumatic hand injuries. The authors aim to address the limitations of conventional physiotherapy and existing virtual rehabilitation technologies by providing a portable, cost-effective, and ergonomic solution that can be used in clinical and home settings. Bouatrous et al. (16) presented a hand rehabilitation game designed with clinicians for hand rehabilitation after a stroke. The game simulates fruit harvesting using a Leap Motion controller for hand tracking and the Unity 3D platform. Bostanci et al. (17) investigated the effectiveness of virtual reality (VR) training in improving manual skills and grip strength in the non-dominant hand of healthy individuals. The study concludes that VR games can effectively enhance hand function and grip strength in the non-dominant hand, suggesting that such interventions could serve as engaging and beneficial tools in rehabilitation settings.

M. Alimanova et al. (18) focused on gamifying hand rehabilitation using Leap Motion technology to create a game that helps train hand muscles through virtual reality tasks mimicking daily life actions. Chiu et al. (19) investigated the effectiveness of the Wii Sports Resort training in improving coordination, strength, and hand function in children with hemiplegic cerebral palsy. Lansberg et al. (20) investigated the feasibility and effectiveness of using the Neofect Smart Glove, a virtual reality device, for unsupervised home-based hand rehabilitation in chronic stroke patients. Luna-Oliva et al. (21) investigated the potential of using the Xbox 360 Kinect as a therapeutic tool for children with cerebral palsy (CP) in a school environment. The study found significant improvements in the children's balance and activities of daily living (ADL) after the video game treatment.

Considering the above literature, we have identified the gap in understanding the utilization of deep learning models for accurate gesture recognition that ensures the system is adaptive, engaging, and capable of real-time processing. This approach can provide an effective and motivating rehabilitation tool for patients with hand function disorders.

This research uses the Mediapipe Library (9, 10), particularly the javascript version of the library, for hand tracking and gesture recognition. Deep Learning methods only work when provided with gigantic amounts of data. E.g., the well-known ImageNet dataset consisted of more than 1 million images, and MS-COCO, another well-known dataset for object detection dataset, consisted of more than 320 K images. These datasets were state of the art about 5–7 years ago. The field has since moved on to larger and more diverse datasets, which is a major reason for the success of Deep Learning Methods. Therefore, with these resource constraints, the best approach seems to be not to collect a custom dataset and annotations but to use well-tested open-source packages trained on a huge dataset. The added advantage is that this mitigates any chances of dataset bias for our particular use case.

For the hand-tracking task, the camera input is read through the OpenCV javascript library and fed as input to the hand-tracking module from the MediaPipe library, which gives us the required output. OpenCV helps in frame capture and preprocessing. For Gesture Recognition Module Applies a CNN classifier trained to detect gestures such as swipe, pinch, tap, and finger point. PyGame is a desktop game prototyping. This output is then overlaid on top of the input image and displayed to the user. Due to low inference time, this process is nearly real-time. The gesture recognition module used for this research is not the default module provided with the MediaPipe library, but a composite of the hand-tracking module and some custom code. This is because custom gestures are implemented for targeted dexterity exercises. The following block diagram, shown in Figure 1, aims to provide some clarity.

The work consists of different modules, which all talk to each other via the web UI. The host server accomplishes this work. Upon visiting the website, the users are prompted by a login page. The Username and Password are stored in a local JSON file, which is queried by the website server at the time of login. The login process creates a separate profile for each user, enabling a personalized experience as stated in the research objectives.

After successful authentication, the user is directed to a page with a dashboard, which provides the user with an interface to the various modules developed, such as hand tracking, gesture recognition, and playing the games developed. After selecting a particular task, the web UI turns on the webcam. The webcam outputs an image, which is fed into the hand-tracking module. In the case of the hand-tracking module, this output is then overlaid on top of the original image captured by the webcam and displayed on the webpage. If the games are selected, the hand-tracking outputs of the hand-tracking module are taken as inputs to the gesture recognition module, which is automatically turned on, and the gestures recognized are fed as input to the games, which provides a personalized gamified experience to the user. The specifics of the gesture recognition module are discussed in the following chapter.

The user's authentication details are stored in a local JSON file to simplify the authentication process. It must be noted here that this is not a safe process, but it exists only as proof of concept of how each user's personal details may be stored for personal experience. In standard professional settings, this may be done by storing details on a different server, and details encrypted end-to-end; however, this is out of the scope of the effort. Fifteen participants completed the 10-items, 5 point Likert scale System Usability Scale (SUS), after interacting with the hand-tracking games (Pong, Tetris, Fruit Ninja, Virtual Keyboard). The SUS, developed by John Brooke (22), was used to evaluate usability of the game beyond gesture recognition accuracy. The mean SUS score was compared to the benchmark value of 68 using a one-sample t-test to determine significance.

In this study, MediaPipe and Phaser.js libraries provide the right level of abstraction for the work, thus making them an ideal choice. The Phaser.js is a web UI game deployment making it available for players to download, install, or play native to the JavaScript framework. The authentication web UI and home pages were designed using HTML, CSS, and Javascript. The web UI brings together the different modules of the project and makes them work together as a coherent system, as shown in Figure 1. The main reason for implementing a Login system for the website is to provide a personalised score-based report for the user achieved through the login system. The authentication page is shown in Figure 2.

Upon successful authentication, the user is greeted with the page shown in Figure 3. The following Figures 4a,b demonstrate the use of the hand recognition module, which forms the basis for the gesture recognition system and the control for the games. Figure 5 shows the gesture recognition module in action. These figures demonstrate that the gesture is recognized and displayed on the screen for each hand, whether both hands perform the same gesture or different gestures.

The experimental dataset was collected from fifteen healthy volunteers (aged 18–60 years) after obtaining informed consent, with approval from the institutional ethics committee. Each participant engaged with all four gesture-controlled rehabilitation games (Pong, Tetris, Fruit Ninja, and Virtual Keyboard). For each game, gesture recognition outputs, accuracy scores, and gameplay performance metrics were recorded. In total, the dataset comprised approximately 3,500 gesture instances across participants, with recognition accuracy ranging between 85%–95%. The dataset was created exclusively for proof-of-concept evaluation and is not intended as a large-scale training corpus. Each participant's performance metrics were recorded as shown in Table 1. Gesture accuracy for each participant per game is calculated using Equation 1,Accuracy(%)=CG×100 (1)

G = Total number of gestures performed by the participant in a game.

Across participants, gesture recognition accuracy ranged from 85%–95%.

This research mainly involved developing the gaming console without the need for complex multi-sensor setups. It aims to boost cognitive functions and hand-eye coordination through engaging, game-like exercises that accurately track hand gestures. Implementing games and integrating the controls for these games with the gesture recognition module seems to prove that monocam-based computer vision techniques can be used for dexterity exercises. The design emphasizes simplicity and user-friendliness to ensure easy interaction. Additionally, this research implements a web-based user interface to enhance accessibility as a proof-of-concept feasibility study. The current effort can be further extended to test its efficacy by conducting trials in a randomized control trial in a clinical setting, incorporating adaptive difficulty levels and personalized therapy plans, integrating multimodal sensors (e.g., wearable devices) to complement camera-based tracking, and expanding the platform with additional rehabilitation-focused games designed in collaboration with clinicians. This would lead to identifying potential drawbacks in the current approach and can modify the developed game based on clinician recommendation. SUS provides evidence that your deep learning–based hand-tracking solution meets these real-world constraints, not just laboratory accuracy. The t-test confirmed that the developed rehabilitation platform achieved above-average usability, supporting its feasibility as a patient-friendly system. This statistical validation demonstrates that the system is not only technically viable but also meets the user-experience standards necessary for rehabilitation contexts.