Activity recognition recognizes important occurrences in huge video collections. In order to identify clinically significant patterns from the photographs and videos and categories engaging activities for ASD youngsters, machine learning (ML) and computer vision (CV) have enhanced several features of human visual perception(Stevens et al., 2019; Zhao et al., 2022; Gu et al., 2018). Marinoiu et al. were presented with one of the largest multimodal autistic interaction datasets. Additionally, they suggested a fine-tuned action and emotion classification based on data gathered from children with ASD during robot-assisted therapy sessions. Their findings demonstrated a good agreement between machine-predicted scores and expert human diagnosis (Marinoiu et al., 2018). Rehg et al. presented a novel action recognition dataset to analyze children's social and communicative Behaviors using video and audio data. Their early experimental findings showed this dataset's potential to facilitate multi-modal activity detection (Rehg et al., 2013). Washington et al. suggested a deep learning-based computer vision classifier for identifying head banging in home recording videos. They use a head banging detector to extract the target head posture from videos, and then use a CNN + LSTM architecture to analyze the head banging motion. The experimental result enhanced deep learning models by attaining the accuracy of 90% in correctly identifying the head bagging and no head bagging (Rehg et al., 2013). They explored two methods and contrasted a bag-of-visual-words method with RNNs and CNNs. This demonstrates that deep learning architectures give outstanding results for detecting four activities: spinning, head banging, arm flapping, and other hand and arm movements (Washington et al., 2021). Ali et al. proposed a framework to recognize actions in videos of ASD children, using 3D Convolutional Neural Networks (3D-CNN), combined with target person identification and monitoring techniques. The experimental results show that deep learning models can attain an accuracy of 75% in correctly identifying autism Behavior actions in children (Negin et al., 2021). Rajagopalan et al. suggested the SSBD, which comprises videos of autistic kids carrying out everyday activities. They combined a histogram of optical flow alongside a histogram of dominating motions. Their proposed binary classification framework for headbanging and spinning has an accuracy of 86.6%, and for multi-classification, headbanging, arm flapping, and spinning have an accuracy of 76.3% (Rajagopalan and Goecke, 2014). Lakkapragada et al. proposed a deep learning-based model to classify autistic children abnormal hand movement and control hand movement in autistic children. This research aims to show that deep learning algorithms can successfully identify hand flapping in uncontrolled home videos. His work used the SSBD dataset and the deep learning model MobileNet. The experimental results show that the model has attained the highest accuracy of 84.0% (Lakkapragada et al., 2022). Tang et al. proposed a deep learning approach to detect emotion using smile facial expression. In this study, he presented the RCLA&NBH_Smile dataset, a novel dataset. Thirty-four baby face expression recordings of their interactions with their mothers were captured, and more than 77,000 frames were manually labeled. The experimental results show that the proposed model has attained the highest accuracy of 87.16% (Tang et al., 2018). Manocha et al. suggested a system for monitoring autistic children's physical activity to identify abnormalities. His study proposes an activity prediction algorithm constructed using a deep 3D CNN and LSTM for detecting physical anomalies. The experimental results show that the suggested approach has attained an accuracy of 92.89% (Manocha and Singh, 2019). Ali et al. created a Behavior diagnostic paradigm for ASD. The stereotyped Behavior of the children was recorded in an uncontrolled setting during their ASD diagnosis, and they gathered and interpreted a set of these recordings. Children with ASD will be classified and their performance assessed using a multi-modality-based late fusion network. The findings showed that the proposed approach achieves better results and an accuracy of 85.6–86.04% (Ali et al., 2022).

People can communicate orally and nonverbally using facial expressions and eye contact. It can be distressing and lead to social anxiety for some individuals with ASD to maintain eye contact. It is difficult for kids with ASD to detect nonverbal clues, respond to them, and understand their gestures and emotions. Carpenter et al. used a trained computer vision model to extract positive, neutral, and various other facial features from a database of facial expressions. He argues that mimicking facial expressions is a crucial sign of social interaction abilities, supported by his discovery that kids with ASD exhibit more neutral facial expressions (Carpenter et al., 2021). He proposed a transfer learning-based approach to analyzing the facial features of autism patients. Their research showed that their method could more accurately identify the emotional expressions of children with ASD (Han et al., 2018). He developed an automated approach to identify emotions in youngsters engaging with robots to treat ASD. According to their research, computer vision could improve Behavior analysis when people engage with robots (Leo et al., 2015). Leo et al. suggested a machine learning approach for examining the facial expressions of kids with TD and ASD. The suggested approach may be effectively used to analyse the facial expressions made by kids with ASD thoroughly. Their results showed that their approach may yield an F1-score of 0.86% (Leo et al., 2019). Chintan et al. suggested an approach for digitally interpreting facial expressions to analyze human Behavior. He predicted the emotion across seven distinct categories using a deep learning network and the CK+ dataset (Thacker and Makwana, 2019). Ahmed et al. developed an automated approach to identify kids' facial expressions in videos recorded while they were using computer-assisted learning software. Their results showed how computer vision evaluation can automatically quantify Behavioral and emotional participation (Ahmed and Goodwin, 2017). Furthermore, a deep learning technique for facial image analysis has been used to identify cognitive developmental issues. Li et al. introduced a CNN-based approach for classifying ASD based on face features. Its results show that several face features improve autism classification (Li et al., 2019). Every methodology covered in the literature review section has been employed in various deep learning or machine learning models, except for a few limitations that are described in the comparison in Table 1. Most research tested their suggested model utilizing performance assessment parameters; there was room for improvement, evident in our results section.

In data acquisition, we collected the dataset used in our experiment. There is no previous dataset available that fully meets our condition. The SSBD data set was used to train the deep learning models. The SSBD is a freely accessible dataset gathered from autistic children. Figure 2 presents the detail steps for data acquisition and depict the flow of converting video into frames. The data consists of Videos that capture autistic children performing actions like spinning, headbanging, and shaking their hands. Parents and caretakers posted their videos on websites that are accessible to the public. Figure 3 presents the details of classes in dataset.

There are 75 videos in the preliminary dataset, which was gathered from the social media platform YouTube (Rajagopalan et al., 2013). Only one dataset, SSBD, is freely available, but this data is somewhat helpful. However, it does not fully meet our conditions. The SSBD dataset was developed by collecting the naturalistic video recordings of autistic children.

The MobileNet model is a lightweight model designed explicitly for embedded and mobile devices with constrained computational capabilities, designed by Andrew G in 2017. The MobileNetV2 paradigm was chosen based on several different considerations. Due to the small dataset used to train the model, over-fitting was possible. However, utilizing a smaller but more expressive system, such as MobileNetV2, greatly reduced this effect. The primary concept included employing depth-wise distinct convolutions to minimize processing demands and parameter values while achieving high precision (Howard et al., 2017).

University of Oxford researchers introduced the VGG (Visual Geometry Group) network as their deep convolutional neural network model in 2014. The VGG network model features 19 distinct layers, where 16 belong to convolutional operations, followed by fully connected layers as the final three. The stride measures at one pixel, and the pad measures at one pixel with a filter size set to 3 × 3. A small kernel size enables parameters to cover the entire image space while reducing the number of parameters. The 2 × 2 max pooling function of VGG-19 executes through a stride of 2. VGG Net further supported the theory that convolutional neural networks need an extensive layer structure because they understand visual patterns through structured systems. Feature extraction takes place through 16 layers of the network, following classification functions operated by the first three convolutional layers. Each feature extraction layer group contains five sections where max pooling operates as the final step. The system requires images with 224 × 224 dimensions (Simonyan and Zisserman, 2015).

Tan and Le proposed the EfficientNet-B4 as a convolutional neural network (CNN) model in 2019, which became part of the EfficientNet family. The image classification domain has recognized the significant accomplishments of EfficientNet-B4. The platform contains two scaling elements for reliable dimension expansion and resolution enhancement, resulting in cutting-edge performance. The neural network contains depth-wise separable inverted bottleneck convolutions named MBConv and squeeze-and-excitation SE blocks for feature recalibration improvement (Szegedy et al., 2015).

An LSTM model (also known as a Long Short-Term Memory) is a type of recurrent neural network that is specifically trained to acquire long-term dependencies in sequential data. It solves the vanishing gradient issue on the basis of memory cells and gating. Time-series analysis, and natural language processing are the most effective tasks with which LSTMs work. The model has input, forget and output gates that selectively remember or forget information. This is an advantage because LSTM models are highly appropriate in modeling intricate patterns of time-varying data (Saar-Tsechansky and Provost, 2007).

The 3D CNN is a continuation of the traditional 2D CNN, which operates on a three-dimensional dataset. It takes into account both time and space using 3D convolutional kernels. The applications of 3D CNNs include video analysis, action recognition and medical imaging. They model dynamic behaviors by learning motion patterns across sequential frames. This is why 3D CNNs are applicable in those tasks that require spatiotemporal features mentions (Ali et al., 2023).

The GRU model, a particular kind of recurrent neural network (RNN), is a widely used deep learning technique. GRU was developed to address the vanishing gradient issue with RNN. GRU controls the transmission status according to the state of the gates to remember the information that needs to be stored for an extended period, and less important information. Figure 5 illustrates a GRU cell's basic structure, consisting of two gates: an update gate and a reset gate. Information moving through the cell is rejected and accepted by the two gates. The reset gate determines what quantity of the previous data should be forgotten based on the decision taken by the sigmoid function σ. The output of the sigmoid is r_t. The GRU model will process the data if the sigmoid function value is 1; if it is 0, the data will not be processed. The current input x_t and the prior hidden state (ht-1) serves as the input for the reset gate. The Update gate determines the information that is going to be modified to convey a future state. Thus, the update gate requires a fundamental aspect of the prior state. To modify the cell state, the update gate additionally incorporates a sigmoid activation function (Zhang et al., 2015; Manocha and Singh, 2019; Bansal et al., 2023).

In this investigation of autism action classification, CNN in conjunction with GRU was employed as a classifier. Compared to the LSTM architecture, the GRU architecture is simpler and requires fewer parameters to be configured. In Figure 6, we show the proposed CNN GRU model architecture. The CNN GRU model uses a CNN convolutional layer to extract some key features from an image while preserving the image's original feature layout. Additionally, to avoid the issue of overfitting the model, the max pooling layer is utilized to shave the weak feature values and choose the deeper feature values from the key instances. This work also employed the rectified linear unit to trim down the eigenvalues smaller than 0 between the convolution layer and the layer that performs max pooling to speed up model training. The gated recurrent unit's (GRU) update and reset gates are then used to process the eigenvalues, speeding up the model's computation and improving its accuracy. For the convenience of the fully linked layer's use later on, the feature value is converted into a single-dimensional data by connecting the flattened layer. Lastly, the output is generated using the Softmax activation function.

The k-fold validation results and the comparative performance analysis, which is strongly support the selection of CNN-GRU architecture. Although layers in CNN are useful in extracting spatial features of single frames, autism-related Behavior is a temporal phenomenon that need sequence modeling across frames. GRU was chosen in place of LSTM because it uses a simpler gating operation and fewer parameters to be trained, resulting in better training stability and less overfitting during a k-fold training where the same model is trained repeatedly on varying data splits. Based on the results of the performance tables, CNN-GRU model is always more precise, recall, and F1-score per-fold than standalone CNN models, which means that it generalizes better and its performance is less variable. This equal performance at least in recall and F1-score indicates that GRU is very good at capturing the temporal dependencies, but with no computational cost of LSTM. Thus, the k-fold validation findings are empirical reasons to use CNN-GRU as an effective and strong architecture to assess autism behavior.

The same hyperparameter settings were used to train all the models to facilitate a fair comparison of them: VGG16, CNN-LSTM, MobileNet, CNN-GRU and EfficientNet-B7. The output layer activation function was the Softmax and a dropout rate of 0.4 was used to ensure that it did not overfit. The Adam optimizer was used with the learning rate of 0.001 to train the models to minimize the categorical cross-entropy loss. The evaluation metric was the accuracy and all models were trained with 50 epochs and under the same conditions.

Performance evaluation is necessary in finding out the extent to which a classification model achieves its objectives. The target performance evaluation metrics determine how well and accurately the model works on the test data. An appropriate selection of metrics, such as Accuracy, and F1 score, among others, is essential to carry out a detailed assessment (Zhang et al., 2015). The following are the formulae generally used to calculate the above performance measures.

Accuracy is calculated as the sum of the value of TP and TN, divided by the total number of TP, TN, FP, and FN values.

The percentage of correctly determines the degree of precision recognized positive instances and dividing the total number of positively and mistakenly identified positive instances, as formulated in Equation 2.

Recall is calculated as the percentage of correctly recognized positive instances, divided by the total number of true positive and identified false positive instances, as formulated in Equation 3.

F1 score is calculated by multiplying the mean of Recall and Precision and dividing by the total number of Recall and Precision, as expressed in Equation 4.

In Table 2 presents the results of different deep learning models. We implemented five deep learning models on the SSBD dataset and assessed their performance using validation accuracy and loss. Our proposed customized CNN-GRU model performs better than other models and attains the highest validation accuracy of 96% with a minimum validation loss of 0.16. Moreover, VGG16 performs the worst compared to others and achieves a minimum validation accuracy of 72.37% with a validation loss of 4.75. Our suggested approach has attained the highest accuracy score of 0.96 for arm flapping and head banging classes. In contrast, normal Behavior and spinning classes have attained low performance metrics scores of 0.96 and 0.97.

The graphical representation of the experiment results of our proposed CNN-GRU model is shown in Figure 7. We clearly see that the Arm flapping class has attained the highest bar because it attains the highest F1 score, 0.98, and the head banging class achieves the same result. Normal Behavior and spinning classes achieve the lowest bar because they earn low F1 scores of 0.93 and 0.94, respectively.

All the models were tested by means of k-fold cross-validation to ensure a high quality and objective evaluation. In Table 4 presents the results of different deep learning models. The findings suggest that the CNN-GRU model has the best overall performance with the average accuracy of 0.92 and equal precision, recall and F1-score of 0.91. MobileNet is also highly and steadily performing, with 0.91 on all evaluation measures, which demonstrates its efficiency and the ability to generalize in conditions of cross-validation. VGG16, is also highly and steadily performing, with achieving score 0.91. EfficientNet-B7 achieves the competitive performance of 0.88 and the standalone 3dCNN-LSTM model achieves the same of 0.89. Comprehensively, the stability and the generalization capabilities of the hybrid CNN-GRU model are supported by k-fold cross-validation and determine it as the most trustworthy strategy among the considered ones.

The significance level (alpha) is set to 0.05. The proposed CNN-GRU model is compare to baseline models by using four paired t-tests. All the comparisons had p-values that were significantly lower than the adjusted significance value of 0.0125 as indicated in Table 5, which proved to be statistically significant. Precisely, CNN-GRU had tremendous improvements compared to MobileNet (p = 0.0028) and VGG16 (p = 0.0051). Greater significance is found as compared to 3D CNN-LSTM and EfficientNet-B7, and lower p values. These findings suggest that the high performance of CNN-GRU model is not random and is not due to chance alone. Bonferroni's correction is use to regulate the family wise error rate. The significance value is adjusted using Equation 5.

In this ablation study, the value of each specific architecture element is analyzed by comparing the proposed CNN-GRU model with a variety of baseline and hybrid networks, i.e. VGG16, EfficientNet-B7, MobileNet, 3D CNN-LSTM. We measures the importance of various architectural elements through a comparison of the proposed CNN-GRU model and various baseline and hybrid networks such as VGG16, EfficientNet-B7, MobileNet, and 3D CNN-LSTM. Table 4 summarizes quantitative results in terms of accuracy, precision, recall and F1-score whilst Table 5 displays statistical significance of performance differences using paired t-tests. As Table 4 indicates, the proposed CNN-GRU model is the most overall winner in terms of the accuracy of 0.9294 ± 0.0038 and F1-score of 0.9284 ± 0.0039, surpassing all other competing models. Also, the t-test results in Table 5 statistically prove the improvements, with CNN-GRU showing significant performance improvements over MobileNet (t = 3.6811, p = 0.0028) and VGG16 (t = 3.5239, p = 0.0051). Moreover, comparatively with spatiotemporal architectures, CNN-GRU demonstrates a substantial and statistically significant edge over the 3D CNN-LSTM (t = 10.8180, p < 0.001), which means that GRU offers to represent time more efficiently and more reliably than LSTM in such a scenario. The most significant statistical support is seen against EfficientNet-B7 (t = 16.6697, p < 0.001) indicating that the lack of explicit temporal modeling is not offset by the increase in depth of the network.

It is a challenging endeavor due to autism diagnosis relies on long-term observation of human behavior. In this study, we presented an intelligent approach to detect autism related Behavior such as hand flapping, head banging, spinning, and normal Behavior. Our proposed approach uses deep learning and computer vision techniques. An analysis of comparisons is necessary to demonstrate the effectiveness of the suggested model. In Table 6, we present the details of the comparative analysis of our suggested models with various current models used in recent studies. After examining and comparing the performance of our suggested model with various current models, we found that the proposed model achieves a 92.94% accuracy score, which is far better than other approaches used in recent studies.

Instead of being a stand-alone diagnostic system, the suggested system shows tremendous potential for real-world clinical implementation as a decision-support tool. In practice the proposed CNN-GRU-based system to be use as a decision support system for clinicians, integrated with the existing screening system. In the regular Behavioral screening of patients, video clips of the children recorded with the use of cameras in clinical setting and interpreted by the system to provide risk scores and behaviors for Autism. These risk scores and Behaviors can easily integrated into existing electronic health records (EHRs) software systems/clinical dashboards for the clinician to interpret the predictions of the system alongside existing screening tools. In terms of system implementation, the system would not be hardware-dependent; it would be able to run on regular clinical computer workstations or be deliver through a cloud-based system for easy accessibility. The system also would be capable of near-real-time predictions due to the computational efficiency of the CNN-GRU architecture, making it ideal for use in early screening settings. The system would be intend to improve the screening process, help with regular screening, and facilitate the clinician with the diagnosis of ASD in the early stages while maintaining control of the diagnosis with the clinician.

The suggested framework does not imply any direct human experimentation and uses entirely anonymized, publicly available dataset the full scope of the ethical implications related to the use of automated diagnostic support systems in clinical settings is taken into consideration. Specifically, there are risks that these systems will be abused or overused as the sole diagnostic systems. To address this issue, the suggested CNN-GRU model will be used as a purely clinical decision-support tool, one that will help a professional to identify possible patterns of Behavior related to ASD, but not to substitute his/her judgment. The protective measures must involve upholding human-in-the-loop decision making where ultimate diagnoses must be controlled by qualified health care practitioners. Functional audits also require regular performance audits, monitoring biases, and retraining using more diverse datasets to minimize the risk of systematic errors and enhance generalization. Lastly, there is need to establish explicit clinical usage rules and ethical controls to help avoid misuse and have automated predictions used with care, especially in delicate pediatrics diagnosis cases.

Although the SSBD dataset is publicly available and annotated, the systematic form of annotation, this study uses a single dataset consisting of video recordings of children acting in a predetermined Behavior sequence. Although this allows the ability to control the experimentation, it might not represent the full range of diversity and complexity of real-world manifestations of autism spectrum disorder. The Behavioral patterns of ASD may be diverse among different people, at different developmental stages, in different environmental situations, and across cultures. While we use stratified k-fold cross-validation to minimize sampling bias and to ensure that the proposed CNN GRU model is tested on similar subsets of the data, we acknowledge that the k-fold validation will not be able to offset the limited Behavioral and contextual diversity within one dataset. Such a restriction may affect the model performance when used in unconstrained real-life situations or even in the clinical environment. The same hyperparameters were used to train all architectures to create consistency and fairness of experiments. Nonetheless, various models such Efficient Net and 3D CNN-LSTM are normally prone to be optimally tuned to the architecture. This single arrangement may not wholly utilize the strengths of some of them and may rather work against them. The future work will be done to conduct systematic hyperparameter optimization specific to each model to have a more balanced and fully optimized comparison. In addition, the validation of the offered methodology on various datasets gathered across a variety of sources, such as clinical settings and home based to enhance the rigor and external validity.