Human-object interaction behavior recognition aims to identify dynamic interaction relationships between humans and objects in images or videos, such as actions like “climbing ladders,” “holding ladders,” or “grasping objects” (Tu et al., 2023). The challenge of this task lies not only in accurately identifying the interacting subjects and objects but also in understanding the semantic relationships between the actions. Existing methods are mainly divided into two-stage methods and single-stage methods: the former relies on intermediate object detection results, making the inference process complex and susceptible to errors (Gkioxari et al., 2018; Zhang, et al., 2021; Gu, et al., 2022); the latter uses end-to-end network architectures to jointly predict the human, object, and interaction types, simplifying the detection process and improving efficiency (Zhu et al., 2024; Kim, et al., 2020). In recent years, Literature Zou et al. (2021) proposed the HOI Transformer model, which utilizes a Transformer encoder-decoder structure to achieve efficient human-object interaction detection within an end-to-end framework. Literature He et al. (2023) further proposed the SG2HOI+ framework, which constructs a dual-layer interactive Transformer network to jointly model scene graph generation (SGG) and human-object interaction detection tasks, significantly improving the contextual understanding of interaction relationships. Building upon this, the rapid development of visual-language pretraining models has provided a new research direction for HOI detection. Multimodal pretraining methods, represented by CLIP (Contrastive Language-Image Pretraining), construct a cross-modal unified semantic space through contrastive learning, and have been widely applied in tasks such as open-vocabulary recognition and cross-modal retrieval (Radford et al., 2021). HOI detection methods based on such models have begun to explore the use of language priors to enhance interaction semantic modeling capabilities. For instance, HOICLIP introduced a query-based knowledge retrieval mechanism, enabling efficient knowledge transfer from the pre-trained CLIP model to HOI detection tasks (Ning et al., 2023). Literature Liao et al. (2022) proposed a method called Gen-VLKT, which introduces a Visual-Linguistic Knowledge Transfer mechanism to effectively fuse visual features with semantic priors, thereby enhancing the model’s semantic understanding of interaction behaviors.

Based on the aforementioned research, this paper proposes a one-stage method that integrates visual and linguistic knowledge. It combines CLIP Transformer to extract global interaction semantics from the linguistic modality and designs HOI Vision Transformer to strengthen local interaction analysis in the visual path, thereby significantly improving the model’s performance in detecting human-object interactions in complex power scenarios.

Prompt learning was originally applied in the field of Natural Language Processing (NLP), where its core idea is to construct specific contextual prompts, leveraging a small amount of labeled or unsupervised data combined with manually designed prompt templates or guiding sentences to enhance model performance without the need for large-scale annotations (Khattak et al., 2023). In human-object interaction recognition tasks, introducing prompt learning not only helps improve the model’s understanding of action semantics but also strengthens the collaborative guidance relationship between the language and visual modalities. Especially in scenarios with imbalanced class distributions, scarce samples, or ambiguous interaction behaviors, guided prompt modeling can provide more discriminative feature support (Zang et al., 2022).

Multimodal models such as CLIP align image and text feature embeddings through contrastive learning during the training phase, constructing a shared semantic space across modalities, which gives the model good generalization ability. Among them, the CoOp (Context Optimization) method pre-appends learnable vectors to the category words, enabling CLIP to be applied to a variety of downstream tasks. The MaPLe (Multi-modal Prompt Learning) method combines learnable text prompts and visual prompts, leveraging the high coupling between the image and text encoders to enhance the semantic expression and cross-modal information sharing between the two modalities (Xue et al., 2024).

In contrast to the aforementioned methods, the approach presented in this paper introduces a prompt learning mechanism in both the language and visual modalities. It constructs a unified semantic prompt vector, and through the language prompt network and visual prompt network, maps this vector into respective prompt representations adapted to each modality, ensuring the modality specificity, interactivity, and adaptability of the prompt information.

This section introduces the proposed multi-element interaction analysis method based on semantic prompts and locally-aware enhanced Transformer. The overall framework consists of a multimodal semantic prompt module (MSP), a contrastive learning module, and two locally-aware enhanced transformer modules (LAET). Figure 1 illustrates the overall structure of the model.

First, in the semantic prompt generation phase, a unified high-dimensional prompt vector is constructed to express richer semantic information. This vector is then mapped into language and visual prompts via the prompt learning network. The language prompt expresses specific operational interactions in textual form, while the visual prompt is embedded based on the behavior objects extracted from the image, guiding the subsequent modality feature extraction process. In the language modality, the input semantic prompts, along with human-object interaction text pairs (e.g., “climb, ladder,” “check, test rod”), are fed into the CLIP Transformer to extract global language features. Simultaneously, a gated recurrent unit (GRU) network is employed to extract local features from the semantic sequence. After fusion, a complete language modality representation is formed. In the visual modality, the visual prompt is combined with the input image, extracting both local region features (e.g., “person, ladder, test rod”) and global structural features of the entire image. Local features are extracted using a pre-trained ResNet-50 network, while global features are encoded by the HOI Vision Transformer. Finally, all modality features are input into the contrastive learning module, where the consistency between semantic information and the visual modality is optimized. This enables the model to accurately associate each set of interaction features with its corresponding HOI category labels (e.g., “person climb ladder,” “person check electricity “), thereby achieving precise recognition, semantic analysis, and risk assessment of human-tool interactions in complex operational scenarios.

To address recognition challenges arising from complex action semantics and high similarity in operation and inspection tasks, this paper proposes an MSP inspired by the Unified Prompt Tuning method (Jia et al., 2021) to enhance multimodal collaborative understanding in complex human–object interaction scenarios. The core idea of this module is to introduce a unified prompt vector to collaboratively guide information interaction between the language modality and the visual modality in a shared semantic space, as shown in Figure 2.

Although unified prompts help establish a shared semantic bridge between different modalities, due to significant differences in the information structure and expression between the language and visual modalities, directly sharing the prompt vector can lead to high generalization but insufficient specificity. This makes it difficult to simultaneously meet the task requirements of both modalities. To address this issue, this paper further proposes a prompt learning network, which introduces a modality adaptation mechanism. This allows the unified prompt to adaptively adjust according to the feature distribution and representation space of different modalities, enhancing the prompt’s adaptability and discriminative power in multimodal tasks and enabling effective mapping from the unified semantic space to modality-specific feature spaces.

Specifically, we define a set of learnable vectors U ∈ R n × d as the unified prompt, and assign prompt vectors U V ∈ R n × d V and U L ∈ R n × d L to the visual and language modalities, respectively, where n represents the length of the prompt vector and d represents the dimension of each prompt vector. During the initialization stage, we first predefine the categories of operators, tools, and actions involved in typical inspection and maintenance tasks, and construct corresponding human–tool interaction prototypes (e.g., “an operator climbing a ladder” and “an operator grasping a test rod”). These natural language descriptions are then encoded using a frozen CLIP text encoder to obtain semantic embeddings, which are subsequently aggregated via mean pooling to form the initial unified prompt representation. This process injects explicit operational semantic priors into the model, thereby facilitating subsequent learning.

After semantic initialization, to further adapt to heterogeneous feature distributions across modalities and enhance cross-modal interaction capability, we introduce a lightweight Transformer layer to perform structured transformations on the unified prompts. In this module, the unified prompt U undergoes self-attention, a feed-forward neural network, and layer normalization, resulting in a structurally enhanced prompt representation U ^ . The self-attention mechanism in the lightweight Transformer facilitates effective interaction between the language and visual modalities, enhancing the expressive power and complementarity of the prompt information across modalities. Furthermore, we represent the prompt learning networks for the language modality and visual modality as Π V and Π L , respectively, which are used to learn the prompt vector representations within each modality. The prompt representations U ^ V and U ^ L are defined in Equations 1, 2, respectively. U ^ V ← ∏ V U V (1) U ^ L ← ∏ L U L (2)

Next, we concatenate the multimodal prompts with the inputs to construct the final multimodal input. In the language modality, we first define the input for the i-th sample as T i = t i 1 , t i 2 , ⋯ t i L H O I , 0 ∈ R m × d L , where m represents the length of the Transformer input sequence, and L H O I equals the combined length of action and object tokens. Accordingly, m − L H O I zero vectors are introduced as padding. By introducing the language modality prompt, the input sequence U L = U L 1 , U L 2 , ⋯ U L n ∈ R n × d L is reconstructed, and the final transformed language input sequence, and the final transformed language input sequence can be represented as shown in Equation 3. T ^ i = U L 1 , ⋯ , U L k , t i 1 , ⋯ , t i L a c t , U L k + 1 , ⋯ , U L n , t i L a c t + 1 , ⋯ , t i L H O I , 0 (3)

Here, L a c t represents the length of the action word. The language modality prompt is divided into prefix prompt U L p r e f i x = U L 1 , ⋯ , U L k ∈ R k × d L and concatenation prompt U L c o n j u n = U L k + 1 , ⋯ , U L n ∈ R n − k × d L , where k denotes the length of vectors in the prefix section of the prompt. Therefore, the number of padding zero vectors is m − n − L H O I . For the visual modality, the i-th input sample is defined as an image I i ∈ R H × W × 3 . The image is then divided into several image patches, resulting in an image patch sequence x i 0 , x i 1 , ⋯ , x i N ∈ R N + 1 × d V . A special token, z i , serves as the global representation of the image and is prepended to the sequence of image patch. As a result, the image input sequence becomes z i , x i 0 , x i 1 , ⋯ , x i N ∈ R N + 2 × d V . Finally, the visual modality prompt is concatenated with the above image input sequence to form the complete visual modality input sequence, as defined in Equation 4. I ^ i = z i , U V 1 , ⋯ , U V n , x i 0 , x i 1 , ⋯ , x i N ∈ R N + n + 2 × d V (4)

To address the challenges of extracting fine-grained interaction features and modeling complex multi-person collaboration in operation and inspection tasks, this paper introduces LAET modules into both the language and visual branches, thereby strengthening the model’s capability in analyzing local features and collaborative interactions. This module employs a dual-branch parallel architecture design, where one branch is based on the Transformer structure, used to extract global features. This branch calculates the relationships between positions in the input sequence using the self-attention mechanism, enabling the model to better capture long-range dependencies and contextual semantic information during both the encoding and decoding phases. The other branch is based on a convolutional neural network (CNN) or a gated recurrent unit (GRU) for local feature extraction, focusing on key regions and fine-grained local information using a fixed receptive field. Although this branch has advantages in capturing local details, its limited receptive field and weight sharing make it difficult to independently model global contextual dependencies (Li et al., 2021). To address this limitation, we propose the fusion of Transformer and CNN/GRU structures, achieving efficient integration of global and local features through joint modeling, significantly improving the model’s understanding and expression of interaction relationships under multimodal input conditions.

To extract more discriminative multimodal features, we use the outputs calculated by Equations 3, 4 as inputs and pass them to the LAET module to extract the fused multimodal enhanced features. The global feature extraction branch for the language modality (LGF) is used to extract the global semantic representation of the text, as defined in Equation 5: f LGF i = CLIP − TextEncoder frozen T ^ i ∈ R d L (5)

In this context, frozen indicates that the feature extraction module is in a frozen state during the training process, meaning its parameters do not participate in backpropagation and updating.

The language modality’s local extractor captures the local semantic details from the text, as defined in Equation 6. f L L F i = GRU T ^ i ∈ R d L (6)

Furthermore, we use the hyperparameter λ l ∈ 0 , 1 to fuse the global and local features, resulting in the final feature representation f L i for the language modality. f L i = λ l · f L G F i + 1 − λ l · f L L F i (7)

In the visual modality, we first represent the set of HOI instances in image I ^ i as a set of feature vectors H i = h i 0 , h i 1 , ⋯ , h i N h ∈ R N h + 1 × d V , where N _h denotes the number of HOI instances detected in the image, and h i j represents the visual feature representation corresponding to the j-th instance. The global feature extraction branch of the visual modality based on the pre-trained HOI vision Transformer (VGF) is defined as shown in Equation 8: f V G F i , H i = HOI − ViT I ^ i ∈ R d V (8)

The visual modality’s local extractor is applied to retrieve local visual details from the image, as defined in Equation 9. f V L F i = ResNet − 50 I i ∈ R d V (9)

Furthermore, we use the hyperparameter λ v 1 ∈ 0 , 1 to combine global and local features for constructing the final visual embedding f V i for the visual modality. f V i = λ v 1 · f V G F i + 1 − λ v 1 · f V L F i (10)

Subsequently, we use the hyperparameter λ v 2 ∈ 0 , 1 to further fuse the HOI features with the local visual features, resulting in the final HOI instance feature representation, as shown in Equation 11. f H O I i = λ V 2 · H i + 1 − λ V 2 · f V L F i (11)

To achieve deeper fusion and consistent alignment of multimodal information, this study introduces a contrastive loss function during the training phase to guide the learning process. Specifically, the model jointly maps visual features, HOI features, and textual semantics into a shared semantic space, where alignment is facilitated through the construction of positive and negative sample pairs. In this process, matched image–text pairs (positive samples) are pulled closer, while unmatched image–text pairs (negative samples) are pushed apart. This design effectively enhances the cross-modal consistency between human action semantics and tool interaction semantics. Based on this principle, the following contrastive learning objective and corresponding loss function are defined.

First, the similarity score logits between the image-text pairs are computed by combining the feature representations obtained from Equations 7–11: l o g i t s = f H O I f L * T , l o g i t s ∈ R N H O I × N L (12)

In this equation, N H O I represents the total number of human-object interaction queries, N L indicates the count of text label inputs, and T serves as the temperature parameter to control distribution smoothness.

Specifically, during the prediction stage, the HOI detection head f H O I produces a fixed set of HOI queries, where each query predicts a human–object pair associated with a human bounding box, an object bounding box, and the corresponding interaction classification scores. These query-based predictions are decoded to form the HOI instance candidate set H i n s . The generated HOI instances are then jointly evaluated with the similarity logits from Equation 12 and the ground-truth labels (GT) through the Hungarian matching algorithm (Matcher) to select the optimal HOI instance set with the lowest matching cost, as shown in Equation 13. H ^ ins , l o g ^ i t s = M a t c h e r H ins , l o g i t s , G T (13)

Finally, the overall loss function F is designed to capture the relationship among the predicted instance set H i n s , l o g ^ i t s , and the ground truth labels, as formulated in Equation 14. F = C E _ L o s s l o g ^ i t s , G T + B C E _ L o s s H ^ i n s + L 1 _ L o s s H ^ i n s , G T + G I O U _ L o s s H ^ i n s , G T (14) where CE_Loss denotes the cross-entropy loss used to supervise the interaction classification based on the similarity logits, BCE_Loss represents the binary cross-entropy loss for HOI instance existence prediction, L1_Loss measures the regression error between the predicted and ground-truth bounding boxes, and GIoU_Loss further constrains the spatial alignment between predicted and ground-truth boxes by optimizing their geometric overlap.

This paper constructs a representative human-object interaction dataset based on typical operation and inspection scenarios in a certain city, covering five types of interaction relationships as detection targets: climbing ladder, holding ladder, lifting ladder, electricity checking, and grasping test rod. To meet the requirements of interaction risk identification tasks in operational scenarios, this paper constructs a dataset based on the HICO-DET dataset format, primarily involving the following steps: (1) To ensure data coverage and representativeness, this paper captures key frame images at a frequency of one frame every 30 s, prioritizing images with relatively complete perspectives and minimal obstructions as positive samples, and appropriately excluding negative samples with no interaction relationships or major obstructions; (2) To further improve sample quality, brightness enhancement and other data preprocessing operations were applied to some images; (3) Referring to the HICO-DET dataset structure, manual annotations were made using annotation tools to label people, tools, and their interaction relationships, generating standard person-object-interaction triplet formats. The final annotated dataset files, trainval_hico_ann.json and test_hico_ann.json, were obtained.

The final dataset contains a total of 1,800 sample images, each of which typically contains one or more interaction targets, with a total of 2,096 annotated interaction targets, as detailed in Table 1. To improve model recognition performance, this paper uses the public dataset HICO-DET for pre-training during the model training phase and as the initialization weights for the model to enhance feature perception capabilities. In terms of dataset partitioning, 80% of the self-built dataset was randomly selected for training, with the remaining portion used for testing. Specifically, 1,440 images were used as the training set, and 360 images were used as the test set.

The research experimental platform is based on a 64-bit Windows 11 operating system, with an NVIDIA GeForce RTX 4060 Ti graphics card featuring 8 GB of video memory. The Python version used is 3.8, and the deep learning framework is Pytorch 1.10.0. The GPU acceleration libraries are CUDNN 8.2.0 and CUDA 11.3. The model was trained for a total of 200 iterations, with a batch size of 4 and an initial learning rate of 0.0001. The Adam optimizer was used for training, along with decoupled weight decay regularization. Additionally, the hyperparameter was set to 0.7 to control the fusion ratio of global and local features.