Throughout this paper, vectors are denoted by lowercase bold letters (e.g., z ), matrices are represented by uppercase bold letters (e.g., Z ), and the transposition of the matrix Z is expressed as Z T . For the matrix Z, the ith row, the jth column, the element located in ith row and jth column and the Frobenius norm are denoted by Z _i*, Z _*j , Z _ij and Z F , respetively. The sign function represented by s i g n x is that the value is −1 when x is less than 0, otherwise, the value is 1.

Assume that O = O n n = 1 N denotes the image-text pair dataset, each sample o n = ( x n , y n , l n ) includes three parts: one part x n ∈ R D x represents an image feature vector, another part y ∈ R D y denotes a text feature vector, and the last part l _n ∈ R ^C denotes the corresponding category labels, where D _x , D _y and C are the dimensions of these two modal data’s feature and the number of the category labels respectively. S ∈ 0,1 N × N is the matrix to measure the similarity degree between different modalities, called the similarity matrix. S _ij = 0 means that x _i and y _j are not similar to each other and S _ij = 1 denotes that these two data have at least one same category label. The input data is transformed into the corresponding hash codes and the similarity degree between different hash codes is obtained by calculating their Hamming distance in our model. The more similar the hash codes, the smaller the Hamming distance; the greater the difference between hash codes, the greater the Hamming distance. The formula for calculating Hamming distance is d c i , c j = 1 2 k − c i , c j , (1) In Eq. 1, c _i and c _j are the hash codes for the vector x _i and y _j , c i , c j represents their inner product and k is the length of hash codes.

MDFFH aims to obtain two hash functions through training, one is f x i ; θ x for images, and the other is g y j ; θ y for texts while maintaining the similarity degree of the original data. Here, θ _x and θ _y denote parameters in the different networks. These hash functions can convert the data into hash codes with unified dimensions for comparison.

The specific details of the networks in our model are as follows.

Image network: Image network is mainly composed of an image multi-dimensional fusion module and a fully connected neural network. Specifically, the multi-dimensional image fusion module includes a Vision Transformer network and a convolutional neural network. In the Vision Transformer network, the ViT-B/16 model is chosen as the basic framework and fine-tuned on this basis. We replace the last MLP Head used for the image classification in the ViT-B/16 model with a single-layer completely connected network with 4,096 neurons where the size of each image patch is 16 × 16. The transformer Encoder has 12 Encoder Blocks, which are shown in Figure 2. At the same time, the first six layers of CNN-F [48] are selected as the model of a convolution neural network. In addition, these two networks are pre-rained on ImageNet [49] to obtain initialization parameters. Finally, the output results of these two networks are fused into the multi-dimensional semantic features learned by the image fusion module by vector concatenation. The fully connected neural network has three layers, in which the number of neurons is 8,192, 4,096, and the hash code length in turn.

Text network: Bag-of-Words (BoW) is usually used to convert text into vectors, but the sparsity of vectors makes it impossible to fully capture the text’s semantic information. Inspired by [28], we adopt a text multi-dimensional fusion module to solve this problem. The text multi-dimensional fusion module extracts the text semantic features in different dimensions through five average pool layers (the scales are 1a, 2a, 3a, 6a, and 10a, where “a” represents the parameter), and uses 1 × 1 convolution layer to integrate multiple features. At the end of this network, there is a three-layer completely connected network to extract the text’s hash codes and the numbers of neurons in every layer are 4,096, 4,096, and the hash code length.

The performance of the cross-modal hashing model depends on whether generated hash codes can effectively reflect the similarity degree between different modalities. Generally speaking, the Hamming distance of hash codes generated by similar original data should be small, and vice versa. To ensure that MDFFH can achieve excellent retrieval performance, we have established an objective function composed of two terms: semantic similarity loss and hashing code quantization loss. We apply P * i = f ( x i ; θ x ) to denote the learned feature from the image network, where θ _x presents the network parameters. Let Q * i = g ( y i ; θ y ) denote the learned feature from the text network, where θ _y refers to the network parameters.

To minimize the semantic gap, we transform different modal data to the same common semantic space to measure similarity. Here, the formula of the likelihood function can be written as follows: p S i j | P * i , Q * j = σ Φ i j , S i j = 1 1 − σ Φ i j , S i j = 0 (2) In Eq. 2, Φ i j = 1 2 P * i T Q * j and σ ( Φ i j ) = 1 1 + e − Φ i j . When S _ij = 1, the inner product of P _*i and Q _*j will be bigger, which is equivalent to that the two data are more similar. On the contrary, the more dissimilar the two data are when S _ij = 0.

The maximization of the likelihood function is equal to the maximization of the negative log-likelihood function. To facilitate the training of MDFFH, the above formula can be converted into the following formula: J similarity = − ∑ i , j = 1 N S i j Φ i j − log 1 + e Φ i j , (3) where Φ i j = 1 2 P * i T Q * j .

Since the output of the continuous variables from the network is converted into hash binary codes through symbolic functions, there is a certain quantization loss. Therefore, we set the quantization loss term of hash binary codes to reduce this error: J quantization = H x − P F 2 + H y − Q F 2 , (4) where H x = sign ( P ) and H y = sign ( Q ) .

From Equations 3, 4, we can get the objective function for optimizing MDFFH as follows: min H , θ x , θ y J = J similarity + η J quantization = − ∑ i , j = 1 N S i j Φ i j − log 1 + e Φ i j + η H x − P F 2 + H y − V F 2 , (5) In Eq. 5, η denotes the hyper-parameter of the hash code quantization loss. Inspired by Jiang et al. [31], we set H = H x = H y during model training.

Given the discreteness of hash codes, we apply an alternating learning strategy to optimize MDFFH: at one time, only one parameter is optimized while the rest of the parameters are unchanged. In the optimization process, the model parameters are updated by the back-propagation with stochastic gradient descent (SGD). The optimization steps are shown in Algorithm 1. Generally, it includes three steps:

1. Optimize θ x with θ y and H fixed.

Select any image data x _i , and obtain the partial derivative of our objective function as following in Eq. 6: ∂ J ∂ P * i = 1 2 ∑ j = 1 N σ Φ i j Q * j − S i j Q * j +   2 η P * i − H * i . (6)

Then through the chain derivation rule, we can get ∂ J ∂ θ x from ∂ J ∂ P * i and optimize θ x according to BP.

2. Optimize θ y with θ x and H fixed.

Select any data y _i , and obtain the derivative of the objective function as following in Eq. 7: ∂ J ∂ Q * j = 1 2 ∑ i = 1 N σ Φ i j P * i − S i j P * i +   2 η Q * j − H * j . (7) Then through the chain derivation rule, we can get ∂ J ∂ θ y from ∂ J ∂ Q * j and optimize θ y according to BP.

3. Optimize hash codes H .

The objective function can be converted into the formula as follows: max H t r H T η P + Q = t r H T R = ∑ i , j H i j R i j , s . t . H ∈ − 1 , + 1 k × N (8) In Eq. 8 R = η P + Q . At last, the hash code matrix H is updated according to the feature matrixes of images and text as following in Eq. 9: H = s i g n η P + Q . (9)

The hash codes of the data not used for training are generated by the hash functions learned by MDFFH. For example, given the query image x q , we can get its hash codes by the hash function as following in Eq. 10: h q x = s i g n f x q ; θ x (10)

Similarly for text data y q , we can get its hash codes by the hash function as following in Eq. 11: h q y = s i g n g y q ; θ y (11)

MIRFLICKR-25K [50]: There are 25,000 images from the Flickr website in this dataset, and every image has text descriptions and labels, thus forming data pairs. During the experiment, we only retain 20,015 data pairs, because there are too few text descriptions for some data pairs. For each text description, the Bag-of-Word model is applied to convert it into 1386-dimensional vector form, and the corresponding label is transformed into 24-dimensional vector form. 2000 data pairs are randomly selected for querying and the rest for retrieval. For model training, we select 10,000 data pairs from retrieval.

NUS-WIDE [51]: There are 269,648 data pairs in this dataset, and each includes images, text descriptions, and data labels. There is a total of 81 categories of original data labels in this dataset. We selected 21 of the most common data labels as the experimental dataset and finally retained 195,834 data pairs after processing. Text descriptions and data labels in each data pair are converted into 1,000 and 21-dimensional vector forms through the Bag-of-Word model. The partition of different sets for model training in this dataset is consistent with the MIRFLICKR-25 dataset.

Evaluation: For cross-modal retrieval, researchers usually study two typical tasks: retrieving text with images and retrieving images with text.

To evaluate MDFFH’s performance, we select the two most commonly used evaluation criteria, namely, the Precision-Recall (PR) Curve and Mean Average Precision (MAP) [52]. The average accuracy (AP) of any query data is calculated as follows: A P = 1 K ∑ s = 1 M U s V s , (12) where K and M are the numbers of retrieved relevant data and the retrieval set, U(s) denotes the proportion of the first s retrieved data related to the query data, and V(s) shows whether the retrieved sth data is related to the query data, which can be judged by the category label. If two data are related, V(s) = 1, otherwise, V(s) = 0. The MAP value can be calculated by averaging the APs of all query data and is positively correlated with model performance.

In addition, the PR curve is another indicator for evaluating the model performance. The performance can be directly judged by drawing a PR curve of this model: if the area under this curve is larger, the model performance is better. Moreover, the corresponding recall and precision can be obtained by altering the Hamming radius and drawing the PR curve.

Baselines: We compare our MDFFH with nine representative models, which are CCA, CMFH, SCM, STMH, SePH, DCMH, DDCH, DCHUC, and UCCH. The first four models belong to hand-crafted models and the rest are deep network models.

We use PyTorch, which is a deep-learning framework based on dynamic tensors, to implement our MDFFH on the NVIDIA RTX 3090 server and the iteration number is set to 300. In the iteration, the learning rate gradually decreases from 0.03 initialized to 10^–6. The hyper-parameter η is set to 1, and the detailed parameter analysis is in the section Parameter Analysis. For each model result, experiments have been run five times and the average value is obtained as a representative.

The MAP scores of MDFFH and nine baseline models based on two general datasets are shown in Table 1, where “I → T” represents from image to retrieve text and “T → I” represents from text to retrieve images. We can find that for hash codes with different lengths, our model is superior to baseline models. For example, when we select the MIRFLICKR-25K dataset, compared with DCMH which is the most representative deep cross-modal hashing model, MDFFH on “I → T” tasks increased by 3.05% on average, and its MAP score on text retrieval image tasks increased by 1.04% on average. On the NUS-WIDE dataset, compared with DCMH, MDFFH’s MAP score on image retrieval text tasks increased by 5.52% on average, and its MAP score on text retrieval image tasks increased by 0.95% on average. In particular, compared with these five hand-crafted baseline models, MDFFH has been greatly improved. This proves that better performance can be achieved by integrating feature learning and the generation of hash codes into a unified end-to-end network. At the same time, MDFFH has a better performance compared with DCMH and DDCH. The reason is that DCMH and DDCH generate hash codes only using single-dimensional semantic features, ignoring the information complementation between multi-dimensional semantic features, which has certain limitations. On the contrary, MDFFH applies the image multi-dimensional fusion module and the text multi-dimensional fusion module to get the multi-dimensional semantic features of different modal data, which can mine richer semantic associations and establish more accurate modal relationships, thus helping to narrow the modal gap to greatly improve the retrieval accuracy.

When the hash code length is set to 16 bits, the PR curves of MDFFH and baseline models under MIRFLICKR-25K and NUS-WIDE datasets are demonstrated in Figure 3. For PR curves of different models, which curve has a larger area represents better performance. From this figure, it is clear that the performance of MDFFH outperforms other baselines, which is consistent with the application of MAP as a performance evaluation index.

The influence of hyper-parameter values in the model based on the MIRFLICKR-25K dataset is studied in this section. The hash code length is uniformly 16 bits and the experimental results are shown in Figure 4. The MAP scores of two cross-modal retrieval tasks change with the hyper-parameter. During the manual adjustment of the hyper-parameter, the range of values is 0.01, 0.1, 1, and 2. The experimental results demonstrate the MDFFH performance can reach the best under the setting of γ = 1. The initial values of other network parameters are randomly generated and then determined through network learning.

We have designed one variant and carried out experiments to verify whether the innovative module in MDFFH improves the overall performance. MDFFH-1 is a variant of MOFFH without a Vision Transformer. The variant aims to check the important influence of the innovative image multi-dimensional fusion module on our model’s retrieval performance. Table 2 shows the comparative results. From this table, it is clear that MDFFH’s performance is better than MDFFH-1’s performance on the MIRFLICKR-25K dataset because of the effective role of the image multi-dimensional fusion module. The image multi-dimensional fusion module effectively combines the global image information concerned by the Vision Transformer with the local image information concerned by the convolutional neural network to generate more representative multi-dimensional semantic features. This can more effectively get the semantic similarity between different data to learn more accurate hash mapping functions, and so improve our model performance.

For analyzing MDFFH’s convergence, experiments are conducted on MIRFLICKR-25K and NUS-WIDE datasets. During the experiment, the hash code length is 16 bits and the relative loss is used as an evaluation criterion. The relative loss of the ith iteration is the ratio of the loss function value of the ith iteration divided by the loss function value of the first iteration and the experimental results are shown in Figure 5. With the number of iterations increasing, the relative loss value decreases rapidly and becomes stable, which means our optimization algorithm is effective.