A matrix is deemed structured if it can be exploited to create effective algorithms [1] and has a small displacement rank [2]. Kissel and Diepold [3] have explored four main matrix structure classes, namely, semiseparable matrices, matrices of low displacement rank, hierarchical matrices and products of sparse matrices, and their applications in neural network. Toeplitz, Hankel, Vandermonde, Cauchy, and Circulant matrices are among the possibly most well-known matrix structures that are all included in the class of matrices with Low Displacement Rank (LDR) in [4].

Circulant matrices are structured matrices that have several features, including identical rows, but are shifted one step to the right [5]. It can be decomposed unitarily into a diagonal matrix whose diagonal entries come from its eigenvalues [6]. The eigenvalues of such matrices are derived in terms of the eigenvalues of matrices of decreased dimension, and linear equation systems involving these matrices are easily solved using fast Fourier transforms [7]. A block circulant matrix is formed by a circulant matrix containing circulant matrix entries. The block-circulant matrices, as circulant matrices, have some unique properties. They have Schur decomposition [8] that can be related to some algorithm of its multiplication [9].

The use of structured matrices as a neural network weight matrices has been demonstrated in a number of earlier research as one method of reducing memory, particularly for memory models and optimizers. The most well-known example is the sparse Toeplitz matrix-based convolutional neural network (CNN) architecture [10]. Convolutional neural networks are currently the top choice for machine learning tasks involving images due to their effectiveness and prediction accuracy [11–13]. The connections between the neurons in CNNs often encode the structure in an implicit manner. There are other intriguing strategies for enhancing conventional CNNs. When performing operations on images represented in the quaternion domain, for instance, Quaternion CNNs [14–16] outperform conventional real-valued CNNs on a number of benchmark tasks. In addition, Cheng et al. [17] substituted circulant projections for the linear ones in fully connected neural networks, while Liao and Yuan [18] proposed using matrices with a circulant structure in convolutional neural networks. Block Toeplitz matrices used in discrete convolutions were merged with the effective weight representation used in neuromorphic hardware by Appuswamy et al. [19], resulting in a family of naturally hardware efficient convolution kernels. The use of generic matrices with low displacement rank in place of weight matrices in neural networks has also been suggested. Toeplitz-like weight matrices, such as circulant matrices and Toeplitz matrices and their inverses, are used, as in the study by Sindhwani et al. [20]. Additionally, Thomas et al. [21] presented a class of low displacement rank matrices for which they trained the operators and low-rank components of the neural network. The theoretical characteristics of neural networks with low displacement rank weight matrices are the subject of several studies. The universal approximation theorem holds for these networks, as demonstrated, for instance, by Zhao et al. [22]. Using Toeplitz or Hankel weight matrices, Liu et al. [23] provide yet another demonstration that the universal approximation theorem remains true for neural networks.

The transformer is one of the well-known neural network models for machine translation that was first presented by Vaswani et al. [24]. This model has been developed up to this point for a variety of uses, such as text summarization [25], video text and images [26], chat bots [27], and speech recognition [28]. One of the improvements is the swapping out of the transformer weight matrices with a structured matrices. Li et al. [29] proposed an efficient acceleration framework, Ftrans, for transformer-based large-scale language representations. Their framework includes an improved block-circulant matrix (BCM)-based weight representation, which allows for model compression on large-scale language representations at the algorithm level with little accuracy. The results of their experiments show that their model significantly reduces the size of NLP models by up to 16 times. Their FPGA design improves performance and energy efficiency by 27.07 and 81 times, respectively, when compared to the CPU and 8.80 times when compared to GPU degradation, with an acceleration design at the architecture level. Moreover, Liao et al. [30] also applied block-circulant matrices for DNNs (deep neural networks), which enabled the network to achieve up to 3.5 TOPS computation performance and 3.69 TOPS/W energy efficiency while saving 108× and 116× memory with negligible accuracy degradation.

Structured weight matrix multiplication often entails the use of an algorithm. It is typical to utilize the FFT algorithm when dealing with a structured matrix that is a circulant matrix. In Multi30k Task 1 German to English with 100x compression, Reid [31] demonstrated that the use of a block-circulant matrix in the feed forward transformer layer in conjunction with the FFT algorithm is able to enhance the performance of transformers. The DCT-DST algorithm, which may be used in place of the FFT approach in circulant matrix multiplication with a vector, has been introduced by Liu et al. [9]. In previous studies, the DCT-DST was generally used for image processing and video/image coding [32–34].

In neural networks, particularly transformer models, the DCT-DST algorithm has not been used for weight matrix-vector multiplication. This study investigates the application of the real block-circulant matrix-DCT-DST method in layers of transformer. In summary, the main contribution of this study is 2-fold. First, we explored the eigenstructures of the real block-circulant matrices. They are then used to verify the Schur decomposition that applied in the DCT-DST algorithm. Second, we formulate the orthogonal matrices which are used to decompose the real block-circulant matrices. The multiplication of these orthogonal matrices will then be used in the DCT-DST algorithm. In particular, when compared to the original transformer approach, using the dense matrices on the multihead attention transformer and the real block-circulant matrices with DCT-DST on the feed forward layer takes less number of model parameters.

After this introduction section, we organize the remainder of this study as follows: We outline the fundamental theory related to the real block-circulant matrices and DCT-DST matrices in Section 2. Using these theories, we explored the eigenstructures of the real block-circulant matrices and formulas of the Kronecker product for orthogonal matrices in Section 3. In the same section, we formulate the Schur form for the real block-circulant matrices. In Section 4, we explain the experiment of the real block-circulant transformer in conjunction with the DCT-DST algorithm.

Definition 2.1. A n × n circulant matrix is formed by cyclically permuting its entries of the n-vector c₀, c₁, .., c_n−1, and is of the form [c0c1⋯cn-1cn-1c0⋯cn-2⋮⋮⋱⋮c1c2⋯c0]. The set of all such matrices with real entries of order n is denoted by B_n, whereas a nm × nm block-circulant matrix is generated from the ordered set C₁, C₂, …, C_n, and is of the form [C1C2⋯CnCnC1⋯Cn-1⋮⋮⋱⋮C2C3⋯C1] with C_i an m × m circulant matrix for each i = 1, 2, …, n. If C is a real block-circulant matrix, the set of all such matrices of order nm × nm is denoted by BC_nm.

Definition 2.2. The discrete Fourier transform (DFT) matrix F = F_n is defined by

with ω=e2πni,i=-1.

Any circulant matrix A has a Schur canonical form A = F^*ΛF with F^* is a conjugate transpose of F and Λ is a diagonal matrix, holding the eigenvalues of A. The eigenvalues of a real circulant matrix C can be arranged in the following order

Partitioning F*=[f0,,⋯,fn-1] with f_k = t_k + js_k we have fn-k=f¯k, where f¯k is a conjugate of f_k. For any eigenvalue λ_k, Cf_k = λ_kf_k means C(fk+f¯k)=λkfk+λkf¯k and C(fk-f¯k)=λkfk-λkf¯k. Like circulant matrices over complex numbers, the real circulant matrix C also posses a real Schur canonical form Ω=UnTCUn where U_n is an orthogonal matrix

where t_k and s_k are real and imaginary parts of f_k, k = 0, .., h. The matrix Ω is real and has the form

and

where α_k and β_k are, respectively, a real and imaginary part of eigenvalues of C, k = 0, …, h.

In working with real block-circulant matrices, we will involve a fundamental operation, namely, Kronecker product. This operation will be applied to diagonalize and represent matrix U_bc and the DCT-DST algorithm for a real block-circulant matrix. The diagonalization of the real block-circulant matrices, their eigenvalues, and vectors will be discussed in the next three theorems.

Theorem 2.3. Olson et al. [8] Let C ∈ BC_nm and generated by C₁, C₂, ⋯ , C_n ∈ B_m. If F_n is a Fourier matrix of dimension n × n and I_m is an identity matrix of order m, then

is a diagonal block matrix of dimension nm x nm with

Fn* is the conjugate transpose of F_n, ω_n is the nth primitive root of unity, and ρ(t,τ)=∑k=1n t(k-1)⊗τ with t and τ are any square matrices.

Theorem 2.4. Olson et al. [8] Let C ∈ BC_nm has generating elements C₁, C₂, ⋯ , C_n ∈ B_m. If ci(1),ci(2),⋯,ci(m) are generating elements of C_i, then

is a diagonal matrix of dimension nm x nm with

Theorem 2.5. Olson et al. [8] Let C ∈ BC_nm, ui(p) denotes the eigenvectors of Λ_i and e_i be the i-th column of the DFT matrix. The eigenvectors of C are

with i = 1, ⋯ , n and p = 1, ⋯ , m.

The eigenvalues and eigenvectors from the preceding theorems will be used later to construct a real Schur canonical form of a real block-circulant matrix and will be used in conjunction with DCT-DST matrices to define multiplication of a real block-circulant matrix with any vector. The discrete trigonometric transform family consists of eight DCT and eight DST versions. Two versions of them are used in this study.

Definition 2.6. The DCT-I and DCT-V matrices are defined as follows:

with τl(l=j,k)={12,if l=0 or l=n1,if l otherwise

ιk={12,if k =n−11,if k otherwise

Definition 2.7. The DST-I and DST-V matrices are defined as follows:

Note that all those transformation matrices are orthogonal. In the following theorem, we will see that the matrix U_n as defined in Equation (3) can be partitioned into a matrix that is generated by the DCT and DST matrices.

Theorem 2.8. Liu et al. [9] Let U_n be the matrix stated in Equation (3). Then, U_n can be partitioned into the following form:

Define

with σ1=2n,σ2=12,ph+1=(12,1,⋯,1),qh+1=(12,1,⋯,1,12)T,

vh+1=(12,-1,⋯,(-1)h-1,(-1)h2)T,

and

where Pa,b(xj)j=0n-1=(xj)j=ab-1,b≥a. Then, the multiplication of Q_n and U_n will be

By using this rule, the multiplication of the circulant matrix with any vector only involves (h+1)-vectors of 1 DCT-I and (h−1)-vectors of 1 DST-I if n = 2h, and (h+1)-vectors of 1 DCT-V and h-vectors of 1 DST-V if n = 2h+1 [9].

In this section, we will define the matrix-vector multiplication algorithm for the real block-circulant matrices. For this reason, the DCT-DST algorithm will be adapted from Liu et al. [9] by first defining the orthogonal matrices U_bc, Q_bc, multiplication Q_bcU_bc, and the real Schur form Ω_bc. In defining those orthogonal matrices, we leverage a Kronecker product operation as introduced in Olson et al. [8]. In the following theorem, we will see what U_bc, Q_bc, and Q_bcU_bc look like.

Theorem 3.1. Let C be a real block-circulant matrix of dimension nm × nm, U_n, U_m, and Q_n are orthogonal matrices as denoted in Equations (12) and (13). The matrices U_bc and Q_bc that assosiated with C can be defined as

and

The multipication between Q_bc and U_bc will have the form

The last theorem shows that the multiplication of Q_bcU_bc can be calculated by applying the multiplication of Q_nU_n with an orthogonal matrix U at dimension m. This multiplication gives a fast way to solve the multiplication between a real block-circulant matrix with any vector by using 1 DCT-I for (h+1)-vector and 1 DST-I for (h−1)-vector, if n = 2h and 1 DCT-V for (h + 1)-vector and 1 DST-V for h-vector if n = 2h + 1.

Furthermore, the two theorems below give the structure of the eigenvalues and the real Schur form of block circulant matrices. The eigenvalue structure of the real block-circulant matrices is fundamental like those of a circulant matrices. The knowledge of it is needed to recognize the real Schur form of the real block circulant matrices. The following theorems describe how their structures are.

Theorem 3.3. Let C ∈ BC_nm and λi(p) denotes the pth eigenvalue on the ith block of matrix C, i = 1, ..., n and p = 1, ..., m. If n = 2h, the eigen structure of C is

with λ1(p)≠λh+1(p) and λn+2-s(m+2-r)=λs(r)¯, and for n = 2h+1 we have

with λ1(p)≠λj(p) for j ≠ 1 and λn+2-s(m+2-r)=λs(r)¯.

Proof. The eigenvalue of C can be written as

It is clear by tedious straightforward calculation that λ1(p)≠λh+1(p). We will show that λn+2-s(m+2-r)=λs(r)¯,p=1,⋯,m.

Theorem 3.3. Let C = circ(C₁, C₂, ⋯ , C_n) be a real block-circulant matrix of dimension nm × nm with Ck ∈ ℝm × m and U_bc as defined in Equation (14). Define

with σn1=circ(1,0,0,...,0,0),σn2=circ(0,1,0,...,0,0),...,σnn=circ(0,0,0,...,0,1) and σnn+1=σn1=circ(1,0,0,...,0,0). Let t_k = a_k + jb_k and λ_k = α_k + j β_k are the eigenvalues of σnk and C_k, respectively. Then, UbcTCUbc=Ωbc is real and for n = 2h it has the form

and for n = 2h + 1 it will be

with

and

We will use the above theorems to define a real block-circulant matrix multiplication algorithm with any vector below.

Algorithm 3.4 The Multiplication of Cx

The following is an example of implementing the above algorithm. Let C be a real block-circulant matrix with n = 4 and m = 3,

The eigenvalues of C are as follows:

λ1(1)=2 λ2(1)=3-i

λ3(1)=0 λ4(1)=3+i

λ1(2)=2+123i λ2(2)=3+0.732i

λ3(2)=123i λ4(2)=3+2.732i

λ1(3)=2+123i λ2(3)=3-2.732i

λ3(3)=-123i λ4(3)=3-0.732i

and the eigenvectors are

U_bc matrix will be in the form

By following the above DCT-DST algorithm, if x = [1 2 3 4 5 6 7 8 9 10 11 12]^T, then Cx=QbsTy4=[0 -1 4 6 5 10 24 23 28 18 17 22]T.

The use of the real block-circulant matrices as a transformer weight matrix combined with the DCT-DST algorithm for multiplication with any vector provides advantages in saving model memory and increasing the BLEU score. In general, based on this study, it was found that the real block-circulant matrix of dimension 128 provides relatively better performance compared to others. However, it needs to be studied further, whether a larger weight matrix size can provide better performance or not.

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

EA: Data curation, Formal analysis, Investigation, Software, Visualization, Writing—original draft, Writing—review & editing. IM-A: Conceptualization, Funding acquisition, Methodology, Resources, Supervision, Validation, Writing—review & editing. AP: Data curation, Resources, Software, Supervision, Validation, Writing—review & editing.