Over the last few years, there has been significant progress on the cryptographic protocols and theories related to ciphertext packing, which promises substantial improvements in the application of HE in complex, distributed applications such as Deep Learning for healthcare remote analysis and monitoring. However, adapting existing operations for SIMD over ciphertext packing is non-trivial.

Unfortunately, the application of ciphertext packing in DL is not straightforward. Indeed, the linear algebra operations used in the internal structures highly affect the performance (18). Understanding the impact of the ordering and election of internal operations on global performance is critical for designing efficient algorithms. However, this complex task requires a proper understanding of cryptographic protocols.

Existing works have attempted to automatically transform linear algebra algorithms so they can be applied using SIMD over packed ciphertexts (18–20) (see Section 6). However, these works provide simplistic views of the required algorithms, which limit their reproducibility and implementation in real-world architectures. Moreover, the algorithms are described and tested for isolated computations, far from the complex workflow and interconnections from the internal layers of Deep Learning. Since the input and output encoding of ciphertext packing differs from regular operations, it is necessary to account for the different formats of the inputs and outputs in each layer of the DL architecture and how these affect the overall performance. For example, when using SIMD operations, it is required to transform the output of a Convolutional Layer to be a valid input for subsequent Dense Layer. These transformations have not been described or accounted for the overall overhead in previous works. Also, depending on specific details of the DL layer, e.g., the use of stride or padding, this transformation requires different adaptions (we provide background on CNNs in Section 2.3). Furthermore, in rotations with packed ciphertexts, the last entries overflow to the first positions. Our work shows how some algorithms profit from this overflow to perform computation.

Overall, existing proposals either leave the adaptation to HE on the user (19) or provide algorithms that are not general for inputs of any size (18). Accordingly, we formulate the following research question:

To tackle the question mentioned above, in this paper we describe a set of algorithms to efficiently transform the operations of Convolutional Neural Networks for their use on packed vectors. A key aspect in the design of the algorithms for cardiovascular diseases is that they work on arbitrary-sized input matrices and vectors, and thus they can adapt to medical images of any size. We provide a holistic view of the DL inference process by not only understanding the individual linear operations required by the algorithms but also the collective relationship of the different layers. For the first time, we show that the transformations required to interconnect the layers pose a significant overhead, which should be considered in the design of the DL architecture.

In summary, the main contributions proposed in this research are the following: 1.

We describe algorithms for executing each layer of a Convolutional Neural Network (CNN) using SIMD HE over packed ciphertexts (Section 3). We include algorithms to adapt the input and output encodings to meet the format required on each layer. As a fundamental contribution, we illustrate these algorithms to consider as inputs arbitrary size matrices, allowing their application to different existing architectures. Among these, we provide a streamlined version of the convolution algorithm that significantly improves the efficiency of the previously considered algorithms.

The rest of the paper is organized as follows. First, we provide background information and describe the adversarial model in Section 2. Second, we describe the different algorithms in Section 3. Third, we conduct a formal analysis for the efficiency and performance of the algorithms in Section 4. Fourth, we empirically evaluate a working prototype in well-defined tests in Section 5. Finally, we discuss conclusions and future work in Section 7.

In this work, we consider an Honest-but-Curious adversary (21–23), a passive adversary that complies with the protocol and does not tamper with the data for malicious purposes. However, it tries to learn as much information from data exchanges. Homomorphic Encryption inherently guarantees input privacy (i.e., the privacy of the data sent to the cloud during the inference process is guaranteed). However, HE does not account for the output privacy of the model. Thus, in our work, we assume that the service provider either is proprietary or has access to the model.²

Furthermore, for all of our use cases, we consider the set of parameters established for the Learning with Errors problem following the guidelines described in the Homomorphic Encryption Security Standard (24). Thus, these parameters provide a secure environment for the execution of the algorithms.

Homomorphic Encryption (HE) is a property of an encryption scheme that permits operating with the ciphertext while translating those changes to the underlying plaintext. This work focuses on widely used HE schemes based on the Ring Learning with Errors (RLWE) problem (25).

Concretely, we focus on Levelled Homomorphic Encryption (LHE) Schemes (16). In these schemes, we represent the RLWE polynomial coefficient modulus Q with a Chinese Remainder Theorem (CRT) coefficient moduli chain qi. The representation allows performing a rescaling operation after each multiplication, thus reducing the incurred noise. Also, it reduces the size of the numbers that schemes have to treat. We note that these techniques do not restrict their application to FHE Packed Schemes, but their application to LHE Schemes remains more constraining and thus relevant.

A key point of LHE is the cost of individual operations on the underlying ciphertext representation, as ciphertext are big polynomials (17). Ciphertext Packing arose as a solution by enabling the introduction of more than one plaintext element per ciphertext (17). It permits executing Single-Instructions-Multiple-Data (SIMD) operations (i.e., the execution of an HE operation on a ciphertext propagates to the underlying plaintext vector).

An essential aspect of LHE design is parameter selection. The parameter N, or polynomial degree, affects the various matrix operations presented in this paper. This parameter establishes the degree of the polynomial. Also, packed schemes define the number of plaintext elements n that a ciphertext can accommodate, i.e., the maximum length of a vector that can be encoded (see Figure 1). Schemes such as BFV (26) allow packing as many elements as the length of the polynomial (i.e., n≤N). In the scheme CKKS (27), though, due to the complex number packing, it is only possible to pack half of the vector size (i.e., n≤N/2). This paper treats parameter N independently of the scheme (i.e., n=N). However, all the conclusions are valid to CKKS by substituting N by N/2. In Section 4.1, we analyze the implications of different parameter selections on the efficiency of the algorithms. Next, we describe the basic routines of an HE scheme: ∙

A key generation routine produces a public key pk and its corresponding private key sk, defined by KG(Q,N)→pk,sk, where N and Q are the homomorphic encryption parameters.

Deep Learning is a set of statistical algorithms based on Deep Artificial Neural Networks. These algorithms have shown proficiency when learning from large amounts of data (6). The basic building blocks of DL models are layers (i.e., the different arrangements of neurons in an architecture). Each layer operates with the data differently, depending on its type (e.g., convolution or dot-product). The basic building blocks of layers have been ported to hardware acceleration and multiple libraries (28–30). This work focuses on Convolutional Deep Neural Networks (CNN). These are nowadays the most prominent networks to deal with images (3, 31) (e.g., classification, object detection, or segmentation). Thus, the use of CNN with privacy-preserving techniques is of paramount importance. For further information on CNN, we refer the reader to the book by Goodfellow et al. (32). Next, we overview the basic structures that require adaptation to work with SIMD operations.

Convolutional layers compute the correlation between a multidimensional input matrix and a kernel or filter. When dealing with 2D images, we consider it to be the relation between an input image of dimensions M∈Rh×w×c and a set of kernels or filters F∈Rfx×fy×c′. The convolution iterates through regions of size (fx,fy) and computes the convolution with each filter F (i.e., multiplication and sum). Usually, in combination with convolutional layers, padding, and stride mechanisms are paired. Considering a picture over which we place a filter, the stride defines the axial displacement of the filters (sx,sy). Padding a matrix consists of uniformly adding a value p times at the beginning and end of each dimension. In CNN, padding highlights potential features existing on the borders and corners of the image that otherwise would disappear.

Pooling layers act as aggregation component. They reduce the size of previous layers and help keep the number of parameters low. They perform a similar operation to the convolution, taking (px,py) regions of the input 2D matrix M∈Rh×w and computing an operation over the region. The most common type of pooling layer is the max pooling layer. It involves computing the maximum pixel of the extracted region. Unfortunately, the max function is non-linear. With current LHE schemes, non-linear functions require polynomial approximations (e.g., Taylor or Chebyshev approximations (33)), incurring extra complexity. Thus, it is also possible to use another classical, yet-effective approach to the Average Pooling Layer, which applies the following operation:1n∗∑i=xx+px∑j=yy+pyMi,jFully connected or dense layers link each input element to all output activations. Dense layers usually translate to matrix multiplication between weights matrix (W∈Rh×w) and the input vector (x∈Rw) and adding a bias element (b∈Rh) (i.e., z=Wx+b). In the weights matrix, each entry represents the relation between the input element y and output element x (i.e., each row relates to an input element, and the specific column represents the relationship with the output element).

Activation functions are often non-linear functions that involve introducing non-linear behaviors in the approximated functions. The rationale of these functions is to resemble synaptic signals transmitted by biological neurons (34). Thus, without these, Neural Networks would reduce to complex polynomials. In this work, we do not cover activation function algorithms, as they involve operating on all entries of the ciphertext (i.e., it is straightforward with SIMD). Regarding the existing approximations of non-linearities, we refer the reader to different works (35, 36).

In this paper, we consider a use case where a client-server scenario, where a client needs to outsource the computation of DL inference to a not necessarily trusted server. For that, the client and server rely on public-key FHE. The client performs the key generation, encrypts the data with his public key, and sends the ciphertext to the server. The server receives the public and relinearization keys and the ciphertext. Through privacy-preserving processing, the server can operate on the data and return the result to the client. The client owns a private key which he never released, therefore is the only person able to decrypt the information. Also, we consider that the client performs no computation except light tasks before encrypting or after decryption. For example, the client can perform padding to reduce the load of convolutions since it is a light task. Also, after decrypting the information, the client can retrieve the relevant information entries instead of having the server post-process the result.

We use M∈Rh×w to represent a matrix M of dimensions h×w. Also, we use Mi,j to refer to the entry on row i and column j of the matrix.

In Section 2.2, we described the encoding, encryption, decoding, and decryption routines. For simplicity, in the algorithms, we assume that a vector’s encryption routine also comprises the previous encoding. Similarly, the decryption routine comprises the decoding after decryption. Thus, we denote ct=E(pk,M∈Rh×w), as the encryption of the encoded matrix M, whose layout in the encrypted vector may be defined depending on the algorithm.

Many of the algorithms rely on binary bitmasks to obtain relevant information. These are composed of binary values. In the description of the algorithms, we assume bitmasks are initially filled with 0, and we express a condition to get the positions (indexes) where entries are set to 1. We use the parameter t (bitmask[t]∈Z2N), which defines such indexes. For example, for a bitmask where even indexes are 1, we denote the mask as bitmask[t]←{tmod2=0}.

In HE, the ciphertext representation uses integer ring polynomial representation, unlike Deep Learning floating point representation. This problem has multiple approaches, such as fixed point representations or CKKS encoding (27). The algorithms provided here are represented generically without discussing the specific representation (i.e., the only requirement is the availability of the operations defined in Section 2.2).

In the following sections, we provide the general algorithms that allow the adaptation of common layers in CNNs, i.e., Convolutional and Dense blocks. Then, we provide a discussion on how to use activation functions (which are non-linear) in the context of Packed Homomorphic Encryption. Figure 3 provides a summary overview of the application of the different algorithms in a CNN pipeline.

Computing a classical 2D Convolutional Layer involves a relationship between an input bi-dimensional matrix region and a bi-dimensional filter. As explained in 2.3, the convolution can combine structures such as padding, stride, or pooling layers. Accordingly, the algorithms defined for LHE should also account for the use of these variants.

This section presents a new algorithm for convolution, dubbed the Streamlined Convolution Algorithm. It allows combining convolutional, pooling, and activation layers in subsequent blocks, neglecting the cost of initial representation and executing a single result transformation algorithm (i.e., we can use the output representation of the algorithm arbitrarily). We first describe the convolution algorithm with stride and the result transformation algorithm (Section 3.2.1). Then, we provide its integration with padding (Section 3.2.2). and Average Pooling Layers (Section 3.2.3. For reference, Appendix A details the convolution algorithms that extend and generalizes previous work (18), which we use as a baseline comparison in Section 5.

The Dense or Fully-Connected Layer of a Neural Network performs a weighted connection of all the inputs to all outputs. It is a linear transformation that a matrix-vector multiplication can represent, where a weight matrix W∈Rh×w is multiplied by the vector x∈Rn. The weights matrix contains parameters fine-tuned during the training. The input vector comprises all the outputs from the n neurons in the previous layer.

This section proposes algorithms for efficiently applying SIMD operations over encrypted data on Dense Layers. These algorithms generalize two previously proposed algorithms for DL inference: a Diagonal Matrix-Vector multiplication (19) and a Matrix-Matrix multiplication (18). While these matrix multiplication algorithms report simplistic examples, in our work, we describe a generalization together with all the transformation algorithms required for the internal connections.

Neural Networks have excelled at classification and regression tasks because they can map non-linear distributions. Activation functions are a crucial component of such success, and their integration within FHE schemes is a hot research topic in the literature. Linear approximations of various kinds are among the most successful yet straightforward proposals to introduce activation functions in HE-based DL. Authors have proposed solutions ranging from alternative polynomials (35, 38), Taylor and Chebyshev Polynomials (36) or simple linear regressions (39, 40). The common goal of these works is to obtain a low-degree polynomial to approximate the non-linear behavior as accurately as possible. This paper assumes that the activation functions have somehow been approximated to polynomials using any of the existing proposals. Thus, computing an activation function to Packed Homomorphic Encryption does not require any particular format or construction as it applies the transformation to all the slots of the ciphertext. We can often insert the activation layers with other layers or building blocks of the layers (i.e., algorithm execution and result transformation) without changing the final result. In Section 4.3, we provide insights on where introducing activation functions for the convolutional and dense blocks would be more or less desirable.

Generally, HE operations are performance-wise heavy to execute over ciphertexts. Our analysis focuses on the transformations applied to ciphertexts. Plaintext operations have a negligible impact on the computation; thus, we do not account for them in the analysis. For evaluating these algorithms, we rely on four metrics that define the efficiency of a circuit C:

Multiplication depth (DC) defines the maximum number of consecutive products that an HE ciphertext needs to apply in a given circuit C. The multiplication depth directly impacts the parametrization of HE schemes, specifically in N and Q. In terms of cleartext operations, N defines the polynomial degree. Thus, a bigger N would involve operating over larger degree polynomials (i.e., more coefficients to compute per ciphertext operation). Also, working with bigger Q involves computing more remainders. In Levelled Homomorphic Encryption Schemes, each multiplication usually requires a rescaling operation to reduce the underlying noise. Therefore, we consider the need for one rescaling per DC (i.e., per multiplication). In this case, we need DC different moduli (qi) in a polynomial coefficient modulus Q. In direct relation with Q, N often defines a maximum capacity for a Q (i.e., increasing the DC may not only involve increasing Q but also N). The optimization of these parameters is of paramount importance to obtain better runtimes. For all these reasons, keeping a minimal depth of the circuit is very important for achieving efficiency in the desired computation, which justifies why DC is one of the metrics analyzed for the efficiency of the algorithms.

Operation cost differs across the different available computations in HE. Multiplication is the most costly since it not only requires multiplication but is paired with a relinearization phase (i.e., preventing the polynomial degree from growing) and a rescaling phase (i.e., reducing the noise scale). The next more costly operation is rotation, which involves generating different Galois Keys. In CKKS, if the encoding scale s is chosen the same as the smallest modulus prime qi, we can neglect the noise and depth cost. Element-wise additions are the lowest cost operation and are considered linear in computation and noise growth. For the rest of the analysis, we denote the addition and subtraction complexity (Osum) as the number of sums (and subtractions) required by a circuit. Likewise, we consider the multiplication complexity (Omul) and rotation complexity (Orot) as the number of multiplications and rotations in the circuit.

Memory complexity (Omem) accounts for the number of ciphertexts needed in memory to execute one of the algorithms. Given the large memory size of ciphertexts, minimizing the number of ciphertexts simultaneously residing in the main memory is essential.

Memory constraints (Ocon) determines the constraints that an algorithm imposes on the size of plaintext vectors n it operates with, so these can fit in ciphertext with N slots (i.e., n<N). If the plaintext vectors do not fit in the ciphertext, the circuit would require an extended vector representation (i.e., the plaintext vectors are packed within multiple ciphertexts). In general, for most algorithms, we consider that for an input matrix M∈Rh×w, we can compute the algorithm if h⋅w≤N (i.e., if the full matrix size fits in a ciphertext vector slot). However, some specific algorithm representations of information may define harder or softer limits for the execution. As we detail in the following section, the memory constraints Ocon directly impact the operation cost. In the following sections, we use r to define the number of ciphertext vectors of size N needed to host a plaintext vector of size n.

The effect of the “Memory Constraints” Ocon is important for the Rotation operation. In HE programming, most algebra circuits present memory constraints, given the difficulty of packing all the information within the same ciphertext. In parallelism with classical (non-HE) programming, we could consider when a program does not fit into the available memory and uses swap space. At that point, computation becomes a constraint and is more expensive. In HE circuits with packing, when we need multiple ciphertexts to represent the plaintext data, rotations have a worse impact on the efficiency of algorithms. Indeed, many algorithms rely on rotations to benefit from SIMD operations (e.g., to transform output layouts). Previous works assume rotations as “cost-free” operations and thus use them arbitrarily (18). We observe, however, that when an algorithm is generalized to work on arbitrary-size plaintext inputs (often large scale), the assumption does not hold anymore. Suppose the plaintext vector entries n extend over the available slots N. In that case, multiple ciphertexts are required, and the plaintext vectors and rotation cost are no longer neglectable since at least r=⌈n/N⌉ ciphertext vectors are required.

To demonstrate this performance decrease, we depict in Figure 4 the rotation procedure for n>N, i.e., when more than one ciphertext is needed. In the example, we represent one input vector with two ciphertexts. Considering a 2-left rotation (≪2), we observe that individual rotations of the vectors are partial. Thus, this involves further modifications, such as an additional multiplication of the vectors by a mask, which increases DC. Table 1 shows the overall complexity of this process. We provide the details in Algorithm 10.

In this section, we provide a detailed analysis of the algorithms presented in Section 3 concerning the efficiency metrics proposed. Table 2 presents the performance formal complexity extracted from the different algorithms. Next, we give key insights extracted from the analysis and discuss future directions and best practices to apply the algorithms for Deep Learning Inference.