We focus on an existing algorithm called Latent Dirichlet Allocation to train a topic model for a set of documents. Intuitively, a topic model categorizes documents into different topics, where each document is assigned a combination of one or more topics. Furthermore, this gives insights into what words are often associated with these topics. Latent Dirichlet Allocation (LDA) is one of many topic modelling techniques. Among the most common topic modelling techniques, LDA is the most consistent performer over several comparison metrics, making it the most suitable algorithm for most applications (3). In particular, we consider LDA and use a technique called Gibbs sampling to train the model. Gibbs sampling is an iterative method to estimate latent distributions of a dataset based on observations from that dataset.

This means that we iterate over all the words in all the documents and observe what topic it most likely belongs to. With this topic, we then update the parameters of the topic model. This is done until the parameters converge to a stable representation of the topic model. There are also other methods to train latent parameters, but Gibbs sampling was chosen because it often yields relatively simple algorithms for approximate inference in high-dimensional models such as LDA (4, Figure 8).

Some research has already been done on privacy-preserving Latent Dirichlet Allocation. We can distinguish two lines of research: work that enables privacy-preserving LDA on centralized textual data, such that the final model does not leak information about the inputs (5), and work that enables LDA on distributed textual data, such that the information sent throughout the protocol does not leak information about the inputs (6–8).

Our work falls into the latter category and therefore distinguishes itself from the work in the former category by enabling LDA on decentralized data instead of centralized data. We present several new secure protocols to perform each step of the LDA algorithm in a privacy-preserving way. We now provide more explanation of the other works in the latter category. A comparison between our work and related work can be found in Table 1.

The first work on privacy-preserving LDA on distributed data was published in 2010 by Yang and Nakagawa (8). Similar to us, they use homomorphic encryption. They use a custom protocol to draw the topics, which reveals the distributions to all parties. Additionally, they use a slightly altered version of the LDA algorithm, as do we. Whereas they argue the validity of their alteration with a notion of convergence based on the number of changes the algorithm makes, we use a more robust analysis using the perplexity score, showing that our alteration retains the quality and convergence rate of regular LDA.

Wang, Tong and Shi (7) propose a privacy-preserving LDA solution using federated learning and differential privacy. Their solution makes it possible to do local sampling, as the intermediate values are perturbed using differential privacy techniques. As their experiments show, this comes at a quality cost, as the perplexity score is higher for their solution than for regular LDA. Instead, we use homomorphic encryption to keep all information hidden, including intermediate values.

Colin and Dupuy (6) propose a solution to decentralized LDA with varying network topologies. They claim that their solution attains privacy of the textual documents, but no privacy arguments are given. In each iteration, two nodes, each holding a number of documents, exchange (and average) their local statistics. This is similar to sharing the matrix nm(k), which we avoid in our solution for privacy reasons.

We present a novel solution for decentralized topic modelling in a privacy-preserving manner using latent Dirichlet allocation. This is the first solution that does not leak anything about the content of the documents while at the same time maintaining the accuracy of non-private versions of LDA. This way, we bridge the gap between accuracy and security in distributed LDA training by presenting a solution that is both highly accurate as well as secure. Furthermore, we present two generic, cryptographic building blocks of independent interest: –

Securely drawing a random number from a finite set without revealing the drawing probabilities, as described in Sections 3.4, 3.5.

In this work, we consider the scenario where the documents are not stored in a single database, but are distributed among multiple parties that want to train a joint topic model, but do not wish to simply share these documents with each other. Concretely, our goal is to mimic the existing LDA algorithm in a privacy-preserving manner while maintaining the same accuracy as the non-private version of the algorithm.

Suppose we have M documents, document m, 1≤m≤M, containing Nm words. We consider the setting where we have multiple parties, each having one or more (sensitive) documents. Let K be the number of topics, and V the number of terms¹ in our vocabulary. Let α=(α1,…,αK) be the Dirichlet hyperparameters for the topics in the topics-document distribution, and β=(β1,…,βV) the Dirichlet hyperparameters for the terms in the terms-topic distribution. All these parameters are public.

During the distributed algorithm, we need to manage the secret matrix elements nm(k), representing the number of words in document m that have topic k, and nk(t), representing the number of words with term t that have topic k. Note that {nm(k)}m∈{1…M},k∈{1…K} is a matrix, which will be referred to as the document-topic matrix. Furthermore, {nk(t)}k∈{1…K},t∈{1…V} will be referred to as the topic-term matrix. The document-topic matrix can be split into M vectors, such that each party can manage and store only the vectors corresponding to its own documents. For the second matrix we need a different solution to avoid sharing sensitive data, see Section 3.

The purpose of the algorithm is to train the latent variable zm,n, denoting the topic of the nth word of document m. In each iteration, for each document, and for each word within that document, a new topic is sampled for that word from a dynamic multinomial distribution. Given the word with index i=(m,n) and term t, this distribution is proportional to:(1)Pr(zi=k)∝nk,¬i(t)+βt∑τ=1Vnk,¬i(τ)+βτ⋅nm,¬i(k)+αk∑κ=1Knm,¬i(κ)+ακ,where nk,¬i(t) indicates the count nk(t), excluding the current word with index i, and similarly nm,¬i(k) (4). The first ratio can be roughly interpreted as the empirical probability that a word (not the current word) with topic k has term t. The second ratio can be roughly interpreted as the empirical weight of topic k in document m. The hyperparameters α and β are often called pseudo-counts (from prior belief) and contribute too.

We denote the encryption of a message or plaintext m by [m]. We use the Paillier encryption scheme (9), which gives us the operations ⊕ and ⊗ such that:[x]⊕[y]=[x+y] and c⊗[x]=[c⋅x],for any public constant c, and secret messages x and y. That is, given encryptions [x] and [y] of x and y, we can obtain an encryption [x+y] of the sum x+y without decrypting the ciphertexts. The resulting ciphertext can be decrypted to yield the result, or be input for further encrypted operations.

Secret Sharing has similar properties but works in a fundamentally different, key-less way. Suppose we have a secret s and wish to use this in a computation with a set of parties P1,…,Pn. The party holding the secret s can split this secret up into a number of shares s1,…,sn and send each si to party Pi. We denote the sharing of s by ⟨s⟩=s1,…,sn.

Each party Pi can then compute operations for a public constant c and secret sharings ⟨x⟩=x1,…,xn and ⟨y⟩=y1,…,yn for secrets x and y such that:⟨x⟩⊞⟨y⟩=⟨x+y⟩,\; c⋅⟨x⟩=⟨c⋅x⟩and⟨x⟩⊠⟨y⟩=⟨x⋅y⟩.In this work, we use the Shamir secret sharing scheme (12), which is a linear secret sharing scheme. This means we can compute the linear additions and multiplications with a public constant without interaction between the parties. Multiplication of two secrets is additionally possible with communication between the parties.

For both techniques, we assume the semi-honest setting, where each entity tries to learn as much information about the other entities’ data as possible, but does follow the steps of the protocol. For most use cases, this security model will suffice, as it is likely that honest participation will be agreed upon within a contractual agreement between the entities. Furthermore, since LDA already has some inherent privacy properties (5), it is unlikely that during execution a dishonest entity can retrieve a significant amount of information about other entities’ documents. However, we acknowledge this security model might not be appropriate for large-scale deployments with many potentially dishonest entities.

As highlighted in Section 1.4, LDA essentially manages and updates two matrices: a document-topic matrix and a topic-term matrix. The document-topic matrix keeps track of the topic distribution of each document and consists of elements nm(k), representing the portion of document m belonging to topic k. The topic-term matrix keeps track of the topic distribution of each term in the vocabulary and consists of elements nk(t), representing the portion of term t belonging to topic k over all documents.

However, these matrices are precisely the sensitive information that completely leaks the content of the documents of a party when simply giving it away. Therefore, we need to find a secure way to store these matrices without (significantly) decreasing the accuracy of the algorithm.

A crucial observation is that during the LDA algorithm, the matrix elements nm(k) of the document-topic matrix are only needed by the party actually holding document m. Therefore, it is not needed to maintain a complete, joint matrix of all the documents, but it suffices to let each party locally maintain a part of that matrix corresponding to only its own documents.

On the other hand, the topic-term matrix depends on the distribution over all the documents and should therefore be available to all the parties in an oblivious way. Maintaining this matrix comes down to adding to, and subtracting from, the elements in the matrix, which suggests the use of additively homomorphic encryption for this. To avoid individual parties from decrypting and learning the entries, we furthermore need threshold decryption (10). This ensures that a decryption can only be done if all the parties participate. Note that if we were to do this with secret sharing, each party would need to keep track of the entire matrix, which would introduce a lot of computational overhead.

A formal description of our Secure LDA solution for securely computing the topic-term matrix nk(t) and the document-topic matrix nm(k) can be found in Algorithm 1. In Figure 1, we present an intuitive overview of how our algorithm works. Roughly speaking, our Secure LDA solution consists of three phases: initialisation (blue), sampling (green) and updating (orange). Finally, the results are decrypted in a joint decryption phase (red).

In the initialisation phase, the goal is to initialise the two matrices with a random distribution that will be refined. To this end, all the parties sample random topics for each word in each document, and use these to fill in an initial (local) view on the document-topic matrix and the topic-term matrix. Next, the parties need to build a global view of the complete topic-term matrix. To achieve this, the parties encrypt all the elements in their local topic-term matrix and combine these by sending the encrypted elements to each other and aggregate them into a global matrix by adding the (encrypted) matrices of all the parties element-wise.

After the initialisation, for a fixed number of iterations, the parties perform a sampling and an updating phase. During the sampling phase, the parties use the (secret) matrices as they are at the start of the iteration, to compute, for each word in each document, a probability distribution over the topics. The secure sampling procedure ensures that the distributions remain hidden from the parties and is outlined in Sections 3.4, 3.5. For each party, the secure sampling procedure yields a new topic for each word in each document. A party uses this information to update her local version of the encrypted topic-term matrix and local document-topic matrix.

The distribution that is drawn from is proportional to Equation 1. Note that these distributions are in an encrypted form and the actual probabilities can thus not be seen by the parties. First, we compute the encrypted weights for all the topics using the procedure presented in Section 3.5. After that, we can perform a secure draw from the encrypted weights using our novel algorithm to draw from a secret probability distribution as presented in Section 3.4. This way, the parties obtain for each word in each document a newly sampled topic. During this sampling, the parties locally keep track of the matrix updates, which means that they decrease their local counters corresponding to the matrix elements of the old word topic by one, and increase the counters for the new topic by one.

The second part of each iteration then consists of each party updating its local document-topic matrix and the parties together updating the global topic-term matrix using the locally tracked changes. To this end, each party encrypts their local changes to the topic-term matrix and sends this to all the other parties. Then the parties can simply add these encrypted counters to their encrypted topic-term matrix to get the new, consistent, topic-term matrix. The document-topic matrix can be updated locally by each party without any communication.

We observe that the LDA algorithm requires linear computations, except for the computation of the probability Pr(zi=k) and the secure draw that uses these probabilities in the sampling step. Therefore, we perform most of the operations for tracking the topic-term matrix using AHE, and introduce a novel mechanism to switch between AHE and secret sharing in Sections 3.6, 4.2 to obtain the best performance. Concretely, we use AHE for the linear operations and only switch to (Shamir) secret sharings for securely drawing the new topics.

Typically, convergence of an LDA algorithm is checked by monitoring the changes in the model parameters, or monitoring how well the model fits a separate set of documents. In the encrypted domain, this can be quite costly to check after each iteration. Therefore, we simply iterate a sufficiently large, fixed number of times.

An important building block of secure LDA is a method of drawing a new topic k~∈{1,…,K}, given secret weights wk∈N, such thatPr(k~=k)=wk∑iwi,1≤k≤K.The new, randomly chosen topic will be revealed to party p, the holder of the current document. The intuition behind our solution is to compute cummulative weights Sk, k∈{1,…,K} such that Sk=∑i=1kwi. For notational convenience, we define an “extra” weight S0=0. Next, the parties sample a random value r in the range {0,SK−1} and find between which two cumulative weights this value r lies, which then corresponds to the sampled topic. Since r is sampled uniformly at random in the total range, the probability of r precisely ending up between cumulative weights Sk−1 and Sk is exactly (Sk−Sk−1)/SK=wk/∑iwi. This can be implemented with only log2⁡K secure comparisons between r and thresholds t=Sk (with varying k) by traversing a binary tree from the root to the leaf representing the new topic. Note that our solution assumes that the weights are integers. In Section 3.5, we explain how we securely transform fractional weights into integer weights.

Formally, the parties do the following for every word w in each document: 1.

The parties generate a secret random number ⟨r⟩, r∈{0,…,SK−1}: (a)

They generate a secret random number ⟨R⟩, R∈{0,…2ℓ−1} for sufficiently large ℓ.

The first threshold choice will be t=KK÷2, each iteration adapting the threshold following the binary search principle. This means that if r<t, we go to the left and otherwise to the right. As the numbers ⟨wi⟩ are secret-shared, party p needs to generate a secret-shared binary indicator vector ⟨δ1⟩…⟨δK⟩, such that the threshold can be computed by ⟨t⟩=∑i⟨δi⟩⋅⟨wi⟩. Party p is the only party that can determine the binary indicator vector, because it is the only party that is allowed to learn k~.

A key element of Algorithm 1 is the secure, random sampling of new topics for all of the words. As explained in Section 3.3, this is done in two steps: computing the integer weights and performing the secure draw. This subsection will introduce the steps required to compute the integer weights for Equation 1 given the matrices.

We assume we are given matrices [nk,¬i(t)]=[nk(t)+Δk(t)] and nm,¬i(k), the first one encrypted and the second one privately known to party p, the holder of document m. We omit the index ¬i for convenience.

To sample a new topic, first the weights have to be computed that determine the probabilities according to Equation 1, which we denote as Pr(zi)∝[wkn]/[wkd] for simplicity. The weights consist of numerators[wkn]=[(nk(t^)+βt^)⋅(nm(k)+αk)],and denominators[wkd]=[∑τ=1V(nk(τ)+βτ)⋅(∑κ=1Knm(κ)+ακ)].The encrypted numerators and denominators can easily be computed by party p due to the additively homomorphic property of our encryption scheme.

The only problem is that the hyperparameters α and β are not integers, while the secret sharing scheme requires the plaintexts to be integers. For this work, we chose symmetric priors, meaning αi=α, 1≤i≤K, and βi=β, 1≤i≤V (see Section 5.3). We then approximate the fractions α=αnαd and β=βnβd, where αn, αd, βn and βd are integers. Then the numerators wkn and denominators wkd are converted to integers w~kn and w~kd by multiplying both with αdβd.

Eventually, we want to obtain integer weights for the secure draw (see Section 3.4). To avoid costly secure integer divisions w~knw~kd, we multiply these fractions with W=∏kw~kd to obtain w~k=w~kn⋅∏κ≠kw~κd as follows: 1.

Party p computes the encryptions [w~kn]=[wkn⋅αdβd]=([nk(t^)]βd⋅[βn])αd⋅nm(k)+αn and [w~kd]=[wkd⋅αdβd]=([βn]⋅[V]⋅∏τ=1V[nkτ]βd])αn⋅V+αd⋅∑κnm(κ), which are converted to secret sharings (see Section 3.6) for efficiency reasons.

During the execution of Algorithm 1, we need to transform the encrypted weights [w] to Shamir secret sharings ⟨w⟩ to randomly draw new topics more efficiently. Suppose we have precomputed pairs ([R],⟨r⟩), such that R contains σ more bits than w, and r=RmodN, where N, N>w, is the modulus of the Shamir secret sharing scheme. Then a conversion from [w] to ⟨w⟩ is relatively straightforward: 1.

Compute [w+R]=[w]⋅[R], and (jointly) decrypt it.

We combine the sampling of all new topics of one party [step 2(a)iiB], such that we can parallelise each step of the binary search (see Section 3.4), and drastically reduce the number of communication rounds. This means that the probabilities from Equation 1 are not recomputed after each single topic sampling, but only when during one iteration all words of all documents of a certain party have been assigned a new topic. This version, which we will refer to as batched LDA, enables us to execute all secure comparisons at the same level of the binary tree (see Section 3.4) in parallel, and significantly reduce the total number of communication rounds. The disadvantage is that the drawing probabilities are not constantly adjusted, which might lead to accuracy loss, see Section 5.4.1.

We have multiple conversions that can be efficiently combined into one protocol. Suppose we have weights w1,…,wω, and corresponding pairs ([Ri],⟨ri⟩), 1≤i≤ω, such that ω⋅(σ+⌈log2⁡w⌉+⌈log2⁡n⌉+1)<⌈log2⁡N⌉, where ⌈log2⁡w⌉ is an upper bound on the bit size of the weights, ⌈log2⁡n⌉ is the bit size of number of parties n and ⌈log2⁡N⌉ the bit size of the encryption modulus. Then the ω conversions can be combined as follows. 1.

[C]=[wω+Rω]=[wω]⋅[Rω]

Because topic sampling is performed in a secure, but joint way, the parties learn the total number of words in all documents of a single party. However, nobody learns the sampling probabilities, and only the document holder learns the new topics (of the words in his documents). Our solution is secure in the semi-honest model, i.e., parties are expected to exactly follow the protocol steps, but are allowed to compute with any data that is received during execution in an attempt to gain additional insights in other parties’ data.

As we use standard building blocks, such as secure comparison and random number generation, of the MPyC platform, which is known to be secure in the semi-honest model, our computations with secret-sharings are secure too. Similarly, Paillier is known to be semantically secure, and since we use threshold decryption, encrypted information will never fall in strange hands.

Therefore, we only need to investigate the conversions from encryptions to secret-sharings, as described in Section 3.6. Because the numbers R contain σ more bits than the weights, where σ is the statistical security parameter, we know that the sum w+R is statistically indistinguishable from a large random number, and can be safely revealed. Furthermore, as each party i generates its own Ri and ri, the sums ∑iRi and ∑iri can be considered as secret random numbers.

We have implemented our secure LDA approach in Python 3.8. For the homomorphic encryption functionalities, we have used the Paillier implementation available in the TNO MPC Lab (15). This implementation is based on the distributed Paillier solution presented in (10). For the functionalities based on secret sharing, we have used the MPyC framework (16). This framework implements a number of functionalities based on Shamir secret sharing. We performed all of our experiments with three parties, but stress that our implementation also works for more parties.

For our experiments, we used the Amazon reviews dataset presented by Ni, Li and McAuley (17). In total, this dataset consists of over 200 million reviews. However, we only used the first 150 entries. Furthermore, we split these 150 entries into three separate datasets of 50 documents for the three different parties. In total, this results in a vocabulary length of V=1492 terms and a total number of 2,965 words in the distributed corpus. For the experiments, we used 5, 10, 20, 30, 40 and 50 documents per party. As the number of words is not the same for every document, we compared the number of words over all documents for the actual experiments, which is 16, 406, 873, 1549, 2,197 and 2,965 respectively. Furthermore, we chose the symmetric priors α=β=1K. This corresponds to the default parameter choices in the scikit-learn implementation of LDA.

All experiments have been run on a single server running an Intel Broadwell CPU at 2.1 GHz with 4 cores and 32 GB RAM. The parties communicated via (local) HTTPS connections.

We evaluate the performance of our solution in terms of accuracy and runtime.

As explained in Section 1.2, there are three works that also consider decentralized, privacy-preserving LDA. In Table 1, we highlight the most important differences between our works and these related works. Due to the lack of comparable runtime measurements in these works it is hard to compare our work in that regard. Instead, we turn to a conceptual comparison.

In terms of accuracy, it is unclear how the altered algorithm of (8) impacts the accuracy exactly since they do not provide metrics such as perplexity. We do know that their convergence notion influences the resulting model accuracy to some extend. Furthermore, they leak the probability distributions for the topics in every round, which is a privacy risk as this reveals information about other parties’ data. Our solution keeps Pr(zi=k) secret throughout the entire protocol. Furthermore, they do not provide a security argument for their solution, which we do.

Due to the use of differential privacy, (7) is not able to match the accuracy of non-private LDA like we are able to do using MPC. Furthermore, this is a weaker security guarantee and might still leak some (statistical) information about the data of the other parties. This solution is, however, faster than our solution.

Finally, in (6) an approach is used where statistical information about the documents of the parties is shared in every round. This way, they are able to learn models with high accuracy and obtain a high performance at the cost of very low security guarantees as this essentially comes down to sharing your document-topic matrix.

All in all, our solution is very secure and accurate, at the cost of a lower performance. However, our solution scales linearly in both the number of words and the number of topics, which makes it scalable in practice.