The first phase of the SP is used to compute the overlap between each column and its respective input, as shown in Algorithm 1. In Algorithm 1, the SP is represented by the object sp. The method col.connected_synapses() returns an instance to each synapse on col’s proximal segment that is connected, i.e., synapses having permanence values greater than the permanence connected threshold, psyn_th. The method syn.active() returns “1” if syn’s input is active and “0” otherwise. pseg_th¹ is a parameter that determines the activation threshold of a proximal segment, such that there must be at least pseg_th active connected proximal synapses on a given proximal segment for it to become active. The parameter col.boost is the boost value for col, which is initialized to “1”.

The second phase of the SP is used to compute the set of active columns after they have been inhibited, as shown in Algorithm 2. In Algorithm 2, kmax_overlap(C, k) is a function that returns the k-th largest overlap of the columns in C. The method sp.neighbors(col) returns the columns that are within col’s neighborhood, including col, where the size of the neighborhood is determined by the inhibition radius. The parameter k is the desired column activity level. In line 2 in Algorithm 2, the k-th largest overlap value out of col’s neighborhood is being computed. A column is then said to be active if its overlap value is greater than zero and the computed minimum overlap, mo.

The third phase of the SP is used to conduct the learning operations, as shown in Algorithm 3. This code contains three parts – permanence adaptation, boosting operations, and the inhibition radius update. In Algorithm 3, syn.p refers to the permanence value of syn. The functions min and max return the minimum and maximum values of their arguments, respectively, and are used to keep the permanence values bounded in the closed interval [0, 1]. The constants syn.psyn_inc and syn.psyn_dec are the proximal synapse permanence increment and decrement amounts, respectively.

The function max_adc(C) returns the maximum active duty cycle of the columns in C, where the active duty cycle is a moving average denoting the frequency of column activation. Similarly, the overlap duty cycle is a moving average denoting the frequency of the column’s overlap value being at least equal to the proximal segment activation threshold. The functions col.update_active_duty_cycle() and col.update_overlap_duty_ cycle() are used to update the active and overlap duty cycles, respectively, by computing the new moving averages. The parameters col.odc, col.adc, and col.mdc refer to col’s overlap duty cycle, active duty cycle, and minimum duty cycle, respectively. Those duty cycles are used to ensure that columns have a certain degree of activation.

The method col.update_boost() is used to update the boost for column, col, as shown in Algorithm 4, where maxb refers to the maximum boost value. It is important to note that the whitepaper did not explicitly define how the boost should be computed. This boost function was obtained from the source code of Numenta’s implementation of HTM, Numenta platform for intelligent computing (NuPIC) (Numenta, 2016).

The method sp.update_inhibition_radius() is used to update the inhibition radius. The inhibition radius is set to the average receptive field size, which is average distance between all connected synapses and their respective columns in the input and the SP.

Competitive learning networks typically have each node fully connected to each attribute in the input. The SP follows a different line of logic, posing a new problem concerning the visibility of an input. As previously explained, the subset of input attributes that connect to a particular column are determined randomly. Let c⇀∈{0,1,…,nc−1}1×nc be defined as the set of all columns indices, such that c⇀i is the column’s index at i. Let U∈{0,1}ns×na be defined as the set of attributes for all inputs, such that U_s,a is the attribute for input s at index a. Let Λ∈{0,1,…,na−1}nc×nps be the source column indices for each proximal synapse on each column, such that Λi,k is the source column’s index of c⇀i’s proximal synapse at index k. In other words, each Λi,k refers to a specific index in U_s.

Let ac⇀∈{0,1}1×na be defined as vector of events, such that ac⇀a≡∃!a∈Λi∀a. Therefore, ac⇀a is the event of attribute a connecting to column a, where ∃! is defined to be the uniqueness quantification. The probability of a single attribute, U_s,a, connecting to a column is the ratio of n_ps and n_a, as shown in equation (1).

It is also desired to know the average number of columns an attribute will connect with. To calculate this, let λ⇀∈{0,1,…,nc+nps}1×na be defined as a vector of random quantities, such that λ⇀a≡∑i=0nc−1∑k=0nps−1I(a=Λi,k). Therefore, λ⇀a is the count of connections between each attribute and all columns. Recognizing that the probability of a connection forming in n_c follows a binomial distribution, the expected number of columns that an attribute will connect to is simply equation (2).

Using equation (1), it is possible to calculate the probability of an attribute never connecting, as shown in equation (3). Since the probabilities are independent, it simply reduces to the product of the probability of an attribute not connecting to a column, taken over all columns. Let λ′≡∑a=0na−1I(λ⇀a=0), the random variable governing the number of unconnected attributes. From equation (3), the expected number of unobserved attributes may then be trivially obtained as equation (4). Using equations (3) and (2), it is possible to obtain a lower bound for n_c and n_ps, by choosing those parameters such that a certain amount of attribute visibility is obtained. To ensure, with high probability, the observance of all attributes, equation (3) must be sufficiently close to zero. This is obtained by fully connecting the SP columns to the input columns or by having a sufficiently large number of columns. Once that is satisfied, the desired number of times an attribute is observed may be determined by using equation (2).

Once each column has its set of inputs, the permanences must be initialized. As previously stated, permanences were defined to be initialized with a random value close to ρ_s but biased based off the distance between the synapse’s source (input column) and destination (SP column). To obtain further clarification, NuPIC’s source code (Numenta, 2016) was consulted. It was found that the permanences were randomly initialized, with approximately half of the permanences creating connected proximal synapses and the remaining permanences creating potential (unconnected) proximal synapses. Additionally, to ensure that each column has a fair chance of being selected during inhibition, there are at least ρ_d connected proximal synapses on each column.

Let Φ∈Rnc×nps be defined as the set of permanences for each column, such that Φi is the set of permanences for the proximal synapses for c⇀i. Let max(v⇀) be defined as a function that will return the maximum value in v⇀. Similarly, let min(v⇀) be defined as a function that will return the minimum value in v⇀. Each Φi,k is randomly initialized as shown in equation (5), where Unif represents the uniform distribution. Using equation (5), the expected permanence value is ρ_s; therefore, half of the proximal synapses connected to each column will be initialized in the connected state. This means that ρ_d should be chosen such that ρ_d ≤ n_ps/2 to sufficiently ensure that each column will have a greater than zero probability for becoming active. Additionally, the columns’ inputs must also be taken into consideration, such that ρ_d should be chosen to be small enough to take into consideration the sparseness in the input.

It is possible to predict, before training, the initial response of the SP to a given input. This insight allows parameters to be crafted in a manner that ensures a desired amount of column activity. Let X∈{0,1}nc×nps be defined as the set of inputs for each column, such that X_i is the set of inputs for c⇀i. Each X_i,k is mapped to a specific U_s,a, such that X_i ⊂ U_s when n_ps ≠ n_a and X_i ⊆ U_s otherwise. Let A∈{0,1}nc×nps be defined as a matrix of events, such that A_i,k ≡ X_i,k = 1. A_i,k is therefore the event that the attribute connected via proximal synapse k to column i is active. Let P(Ai,k) be defined as the probability of the attribute connected via proximal synapse k to column i being active. Therefore, P(∃Ai,k∈Ai) is defined to be the probability of any proximal synapse connected to column i being active. Similarly, P(∃Ai,k∈A) is defined to be the probability of any proximal synapse on any column being active. The relationship between these probabilities is given in equations (6) and (7).

Let as⇀∈{0,1,…,nps}1×nps be defined as a vector of random quantities, such that as⇀i≡∑k=0nps−1Xi,k. Therefore, as⇀i is the number of active proximal synapses on column i. The expected number of active proximal synapses on column i is then given by equation (8). Let as≡1nc∑i=0nc−1∑k=0nps−1Xi,k, the random variable governing the average number of active proximal synapses on a column. Equation (8) is then generalized to equation (9), the expected number of active proximal synapses for each column. If E[as]<ρd, it is very unlikely that there will be enough active proximal synapses on a given column for that column to become active. This could result in an insufficient level of column activations, thereby degrading the quality of the SP’s produced output. Therefore, it is recommended to select ρ_d as a function of the sparseness of the input, such that ρd<E[as]. Reducing ρ_d will increase the level of competition among columns; however, it will also increase the probability of including noisy columns, i.e., columns whose synapses’ inputs are connected to attributes within the input that collectively provide little contextual meaning.

Let AC∈{0,1}nc×nps be defined as a matrix of events, such that ACi,k≡Ai,k∩I(Φi,k≥ρs). Therefore, ACi,k is the event that proximal synapse k is active and connected on column i. Let ac⇀∈{0,1,…,nps}1×nps be defined as a vector of random quantities, such that aci⇀≡∑k=0nps−1ACi,k. Therefore, aci⇀ is the count of active and connected proximal synapses for column i. Let P(ACi,k=1)≡P(Ai,k)ρs, the probability that a proximal synapse is active and connected.⁴ Following equation (8), the expected number of active connected proximal synapses on column i is given by equation (10).

Let Bin(k; n, p) be defined as the probability mass function (PMF) of a binomial distribution, where k is the number of successes, n is the number of trials, and p is the success probability in each trial. Let at≡∑i=0nc−1I((∑k=0nps−1Xi,k)≥ρd), the random variable governing the number of columns having at least ρ_d active proximal synapses. Let act≡∑i=0nc−1I((∑k=0nps−1ACi,k)≥ρd), the random variable governing the number of columns having at least ρ_d active connected proximal synapses. Let π_x and π_ac be defined as random variables that are equal to the overall mean of P(A) and P(AC), respectively. The expected number of columns with at least ρ_d active proximal synapses and the expected number of columns with at least ρ_d active connected proximal synapses are then given by equations (11) and (12), respectively.

In equation (11), the summation computes the probability of having less than ρ_d active connected proximal synapses, where the individual probabilities within the summation follow the PMF of a binomial distribution. To obtain the desired probability, the complement of that probability is taken. It is then clear that the mean is nothing more than that probability multiplied by n_c. For equation (12), the logic is similar, with the key difference being that the probability of a success is a function of both X and ρ_s, as it was in equation (10).

Let b⇀∈R>01×nc be defined as the set of boost values for all columns, such that b⇀i is the boost for c⇀i. Let Y≡I(Φi≥ρs)∀i, the bit-mask for the proximal synapse’s activations. Therefore, Y_i is a row-vector bit-mask, with each “1” representing a connected synapse and each “0” representing an unconnected synapse. In this manner, the connectivity (or lack thereof) for each synapse on each column is obtained. The overlap for all columns, α⇀∈R≥01×nc, is defined by equation (13), which is a function of α^⇀∈N>01×nc. α^⇀ is the sum of the active connected proximal synapses for all columns and is defined by equation (14).

Comparing these equations with Algorithm 1, it is clear that α^⇀ will have the same value as col.overlap before line five and that the final value of col.overlap will be equal to α⇀. To help provide further understanding, a simple example demonstrating the functionality of this phase is shown in Figure 2.

Let H∈{0,1}nc×nc be defined as the neighborhood mask for all columns, such that H_i is the neighborhood for c⇀i. c⇀j is then said to be in c⇀i’s neighborhood if and only if H_i,j is “1”. Let kmax(S, k) be defined as the k-th largest element of S. The set of active columns, c^⇀∈{0,1}1×nc, is defined by equation (15), such that c^⇀ is an indicator vector representing the activation (or lack of activation) for each column. The result of the indicator function is determined by γ⇀∈N1×nc, which is defined by equation (16) as the ρ_c-th largest overlap (lower bounded by one) in the neighborhood of c⇀i∀i.

Comparing these equations with Algorithm 2, γ⇀ is a slightly altered version of mo. Instead of just being the ρ_c-th largest overlap for each column, it is additionally lower bounded by one. Referring back to Algorithm 2, line 3 is a biconditional statement evaluating to true if the overlap is at least mo and greater than zero. By simply enforcing mo to be at least one, the biconditional is reduced to a single condition. That condition is evaluated within the indicator function; therefore, equation (15) carries out the logic in the if statement in Algorithm 2. Continuing with the demonstration shown in Figures 2 and 3 shows an example execution of phase two.

Let clip(M, lb, ub) be defined as a function that will clip all values in the matrix M outside of the range [lb, ub] to lb if the value is less than lb, or to ub if the value is greater than ub. Φ is then recalculated by equation (17), where δΦ is the proximal synapse’s permanence update amount given by equation (18).⁵

The result of these two equations is equivalent to the result of executing the first seven lines in Algorithm 3. If a column is active, it will be denoted as such in c^⇀; therefore, using that vector as a mask, the result of equation (18) will be a zero if the column is inactive, otherwise it will be the update amount. From Algorithm 3, the update amount should be ϕ₊ if the synapse was active and ϕ₋ if the synapse was inactive. A synapse is active only if its source column is active. That activation is determined by the corresponding value in X. In this manner, X is also being used as a mask, such that active synapses will result in the update equaling ϕ₊ and inactive synapses (selected by inverting X) will result in the update equaling ϕ₋. By clipping the element-wise sum of Φ and δΦ, the permanences stay bounded between [0, 1]. As with the previous two phases, the visual demonstration is continued, with Figure 4 illustrating the primary functionality of this phase.

Let η(a)⇀∈R≥01×nc be defined as the set of active duty cycles for all columns, such that ηi(a)⇀ is the active duty cycle for c⇀i. Let ηi(min)⇀∈R≥01×nc be defined by equation (19) as the set of minimum active duty cycles for all columns, such that ηi(min)⇀ is the minimum active duty cycle for c⇀i. This equation is clearly the same as line 9 in Algorithm 3.

Let update_active_duty_cycle (c⇀) be defined as a function that updates the moving average duty cycle for the active duty cycle for each c⇀i∈c⇀. That function should compute the frequency of each column’s activation. After calling update_active_duty_cycle (c⇀), the boost for each column is updated by using equation (20). In Equation (20), β(ηi(a)⇀, ηi(min)⇀) is defined as the boost function, following equation (21).⁶ The functionality of equation (20) is therefore shown to be equivalent to Algorithm 4.

Let η(o)⇀∈R≥01×nc be defined as the set of overlap duty cycles for all columns, such that ηi(o)⇀ is the overlap duty cycle for c⇀i. Let update_overlap_duty_cycle(c⇀) be defined as a function that updates the moving average duty cycle for the overlap duty cycle for each c⇀i∈c⇀. That function should compute the frequency of each column’s overlap being at least equal to ρ_d. After applying update_overlap_duty_cycle(c⇀), the permanences are then boosted by using equation (22). This equation is equivalent to lines 13–15 in Algorithm 3, where the multiplication with the indicator function is used to accomplish the conditional and clipping is done to ensure the permanences stay within bounds.

Let d(x, y) be defined as the distance function⁷ that computes the distance between x and y. To simplify the notation,⁸ let pos(c, r) be defined as a function that will return the position of the column indexed at c located r regions away from the current region. For example, pos(0, 0) returns the position of the first column located in the SP and pos(0, −1) returns the position of the first column located in the previous region. The distance between pos(c⇀i,0) and pos(Λi,k,−1) is then determined by d(pos(c⇀i,0),pos(Λi,k,−1)).

Let D∈R>0nc×nps be defined as the distance between an SP column and its corresponding connected synapses’ source columns, such that Di,k is the distance between c⇀i and c⇀i’s proximal synapse’s input at index k. D is defined by equation (23), where Y_i is used as a mask to ensure that only connected synapses may contribute to the distance calculation. The result of that element-wise multiplication would be the distance between the two columns or zero for connected and unconnected synapses, respectively.⁹

The inhibition radius, σ₀, is defined by equation (24). The division in equation (24) is the sum of the distances divided by the number of connected synapses.¹⁰ That division represents the average distance between connected synapses’ source and destination columns and is therefore the average receptive field size. The inhibition radius is then set to the average receptive field size after it has been floored and raised to a minimum value of one, ensuring that the radius is an integer at least equal to one. Comparing equation (24) to line 16 in Algorithm 3, the two are equivalent.

Two types of inhibition exist, namely global and local inhibition. With global inhibition, the neighborhood is taken to be all columns in the SP, i.e., every value in H would be set to one regardless of the system’s topology. With local inhibition, the neighborhood is determined by the inhibition radius. Using the desired inhibition type, the neighborhood for each column is updated. This is done using the function h(c⇀i), which is dependent upon the type of inhibition as well as the topology of the system. This function is shown in equation (25), where ζ⇀ represents all of the columns located at the set of integer Cartesian coordinates bounded by an n-dimensional shape. Typically the n-dimensional shape is a represented by an n-dimensional hypercube. Note that if global inhibition is used, additional optimizations of equations (16) and (19) may be made, and the need for equations (24) and (25) is eliminated. For simplicity, only the generalized forms of the equations, supporting both global and local inhibition, were shown.

It is convenient to think of a permanence value as a probability. That probability is used to determine if a synapse is connected or unconnected. It also represents the probability that the synapse’s input is important. It is possible for a given synaptic input to be represented in multiple contexts, where the context for a specific instance is defined to be the set of inputs connected, via proximal synapses, to a column. Due to the initialization of the network, it is apparent that each context represents a random subspace; therefore, each column is learning the probability of importance for its random subset of attributes in the feature space. This is evident in equation (18), as permanences contributing to a column’s activation are positively reinforced and permanences not contributing to a column’s activation are negatively reinforced.

If all contexts for a given synaptic input are observed, the overall importance of that input is obtained. Multiple techniques could be conjured for determining how the contexts are combined. The most generous method is simply to observe the maximum. In this manner, if the attribute was important in at least one of the random subspaces, it would be observed. Using those new probabilities the degree of influence of an attribute may be obtained. Let ϕ^⇀∈(0,1)1×na be defined as the set of learned attribute probabilities, such that ϕ^⇀a represents the probability of U_s,a being a meaningful attribute. One form of ϕ^⇀ is shown in equation (26).¹³ In equation (26), the indicator function is used to mask the permanence values for U_s,a. Multiplying that value by every permanence in Φ obtains all of the permanences for U_s,a.

Using ϕ^⇀ it is possible to extract a set of meaningful features from the original set of input attributes. This is done by selecting attributes whose corresponding ϕ^⇀a is above some threshold. In the case of equation (26), the threshold is ρ_s. This is because ϕ^⇀ is representative of the maximum permanence for each attribute in U_s, and for a given U_s,a to be observed it must be connected, which may only happen when its permanence is at least equal to ρ_s. The original set of attributes may then be reduced to a new set of meaningful features. This is done by selecting all attributes whose indexes are in u⇀, which is obtained by equation (27). Therefore, u⇀ will contain the set of all indexes in {0,1,…,na−1} whose ϕ^⇀a≥ρs. The dimensionality of U_s may be reduced to the size of u⇀, which will have a length of n_a or less.

Building off the feature selection technique, it is possible to obtain the SP’s learned representation of a specific input. To reconstruct the input, the SP’s active columns for that input must be captured. This is naturally done during inhibition, where c^⇀ is constructed. c^⇀, a function of U_s, is used to represent a specific input in the context of the SP.

Determining which permanences caused the activation is as simple as using c^⇀ to mask Φ. Once that representation has been obtained, ϕ^⇀ is used to obtain the overall permanence for the attributes. Those steps will produce a valid probability for each attribute; however, it is likely that there will exist probabilities that are not explicitly in {0, 1}. To account for that, the same technique used for dimensionality reduction is applied, by simply thresholding the probability at ρ_s. This process is shown in equation (28),¹⁴ where u^⇀∈{0,1}1×na is defined to be the reconstructed input.

Thornton and Srbic (2013) used a similar approach for showing how the representation of the input to the SP is understood. The key advantage of this approach is that the input is able to be reconstructed based solely off the permanences and the active columns. This is a novel approach that allows for the input to be decoupled from the system.

The SP is a spatial algorithm, as such, it should perform well with inherently spatial data. To investigate this, the SP was tested with a well-known computer vision task. The SP requires a binary input; as such, it was desired to work with images that were originally black and white or could be readily made black and white without losing too much information. Another benefit of using this type of image is that the encoder¹⁶ may be de-emphasized, allowing for the primary focus to be on the SP. With those constraints, the modified National Institute of Standards and Technology’s (MNIST’s) database of handwritten digits (LeCun et al., 1998) was chosen as the dataset.

The MNIST images are simple 28 × 28 grayscale images, with the bulk of the pixels being black or white. To convert them to black and white images, each pixel was set to “1” if the value was greater than or equal to 255/2 and “0” otherwise. Each image was additionally transformed to be one-dimensional by horizontally stacking the rows. The SP has a large number of parameters, making it difficult to optimize the parameter selection. To help with this, 1,000 independent trials were created, all having a unique set of parameters. The parameters were randomly selected within reasonable limits.¹⁷ Additionally, parameters were selected such that E[λ′]=0. To reduce the computational load, the size of the MNIST dataset was reduced to 800 training inputs and 200 testing inputs. The inputs were chosen from their respective initial sets using a stratified shuffle split with a total of five splits. To ensure repeatability and to allow proper comparisons, care was taken to ensure that both within and across experiments the same random initializations were occurring. After performing the initial experimental batch, the final parameters were selected. Using those parameters, the full MNIST dataset was used. To perform the classification, a linear support vector machine (SVM) was utilized. The input to the SVM was the corresponding output of the SP.

Three comparisons were explored for both global and local inhibition: using the set of active columns as the features (denoted as “column”), using ϕ^⇀ as the features (denoted as “probabilistic”) and using the dimensionality reduced version of the original input as the new input (denoted as “reduction”). The results are shown in Table 2¹⁸ and Table 3¹⁹ for global and local inhibition, respectively. For reference, the same SVM without the SP resulted in an error of 7.95%. The number of dimensions was reduced by 38.01 and 35.71% for global and local inhibition, respectively. Both the probabilistic and reduction methods only performed marginally worse than the base SVM classifier. Considering that these two techniques are used to modify the raw input, it is likely that the learned features were the face of the numbers (referring to attributes equaling “1”). In that case, those methods would almost act as pass through filters, as the SVM is already capable of determining which attributes are more/less significant. That being said, being able to reduce the number of attributes by over two thirds, for the local inhibition case, while still performing relatively close to the case where all attributes are used is quite desirable.

Using the active columns as the learned feature is the default behavior, and it is those activations that would become the feedforward input to the next level (assuming an HTM with multiple SPs and/or TMs). Both global and local inhibition outperformed the SVM, but only by a slight amount. Considering that only one SP region was utilized, that the SP’s primary goal is to map the input into a new domain to be understood by the TM, and that the SP did not hurt the SVM’s ability to classify, the SP’s overall performance is acceptable. It is also possible that given a two-dimensional topology and restricting the initialization of synapses to a localized radius may improve the accuracy of the network. Comparing global to local inhibition, comparable results are obtained. This is likely due to the globalized formation of synaptic connections upon initialization, since that results in a loss of the initial network topology.

To explore the input reconstruction technique, a random instance of each class from MNIST was selected. The input was then reconstructed as shown in Figure 8.²⁰ The top row shows the original representation of the inputs. The middle row shows the SDR of the inputs. The bottom row shows the reconstructed versions. The representations are by no means perfect, but it is evident that the SP is indeed learning an accurate representation of the input.

One of the main purposes of the SP is to create a spatial representation of its input through the process of mapping its input to SDRs. To explore this, the SP was tested on Bohanec and Rajkovic (1988) and Lichman (2013) car evaluation dataset. This dataset consists of four classes and six attributes. Each attribute has a finite number of states with no missing values. To encode the attributes, a multivariate encoder comprised of categorical encoders was used.²¹ The class labels were also encoded, by using a single category encoder.²²

The selection of the SP’s parameters was determined through manual experimentation.²³ Cross validation was used, by partitioning the data using a stratified shuffle split with eight splits. To perform the classification an SVM was utilized, where the output of the SP was the corresponding input to the SVM. The SP’s performance was also compared to just using the linear SVM and using a random forest classifier.²⁴ For those classifiers, a simple preprocessing step was performed to map the text-based values to integers.

The results are shown in Table 4.²⁵ The SVM performed poorly, having an error of 26.01%. Not surprisingly, the random forest classifier performed much better, obtaining an error of 8.96%. The SP was able to far outperform either classifier, obtaining an error of only 2.32%. From literature, the best known error on this dataset was 0.37%, which was obtained from a boosted multilayer perceptron (Oza, 2005).

This result shows that the SP was able to map the input data into a suitable format for the SVM, thereby drastically improving the SVM’s classification. Based off this, it is determined that the SP produced a suitable encoding.

Comparing the SP’s performance on spatial data to that of categorical data provides some interesting insights. It was observed that on spatial data the SP effectively acted as a pass through filter. This behavior occurs because the data is inherently spatial. The SP maps the spatial data to a new spatial representation. This mapping allows classifiers, such as an SVM, to be able to classify the data with equal effectiveness.

Preprocessing the categorical data with the SP provided the SVM with a new spatial representation. That spatial representation was understood by the SVM as readily as if it were inherently spatial. This implies that the SP may be used to create a spatial representation from non-spatial data. This would thereby provide other algorithms, such as the TM and traditional spatial classifiers, a means to interpret non-spatial data.

It is noticed that ΔΦ represents an unlearned function of a random variable coming from a prior distribution. That random variable is nothing more than X. It is required that Xi,k∼Ber(P(Ai,k)), where Ber is used to denote the Bernoulli distribution. Recall that the initialization of the SP is performed such that each X_i,k ∈ X_i is obtained by randomly sampling without replacement from U_s. Each X_i ∈ X is independent, since they are obtained by sampling from the same U_s. This initialization scheme means that each X_i ∈ X is independent and identically distributed (i.i.d.), but that each X_i,k ∈ X_i is not i.i.d.

If the initialization scheme is redefined such that X_i,k ∈ X_i is obtained by sampling with replacement, then each X_i,k ∈ X would be i.i.d. This slight change to the initialization would mean that a given input attribute may be observed by a column zero up to n_ps times. If n_c is set to be sufficiently large, then each U_s,a will still be observed and the overall impact of this change should be minor. Assuming that this new initialization scheme is utilized, then Xi.i.d.˜ Ber(θ), where θ is defined to be the probability of an input being active. Using the PMF of the Bernoulli distribution, the likelihood of θ given X is obtained in equation (30), where t ≡ n_cn_ps and X¯=1t∑i=0nc−1∑k=0nps−1Xi,k, the overall mean of X. The corresponding log-likelihood of θ given X is given in equation (31).

Taking the gradient of the log-likelihood of equation (31) with respect to θ results in equation (32). Similarly, the partial derivative of the log-likelihood for a single X_i,k results in equation (33). Equation (33) suggests by its form that an unconstrained local update of θ may be obtained. Let Θ∈[0,1]nc×nps≡P(∃Ai,k∈A), such that Θ_i,k = ℙ(A_i,k). Recall that X is populated based off the entries in a single U_s, i.e., X represents the current input to the SP, not all of the inputs. Solving equation (32) for the maximum-likelihood estimator results in maximizing the likelihood of the current sample rather than the population. Realizing this, an approach similar to stochastic gradient ascent is used to obtain equation (34), where δϕ+(Θi,k) and δϕ+(Θi,k) are functions of Θi,k that compute the permanence increment and decrement amount for each Θi,k, respectively.

Relating equation (34) to equation (29), δϕ+(Θi,k) and δϕ+(Θi,k) were defined to return the constants ϕ₊ and ϕ₋, respectively. If the probabilities of each X_i,k = 1 are sufficiently close in value for all U_s ∈ U, then global constants may be sufficient. Using constants also simplifies the permanence update equation, which is likely why the original permanence update method, equation (18), used this approach. For more complicated inputs, it is unlikely that this assumption will hold true. To address that, this new method allows for a local and adaptive way to tune the permanences.

The set of active columns is the learned component in equation (18). Those columns are used to determine which permanences are selected for update. Obtaining the set of active columns is done through a process similar to competitive learning (Rumelhart and Zipser, 1985). In a competitive learning network, each neuron in the competitive learning layer is fully connected to each input neuron. The neurons in the competitive layer then compete, with one neuron winning the competition. The neuron that wins sets its output to “1” while all other neurons set their output to “0”. At a global scale, this resembles the SP with two key differences. The SP permits multiple columns to be active at a time and each column is connected to a different subset of the input’s attributes.

Posit that each column is equivalent to a competitive learning network. This would create a network with one neuron in the competitive layer and n_ps neurons in the input layer. The neuron in the competitive layer may only have the state of “1” or “0”; therefore, only one neuron would be active at a time. Given this context, each column is shown to follow the competitive learning rule.

Taking into context the full SP, with each column as a competitive learning network, the SP could be defined to be a bag of competitive learning networks, i.e., an ensemble with a type of competitive learning network as its base learner. Recalling that X ⊆ U_s, each X_i is an input for c⇀i. Additionally, each X_i is obtained by randomly sampling U_s without replacement. Comparing this ensemble to attribute bagging (Bryll et al., 2003), the primary difference is that sampling is done without replacement instead of with replacement.

In attribute bagging, a scheme, such as voting, must be used to determine what the result of the ensemble should be. For the SP, a form of voting is performed through the construction of α⇀. Each base learner (column) computes its degree of influence. The max degree of influence is n_ps. Since that value is a constant, each α⇀i may be represented as a probability by simply dividing α⇀i by q. In this context, each column is trying to maximize its probability of being selected. During the inhibition phase, a column is chosen to be active if its probability is at least equal to the ρ_c-th largest probability in its neighborhood. This process may then be viewed as a form of voting, as all columns within a neighborhood cast their overlap value as their vote. If the column being evaluated has enough votes, it will be placed in the active state.