Equivalence structure (ES) extraction is conducted for focusing on multidimensional temporal patterns in a multidimensional sequence that is similar to those in a different multidimensional sequence (Satoh and Yamakawa, 2017). The extraction is a task, where the input is a set of sequences and the output is a set of ESs. A K-dimensional ES is a set of K-tuples with elements that are K IDs for K sequences, and each of the K-tuples specifies a K-dimensional sequence specified by the IDs of its K-tuple. The K-tuples of an ES indicate which K-dimensional sequences are considered equivalent. The standard for determining whether K-dimensional sequences are equivalent is based on the subsequences of the K-dimensional sequences. If we consider a tuple of an ES to represent a relation among sequences specified by its IDs, then an ES can be seen as an analogous relation of such relations. In this sense, ES extraction can be considered metarelation mining.

ES extraction can be used for multidimensional temporal feature extraction, as well as preprocessing for transfer learning and imitation learning. For examples of imitation learning, in experiments conducted by Katz et al. (2016) and Delhaisse et al. (2017), the correspondence relation between a teacher and student must be known, or it is necessary to tell key poses to a student before conducting imitation learning. However, there can be some situations, where these conditions are impossible. In such situations, ES extraction can be used to find correspondence relations between a student and teacher.

Motif discovery techniques (Lonardi and Patel, 2002; Tanaka et al., 2005; Minnen et al., 2007; Ghoneim et al., 2008; Vahdatpour et al., 2009; Mueen and Keogh, 2010; Liu et al., 2012) can be considered similar to ES extraction. The techniques allow for obtaining multidimensional temporal repeated patterns. However, the patterns must be repeated in the same multidimensional sequence. In other words, the techniques do not reveal multidimensional temporal patterns that appear in a multidimensional sequence and in a different multidimensional sequence, whereas ES extraction focuses on such patterns.

There are two methods to extract ESs (Satoh and Yamakawa, 2017). One is brute-force search (BFS), where all possible K-tuples are considered. BFS is usually infeasible, since a serious combinatorial explosion occurs in the process. The other method is the recently proposed ES incremental search (ESIS) (Satoh and Yamakawa, 2017). In the process of ESIS, K-dimensional ESs are obtained from candidates generated from (K − 1)-dimensional ESs, and ESIS obtained ESs much faster than BFS in experiments. However, it has been unclear why ESIS worked and can be reliable, because the proofs necessary to derive ESIS have not been sufficiently obtained.

This paper presents proofs to validate ESIS, as well as a property of the solution of ESIS that could be useful for developing a faster method. We investigated the property by conducting an experiment using motion capture datasets. ESIS can be derived by making an assumption, and a previous study experimentally confirmed that the assumption was satisfied using only noise-free artificial data (Satoh and Yamakawa, 2017). Therefore, in this study, we conducted another experiment to confirm that the assumption is satisfied when using real datasets with noise.

This section clarifies the theorem and proof to verify ESIS. To begin with, we define the following function: (11)dropU(K),k=u1,…,uk−1,uk+1,…,uK|⟨u1,…,uK⟩∈U(K), which drops the kth elements of all the K-tuples of a set U^(K). The following assumption is necessary to derive ESIS (Satoh and Yamakawa, 2017).

Assumption 1: (12)∀k∈[K]:∀c∈[CK∗]:∃c(k)∈[CK−1∗]:drop(S∗c(K),k)⊆S∗c(k)(K−1) where S∗1(K),…,S∗CK∗(K) are complete K-dimensional ESs, and S∗1(K−1),…,S∗CK−1∗(K−1) are complete (K − 1)-dimensional ESs.

For example, if S∗c(K)= {⟨#1, #2, #3⟩, ⟨#7, #6, #4⟩}, then sets {⟨#2, #3⟩, ⟨#6, #4⟩}, {⟨#1, #3⟩, ⟨#7, #4⟩}, and {⟨#1, #2⟩, ⟨#7, #6⟩} are all subsets of (K − 1)-dimensional ESs. Figure 3 shows an illustration of the example. How much this assumption holds depends on conditions such as the standard for the dissimilarity between multidimensional sequences and its parameters, but it is considered that the assumption holds to a certain extent.

The following corollary obviously holds under Assumption 1.

Corollary 1: (13)∀c(K)∈[CK−1∗]:∃c∈[CK∗]:S∗c(K)⊆S′c(K)(K) where (14)S′c(K)(K)≡⟨s1,…,sK−1,u⟩|u∈[N]\{sk}k∈[K−1],⟨s1,…,sK−1⟩∈S∗c(K)(K−1), (15)drop(S∗c(K),K)⊆S∗c(K)(K−1).

PROOF. Under Assumption 1, the following is clear: (16)∃⟨s1∗,…,sK∗⟩∈S∗c(K):∃⟨s1,…,sK−1⟩∈S∗c(K)(K−1):∀k∈[K−1]:sk∗=sk.

From the definition of K-tuples to specify K-dimensional sequences, ∀k∈[K−1]:sK∗≠sk∗ and sK∗∈[N]\{sk∗}k∈[K−1].

Hence, Equation (13) holds. □

The summation of the cardinalities of all candidates S′1(K),…,S′CK−1∗(K) for K-dimensional ESs satisfies the following inequality: (17)∑c∈[CK−1∗]S′c(K)=∑c∈[CK−1∗]S∗c(K−1)×(N−K+1)≤ NPK where the equality holds if and only if: (18)∑c∈[CK−1∗]S∗c(K−1)= NPK−1.

This means that every possible (K − 1)-tuple belongs to one of (K − 1)-dimensional ESs, which is unlikely to occur. Therefore, using Corollary 1, we can usually reduce the number of tuples to search.

Moreover, we can derive the following lemma under Assumption 1.

Lemma 1: (19)∀c∈[CK∗]:∀i∈[K]:∀j∈[K]\i:∃c({i,j})∈[C2∗]:drop(S∗c(K),[K]\{i,j})⊆S∗c({i,j})(2).

For example, if there is a complete four-dimensional ES {⟨#1, #2, #3, #4⟩, ⟨#5, #6, #7, #8⟩}, then drop(S∗c(4),[4]\{1,2}), drop(S∗c(4),[4]\{1,3}), and drop(S∗c(4),[4]\{3,4}) are sets {⟨#1, #2⟩, ⟨#5, #6⟩}, {⟨#1, #3⟩, ⟨#5, #7⟩}, and {⟨#3, #4⟩, ⟨#7, #8⟩}, respectively, and they are all subsets of two-dimensional ESs.

PROOF. We use mathematical induction to prove Lemma 1. Under Assumption 1, we see that (20)∀c∈[CK∗]:∃c(pK)∈[CK−1∗]:S∗c(pK)(K−1)=drop(S∗c(K),pK)∪Ec,pK(K−1), where ⟨p₁, …, p_K⟩ is a K-permutation of [K], 3 < K, and (21)Ec,pK(K−1)≡S∗c(pK)(K−1)\dropS∗c(K),pK.

Consider a particular complete K-dimensional ES S∗c(K) and a particular complete (K − 1)-dimensional ES S∗c(pK)(K−1) that satisfies drop(S∗c(K),pK)⊆S∗c(pK)(K−1). If we additionally drop the p_K–first elements of all the tuples of S∗c(pK)(K−1), we have (22)∃c({pK−1,pK})∈[CK−2∗]:S∗c({pK−1,pK})(K−2)⊇dropS∗c(pK)(K−1),pK−1.

From Equation (20), the right-hand side can be rewritten and we have (23)∃c({pK−1,pK})∈[CK−2∗]:S∗c({pK−1,pK})(K−2)⊇dropdropS∗c(K),pK∪Ec,pK(K−1),pK−1⊇dropdropS∗c(K),pK,pK−1∪dropEc,pK(K−1),pK−1.

Since the left-hand side is a superset of the right-hand side, the left-hand side is still the superset even after removing the set drop(Ec,pK(K−1),pK−1) from the right-hand side. Thus, we obtain: (24)∃c({pK−1,pK})∈[CK−2∗]:S∗c({pK−1,pK})(K−2)⊇dropdropS∗c(K),pK,pK−1⊇dropS∗c(K),{pK−1,pK}.

Next, we assume: (25)∀c∈[CK∗]:∃c({pK−k,…,pK})∈[CK−k−1∗]:dropS∗c(K),{pK−k,…,pK}⊆S∗c({pK−k,…,pK})(K−k−1) where k∈[K−4]. Under this assumption, it is clear that (26)∀c∈[CK∗]:∃c({pK−k,…,pK})∈[CK−k−1∗]:S∗c({pK−k,…,pK})(K−k−1)=dropS∗c(K),{pK−k,…,pK}∪Ec,{pK−k,…,pK}(K−k−1), where (27)Ec,{pK−k,…,pK}(K−k−1)≡S∗c({pK−k,…,pK})(K−k−1)\dropS∗c(K),{pK−k,…,pK}.

Consider a particular complete K-dimensional ES S∗c(K) and a particular complete (K–k–1)-dimensional ES S∗c({pK−k,…,pK})(K−k−1) that satisfy drop(S∗c(K),{pK−k,…,pK})⊆S∗c({pK−k,…,pK})(K−k−1). If we also drop the pK−k−1th elements of all the tuples of S∗c({pK−k,…,pK})(K−k−1), we have (28)∃c({pK−k−1,…,pK})∈[CK−k−2∗]:S∗c({pK−k−1,…,pK})(K−k−2)⊇dropS∗c({pK−k,…,pK})(K−k−1),pK−k−1.

From Equation (26), the right-hand side can be rewritten and we have (29)∃c({pK−k−1,…,pK})∈[CK−k−2∗]:S∗c({pK−k−1,…,pK})(K−k−2)⊇dropdropS∗c(K),{pK−k,…,pK}∪Ec,{pK−k,…,pK}(K−k−1),pK−k−1⊇dropdropS∗c(K),{pK−k,…,pK},pK−k−1∪dropEc,{pK−k,…,pK}(K−k−1),pK−k−1.

Since the left-hand side is a superset of the right-hand side, the left-hand side is still the superset even after removing the set drop(Ec,{pK−k,…,pK}(K−k−1),pK−k−1) from the right-hand side. Thus, we obtain: (30)∃c({pK−k−1,…,pK})∈[CK−k−2∗]:S∗c({pK−k−1,…,pK})(K−k−2)⊇dropdropS∗c(K),{pK−k,…,pK},pK−k−1⊇dropS∗c(K),{pK−k−1,…,pK}.

From Equations (24) and (30), we obtain: (31)∀k∈[K−3]:∀c∈[CK∗]:∃c({pK−k,…,pK})∈[CK−k−1∗]:dropS∗c(K),{pK−k,…,pK}⊆S∗c({pK−k,…,pK})(K−k−1).

Hence, we see that (32)∀c∈[CK∗]:∃c({pK−k,…,pK})∈[CK−k−1∗]:dropS∗c(K),{p3,…,pK}⊆S∗c({p3,…,pK})(2). □

Lemma 1 tells us that the sets of tuples composed of the ith and jth IDs of the tuples of a K-dimensional ES are subsets of a two-dimensional ES. Considering Lemma 1, we can derive the following theorem.

Theorem 1: (33)∃c1∈[C2∗]:⋯∃cK−1∈[C2∗]:S∗c(K)⊆S′′c(K),c(K−1)(K), where (34)S′′c(K),c(K−1)(K)≡⟨s1,…,sK−1,u⟩u∈u ∧k∈[K−1]⟨sk,u⟩∈S∗ck(2), ⟨s1,…,sK−1⟩∈S∗c(K)(K−1), (35)dropS∗c(K),K⊆S∗c(K)(K−1), (36)c(K−1)≡⟨c1,…,cK−1⟩, and S∗c(K) and S∗c(K)(K−1) are a particular complete K-dimensional ES and a particular complete (K − 1)-dimensional ES that satisfy Equation (35), respectively.

PROOF. From Equation (35), it is clear that (37)∀⟨s1∗,…,sK∗⟩∈S∗c(K):∃⟨s1,…,sK−1⟩∈S∗c(K)(K−1):∧k∈[K−1]sk=sk∗.

From Lemma 1, we see that (38)∀k∈[K−1]:∃ck∈[C2∗]:∀⟨s1∗,…,sK∗⟩∈S∗c(K):drop⟨s1∗,…,sK∗⟩,[K]\{k,K}∈S∗ck(2).

Let ck∗ be a c_k that satisfies (39)∀⟨s1∗,…,sK∗⟩∈S∗c(K):drop⟨s1∗,…,sK∗⟩,[K]\{k,K}∈S∗ck(2).

We then see that (40)∀k∈[K−1]:∀⟨s1∗,…,sK∗⟩∈S∗c(K):⟨sk∗,sK∗⟩∈S∗ck∗(2).

From this equation, we find that (41)∀⟨s1∗,…,sK∗⟩∈S∗c(K):sK∗∈u∧k∈[K−1]⟨sk∗,u⟩∈S∗ck∗(2).

Therefore, we have (42)∃c1∈[C2∗]:⋯∃cK−1∈[C2∗]:∀⟨s1∗,…,sK∗⟩∈S∗c(K):⟨s1∗,…,sK∗⟩∈S′′c(K),c(K). □

The number of possible IDs for each sK∗ in Equation (41) is less than or equal to N−K+1. It follows that (43)∑c∈C(K−1)S′′c(K),c(K)≤S′c(K)(K), where c(K)∈[CK−1∗], (44)C(K−1)≡⟨c1,…,cK−1⟩|c1∈[C2∗],…,cK−1∈[C2∗].

The equality holds if and only if the number of possible values for u in Equation (33) is (N − K + 1). This is unlikely to occur. Therefore, we can usually reduce the number of tuples to search more than just by using Corollary 1. From Equations (17) and (43), (45)∑c(K)∈[CK−1∗]∑c∈C(K−1)S′′c(K),c(K)≤∑c(K)∈[CK−1∗]S′c(K)(K)≤ NPK.

In this section, we consider ESs obtained by ESIS (simply referred to as ESs in this section). We denote a K-dimensional ES by S^c(K) and let K^max be the number of the dimensions of the highest-dimensional ES. Since ESIS was derived under Assumption 1, if 3≤K^max, there are (K–k)-tuples in (K–k) -dimensional ESs that can be expressed in the following form: (46)d(K−k)∈dropS^c(K),{p1,…,pk}, where ⟨p₁, …, p_k⟩ is a k-permutation of [K], 2 ≤ K–k, K≤K^max, and 1 ≤ k. We call such a K-tuple a derivative tuple.

The number of derivative tuples that can be derived from a K-dimensional ES S^c(K) is (47)S^c(K)×∑k=1K−2 KCk.

The existence of derivative tuples may cause a substantial increase in the processing time of ESIS when K^max is large. Therefore, solving this problem may lead to the development of a faster ES search method.

Experiments were previously conducted using a noise-free artificial dataset and two motion capture datasets (Satoh and Yamakawa, 2017). The former experiment confirmed that ESIS can obtain the same ESs as those obtained by BFS, which is possible if Assumption 1 holds. The latter experiment confirmed that ESIS can obtain ESs that are considered plausible as a correspondence relation between two motion capture datasets, but BFS was stopped because the processing time exceeded one day when K was only three. However, no previous investigation has examined whether ESIS can obtain the same ESs as BFS while using real datasets with noise.

This section describes two experiments conducted using the same motion capture datasets used in a previous study (Satoh and Yamakawa, 2017). The main aim in the first experiment is to investigate derivative tuples described in Section 4. The main aim in the second experiment is to investigate whether or not ESIS can obtain the same ESs as those obtained by BFS while using real datasets. Section 5.1 first describes the dissimilarity function used as a standard for equivalence between multidimensional sequences.