Association rules, introduced in the early 1990s (Agrawal et al., 1993b,a), find relationships and dependencies between items that co-occur in a dataset. Typically, association rules are expressed in the form of A → B, where A and B represent sets of items. One of the most well-known applications of association rules is market basket analysis, where the goal is to discover relationships between items frequently purchased together. By identifying such associations, businesses can gain insights into customer behaviour and make informed decisions regarding product placement, cross-selling, and promotions. The original algorithm for mining association rules, known as Apriori, was proposed by Agrawal et al. (1993b). It employs a principle known as the Apriori property, which asserts that all subsets of a frequent itemset must also be frequent. However, due to its iterative approach that requires multiple database scans, it was found to be computationally inefficient in practice. A more efficient algorithm called FPGrowth was later introduced by Han et al. (2004). The FPGrowth algorithm efficiently finds frequent itemsets by using a compact prefix tree (FP-tree) to avoid candidate generation, whereas the Apriori algorithm generates and tests candidate itemsets, making FP-Growth faster and more memory-efficient for large datasets.

To evaluate association rules, support and confidence measures as shown in Equations 1, 2 are used. While support captures the frequency of an association rule, confidence describes the conditional probability of a rule, given a certain prior.

While support and confidence are commonly employed to evaluate association rules, these metrics have notable limitations. In particular, they overlook statistical correlations and may fail to reflect genuine dependencies between events (Morishita and Sese, 2000; Sese and Morishita, 2002; Llinares-López et al., 2015). To address this, Piatetsky-Shapiro (1991) propose the use of lift, defined in Equation 3.

Although lift is useful for identifying deviations from independence, it can still yield association rules that are statistically insignificant (Hämäläinen and Nykänen, 2008). To address this limitation, alternative interestingness measures based on statistical hypothesis testing have been proposed to more rigorously assess the significance of associations (Webb, 2006). These methods typically rely on a 2 × 2 contingency table, as shown in Table 1, which summarises the joint and marginal frequencies of itemset occurrences and non-occurrences. This table serves as the foundation for statistical tests such as the Chi-squared test and Fisher's exact test, both of which evaluate whether item co-occurrence significantly deviates from what would be expected by chance. Like the Chi-squared statistic, the 1-quality measure, which is also known as leverage, can be used as an alternative to assess the association between a pattern and the target variable (Pellegrina and Vandin, 2024). This measure captures how often a pattern and the target variable co-occur, compared to what would be expected if they were independent. In other words, it quantifies the gap between the observed frequency of a pattern appearing in transactions labelled with 1 and the frequency we would expect under the assumption that the pattern and the target are unrelated.

Algorithms such as AprioriSMP (Morishita and Sese, 2000) and TidalSMP (Sese and Morishita, 2002) incorporate the application of these tests, employing horizontal and vertical mining strategies utilising p-values to identify statistically significant itemsets. To combat the multiple testing problem in significant pattern mining,² which searches for the most significant patterns, one can control for false discoveries (e.g., False Discovery Rate (FDR) (Benjamini and Hochberg, 1995; Benjamini and Yekutieli, 2001) or Family-Wise Error Rate (FWER) (Bonferroni, 1936; Holm, 1979)). In their tutorial, Pellegrina et al. (2019) provide a comprehensive overview of existing methods. For instance, Terada et al. (2013a) introduced the “Limitless Arity Multiple-testing Procedure” (LAMP), which enhances efficiency by pruning itemsets based on their minimum attainable p-values. The computational performance of LAMP was further improved in subsequent work by Minato et al. (2014). Alternatively, Terada et al. (2013b); Llinares-López et al. (2015) applied permutation tests to adjust for multiple comparisons.

Despite their effectiveness in discovering associations, the aforementioned approaches do not consider temporal constraints in their rule mining process. In other words, they only look at the frequency of items but ignore the order of their occurrence. To address this limitation, SPM algorithms have been developed. SPM algorithms can be divided into two categories, namely Apriori-based and pattern-growth approaches (Millham et al., 2021). Apriori-based algorithms, such as Sequential PAttern Discovery using Equivalence classes algorithm (SPADE) (Zaki, 2001) and Generalised Sequential Pattern algorithm (GSP) (Srikant and Agrawal, 1996), generate a large number of sequence candidates, which are then tested for a specified minimum support threshold. Pattern-growth algorithms, such as PrefixSpan (Pei, 2001) and FreeSpan (Han et al., 2000), solve the computational issues of large candidate generation by introducing an efficient search space partitioning (Abdullah et al., 2019).

While these algorithms provide a computationally efficient way of mining sequential patterns, they typically result in a large set of sequences that are often redundant and do not necessarily carry information about the underlying classification problem (Gan et al., 2019). One way to minimise redundancy is to use Closed and maximal sequential patterns. A closed sequential pattern (Han et al., 2013) retains frequency information by ensuring no supersequence has the same support, while a maximal sequential pattern ensures no supersequence is frequent, keeping only the longest relevant patterns (Fournier-Viger et al., 2013). However, these approaches fall short in accounting for correlation with external classes since statistical measures such as Chi-squared or the correlation coefficient are anti-monotone (Morishita and Sese, 2000; Sese and Morishita, 2002). In this context, non-maximal or open patterns might still hold high explanatory value. Another solution is high-utility SPM, which considers the utility of each sequential pattern during the mining process (Truong-Chi and Fournier-Viger, 2019). For utility-based pattern mining, its associated algorithms, such as USpan (Yin et al., 2012) and CHUSP (Dinh et al., 2023), generally assume that a utility value is specified for each event in the sequence. However, in a classification setting, such utility measures are not explicitly available, which makes the algorithms unsuitable. Similar to the previous work on statistical testing for itemsets, Dalleiger and Vreeken (2022) also leverage an upper bound during the mining process to increase the efficiency, while Tonon and Vandin (2019) proposes using the Westfall-Young method for multiple hypothesis testing with SPM. Our work complements these existing works on statistical testing by proposing a more streamlined approach that repurposes existing SPM algorithms without in-built statistical testing capabilities to compute a novel measure of interest. This novel measure captures the directional and bounded discriminative effect size that a pattern exhibits with respect to a given binary classification problem and is based on conditional, within-group differences.

A binary time series classification problem involves categorising sequences of time-dependent data (time series) into one of two distinct classes (Lin et al., 2015). Each time series consists of ordered data points collected over time, and the goal is to train a model to predict whether a given time series belongs to one of the two predefined classes, typically based on patterns or trends in the temporal data (Ismail Fawaz et al., 2019). We refer to this time series as categorical when dealing with discrete data points as features.

There are two ways of modelling a binary time series classification problem: (1) using time series models directly or (2) extracting features from the time series and treating them as a tabular and time-invariant dataset as input to a regression problem (Fulcher and Jones, 2014). Compared to numerical time series, categorical time series require learning a numerical representation of the categorical values. One of the earliest approaches to model categorical time series is Markov chains (Gagniuc, 2017), which use a transition matrix to estimate the next event. Lin et al. (2022) introduces a hidden Markov ensemble algorithm that uses the Wasserstein distance and autoencoders to learn discrete features of time series, combined with a hidden Markov model for learning continuous features. An alternative approach is given by Wang et al. (2021), who propose an end-to-end representation learning model for time series classification, utilising temporal convolution, residual networks, bidirectional long short-term memory (LSTM) networks, and a multi-layer perception network. Similarly, the Temporal Fusion Transformer (Lim et al., 2021) proposes a neural network architecture that not only learns a representation of a time series but also combines it with static, time-independent features to solve a classification problem. However, such neural network based architectures are considered black-box models. To interpret such them on an observation level, we need an additional interpretability component, such as SHAP (Lundberg and Lee, 2017) or one of its time-aware derivatives (Nayebi et al., 2023; Raykar et al., 2023). This, however, adds more complexity and runtime, necessitating more straightforward approaches in practice.

An easier-to-interpret and computationally less demanding way of determining significant features for a classification problem is through machine learning, where the importance of features can be learned based on their correlation to the underlying problem (Saarela and Jauhiainen, 2021; Liu et al., 2022). However, the features must be mined manually beforehand, involving domain knowledge and feature engineering to create meaningful features (Dong and Liu, 2018). In the case of numerical time series, such features are overall trends, seasonality, stationarity, lagged values and other measures of central tendency, such as the minimum or maximum value (Mukhopadhyay and Samanta, 2023). A more advanced technique of mining features is time series shapelets, which aims at finding the most representative numerical time series subsequence for a given class (Ye and Keogh, 2009). However, these numerical features do not apply to a categorical time series since categorical values cannot be directly transformed to an ordinal scale (Lin et al., 2015; Fokianos and Kedem, 2003). Instead, one common technique used to encode categorical events is binary encoding, also known as one-hot-encoding (Suits, 1957). Although this technique disregards the temporal relationships within sequences, it identifies whether a specific sequence appears in an observation. As a result, it is commonly used in interpretable machine learning (Davis, 2021; Alkharusi, 2012). However, we obtain a memory-intensive feature matrix by doing so, making it challenging to apply this approach in practice (Xiang et al., 2020). In this study, we propose a memory-efficient method for extracting important sequential patterns from categorical time series data and quantifying their influence on the classification task, thereby making it well-suited for exploratory data analysis and feature selection.

Feature selection aims to reduce dimensionality in machine learning problems. In their survey, Preyanka Lakshme and Kumar (2022) divide the feature selection process into unsupervised and supervised problems. For supervised problems, the authors further distinguish between (1) filter, (2) wrapper, (3) embedding, and (4) hybrid methods. The filter approach describes the selection of features based on statistical properties, such as the missing value ratio, the correlation coefficient, or the permutation feature importance, ANOVA, Mutual information (Kraskov et al., 2004). Wrapper methods iteratively check if a machine learning model has improved its prediction capabilities due to the inclusion or removal of features. Typical examples encompass the forward (Whitney, 1971) and backward selection (Marill and Green, 1963) of features, as well as genetic algorithms (Leardi, 1996). The embedded and hybrid approaches are mixtures of the already mentioned types. Embedded methods use the filter and wrapper methods within the actual prediction model, implementing their own feature selection during the training process of the machine learning model. A typical example is the L1 or L2 regularisation of linear regression models (Ng, 2004). Hybrid approaches, in contrast, use a combination of filter and wrapping methods. A prominent example of such a hybrid system is Boruta (Kursa et al., 2010), which iteratively checks if each feature is more important than randomly shuffled features from the supplied dataset. Based on statistical significance tests, the features are then either retained or discarded. Another advanced hybrid feature selection algorithm is the minimum Redundancy Maximum Relevancy (mRMR) principle (Peng et al., 2005). The idea is to account for redundancy among the important features and thereby maximise the overall discriminative power of the selected features. While the above-mentioned feature selection methods work well with tabular data, they do not consider temporal dependencies of sequential events by design. Hence, to utilise these feature selection algorithms, it is necessary to extract temporal features beforehand. With our work, we address this shortcoming and directly integrate the feature selection into the SPM process. The subsequent section presents a comprehensive delineation of the proposed methodology utilised for the extraction and subsequent selection of temporal features.

The main idea of the feature selection algorithm is to first capture sequential patterns for each class of the binary classification problem separately. Next, we calculate the difference in the confidence measure ∈[0, 1] for each pattern r and select the most important sequences based on this difference.

Equation 4 illustrates the underlying concept formally, where D_pos represents the set of positive examples and D_neg the negative examples. For each sequence r, we calculate a difference δ_r yielding values within the range [-1,1]. This difference represents the contrast in confidence levels between two subsets: one containing only positive class values and the other containing only negative class values. The absolute value of δ_r serves as an indicator of the sequence's impact on classification. A value close to 1 indicates a strong influence, while 0 suggests the sequence is equally influential for both classes. Furthermore, the sign of δ_r provides additional insight: negative values indicate a greater impact on the negative class, whereas positive values indicate a greater impact on the positive class. In this work, we used the PrefixSpan algorithm (Pei, 2001) to mine sequential patterns, but the overall procedure is independent of the SPM algorithm.

We repeat the overall process Z = 10 times with stratified, random sampling to better reflect the underlying aleatoric uncertainty. This means each fold contains the same proportions of classes, but the data points are drawn at random. The choice of Z is arbitrary; however, we select ten as a balance between the computational cost of SPM and predictive accuracy. This choice aligns with findings in cross-validation research, suggesting that 5-10 folds is generally sufficient for model evaluation (Kohavi, 1995; Breiman and Spector, 1992; Hastie et al., 2009). We store the confidence difference between positive and negative classes for each run. As an initial measure to diminish the number of sequences, we employ two minimum support thresholds: an absolute threshold of 100 occurrences, which guarantees the existence of a sufficient number of data points for statistical analysis, thereby precluding the extraction of numerous unimportant patterns. Afterwards, we shrink the number of sequences by only considering unique ones. This is important since sequences may have different antecedents and consequents but share the same sequence of events. For instance, the sequence A → B → C consists of two sequential patterns with different confidence estimations: P(C|A → B) and P(B → C|A). In our case, we have retained the sequences with the highest absolute delta confidence measure and disregarded the remaining sequences.

To select discriminative features that have statistically significant effects on the classification problem, we conduct a Mann-Whitney-U (MWU) test (Mann and Whitney, 1947; Wilcoxon, 1945) on the confidence differences for each sequence. This non-parametric approach was selected based on several methodological considerations. First, the data exhibited non-normal distribution patterns and were measured on an ordinal scale, while also meeting the critical assumption of independence between comparison groups. Additionally, the dataset comprised absolute delta confidence values strictly bounded between 0 and 1, which precluded the use of parametric tests that assume unbounded, normally distributed continuous data. The number of samples in this context depends heavily on the number of bootstrap iterations, which directly influences the stability and reliability of the resampled estimates. Finally, the MWU test offers robust performance with small sample sizes and is relatively unaffected by outliers, an advantage given the constrained range and potential skewness of our measures. In particular, we are interested in sequences with an absolute delta confidence value greater than 0. To further reduce the chances that a sequence is considered important by randomness and to ensure a sufficiently large effect size for practical significance, we require the absolute delta confidence value of the sequences to be above a user-defined delta confidence threshold. To combat the multitesting problem of inferring only based on observed values and to control the false discovery rate (i.e., identify as many significant features as possible while incurring a relatively low proportion of false positives), we adjust the p-values via Benjamini-Yekutieli correction (Benjamini and Yekutieli, 2001). Unless stated otherwise, we have considered the maximum sequence length of 3 as a length constraint, similar to prior work on website/clickstream data based on online retail behaviour (Desai and Ganatra, 2015). Longer sequences generally have lower support and are unnecessary to demonstrate our feature selection algorithm.

In this section, we will introduce the three datasets with sequential patterns, based on which we conducted our evaluation. We primarily use a synthetic dataset to provide a controlled environment and showcase the inner workings, and then utilise two real-world datasets to test the applicability of the proposed algorithm in practice. In particular, we utilise a dataset on malware detection through API call sequences (Oliveira, 2019) and a dataset on customer churn with clickstream data Requena et al., (2020). After a brief description of each dataset and its preprocessing, we compare and provide an overview of the statistical properties of each dataset.

The synthetic dataset consists of 20,000 sequences composed of 15 unique events, which we will denote as separate letters, ranging from “A” to “O” in the Latin alphabet. To generate a sequence, we randomly draw the letters sequentially until we meet the desired sequence length, which we also select randomly to be between 2 and 15. We control for the informativeness by imposing mutually exclusive constraints, that is, we require some sequences to be only indicative for one class and not to appear together with another informative sequence. In our case, A → B, B → C, C → D,D → E are indicative for the positive class and each of the subsequences occurs in 10 % of all sequences. Likewise, F → G, G → H, H → I, I → J signal the negative class, and each subsequence also occurs in 10 % of all sequences. With these sequences, we can predict the classes of 80 % of all sequences, and the classification results of the remaining 20 % are subject to chance.

The malware dataset consists of 44,058 API call sequences resulting from a 3,000-hour-long execution of malware and goodware data points in a Cuckoo Sandbox environment (Oliveira, 2019). It consists of 43,979 malware and 1,079 goodware call sequences, constituting the two classes in this classification task. Each sequence has 100 non-consecutive API calls encoded as integer values, which means the same API call does not occur in direct succession. To avoid class imbalance effects, we downsample the malware class at random.

The raw customer churn clickstream data stems from a fashion e-commerce website over two months in 2018 Requena et al., (2020). It contains 443,652 anonymised sessions of clickstream trajectories of website visitors. Each session represents a series of events that occur within 30 min. The six unique clickstream events, which are “Page view”, “Detail”, “Add”, “Remove”, “Purchase”, and “Click”, describe actions on the website. Notably, only 2.08 % of these clickstream sessions culminate in purchases, thereby rendering it a profoundly imbalanced classification problem. To maintain comparability, we conduct the same preprocessing on the raw data as (Requena et al. 2020). This implies that we keep sequences that are longer than 4 clicks to ensure that the sequence contains enough events for classification, and we trim the sequences by only considering the clicks that occurred before a purchasing decision. Furthermore, we also downsample the majority class to create a balanced classification problem and remove sequences that are unreasonably long for a 30-minute session. Just as (Requena et al. 2020), we choose 155 as the maximum allowed sequence length, equalling a reduction in observations of only 1 %.

Table 2 illustrates key characteristics of the resulting preprocessed datasets. While they all yield a balanced class ratio, the number of unique events and sequence lengths differ significantly. The malware dataset has, on average, the longest sequence length and the highest number of unique events. The synthetic dataset, on the other hand, has the shortest sequences on average, and the churn dataset possesses the fewest unique events.

To analyse the individual steps of our proposed feature selection algorithm, we track the number of sequences remaining after each step in Table 3. For clarity in the discussion and analysis, we refer to the aggregated mined sequences after bootstrapping. Notably, applying SPM on the malware dataset results in a very high number of sequences due to 1) a high number of unique events and 2) a sequence length of 100 for each observation.

Figure 1 demonstrates the correlation between the target variable and the delta confidence measure (RQ1). We observe a statistically significant positive linear correlation, as measured by the Pearson correlation coefficient ρ, for all three datasets, albeit to varying extents. The synthetic and malware datasets exhibit an almost perfect linear correlation of 0.85 and 0.9, while the churn dataset only yields a correlation coefficient of 0.56. Furthermore, as expected, no sequences with a delta confidence around δ = 0 exist. Instead, sequences with δ> 0 have, on average, a higher number of positive observations, whereas the reverse is true for sequences with δ < 0. In the synthetic dataset, the extreme points are represented by their weighted sequences, which also matches our expectations.

As noted by Geng and Hamilton (2006) and Tan et al. (2004), a multitude of interesting measures exist. In this section, we compare the delta confidence criterion against five of the most prevalent measures, namely Chi-squared ∈[0, ∞], entropy ∈[0, 1], the Fisher odds ratio ∈[0, ∞], the Phi-statistic ∈[−1, 1] and the 1-quality ∈[−1, 1]. The previous visualisation of delta confidence values in Figure 1 reveals that only the malware dataset exhibits delta confidence values ranging from -1 to 1. As a consequence, this is the only dataset that can give a complete overview of the relationship between other interesting measures and the delta confidence criterion, which Figure 2 visualises.³ For the Chi-squared and entropy, we observe a parabolic trend, whereas for the Fisher Odds Ratio, we see an exponential correlation. Given the symmetric nature of Chi-squared values and entropy, it is not possible to deduce the influence direction from these metrics alone. The Fisher odds ratio ranges from ∈[0, ∞], whereby a value < 1 indicates a negative influence and a value >1 a positive one. However, given the asymmetric nature of the importance curve, these interestingness measures are difficult to interpret.

A particular interesting comparison is between the delta confidence, the phi coefficient and the 1-quality criterion. While the theoretical bounds of Phi (±1) and 1-quality (±1) are seldom reached in practice, delta confidence more readily approaches its extremes when patterns are highly class-specific. This disparity arises from their distinct methodological underpinnings: delta confidence quantifies directional association by normalising the difference in pattern occurrence across classes, whereas the Phi-statistic is derived from the full confusion matrix, considering both the presence and absence of a pattern. Similarly, the 1-quality measure, derived from statistical independence, relies on joint and marginal probabilities from the contingency table, often yielding smaller values. Hence, both the Phi coefficient and 1-quality provide a more conservative, global assessment of association, reflecting the influence of cases where the sequence does not occur. In contrast, delta confidence highlights local, class-specific associations.

This difference has practical implications, especially in applied domains like e-commerce or fraud detection. Consider a behavioural sequence, e.g., [newsletter → product → checkout], that occurs almost exclusively among premium users. Even if rare in the total dataset, delta confidence will correctly yield a value close to +1, highlighting the pattern's strong class specificity. The phi coefficient or 1-quality measure, on the other hand, will under-represent the sequence's discriminative power due to the dilution from observations that do not comply with the sequence. From a business perspective, such high delta confidence patterns are valuable for tasks like targeted marketing or early customer profiling, where the goal is to identify precise, class-specific signals rather than optimise global prediction performance.

Beyond correlation analysis, further insights emerge when considering p-values, which are illustrated in Figure 3. The figure compares the p-values obtained using the delta confidence combined with the MWU test to those derived from the Chi-squared test and Fisher's exact test. It also includes a comparison with a similar setup to the delta confidence + MWU test, but instead of using delta confidence, it employs the phi coefficient along with the 1-quality criterion. Notably, for a fixed p-value obtained through the delta confidence + MWU approach, we observe differing p-value magnitudes for the Chi-squared and Fisher's exact test. For the alternative setups with the phi coefficient and 1-quality, we cannot observe such a clear pattern. This suggests that, despite their monotonic correlation as indicated by the Spearman coefficient r_s, the delta confidence and conventional subgroup interesting measures may carry different information and therefore complement each other. While delta confidence measures conditional, within-group differences, the subgroup interesting measures assess the significance of observed patterns based on frequency distributions. Importantly, even when the MWU test confirms the statistical significance of the delta confidence values, the Chi-squared or Fisher's exact test can yield different results, due to inherent differences in the data and hypothesis (such as continuous vs. categorical data, or testing for differences in distributions vs. testing for independence). In summary, delta confidence offers directional, subgroup-sensitive properties that are particularly useful for decision-making scenarios that require understanding nuanced behaviours within specific subpopulations.

To elucidate the impact of the hyperparameters on our proposed feature selection process, we conducted ablation studies, controlling for accuracy, number of remaining sequences, and overall runtime on the synthetic dataset. Figure 4 synthesises the effects of the minimum support, maximum sequence length, multitesting, minimum effect size, and the number of bootstrapping rounds. Overall, each threshold value was subjected to five iterations, and for enhanced readability, we report solely the average values of the results. To better show the effect of multiple bootstrap rounds and the maximum sequence length, we have set the minimum support threshold for these ablation studies to 0. As anticipated, the minimum support precipitously reduces the overall runtime and sequence count until no sequences surpass this threshold. The overall classification accuracy remains unaffected. The maximum sequence length also shows the expected effects: for an increasing maximum sequence length, the runtime increases exponentially while the AUC values and the number of significant features do not change significantly. We can also observe that multitesting reduces the overall number of sequences, has no effect on the overall classification accuracy, but introduces a computational overhead in terms of runtime. Elevating the minimum effect size yields a drastic reduction in the overall feature count and runtime, while accuracy is maintained until no sequences are available anymore. With regard to bootstrapping, it is unsurprising that the runtime increases linearly with increasing bootstrap rounds, while overall classification performance remains largely unchanged. However, with fewer bootstrap rounds, the statistical power of the MWU test is low, which can result in no sequences passing the significance test. In contrast, higher numbers of bootstrap rounds increase statistical power, leading to more sequences reaching significance and a more stable delta confidence distribution.

To address RQ2, we conduct a comparative analysis of the feature selection capabilities of the delta confidence measure with respect to existing algorithms. However, instead of directly using the absolute delta confidence measure, we create a new ranking ∈ [0,1] by multiplying the relative support values with the absolute delta confidence values. In this way, we also account for the frequency of each pattern, which is independent of the delta confidence measure. We adhere to a machine-learning workflow as illustrated in Figure 5, partitioning our data into training and testing sets using a stratified 80:20 split. Furthermore, for increased robustness, we repeat each experiment five times with different random seeds each time and employ three well-established algorithms: Naïve Bayes, Logistic Regression, and an eXtreme Gradient Boosting (XGB) (Chen and Guestrin, 2016) classifier. Before training and evaluating each classifier on the area under the curve (AUC) of the receiver operating characteristic (ROC), we utilise a feature selection algorithm to select the most informative features based on the training data. For benchmarking purposes, we deliberately select three established feature selection methods that each belong to a different category introduced in Subsection 2.3: mutual information (filter), random forest feature importance (embedding), and an adaptation of mRMR (hybrid). Furthermore, we have also considered the Chi-squared as an additional filter method to reflect statistical significance testing. By selecting at least one representative from each category, we aim to achieve a broad diversification of different feature selection methods, allowing for a comprehensive comparison with the delta confidence measure. We exclude the wrapper category from consideration due to the high computational costs associated with its iterative process. In terms of the employed mRMR adaptation, we base our mRMR feature selection on the feature importance of a random forest, which was first introduced by Zhao et al. (2019). To control for the redundancy (i.e. high correlation), we use Theil's U ∈ [0,1], an asymmetric correlation measure between categorical variables. In addition to comparing our approach with existing feature selection methods, we conduct a separate analysis on the impact of sequence encoding techniques. Specifically, we contrast one-hot encoded events with one-hot encoded sequential patterns to elucidate the overall significance of sequential patterns in classification tasks. For better readability, we report only the mean AUC values.

Figure 6 shows that the overall effectiveness of sequential patterns versus one-hot encoded events depends on the characteristics of the datasets. The feature selection based on one-hot encoded events for the churn dataset shows better results than our proposed sequential pattern feature selection for all benchmark algorithms. The AUC values on the synthetic dataset indicate the opposite result. On the malware dataset, the sequential patterns initially perform better than the one-hot encoded events, indicating that there is one pattern which is particularly important for the classification problem. However, with more features, the results plateau and the event-based feature selection methods achieve better results.

In Figure 7, we benchmark the delta confidence measure with the other feature selection algorithms on one-hot encoded sequential data. For a fair comparison, we use the same preprocessing as our proposed algorithm but different feature selection algorithms after discarding the uninformative sequences by the statistical tests. Our proposed solution achieves results comparable to those of the churn and synthetic datasets of the benchmark feature selection algorithms. However, the feature selection with the delta confidence-based ranking on the malware dataset performs similarly to the Chi-squared feature selection but worse than the other algorithms.

As an additional analysis to investigate the robustness of overall sequential patterns (RQ2), we assess their importance in our study and contextualise the classification results from the feature selection analysis. Specifically, we compare the performance of various encoding schemes and machine learning models to evaluate the consistency and impact of sequential patterns across different methodological approaches.

In the following, we describe the experimental setup for these scenarios. We use a XGB classifier for the first two cases, and a LSTM model for the third case. For all scenarios, we follow the machine-learning workflow outlined in Section 4.4, with the addition of hyperparameter tuning for the employed models. We use a shallow LSTM model, consisting of three layers, as a proxy for a more complex network. Although we recognise that deeper architectures often produce better results, we opted for a simpler model due to the risk of overfitting for the given sample sizes. The first layer of the employed architecture is an embedding layer that transforms the discrete sequences into a dense representation. Next, the representation is passed to an LSTM layer with a hidden size of n, which is subject to hyperparameter tuning. The final layer is a fully connected layer, which transforms the output of size n to the desired binary format. To avoid overfitting, we additionally apply Dropout (Srivastava et al., 2014) to the fully connected layer with a dropout rate of 20 %. For updating the weights during training, we utilise the Adam optimiser (Kingma and Ba, 2014) in combination with binary cross-entropy loss and train for 100 epochs unless the result has not improved for ten consecutive rounds. Table 4 provides an overview of the possible hyperparameters, which have been selected for each scenario by the Tree-Parzen-Algorithm (Bergstra et al., 2011).

Table 5 shows the mean and standard deviation of F1 and AUC for five experimental runs with different random seeds, for the above three scenarios. The best values are highlighted in bold. The results show that the performance of the selected sequential patterns, which have subsequently been passed to an XGB classifier, achieve the same or greater classification performance than the one-hot encoded events, which have also been passed to an XGB classifier. The classification accuracy of the LSTM is lower than that of the previously mentioned XGB classifier. The exception is the synthetic dataset, where the LSTM achieves on-par results with the sequentially encoded features.

We conduct a qualitative analysis to further distinguish our proposed algorithm from existing feature selection algorithms. We compare our feature selection methodology with a non-exhaustive list of existing solutions, focusing on two primary criteria: interpretability and computational efficiency. To qualitatively evaluate the interpretability dimension, we distinguish between the set of mined features and the direction of influence. To characterise the set of mined features, we use the same naming convention of Kursa et al. (2010), who differentiates between “all-relevant” and “minimal-optimal” feature selection algorithms. While “all-relevant” describes a feature set containing all important features, “minimal optimal” refers to a setting where the optimal subset of features for a given classifier is mined. Table 6 provides an overview of the interpretability aspect. While all classifier-based feature selection solutions, except Boruta, represent minimal-optimal solutions, all filter methods are all-relevant. The direction of influence a feature has on the classification task depends on the chosen model. Our approach is the only solution offering a directed impact quantification with an all-relevant feature set. While this approach might yield a lot of statistically significant features, it is important for exploratory data analysis, where all features that contribute to a given classification problem are important and not just an inferred subset from a covariance matrix.

Regarding computational efficiency, we differentiate between the maximum memory consumption and the time required to select the feature. Since our primary objective is to identify and interpret sequential patterns, we focus solely on mining them and their subsequent feature selection process. In particular, we compare the feature selection process as outlined in Algorithm 1 with the alternative approach of first mining the sequential patterns with the PrefixSpan algorithm and then selecting the important sequential patterns, for example, by means of the feature importance of a random forest or the mutual information criterion. This contrasts with Subsection 4.4, where we only compare the delta confidence measure with existing feature selection solutions and not the entire process.

Figure 8 illustrates the time and peak memory required for increasing sample sizes of the synthetic dataset. As expected, mRMR is the most computationally expensive method in processing time since it incrementally looks for the sequential pattern that minimises the redundancy of the already existing features. While the random forest and mutual information show the fastest execution times for a low number of sequences, the delta confidence measure becomes relatively faster for an increasing number of sequences until it becomes the quickest feature selection method. We attribute this performance difference to two counteracting forces. First, for a sufficiently high bootstrap rate, mining sequential patterns multiple times on bootstrapped datasets generally takes longer than mining them on the full dataset. Second, with the delta confidence measure, it is not necessary to explicitly pass the mined sequences to another feature selection model. However, the random forest feature importance and the mutual information criterion require a separate projection of the mined sequential pattern on the observations. This step is computationally expensive since it involves checking for each sequential pattern if it is contained in an observation. Since this effect depends on the sample sizes and the number of sequential patterns, which increases with increasing sample sizes, the resulting computational time complexity is bilinear. Given that the PrefixSpan algorithm scales linearly with increasing sample sizes (Pei, 2001), our proposed delta confidence criterion scales better for large sample sizes. In terms of peak memory consumption, our delta confidence criterion avoids the costly creation of a one-hot encoded feature matrix, which is needed for subsequent machine learning models to estimate the importance of each sequential pattern. Given that this one-hot encoded matrix requires the majority of the memory, the peak consumption of the mRMR, random forest and the mutual information criterion overlap.