Two traditional regularization methods, LASSO and Pathway LASSO, are utilized to estimate IE for the high-dimensional model. When applied to our model, LASSO can be viewed in terms of Equation 3:

The L₁-norm penalty functions are defined as P1(α)= λ1α∑i=1p|αi| and P2(β)= λ1β ∑i=1p|βi|. These functions impose penalties on the model parameters α and β to enhance the accuracy of parameter selection.

Pathway LASSO can be viewed in terms of Equation 4, which is aimed at stabilizing estimates and minimizing estimation bias:

This method, drawing from the principles of EN, introduces a penalty P1(α,β)=κ∑i=1p[|αiβi|+ν(αi2+βi2)] for the identification of path-specific effect, and this penalty remains convex when ν is greater than or equal to 0.5. An additional L1-norm penalty P2(α,β)=ω∑i=1p(|αi|+|βi|) is included to further shrink the individual parameters αi and βi, thereby improving the selection accuracy. Moreover, to address the overestimation of DE ( γ), an L1-norm penalty for P3(γ)=λγ|γ| is also proposed in this method.

We propose two tuning parameter strategies that relax the L1-norm constraints to address challenges in traditional methods. The first strategy introduces an L1-norm penalty λγ|γ| for DE and establishes a minimum threshold for λγ (specified as λγ≥c) to address the issue of overestimation. This adjustment is particularly effective in reducing DE overestimation by compressing its estimated value toward zero, leading to a lower bias and variance of the estimated IE.

The second strategy aims to further enhance the TPR of IE by critically evaluating parameter estimation in IEi ( αiβi) and improving the detection of mediators with nonzero impacts. This is achieved by reducing the magnitude of the tuning parameter, λ1β, which is linked to the L1-norm penalty for the parameter vector β, under the condition that λ1α>λ1β. The justification for this strategy is detailed subsequently, focusing on the influence of the tuning parameters λ1α and λ1β on parameter estimation.

We initiate our analysis by examining the effectiveness of our first tuning parameter strategy in mitigating the overestimation of DE. In the simulation study, we compare LASSO that utilizes three distinct tuning parameter strategies, specifically, (1) TRL: traditional LASSO method, without applying any penalty to DE (where the tuning parameter λγ=0), (2) MDL: LASSO with a modified L1-norm penalty λγ|γ| (where λγ>0) for DE, and (3) SMDL: LASSO with a strictly modified L1-norm penalty λγ|γ| (where  λγ≥c) for DE. This comparison examines how various tuning parameter strategies impact the accuracy of estimating parameters. Additionally, we report the findings by altering the number of mediators across various levels. The optimal model is determined by BIC.

An example illustrates the overestimation of DE when the number of mediators is 200. In this case, we assume the true value of TE to be 521.736, which consists of a DE of 2 and an IE of 519.736. Using the TR_L strategy, the average estimation of DE tends to be greatly overestimated, reaching 505.382. IE is substantially underestimated at 12.998, leading to a poor TPR of IE (18.6%).

The MD_L strategy significantly reduces the overestimation of DE to 312.117. Even though this estimate is substantially lower than the overestimation observed by the TR_L strategy, it is still undesirable as the true value of DE is 2. The estimated IE increases from 12.998 to 194.406. The MSE of IE decreases from 82.094 to 68.408. The RB is also reduced, moving from -0.975 to -0.626, and the TPR of IE increases from 18.6% to 46.4%.

Among the three tuning parameter strategies evaluated, the overall performance of the SMD_L strategy demonstrates superior overall performance compared to the alternative approaches. Using grid search, when threshold c is set to 0.3, the SMD_L strategy reduces the estimated DE to 1.688, closely approximating the true DE value of 2. As a result, the SMD_L strategy elevates the TPR of IE to 81.1%, reduces the MSE to 64.690, and lowers the RB to -0.059. Thus, adjusting the range of the tuning parameter λ_γ beyond 1 is crucial for achieving the desired level of TPR of IE in high-dimensional mediation models.

Figure 1 presents simulation results with varying numbers of mediators. In a low-dimensional model setting (with n = 50 and p = 30), the TR_L strategy accurately estimates IE. However, as the number of mediators increases, DE obtained by the TR_L strategy is substantially overestimated. In high-dimensional settings (with n = 50 and p ≥ 50), the impact of the tuning parameter strategy on the estimation is consistent with the previous example (p = 200). The TR_L strategy overestimates DE substantially, resulting in an estimation of IE close to zero, a larger estimation of MSE and RB, and a smaller estimation of TPR and TNR. Conversely, the MD_L and SMD_L strategies successfully mitigate this overestimation, enhancing the accuracy of estimation and biomarker detection. Notably, the SMD_L strategy is particularly beneficial in estimating IE with desirable MSE, RB, TPR, and TNR.

The abovementioned findings assumed a small true value of 2 for DE. To further examine the SMD_L strategy, we conducted another simulation study with a sample size of 50 and 200 mediators, mainly focusing on situations with larger DE values. In this simulation, the true value of IE was fixed at 320, while the true values of DE were varied across a spectrum of 36, 80, 137, 213, 320, 480, 747, 1280, 2880, and 6080. This spectrum represents a transition in the mediation proportion (i.e., the ratio of IE to TE) from 90% to as low as 5%.

Simulation results for the MD_L and SMD_L strategies showed an overestimation bias in IE, intensifying as the true value of DE increased. The underlying reason for this phenomenon is that the TE is a sum of DE and TIE; when the estimation of DE is constrained, the estimation of TIE correspondingly increases, leading to a rise in both the MSE and RB of IE. Moreover, a threshold c in the SMD_L strategy was ascertained by identifying the minimal MSE of DE derived from the grid research outcomes. This strategy facilitates an accurate estimation of DE and enhances the TPR of IE.

It is noteworthy that the TPR of IE under the SMD_L strategy ranges between 69% and 81%, which is significantly larger compared to the MD_L strategy (ranging from 46% to 60%) and the TR_L strategy (ranging from 16% to 27%). The TNR of IE for the three strategies exhibits similar values. To conclude, when the research objective is to enhance biomarker detection rather than providing a precise estimate of IE, the SMD_L strategy remains an effective strategy for addressing the problem of overestimation of DE.

Another simulation study examined whether the second tuning parameter strategy further improves model performance. We relaxed the assumption of the L₁-norm equal-sized tuning parameter constraint (where λ_1α= λ_1β) for parameters regarding IE (i.e., α and β) and observed whether this relaxation contributes to the improvement of model performance.

Table 1 illustrates the impact of this relaxation of the constraint using an example with 20 nonzero mediators, with the estimation of parameters obtained via the SMD_L strategy. When penalty sizes are equal, such as λ_1α= 1 and λ_1β= 1, we observe that only 12 out of 20 nonzero mediators are detected, yielding a TPR of IE of 60%. Reducing λ_1α to 0.1 while keeping λ_1β at 1 (where λ_1α< λ_1β) does not improve the TPR of IE, as it does not enhance the TPR of β. However, lowering λ_1β to 0.01 while keeping λ_1α at 1 leads to the detection of 10 additional nonzero β_i, raising the TPR of IE to 95%.

Given that the sum of DE and TIE is a fixed number (TE), the accuracy of estimating DE will inherently influence the estimation outcomes of IE. Consequently, the first tuning parameter strategy (λ_γ|γ| where λ_γ≥ c), designed to determine the estimation of DE, needs to be taken into account when evaluating the second strategy. Thus, a total of 8 methods were compared in this simulation study based on the variants of Pathway LASSO LASSO.

Four variants of Pathway LASSO method were implemented: SMD.S_p (where λγ≥c, λ1α>λ1β), SMD.E_p ( λγ≥c, λ1α=λ1β), MD.S_p ( λγ>0, λ1α>λ1β), and MD.E_p ( λγ>0, λ1α=λ1β), with traditional Pathway LASSO equivalent to MD.E_p. For a consistent comparison, the tuning parameters κ and ν in the path-specific penalty P1(α,β)=κ∑i=1p[|αiβi|+ν(αi2+βi2)] were fixed across all methods. The parameter κ, crucial in determining the magnitude of estimated IE, was set to 0.01 to optimize results, leading to higher TPR and lower MSE of IE, as well as reduced BIC of the model. The parameter ν was fixed at 2, adhering to recommendations from traditional Pathway LASSO. Similarly, another four methods are implemented on LASSO, henceforth referred to as SMD.S_L, SMD.E_L, MD.S_L, and MD.E_L, respectively.

To evaluate the performance of the proposed model for the aforementioned methods, we refrained from identifying an optimal model among the 3,528 tuning parameter combinations ( λγ,λ1α,λ1β). We noted the potential bias induced by the substantial variance in BIC across various penalty configurations ( λ1α,λ1β). Therefore, we employed a two-stage model selection process. The first stage involves selecting the optimal model based on BIC for each sample, considering different penalty configurations ( λ1α,λ1β). The second stage aggregated the results by computing the average performance across the collection of the optimal model and penalty configurations within each method. This selection method aligned with the Minimax property principle, which ensured a more robust and reliable assessment of model performance.

Figure 2 displays the simulation results with varying numbers of mediators. These results indicate that implementing our proposed tuning parameter strategies, SMD.S_P and SMD.S_L, exhibits superior performance. Specifically, as the number of mediators increases, SMD.S_P, SMD.E_P, SMD.S_L, and SMD.E_L, employing our first tuning parameter strategy for DE (i.e., setting λγ≥c) show better performance compared to those that do not apply this strategy, which involves setting a constraint on the tuning parameter for IE (i.e., setting λ1α>λ1β).

We further evaluated the effectiveness of the second tuning parameter strategy in enhancing model performance. Based on the simulation results for a scenario involving 200 mediators, we can infer that the methods implemented with a smaller size of the tuning parameter λ1β, (specifically, MD.S_p, SMD.S_p, MD.S_L and SMD.S_L), demonstrate a better model performance. This improvement is evident showing smaller MSE compared to cases where the penalty size for λ1α and λ1β is equal (namely, methods MD.E_p, SMD.E_p, MD.E_L and SMD.E_L). This finding suggests that our second tuning parameter strategy, which includes reduced λ1β, tends to yield estimates (particularly for parameters β) that are closer to the corresponding actual model parameters. It accurately identifies the mediators with nonzero-effects. Our proposed method, SMD.S_p, outperforms others across IE metrics, with the highest TPR (81.9%), F1 score (0.504), Youden Index (0.614), and lowest MSE (45.240). For signal detection, SMD.S_p excels with TPRs of 86.7% for large signals and 77.1% for small signals, surpassing other methods in identifying both magnitudes.

In this section, we conduct a simulation study to test our hypothesis that the choice of the scaling factor k in SIS procedure is influenced by the ratio of the number of mediators (p) to the sample size (n), referred to as the p/n ratio. For this simulation, we keep the sample size constant at 50 while changing the number of mediators to 200, 500, and 1000. This gives us p/n = 4, 10, 20. We calculated the reduced dimension as d = k[n/log(n)], and adjusted the scaling factor k across a range of values: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 30, and 40.

Figure 3 presents simulated results for the varying scaling factor k. When the number of mediators was set at 200, yielding a p/n ratio to 4, the largest TPR for identifying mediators with nonzero effects is attained at k = 3 and 4. This corresponds to reduced dimensions of 38 and 51, respectively. When the number of mediators is increased to 500, leading to a p/n ratio of 10, the TPR peaks at k = 8, giving a reduced dimension of mediators at 102. In the case of 1000 mediators, where the p/n ratio stands at 20, the best TPR occurs at k = 15 or k = 20, resulting in reduced dimensions of 192 or 256, respectively. These findings support our proposition that the p/n ratio influences the optimal choice of k. The understanding of the motivational forces behind this is that when the reduced dimension d is increased, the sure screening property of SIS method is more likely to capture a greater number of true signals. Based on the findings from our simulations, with a fixed sample size of n = 50, it is advisable for researchers to select the k value in accordance with the p/n ratio.

We evaluated seven regularization approaches: the proposed LASSO (SMD.S_L), the proposed Pathway LASSO (SMD.S_P), LASSO, Pathway LASSO, MCP, DBL, and AdL. Simulations were conducted under both small and large sample scenarios. In the small sample setting, the sample size was fixed at n = 50 and the number of mediators (p) varied from 30, 50, 100, 150, to 200. In the large sample setting, p was fixed at 200 and n varied from 50, 100, 200, 300, to 500. Figure 4 presents the model performance in term of Youden Index, TNR, and TPR.

In terms of TPR, the proposed LASSO and proposed Pathway LASSO substantially outperformed the other five approaches for both small sample and large sample scenarios. In a small sample case with n = 50 and p = 200, their TPRs were 0.741 and 0.819, compared with 0.425 for LASSO and 0.403 for Pathway LASSO. The TPR of IE for MCP, DBL, and AdL were only 0.043, 0.191, and 0.109, respectively. For a large sample case with n = 500 and p = 200, the TPRs of both proposed LASSO and proposed Pathway LASSO reached 0.970, slightly larger than MCP (0.933) and DBL (0.929). In contrast, LASSO, Pathway LASSO, and AdL identified only half of the true signals, with TPRs ranging from 0.513 to 0.600.

In evaluating Youden Index, our proposed LASSO and Pathway LASSO consistently ranked among the top four methods, effectively balancing TPR and TNR across both small and large sample scenarios. This reflects their superior ability to detect true signals while correctly excluding non-signals. Although Pathway LASSO achieved slightly better TNR than our approaches, this improvement came at the expense of markedly lower TPR, resulting in Youden indices that did not exceed 0.452, even under large-sample conditions.

MCP and DBL performed best in large sample, low dimensional settings (e.g., p = 200 with n = 300 or 500), where their Youden indices surpassed those of our proposed methods. However, in small sample scenarios, both methods produced the poorest outcomes in terms of both Youden index and TPR. In contrast, our proposed LASSO and Pathway LASSO consistently delivered the strongest performance in high-dimensional settings (i.e., n < p), across both small and large sample cases, underscoring their robustness and applicability to high-dimensional models