The study was carried out in accordance with the Declaration of Helsinki and the recommendations of the Institutional Review Board at Beth Israel Deaconess Medical Center with written informed consent from all subjects.

Retrospective data was obtained from 36 adults (17 females) of mean age 62.9 years (range: 50–79 years), who had participated in research at the Berenson-Allen Center for Noninvasive Brain Stimulation at Beth Israel Deaconess Medical Center between May 2012 and May 2015. The participants were drawn from different populations: nine participants (four males, mean ± SD age: 67.7 ± 6.9 years) had a probable diagnosis of mild-to-moderate AD (McKhann et al., 2011) with a CDR = 1.0 and an MMSE score between 18 and 23; 15 participants (nine males, mean ± SD age: 63.4 ± 7.3 years) had Type-2 diabetes (T2DM) but were otherwise cognitively intact (MMSE ≥ 27), and the remaining 12 healthy controls (six males, mean ± SD age: 58.6 ± 9.1 years) were both cognitively intact (MMSE ≥ 27) and non-diabetic (hemoglobin A1c < 6.2%). AD participants consisted of individuals who were randomized to a Sham-control group for a proof-of-principle study on the combined impact of daily rTMS and cognitive training (Brem et al., manuscript submitted for publication). T2DM and control participants were recruited for a study on cortical plasticity in T2DM (Fried et al., 2016). None of the participants had any unstable medical condition or comorbidity. Saliva was obtained from 24 participants (10 controls, 10 T2DM, 4 AD) to assess BDNF and APOE polymorphisms. All participants underwent anatomical MRI scan, structured neurological exam, medical history review, formal neuropsychological testing, and two identical TMS visits. Median time between TMS visits was 14 days (range: 2–344 days). Average (±SD) start time for the two TMS visits was 10:57 (±1:07) for Visit-A and 10:37 (±0:55) for Visit-B. Blood glucose levels were assessed in all T2DM subjects at the beginning of each TMS visit for the purpose of establishing that glucose levels were within a normative range defined a priori as 80–200 mg/dL.

A T1-weighted anatomical MRI scan was obtained in all participants and used for neuronavigation. Scans were completed on a 3T scanner (GE Healthcare, Ltd., United Kingdom) using a 3D spoiled gradient echo sequence: 162 axial-oriented slices for whole-brain coverage; 240-mm isotropic field-of-view; 0.937-mm × 0.937-mm × 1-mm native resolution; flip angle = 15°; TE/TR ≥ 2.9/6.9 ms; duration ≥ 432 s.

All parameters used in the study conformed to current recommended guidelines for the safe application of TMS endorsed by the IFCN (Rossi et al., 2009; Rossini et al., 2015). A Navigated Brain Stimulation system (Nexstim Plc, Finland) was used to identify the hand region of the left primary motor cortex and ensure consistent targeting throughout each TMS visit. At the first visit, the hotspot for the FDI muscle was marked on the participant’s MRI. The hotspot was defined as the site from which single-pulse TMS elicited MEPs that were more consistent and higher in amplitude in FDI than in either the abductor pollicis brevis or abductor digiti minimi muscles. The hotspot was reassessed at the second visit using the first visit as a reference. Following IFCN guidelines (Rossini et al., 2015), RMT (using both mono- and biphasic pulses) and AMT (using biphasic pulses) were measured. We defined RMT as the lowest intensity of stimulation that elicited MEPs ≥50 μV in at least five of ten pulses in the relaxed FDI, and AMT as the lowest intensity that elicited MEPs ≥200 μV in at least 5 of 10 pulses with the FDI slightly contracted. Single and paired-pulse TMS trials were separated by a randomized 5000–6000 ms interval to avoid applying patterned repetitive stimulation.

Neurophysiological data included three motor thresholds (mono- and biphasic RMT and biphasic AMT; expressed as percent of maximum stimulator output intensity), two measures of cortico-motor reactivity (unconditioned MEPs elicited with the monophasic coil that was used to assess the effects of the paired-pulse paradigms and baseline MEPs elicited with the biphasic coil that were used to assess the impact of iTBS), three paired-pulse measures (SICI, LICI, ICF; with the average amplitude of the conditioned MEPs expressed as the percent change the amplitude of unconditioned MEPs), and the six post-iTBS time-points (Post05, Post10, Post20, Post30, Post40, and Post50; with the average amplitude of MEPs from each time-point expressed as the percent change from the pre-iTBS baseline average). From the iTBS modulation, three further measures of plasticity were calculated: the Max+, or greatest change in MEP amplitude across all six time-points; the summed area under-the-curve for the first 20-min post-iTBS (AUC_0-20), corresponding to the period of peak effect in neurotypical individuals (Wischnewski and Schutter, 2015); and the summed area under-the-curve across all post-iTBS time-points (AUC_0-50). The area under-the-curve was calculated as the summed products of the average %Δ in MEP amplitude at two consecutive time-points and the time in minutes between them.

For all neurophysiological measures, Cronbach’s α coefficients (Cronbach and Warrington, 1951) were calculated between the two visits to assess test–retest reliability. The α coefficient provides a measure of internal consistency of a set of items (Tavakol and Dennick, 2011), in this case, the same subjects tested on two separate occasions. The α coefficient was calculated for all subjects together and for each group (AD, T2DM, controls) separately using the free online software program Cronbach alpha (v1.0.3¹) (Wessa, 2016).

Reliability coefficients, such as the α, can be used to adjust effect sizes (Baugh, 2002; Wright, 2014). Using this approach, it is possible to predict how the reproducibility of a given measure might affect a hypothetical effect size, which in turn could be used in a sample size calculation that takes into consideration the reproducibility (or lack thereof) of the measure of interest. To illustrate this point and provide a resource for future studies, adjustments for each measure were made to a hypothetical Cohen’s d effect size of 0.5, which corresponds to a within-subject change of half a standard deviation, and is considered a medium effect size (Cohen, 1992). First, a hypothetical, or idealized, Cohen’s d is converted into an r (Cohen, 1988, p. 23):

This idealized r is then adjusted for unreliability using the Cronbach’s α (Wright, 2014):

Finally, the corrected r is converted back into an adjusted d (Friedman, 1968, p. 246):

To investigate factors associated with intraindividual variability, additional analyses were performed in JMP Pro (v12.1.0²) using a normal distribution and a two-tailed 95% confidence interval. Given the exploratory nature of these analyses, individual p-values were not adjusted for multiple comparisons and should be interpreted accordingly. For between-groups comparisons, the sample sizes in the present study provided 0.80 power to detect a medium effect size (Cohen’s d = 0.54). The first set of analyses concerned correlations between variables that were collected at each visit and thus were performed using the net difference between Visits A and B (Δ_B-A) so that the direction of change between visits was taken into account. These analyses included: (1) how differences in baseline MEP amplitude relate to differences in RMT (for both mono- and biphasic pulses); (2) how differences in SICI, LICI, and ICF relate to differences in unconditioned monophasic MEP amplitudes and RMT; and (3) how differences in post-iTBS measures relate to differences in baseline biphasic MEP amplitudes, RMT, and AMT. The second set of analyses concerned factors, such as group, gender, age, inter-visit interval, and BDNF and APOE polymorphisms, that were assessed only once per participant. Multiple linear regression analyses were performed on the absolute value of the inter-visit difference (|Δ_B-A|) to account for the amount of change between visits in either direction.

Figure 1 shows coefficients for all measures and all groups. Hypothetical effect- and sample sizes adjusted for these α coefficients are shown in Table 2. Following criteria for categorizing reproducibility in neurophysiological assessments (Portney and Watkins, 2009), we defined α ≥ 0.75 as high reproducibility, 0.50 ≤ α < 0.75 as moderate reproducibility, 0.25 ≤ α < 0.50 as low reproducibility, and α < 0.25 as very low to no reproducibility.

Considering all groups combined, the three motor thresholds had high reproducibility (α’s > 0.90). For baseline MEPs elicited at 120% of RMT, monophasic (α = 0.78) and biphasic pulses (α = 0.62) showed high and moderate reproducibility, respectively. Among the paired-pulse measures, reproducibility was moderate for SICI (α = 0.71) and LICI (α = 0.54), while ICF had very low to no reproducibility (α = 0.25). All post-iTBS measures demonstrated low reproducibility (α’s = 0.29–0.48); except for Post10 (α = 0.18) and Post50 (α = 0.17), which were not reproducible.

Considering each group separately, α coefficients tended to be higher for the AD group than for T2DM and controls. In particular, the AD group demonstrated high reproducibility for biphasic MEPs (α = 0.90), all three paired-pulse protocols (α’s > 0.81), Post10 (α = 0.81), and AUC_0-20 (α = 0.82). Further, reproducibility in AD was moderate (α’s = 0.53–0.65) for all remaining post-iTBS measures, except Post50, which was not reproducible (α = 0.16). By comparison, both controls and T2DM individuals showed moderate reproducibility (α’s = 0.50–0.67) for baseline monophasic MEPs and no reproducibility (α’s = 0.11–0.19) for baseline biphasic MEPs. One positive outlier was LICI, which showed high reproducibility in controls (α = 0.98).

For measures assessed with a monophasic pulse, there was no significant relationship between the Δ_B-A of baseline MEP amplitudes and the Δ_B-A of RMT, R₃₁ = 0.01, p = 0.934. Similarly, there were no significant relationships between the Δ_B-A of any of the paired-pulses measurements with the Δ_B-A of either RMT or baseline MEP amplitudes (|R|’s < 0.27, p’s > 0.13).

When using a biphasic coil, the Δ_B-A of baseline MEP amplitudes was significantly correlated with the Δ_B-A of RMT, R₃₄ = -0.35, p = 0.035. Specifically, a 1% (maximum stimulator output) increase in the net difference of RMT was associated with a 70-μV decrease in the net difference of baseline MEP amplitude (Figure 2A). Further, the Δ_B-A’s for all iTBS plasticity measures except Post30 and Post40 were significantly correlated with the Δ_B-A of biphasic baseline MEPs amplitudes (R’s<-0.36, p < 0.04). In all cases, an increase in the net difference of baseline MEP amplitudes was associated with a decrease in the inter-visit difference of post-iTBS facilitation. This relationship was most apparent for AUC_0-20, where a 1-mV increase in the inter-visit difference of baseline MEP amplitude was associated with a 390 (%Δ^∗min) decrease in the net difference of the AUC (Figure 2B). By contrast, there were no significant relationships between the Δ_B-A of any of the iTBS plasticity measures with the Δ_B-A of either RMT or AMT (|R|’s < 0.32, p’s > 0.07). These results indicate that as much as 23% of the visit-to-visit variability in iTBS plasticity measures can be accounted for by the variability in the baseline MEP amplitude, which in turn is impacted by the variability in RMT.

In the T2DM subjects, blood glucose levels did not differ significantly between visits (p > 0.1) and no significant relationships were observed between changes in blood glucose levels and changes in any TMS measure between visits (p’s > 0.2).

Controlling for the interval between visits, as well as the age and gender of participants, the linear models yielded no difference between groups in the |Δ_B-A| of any neurophysiological measure (F’s < 2.2, p’s > 0.13). These results indicate that the absolute amount of change between visits in motor thresholds as well as baseline reactivity, paired-pulse, and plasticity measures were equivalent across AD, T2DM and control participants at the 0.05 level.

The multiple regression analyses indicated a significant relationship between the |Δ_B-A| of monophasic MEPs and age, controlling for group, gender, and inter-visit interval (F_1,1 = 5.62, p = 0.025). Specifically, a 1-year increase in participant age was associated with a 0.03 mV increase in the absolute visit-to-visit difference in the amplitude of biphasic MEPs. Similarly, there was a significant relationship between the |Δ_B-A| of Post05 facilitation and inter-visit interval, controlling for group, gender, and age (F_1,1 = 6.42, p = 0.017). Specifically, a 1-day increase in the interval between visits was associated with a 0.03 mV decrease in the absolute inter-visit difference in the %Δ in MEP amplitudes at Post05. None of the other relationships were significant (F’s < 4.0, p’s > 0.05).

The multiple regression analyses demonstrated a significant effect of BDNF polymorphism on the |Δ_B-A| of several post-iTBS measures after controlling for Group (Figure 3). Specifically, intraindividual variability was higher for BDNF-Met carriers than BDNF-Val homozygotes for Post05 (F_1,1 = 6.76, p = 0.017) and Post50 (F_1,1 = 6.79, p = 0.017), and AUC_0-20 (F_1,1 = 4.99, p = 0.037). By comparison, there was no significant effect of APOE status on the |Δ_B-A| of any of TMS measures (F’s < 2.8, p’s > 0.1).

Epidural recordings of cortico-spinal volleys in conscious humans receiving TMS have shown that depending on its shape, current direction, and intensity, a TMS pulse can result in direct depolarization of the layer-V pyramidal cell (D-waves) and/or indirect depolarization through local circuits of interneurons (I-waves) (Burke et al., 1993; Sakai et al., 1997). The use of posterior–anterior monophasic pulse waveforms, which primarily elicit early I-waves, yields higher reproducibility in measures of cortico-motor reactivity over biphasic stimulation, which elicits a more complex pattern of D-waves, and early and late I-waves depending on the intensity of stimulation (Di Lazzaro et al., 2003). Future studies should directly explore how the effect size and reproducibility of single-pulse, paired-pulse, and TBS-based TMS measures are influenced by physical TMS parameters such as pulse shape and duration, and induced current direction relative to the motor cortex.

The reproducibility of biphasic MEP amplitude (α = 0.62) was noticeably lower than that of biphasic RMT (α = 0.93). Given that MEPs were assessed using 120% of RMT, there appear to be factors that do not impact RMT but do add variability to batches of MEPs elicited at suprathreshold intensities. Moreover, the inter-visit change in biphasic baseline MEP amplitudes was inversely related to that of biphasic RMT, suggesting that stimulus-response curve (i.e., the relationship between TMS intensity and MEP size) itself varies across visits. It has been speculated that, at threshold intensities, the second half of the biphasic pulse (posterior–anterior in the present study) contributes primarily to the MEP, while at suprathreshold intensities, there is increasing influence of the first half of the pulse (anterior-posterior in the present study) (Di Lazzaro et al., 2003; Barker, 2017). There is at least some theoretical evidence that biphasic TMS pulses might be less effective than monophasic (posterior–anterior) at probing the neuromodulatory effects of TBS (Di Lazzaro and Rothwell, 2014). However, to our knowledge, this has never been directly investigated. Regardless, the relatively low reproducibility of baseline MEP amplitudes is an area of concern given that it is the basis on which post-iTBS measures are derived. Furthermore, a significant portion of the inter-visit variance in post-iTBS measures is accounted for by visit-to-visit difference in baseline MEP amplitudes. These results are consistent with a recent study showing variability in MEPs within a session is predictive of the response to cTBS (Hordacre et al., 2017). Furthermore, the present results imply that improving the consistency of baseline measures (within and across sessions) would decrease the variability of post-iTBS measures as well. Given that changes in stimulus-response curves might contribute to the changing relationship between RMT and suprathreshold MEP amplitudes, future studies should explore whether the reproducibility of these measures could be improved by choosing stimulation intensities based on individual stimulus-response curves rather than a fixed percent of RMT.

The use of neuronavigation has been shown to increase the consistency of MEPs (Julkunen et al., 2009). However, even with neuronavigation, handheld TMS remains prone to slight deviations in the position, orientation, and inclination of the TMS coil. Robot arms, such as the TMS Robot (Axilum^® Robotics, Strasbourg) have been shown to improve the consistency of trial-to-trial MEPs over handheld TMS (Foucher et al., 2012; Ginhoux et al., 2013). Typically, MEP trials are elicited with individually spaced TMS pulses at a specific frequency range (e.g., 5000–6000 ms in the present study) with some random jitter incorporated to reduce the likelihood of train effects. Several recent studies combining TMS with concurrent EEG have highlighted to role of pre-stimulus oscillatory activity on cortico-motor excitability. Mäki and Ilmoniemi (2010) demonstrated that MEP amplitudes are inversely correlated with the amplitudes of pre-stimulus midrange-beta oscillations (15–18 Hz) over the stimulated motor cortex. Similarly, Iscan et al. (2016) showed that the variability of pre-stimulus power in the upper alpha band (10–12 Hz) was predictive of variability in ICF trials. Alternatively, Berger et al. (2014) suggest that the instantaneous phase of EEG oscillations across a range of frequencies is more predictive of MEP amplitude than spectral power. Together, these studies suggest that technological advances that allow for closed-loop systems to trigger TMS pulses timed to real-time EEG rhythms should result in more consistent MEPs. While these approaches offer the potential to improve the trial-to-trial consistency of MEPs, whether they would translate to greater reproducibility across visits remains to be explored.

While the multiple regression analyses did not show any significant difference between groups in terms of the absolute difference of the measures, participant age was significantly related to the absolute difference of monophasic baseline MEP amplitudes and inter-visit duration was significantly related to the absolute difference in Post05 facilitation, controlling for other factors such as group and gender. These results must be interpreted cautiously given the potential for Type-2 error in the present analysis. That the variability in baseline MEPs increases with age is not particularly surprising; however, its influence is not easily controlled, especially if the focus of the study is aging. It is more surprising that immediate iTBS aftereffects would be more consistent with greater time between visits. One possibility is that visits repeated under shorter intervals might be influenced by the iTBS from the previous visit, a phenomenon known as metaplasticity. It has been shown that the neuromodulatory effects of rTMS increase with consecutive daily application (Maeda et al., 2000; Valero-Cabré et al., 2008). Moreover, these metaplastic effects and state-dependent interactions may be altered by age (Opie et al., 2017) or neuropsychiatric disorders such as autism and Fragile X syndrome (Oberman et al., 2016). While the impact of multiple sessions separated by more than 24 h has not been well explored, a single application of iTBS was shown to alter the expression of GABA-precursor enzyme GAD67 for up to 7 days in the neocortex of rats (Trippe et al., 2009), suggesting the window for metaplastic effects might be longer than previously understood. Indeed, a follow-up analysis of the current data found that the absolute difference of Post05 facilitation was higher between visits conducted within 7 days than those separated by more than a week (Figure 4).

The multiple regression analyses showed that the absolute difference of several post-iTBS measures was higher in subjects with a BDNF-Met allele. While the generalizability of these findings is limited by the small sample size, they nonetheless provide insight into the debate over the role of BDNF polymorphisms in shaping the effects of neuromodulation. Several studies have reported a reduced impact of repetitive TMS in Met carriers (Cheeran et al., 2008; Cirillo et al., 2012; Lee et al., 2013; Chang et al., 2014; Di Lazzaro et al., 2015), still others have reported no difference (Li Voti et al., 2011; Mastroeni et al., 2013). The current results suggest that this divergence in the literature may be due to the fact that the BDNF Met allele leads to more variability in the response to neuromodulation rather than simply blunting its effects.

The present study was a retrospective analysis of data collected under different study protocols. As such there are a number of limitations, principally, the generally small and unequal sample sizes. While the Cronbach’s α is fairly robust to variations in sample size, the linear regression analyses are susceptible to limitations of sample size. As such, the analyses comparing absolute difference of visits between groups are likely underpowered to detect differences of the magnitude observed in the present study. Future studies with larger samples are needed to confirm the present findings. In addition, a number of the non-diabetic controls had hemoglobin A1c values indicating possible pre-diabetes, which may have contributed to the decreased reproducibility seen in this cohort. Further, HbA1c values were not available from the AD group, so the influence of impaired glucose metabolism could not be investigated in AD subjects, despite reports of high co-morbidity between AD and T2DM (Hölscher, 2011).