<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.3 20070202//EN" "journalpublishing.dtd">
<article xml:lang="EN" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">Front. Cell. Neurosci.</journal-id>
<journal-title>Frontiers in Cellular Neuroscience</journal-title>
<abbrev-journal-title abbrev-type="pubmed">Front. Cell. Neurosci.</abbrev-journal-title>
<issn pub-type="epub">1662-5102</issn>
<publisher>
<publisher-name>Frontiers Media S.A.</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="doi">10.3389/fncel.2022.845832</article-id>
<article-categories>
<subj-group subj-group-type="heading">
<subject>Cellular Neuroscience</subject>
<subj-group>
<subject>Original Research</subject>
</subj-group>
</subj-group>
</article-categories>
<title-group>
<article-title>Predicting the Effects of Repetitive Transcranial Magnetic Stimulation on Cognitive Functions in Patients With Alzheimer&#x00027;s Disease by Automated EEG Analysis</article-title>
</title-group>
<contrib-group>
<contrib contrib-type="author" corresp="yes">
<name><surname>Kayasandik</surname> <given-names>Cihan Bilge</given-names></name>
<xref ref-type="aff" rid="aff1"><sup>1</sup></xref>
<xref ref-type="corresp" rid="c001"><sup>&#x0002A;</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/1371044/overview"/>
</contrib>
<contrib contrib-type="author">
<name><surname>Velioglu</surname> <given-names>Halil Aziz</given-names></name>
<xref ref-type="aff" rid="aff2"><sup>2</sup></xref>
<xref ref-type="aff" rid="aff3"><sup>3</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/1619845/overview"/>
</contrib>
<contrib contrib-type="author">
<name><surname>Hanoglu</surname> <given-names>Lutfu</given-names></name>
<xref ref-type="aff" rid="aff4"><sup>4</sup></xref>
<uri xlink:href="http://loop.frontiersin.org/people/1799586/overview"/>
</contrib>
</contrib-group>
<aff id="aff1"><sup>1</sup><institution>Department of Computer Engineering, School of Engineering and Natural Sciences, Istanbul Medipol University</institution>, <addr-line>Istanbul</addr-line>, <country>Turkey</country></aff>
<aff id="aff2"><sup>2</sup><institution>Department of Women&#x00027;s and Childrens&#x00027; Health, Karolinska Institutet</institution>, <addr-line>Stockholm</addr-line>, <country>Sweden</country></aff>
<aff id="aff3"><sup>3</sup><institution>Functional Imaging and Cognitive-Affective Neuroscience Lab (fINCAN), Regenerative and Restorative Medicine Research Center (REMER), Health Sciences and Technology Research Institute (SABITA), Istanbul Medipol University</institution>, <addr-line>Istanbul</addr-line>, <country>Turkey</country></aff>
<aff id="aff4"><sup>4</sup><institution>Department of Neurology, School of Medicine, Istanbul Medipol University</institution>, <addr-line>Istanbul</addr-line>, <country>Turkey</country></aff>
<author-notes>
<fn fn-type="edited-by"><p>Edited by: Katharina S. Rufener, University Hospital Magdeburg, Germany</p></fn>
<fn fn-type="edited-by"><p>Reviewed by: Sunghan Kim, East Carolina University, United States; Vera Moliadze, University Medical Center Schleswig-Holstein, Germany</p></fn>
<corresp id="c001">&#x0002A;Correspondence: Cihan Bilge Kayasandik <email>cbkayasandik&#x00040;medipol.edu.tr</email></corresp>
<fn fn-type="other" id="fn001"><p>This article was submitted to Cellular Neurophysiology, a section of the journal Frontiers in Cellular Neuroscience</p></fn></author-notes>
<pub-date pub-type="epub">
<day>19</day>
<month>05</month>
<year>2022</year>
</pub-date>
<pub-date pub-type="collection">
<year>2022</year>
</pub-date>
<volume>16</volume>
<elocation-id>845832</elocation-id>
<history>
<date date-type="received">
<day>30</day>
<month>12</month>
<year>2021</year>
</date>
<date date-type="accepted">
<day>11</day>
<month>04</month>
<year>2022</year>
</date>
</history>
<permissions>
<copyright-statement>Copyright &#x000A9; 2022 Kayasandik, Velioglu and Hanoglu.</copyright-statement>
<copyright-year>2022</copyright-year>
<copyright-holder>Kayasandik, Velioglu and Hanoglu</copyright-holder>
<license xlink:href="http://creativecommons.org/licenses/by/4.0/"><p>This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.</p></license></permissions>
<abstract>
<p>Alzheimer&#x00027;s disease (AD) is a progressive, neurodegenerative brain disorder that generally affects the elderly. Today, after the limited benefit of the pharmacological treatment strategies, numerous noninvasive brain stimulation techniques have been developed. Transcranial magnetic stimulation (TMS), based on electromagnetic stimulation, is one of the most widely used methods. The main problem in the use of TMS is the existence of large individual variability in the results. This causes a waste of money, time, and more importantly, a burden for delicate patients. Hence, it is a necessity to form an efficient and personalized TMS application protocol. In this paper, we performed a machine-learning analysis to see whether it is possible to predict the responses of patients with AD to TMS by analyzing their electroencephalography (EEG) signals. For that purpose, we analyzed both the EEG signals collected before and after the TMS application (EEG1 and EEG2, respectively). Through correlating EEG1 and repetitive transcranial magnetic stimulation (rTMS) outcomes, we tried to see whether it is possible to predict patients&#x00027; responses before the treatment application. On the other hand, by EEG2 analysis, we investigated TMS impacts on EEG, more importantly if this impact is correlated with patients&#x00027; response to the treatment. We used the support vector machine (SVM) classifier due to its multiple advantages for the current task with feature selection processes by stepwise linear discriminant analysis (SWLDA) and SVM. However, to justify our numerical analysis framework, we examined and compared the performances of different feature selection and classification techniques. Since we have a limited sample number, we used the leave-one-out method for the validation with the Monte Carlo technique to eliminate bias by a small sample size. In the conclusion, we observed that the correlation between rTMS outcomes and EEG2 is stronger than EEG1, since we observed, respectively, 93 and 79% of accuracies during our data analysis. Besides the informative features of EEG2 are focused on theta band, it indicates that TMS is characterizing the theta band signals in patients with AD in direct relation to patients&#x00027; response to rTMS. This shows that it is more possible to determine patients&#x00027; benefit from the TMS at the early stages of the treatment, which would increase the efficiency of rTMS applications in patients with Alzheimer&#x00027;s disease.</p></abstract>
<kwd-group>
<kwd>Alzheimer&#x00027;s disease</kwd>
<kwd>machine learning</kwd>
<kwd>support vector machine</kwd>
<kwd>repetitive transcranial magnetic stimulation (rTMS)</kwd>
<kwd>personalized treatment</kwd>
<kwd>artificial neural network</kwd>
</kwd-group>
<counts>
<fig-count count="8"/>
<table-count count="5"/>
<equation-count count="2"/>
<ref-count count="54"/>
<page-count count="14"/>
<word-count count="9692"/>
</counts>
</article-meta>
</front>
<body>
<sec sec-type="intro" id="s1">
<title>Introduction</title>
<p>Alzheimer&#x00027;s disease (AD) is the most common type of dementia, characterized by a progressive loss of cognitive ability (Lane et al., <xref ref-type="bibr" rid="B36">2018</xref>). The most prevalent signs of Alzheimer&#x00027;s disease include memory and orientation problems, as well as executive and motor dysfunctions (Guarino et al., <xref ref-type="bibr" rid="B21">2019</xref>). At the molecular level, AD has been linked to synaptic impairment, as well as amyloid-beta and tau protein buildup (Bloom, <xref ref-type="bibr" rid="B11">2014</xref>). As a result of this disease&#x00027;s progressive nature, loss of cognitive and motor function eventually leads to death.</p>
<p>Because there is currently no cure for AD, healthcare providers must rely on treatments aimed at delaying the illness&#x00027;s progression and lowering cognitive damage. The majority of these therapeutic methods rely on pharmacological substances (Cummings et al., <xref ref-type="bibr" rid="B17">2019</xref>). Pharmacological medicines, on the other hand, are unable to provide an effective treatment for Alzheimer&#x00027;s disease, and, as a result of their extensive side-effect profile, they also cause other issues (Sharma, <xref ref-type="bibr" rid="B47">2019</xref>). As a result, scientists have begun to investigate options other than pharmaceutical medications for treating this disease.</p>
<p>Transcranial magnetic stimulation (TMS), which has long been used successfully in the treatment of depression (Sonmez et al., <xref ref-type="bibr" rid="B48">2019</xref>), is now being employed as a possible alternative treatment for other neurodegenerative diseases such as Alzheimer&#x00027;s disease (Freitas et al., <xref ref-type="bibr" rid="B19">2011</xref>; Mimura et al., <xref ref-type="bibr" rid="B41">2021</xref>), Parkinson&#x00027;s disease, and other dementias (Begemann et al., <xref ref-type="bibr" rid="B7">2020</xref>; Hanoglu et al., <xref ref-type="bibr" rid="B24">2021</xref>). TMS is a noninvasive approach that uses a magnetic field created outside the body to stimulate or inhibit specific areas of the brain by applying it to the scalp (Hamid et al., <xref ref-type="bibr" rid="B23">2019</xref>). In some ways, TMS treatment appears to be superior than the pharmaceutical approach. TMS can be favored over pharmaceutical medicines due to its ease of use, low side effect profile, and excellent behavioral and cognitive benefits.</p>
<p>However, the effect of TMS on brain activity has not been fully understood (Sale et al., <xref ref-type="bibr" rid="B45">2015</xref>). It is observed that some patients respond positively to treatment, while other gives no response (Dunlop et al., <xref ref-type="bibr" rid="B18">2016</xref>). Unfortunately, this heterogeneity of treatment responses among patients has no obvious connection with the clinical measurements such as personality, demographic, sex, and age (Bailey et al., <xref ref-type="bibr" rid="B4">2019</xref>). Hence, advanced computational methods are needed to analyze the connection between patients&#x00027; certain properties and the TMS treatment to predict the patients&#x00027; possible response to the treatment. That is necessary to generate efficient treatment strategies.</p>
<p>There are some studies with the purpose of predicting the TMS response of patients with psychiatric diseases. Chekroud et al. attempted to match depression patients with the treatment according to the patients&#x00027; Quick Inventory of Depressive Symptomatology (QIDS) test rates (Chekroud et al., <xref ref-type="bibr" rid="B15">2016</xref>). Another study analyzed structural magnetic resonance imaging results to predict the treatment outcomes of patients with schizophrenia (Koutsouleris et al., <xref ref-type="bibr" rid="B34">2018</xref>). There are other studies focus on predicting the outcomes of treatments other than TMS, which are out of our focus on this study (Garnaat et al., <xref ref-type="bibr" rid="B20">2019</xref>).</p>
<p>On the other hand, there are only a few studies on machine-learning analysis of EEG in predicting the effectiveness of rTMS. Corlier et al. (<xref ref-type="bibr" rid="B16">2019</xref>) generated a predictive model based on the functional connectivity on EEG, and as our purpose, they tried to obtain the most informative features for the rTMS benefit prediction. For the classification task, they used Elastiknet (Zou and Hastie, <xref ref-type="bibr" rid="B54">2005</xref>), which randomly selects the subset of features and tries to maximize the classification accuracy over all random trials. Hasanzadeh et al. combined linear and non-linear features from EEG to distinguish TMS responders from non-responders, again on patients with depressive disorders. They also used the multi-source nature of the EEG data and tried to see the most significant electrode locations beside most significant EEG features. They used kNN classifier which is a primitive and low-capacity method that would fail on slightly complicated tasks (Hasanzadeh et al., <xref ref-type="bibr" rid="B25">2019</xref>). Bailey et al. analyzed the electroencephalogram (EEG) signal of patients with depression to see whether it is possible to predict patients&#x00027; response to rTMS treatment. They used band power and connectivity features with a linear SVM classifier to generate their prediction model (Bailey et al., <xref ref-type="bibr" rid="B4">2019</xref>).</p>
<p>Besides these studies focusing on depressive disorders, to the best of knowledge, there is no significant attempt to generate a computational model for EEG analysis with the purpose of predicting the TMS response of patients with AD. Since AD is a neurodegenerative disease, the TMS therapy for patients with AD have more factors to be considered than depressive disorders. Besides, TMS effect is more variable in patients with AD than patients with depressive disorders (Iimori et al., <xref ref-type="bibr" rid="B31">2019</xref>; Weiler et al., <xref ref-type="bibr" rid="B52">2020</xref>). Hence, our task is more complicated, and the discussed methods above would not be applicable to the current task.</p>
<p>The EEG forms a critically high-dimensional dataset whereas it is highly challenging to collect a large number of samples for neurological disorders. Hence, the EEG data analysis for cognitive neuroscience studies is highly challenging. Due to the limited number of samples, studies in the literature mostly reduce the EEG data to a very low dimension by ignoring the advantage of the multi-source nature of EEG data.</p>
<p>In this study, we examined the possibilities to maximize the efficiency of the TMS treatments by predicting patients&#x00027; responses before the TMS application by focusing on preserving the advantage of the multi-source nature of the EEG data. For that purpose, we analyzed pre-TMS EEG signals, which will be named as EEG1 throughout the paper for simplicity. Besides that, we investigated whether a dose application of TMS would increase the correlation between EEG2 and rTMS. For that purpose, we analyzed the post-TMS EEG signals, which will be named as EEG2 in the paper for simplicity (<xref ref-type="fig" rid="F1">Figure 1</xref>). Throughout our analysis besides generating a prediction model, we seek more insight on EEG and TMS relation in patients with AD. For that purpose, we analyzed EEG1 and EEG2 from various perspectives. We measured the sensitivity and stability of the correlations between EEG1/EE2 and rTMS outcomes with respect to the small sample number. We applied multiple dimension reduction and classification methods, compared their performances, and interpreted the results.</p>
<fig id="F1" position="float">
<label>Figure 1</label>
<caption><p>The framework of the data acquisition process. For each patient, EEG signal collection is applied once before the TMS treatment application, that EEG signal is labeled as EEG1 in the manuscript. Another EEG signal is collected after the TMS treatment application which is labeled as EEG2. In this study, EEG1 and EEG2 were separately analyzed. Through the EEG1, we gather the information if it is possible to predict the patients&#x00027; response to the treatment. On the other hand, by the analysis of EEG2, we sought for the effects of TMS application on patients&#x00027; EEG signal if it is dependent on the patients&#x00027; benefit from the treatment.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fncel-16-845832-g0001.tif"/>
</fig>
<p>The rest of the paper is organized as follows: section Materials and Methods provides the details of the dataset collection, data preprocessing, and the brief explanation of applied feature selection and classification techniques. It explains the process of generating the ground-truth labels and optimizing the machine-learning algorithms. Section Results describes all the applied data analysis steps. It explains our motive for further data analysis for more insight on the effects of TMS treatment on patients&#x00027; EEG signals, the sensitivity measure of the data analysis process, comparative results of feature selection, and classification algorithms. Section Discussion discusses the obtained machine-learning results. It explains the interpretation of these results from a clinical view and lists the future motivation for this study.</p></sec>
<sec sec-type="materials and methods" id="s2">
<title>Materials and Methods</title>
<sec>
<title>Subjects</title>
<p>To determine the minimum sample size of patients with AD to be applied TMS, G<sup>&#x0002A;</sup>power (version 3.1.6.6) software was used. We discovered that for 90% power with a significance level of = 0.05, a sample size of at least <italic>n</italic> = 13 participants is required, and thus, we decided to include at least <italic>n</italic> = 14 subjects in the investigation. Our research included 14 patients with AD (10 women, ages 56&#x02013;84). Based on their responses to a TMS safety-screening questionnaire, all patients were eligible for TMS operations. The Ethical Committee of Istanbul Medipol University accepted our research with the number of Ethical Report: 10840098&#x02013;604.01.01&#x02013;E.45425. In Istanbul Medipol University Hospital, all patients and their relatives submitted written informed consent, and all processes were followed. During the study period, patients were assessed while on usual therapy with no changes to their medical treatment.</p>
<p>The clinical diagnosis of AD was made by an experienced neurologist according to the NINCDS-ADRDA (McKhann et al., <xref ref-type="bibr" rid="B40">2011</xref>) criteria. The Clinical Dementia Rating Scale (CDR) scores of the patients participating in the study were between 1-2. The patients were using acetylcholinesterase inhibitors and/or memantine as a medicine. Exclusion criteria were defined as having metallic implants, unable to walk independently, being physically disabled, history of alcohol or substance abuse, mental illness including schizophrenia and delirium, and epileptic disease, seizure, brain tumor, or trauma.</p></sec>
<sec>
<title>Labeling the Patients as TMS&#x0002B; and TMS&#x02013;</title>
<p>In the direction of deciding whether a patient is positively or negatively affected by the TMS treatment, we considered only the Mini-Mental State Examination (MMSE) scores of neuropsychometric tests (NPTs). For that purpose, if a certain patient&#x00027;s MMSE score increased after the TMS treatment, it was labeled as TMS&#x0002B;; similarly, if a certain patient&#x00027;s MMSE score decreased after the TMS treatment, it was labeled as TMS&#x02013;. In our dataset, there were two patients with constant MMSE scores before and after the TMS treatment. There was no certain procedure for labeling these patients. However, during the classification analysis, we observed that their EEG signals behave like a TMS&#x0002B; patient (by labeling them as TMS&#x0002B; we increased the classification results by 20%). Hence, if the MMSE score was constant, we considered these patients as they benefit the treatment, hence TMS&#x0002B;.</p></sec>
<sec>
<title>Experimental Design</title>
<p>Each patient underwent 10 sessions of rTMS therapy over the course of 2 weeks. Patients provided the following information 1 week prior to TMS. The Turkish version of the Mini-Mental State Examination (MMSE) scores was used to assess the overall cognitive status. We also acquired a resting-state EEG1 and EEG2 recording to assess which patient would benefit from the TMS application using machine learning. The MMSE and EEG data were taken from patients again after 2 weeks of TMS therapy.</p></sec>
<sec>
<title>EEG Recording and Preprocessing</title>
<p>Electroencephalography recordings were applied in a dimly illuminated and electrically shielded room (Faraday cage). According to the international 10&#x02013;20 system, EEG was enrolled with the Brain Amp amplifier DC system. EEG band level was between 0.01 and 250 Hz and sampling rate was 500 Hz. Recording electrodes were used with 32 Ag&#x02013;AgCl electrodes. All electrodes were mounted by means of Easy-cap except reference, ground, and EOG electrodes. According to the head size of patients, we changed Easy-cap and chose a suitable one. One reference electrode was bonded with a latch to the left earlobe, and the other reference electrode was also bonded to the front of the right earlobe. The ground electrode was also stuck together behind the right earlobe. EOG electrodes were bonded to the right side of the forehead and to below the left eye. All electrode impedances were decreased to &#x0003C;15 k&#x003A9; (ground = 1 &#x003A9;).</p>
<p>Before and after TMS therapy, two spontaneous EEG data were collected from patients. The spontaneous EEG protocol was 4-min eyes-open recording and 4-min eyes-closed recording. A video recorder recorded all the EEG processes.</p>
<p>To prepare the EEG data for further analysis, we used Brain Vision Analyser software. Preprocessing steps included six different stages. First of all, using infinite impulse response (IIR) section, we filtered raw data low (0.1 Hz) and high cutoff (60 Hz) and enabled the notch filter (50 Hz). To remove the blink-reflex, we used independent component analysis (ICA) after then we performed inverse ICA command. We segmented the data manually as open eyes&#x02013;closed eyes. Open and closed eyes data were recorded for 4 min. Using again segmentation, we created 1-s size of segments and skipped bad intervals. Finally, we rejected artifacts such as blink, and muscle movement manually and saved data in MAT format.</p>
<p>Manual artifact rejection was performed by examining each 1-s epoch one by one. Only one experienced person (HAV) performed the preprocessing processes to ensure standardization.</p></sec>
<sec>
<title>Identification of Stimulation Locations and rTMS Parameters</title>
<p>We determined a stimulation coordinate for each patient using a 10&#x02013;20 EEG system. We aim to stimulate P3 in other words the left posterior parietal cortex (Herwig et al., <xref ref-type="bibr" rid="B30">2003</xref>).</p>
<p>For the TMS application, we used the MAG &#x00026; More company&#x00027;s PowerMag Research tool. The coil type of the TMS device was an &#x0201C;eight&#x0201D; coil with a winding diameter of 2 x 70 mm. We assessed the motor threshold by stimulating the TMS primary motor region, which needed at least 5 of 10 consecutive pulses to cause contraction in the abductor pollicis brevis muscle, at the start of the first study week. TMS coils were held over the left parietal cortex with the handle-oriented 45 degrees to the midsagittal line, generating a posterior to anterior current (Brown et al., <xref ref-type="bibr" rid="B12">2014</xref>). The coil was manually moved to the coordinates displayed on the neuro-navigation device. During the 20-min stimulation period, rTMS was delivered at a 100% resting-motor threshold (Hermiller et al., <xref ref-type="bibr" rid="B29">2019</xref>). The rTMS procedure was 20 Hz for 2 s of stimulation duration (totaling 1,640 pulses each session), followed by a 28-sinterstimulus period (Wang et al., <xref ref-type="bibr" rid="B50">2014</xref>).</p></sec>
<sec>
<title>Data Analysis</title>
<p>To justify our numerical analysis, we performed multiple feature selection and classification methods and compare their results. The complete analysis framework can be seen in <xref ref-type="fig" rid="F2">Figure 2</xref>.</p>
<fig id="F2" position="float">
<label>Figure 2</label>
<caption><p>Analysis framework: Our analysis is based on frequency band features extracted from EEG1 and EEG2. To justify our numerical analysis framework, we examined and compared the performances of different feature selection and classification techniques. We reported all numerical results.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fncel-16-845832-g0002.tif"/>
</fig>
<sec>
<title>Feature Extraction</title>
<p>The raw form of the EEG signal is highly complicated and difficult to interpret. If we focus only on the analysis task, EEG is very large and possibly stores redundant information. Hence, a dimension reduction or feature extraction step is required before classification. In our study, first, we applied the fast Fourier transform to extract the wavelengths. In the literature, the EEG analysis focuses on five main frequency groups such as delta (0.5&#x02013;4), theta (4&#x02013;7), alpha (7.5&#x02013;13), beta (15&#x02013;28), gamma (29&#x02013;48), and we followed the same procedure. For each of these wavelengths, we calculated the power spectral density as the magnitude squared of the filtered data. Then, we got the average power spectral density for non-overlapping windows with a length of 2 s. The feature extraction window size was numerically determined. We observed that the classification accuracy makes a peak when the feature extraction window size is set to 2 min (<xref ref-type="table" rid="T1">Table 1</xref>; <xref ref-type="fig" rid="F3">Figure 3</xref>). Instead of obtaining the power of the frequency band, which is only a scaler, we used this new signal to represent the EEG data. By that, we formed a denser and more informative representation of EEG while preserving more information about the dataset.</p>
<table-wrap position="float" id="T1">
<label>Table 1</label>
<caption><p>The effect of different feature extraction window sizes on the data classification performance.</p></caption>
<table frame="hsides" rules="groups">
<thead><tr>
<th/>
<th valign="top" align="center"><bold>Feature extraction with window size 1 s</bold></th>
<th valign="top" align="center"><bold>Feature extraction with window size 2 s</bold></th>
<th valign="top" align="center"><bold>Feature extraction with window size 4 s</bold></th>
</tr>
</thead>
<tbody>
<tr>
<td valign="top" align="left">Average accuracy</td>
<td valign="top" align="center">73%</td>
<td valign="top" align="center"><bold>78</bold>%</td>
<td valign="top" align="center">59%</td>
</tr>
</tbody>
</table>
<table-wrap-foot>
<p><italic>The accuracies are recorded during the leave one out validation of EEG2. The maximum accuracy is in bold</italic>.</p>
</table-wrap-foot>
</table-wrap>
<fig id="F3" position="float">
<label>Figure 3</label>
<caption><p>Comparison of performances of leave-one-out validation for different window sizes for feature extraction process. <bold>(A)</bold> Results with window size length 1 s, <bold>(B)</bold> Results with window size length 4 s. The results with window size length 2 s can be seen in <xref ref-type="fig" rid="F5">Figure 5A</xref>. As it is seen in the Figure, 2-swindows give the best classification performance.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fncel-16-845832-g0003.tif"/>
</fig>
<p>After this basis feature extraction step, we examined the performance of different dimension reduction techniques such as wavelet transform, principal component analysis, and weight hierarchy determined by stepwise linear discriminant analysis and SVM classifier. Through these applications, we aimed to validate our numerical analysis and investigate the data from the different perspectives to get the core information from EEG signal. The contribution of these dimension reduction techniques to the performance of the data analysis is given in the Results section.</p>
<sec>
<title>Principal Component Analysis</title>
<p>Principal component analysis (PCA) is one of the most widely applied dimension reduction techniques. The main idea of the PCA is finding the most informative dimensions of the data and projecting the data into the subspace by the determined informative dimensions (Jolliffe, <xref ref-type="bibr" rid="B32">2014</xref>). For that purpose, mathematically, all eigenvalues of the D<sup>T</sup>D are determined, where D<sup>T</sup>D is called the covariance matrix of the dataset D. Covariance matrix basically shows how each feature is related to all others. Therefore, by the eigenvalues, PCA examines the features of the dataset and tries to find the most informative and independent ones. For that purpose, the largest normed eigenvalues are determined, and associated eigenvectors are used to project the dataset into a lower-dimensional space. The ideal dimension of the projected dataset is a crucial point. In this work, we considered the decay of the eigenvalues to determine the optimal dimension for the PCA. Besides that, by considering the small sample number, projections into very low-dimensional spaces were also examined.</p></sec>
<sec>
<title>Wavelet Transform</title>
<p>The theory of wavelets emerged in the 1990s to address some limitations of classical Fourier analysis, namely, its limitations in providing local features of signals. The advantage of the wavelet transform is its ability to capture the local properties of the signals, on the contrary to Fourier transform. Consequently, in the last decade, wavelet transform is mostly preferred for analysis when the local information matters. Besides, it is shown that the wavelet transform is very successful in finding the most informative representation of the data in the classification tasks (al-Qerem et al., <xref ref-type="bibr" rid="B1">2020</xref>). Consequently, in many studies, wavelet coefficients are used both for dimension reduction and feature selection purposes (Hazarika et al., <xref ref-type="bibr" rid="B27">1997</xref>; Qu et al., <xref ref-type="bibr" rid="B42">2003</xref>).</p>
<p>Several properties of the wavelet system, including regularity and fast decay, can be controlled by the choice of mother function (Wojtaszczyk, <xref ref-type="bibr" rid="B53">1997</xref>; Heil, <xref ref-type="bibr" rid="B28">2010</xref>). For the given study, we preferred to use Daubechies as the mother function of order four. These parameters are determined numerically to have maximal efficiency during the computational analysis. After the wavelet coefficients were calculated, the 10% largest coefficients were collected to represent the samples. The optimal number of coefficients to be used was again determined by optimizing the efficiency of the numerical analysis.</p></sec>
<sec>
<title>Stepwise Linear Discriminant Analysis</title>
<p>Linear discriminant analysis (LDA) is one of the basic dimension reduction and supervised data classification techniques. The idea of LDA is finding a separating hyperplane by maximizing the inter-class variance and minimizing the intra-class variance (Bishop and Nasrabadi, <xref ref-type="bibr" rid="B9">2006</xref>). Through that idea, LDA has the capability to form a successful separating hyperplane even in the cases, PCA fails. Hence, LDA has several applications in data analysis fields both for dimension reduction and for data classification. Stepwise linear discriminant analysis (SWLDA) forms this separating LDA hyperplane step-by-step to find the set of features to form the best separating hyperplane. Hence, SWLDA is a linear classifier and determines the most informative features at once.</p>
<p>In our analysis, we applied SWLDA both as a feature selection technique and as a linear classifier. For that purpose, we used the toolbox by brain&#x02013;computer interface (BBCI) (Blankertz et al., <xref ref-type="bibr" rid="B10">2016</xref>). Due to the mathematical theory of LDA, we cannot select more features than the number of samples. Hence, the maximum number of features to be selected was set to the number of training samples.</p></sec>
<sec>
<title>Support Vector Machine Classifier Weights</title>
<p>Support vector machine (SVM) is a model-based supervised data classification technique. The main idea of SVM is to specify a linear subspace that separates samples of two different classes with a maximum margin. It generates a decision function and clusters each subject to one side of the determined subspace. The source of the high success of SVM lies in its ability to handle high-dimensional datasets and the logic called kernel trick. If the data cannot be linearly separated in its own space, it is transferred to another space with a kernel function. This space is where the data can be separated from each other in the most optimal way, theoretically, this space does not need to be known by the user, and mathematical operations can be done without any knowledge of the space (Burges, <xref ref-type="bibr" rid="B13">1998</xref>). SVM outputs the decision function that has the form:</p>
<disp-formula id="E1"><mml:math id="M1"><mml:mtable columnalign="left"><mml:mtr><mml:mtd><mml:mi>f</mml:mi><mml:mrow><mml:mo stretchy="false">(</mml:mo><mml:mrow><mml:mi>x</mml:mi></mml:mrow><mml:mo stretchy="false">)</mml:mo></mml:mrow><mml:mo>=</mml:mo><mml:mstyle displaystyle="true"><mml:munderover accentunder="false" accent="false"><mml:mrow><mml:mo>&#x02211;</mml:mo></mml:mrow><mml:mrow><mml:mi>m</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow><mml:mrow><mml:mi>N</mml:mi></mml:mrow></mml:munderover></mml:mstyle><mml:msub><mml:mrow><mml:mi>a</mml:mi></mml:mrow><mml:mrow><mml:mi>m</mml:mi></mml:mrow></mml:msub><mml:msub><mml:mrow><mml:mi>y</mml:mi></mml:mrow><mml:mrow><mml:mi>m</mml:mi></mml:mrow></mml:msub><mml:mi>K</mml:mi><mml:mrow><mml:mo stretchy="false">(</mml:mo><mml:mrow><mml:mi>x</mml:mi><mml:mo>,</mml:mo><mml:msub><mml:mrow><mml:mi>x</mml:mi></mml:mrow><mml:mrow><mml:mi>m</mml:mi></mml:mrow></mml:msub></mml:mrow><mml:mo stretchy="false">)</mml:mo></mml:mrow><mml:mo>&#x0002B;</mml:mo><mml:mi>b</mml:mi><mml:mtext>&#x000A0;</mml:mtext></mml:mtd></mml:mtr></mml:mtable></mml:math></disp-formula>
<p>Here, N is the number of support vectors, &#x003B1;<sub>m</sub> is the weighted components, y<sub>m</sub> is the label of the sample x<sub>m</sub>, b is the tolerance constant special for each class, and K is the kernel function. The K kernel function is a form of similarity measurement that plays a key role in representing and decomposing the data in space and is defined as follows:</p>
<disp-formula id="E2"><mml:math id="M2"><mml:mtable columnalign="left"><mml:mtr><mml:mtd><mml:mi>K</mml:mi><mml:mrow><mml:mo stretchy="false">(</mml:mo><mml:mrow><mml:msub><mml:mrow><mml:mi>x</mml:mi></mml:mrow><mml:mrow><mml:mi>i</mml:mi></mml:mrow></mml:msub><mml:mo>,</mml:mo><mml:mtext>&#x000A0;</mml:mtext><mml:msub><mml:mrow><mml:mi>x</mml:mi></mml:mrow><mml:mrow><mml:mi>j</mml:mi></mml:mrow></mml:msub></mml:mrow><mml:mo stretchy="false">)</mml:mo></mml:mrow><mml:mo>=</mml:mo><mml:mtext>&#x000A0;</mml:mtext><mml:mo>&#x0003C;</mml:mo><mml:mi>&#x003C6;</mml:mi><mml:mrow><mml:mo stretchy="false">(</mml:mo><mml:mrow><mml:msub><mml:mrow><mml:mi>x</mml:mi></mml:mrow><mml:mrow><mml:mi>i</mml:mi></mml:mrow></mml:msub></mml:mrow><mml:mo stretchy="false">)</mml:mo></mml:mrow><mml:mo>,</mml:mo><mml:mtext>&#x000A0;</mml:mtext><mml:mi>&#x003C6;</mml:mi><mml:mrow><mml:mo stretchy="false">(</mml:mo><mml:mrow><mml:msub><mml:mrow><mml:mi>x</mml:mi></mml:mrow><mml:mrow><mml:mi>j</mml:mi></mml:mrow></mml:msub></mml:mrow><mml:mo stretchy="false">)</mml:mo></mml:mrow><mml:mo>&#x0003E;</mml:mo></mml:mtd></mml:mtr></mml:mtable></mml:math></disp-formula>
<p>In addition, considering the decision function equation, it is seen that the &#x003B1;<sub>m</sub>-weighted components with the kernel function, K(x,x<sub>m</sub>), directly affect the decision. Hence, the absolute value of the feature weights assigned by the SVM model is an indicator of the importance of features on the decision. In our study, this idea played a critical role in investigating which features were important in the decision mechanism. By maximizing the classification accuracy, the features were arranged in the order of importance.</p>
<p>Since we focused on the multi-source nature of the EEG signal, we carefully applied all feature extraction transformations separately for each source of EEG data. By that strategy, we avoided any miscalculation due to the discontinuity between subsequent sources.</p></sec></sec>
<sec>
<title>Data Classification</title>
<p>For the classification task, we focused on SVM classifier due to its multiple advantages over other state-of-the-art classification techniques. Widely used machine-learning techniques in EEG analysis, which are random forest classifier, na&#x000EF;ve Bayes classifier, and decision tree, have limited capability to handle such a complicated task (Wang et al., <xref ref-type="bibr" rid="B51">2019</xref>; Anuragi and Sisodia, <xref ref-type="bibr" rid="B2">2020</xref>; Roy et al., <xref ref-type="bibr" rid="B43">2021</xref>). These methods have a low tolerance to high variance and high-dimensional data. More sophisticated techniques such as artificial neural networks (ANN) have more capability to handle such challenging tasks (Roy et al., <xref ref-type="bibr" rid="B44">2019</xref>). However, these methods tend to overfit when the sample number is limited. Once a large dataset is available, ANN would work great, but it is so difficult to avoid overfitting ANN models for such an asymmetric dataset as EEG. However, SVM has the capability to handle difficult tasks such as high variance and high dimensionality (Bishop and Nasrabadi, <xref ref-type="bibr" rid="B9">2006</xref>). More importantly, SVM allows us to extract the feature hierarchy from the model. Hence, we have access to the information on how significant each feature is for the decision-making process. With all these advantages, we decided that the SVM classifier is the best option for the aimed task. In SVM analysis, we used radial basis function (RBF) kernel, and the sigma value was determined numerically for each case. On the other hand, to validate our strategy for data classification, we compared the performance of SVM with two ANN architectures EEGNet (Lawhern et al., <xref ref-type="bibr" rid="B38">2018</xref>) and Shallow ConvNet (Schirrmeister et al., <xref ref-type="bibr" rid="B46">2017</xref>), which are specialized for EEG analysis, and a more traditional approach SWLDA classifier. All parameters such as filter size, stride, and pooling size were optimized according to the sampling rate by the guide of references. Dropout value was optimized for each run to the value of 25 or 50%. Learning rate was optimized for each run. Each model is trained for 150 epochs.</p>
<p>The total number of 14 patients in our dataset was distributed unevenly into two groups. The number of TMS&#x0002B; patients was 8 whereas the number of TMS&#x02013; patients was 6. Due to the limited sample number and the risk of large biological variance, observing the unbiased results was crucial. For that purpose, we always used the Monte Carlo technique during the analysis. The Monte Carlo technique basically estimates the numerical results of non-deterministic events by relying on multiple randomly selected sample spaces (Bauer, <xref ref-type="bibr" rid="B6">1958</xref>). It bases on the randomness to observe the actual routine of the events, which is a core concept in probability theory. It is widely used for data analysis especially for biological data science when the sample number is limited (Manly, <xref ref-type="bibr" rid="B39">2018</xref>). Here, the main goal is covering all possible outcomes through the random trials. Hence, using the combination theory, we calculated the optimum number of random trials that needed to be performed to collect information from all possible sampling. All reported results are the average of the success rates of the optimum number of trials with random data selections. All random sets are selected by the normal distribution, so we can assume that all possible combinations of the samples were considered. As a result, given numerical values are the best estimation of the actual performance of the classification model. Furthermore, this technique based on randomness enables us to calculate the variation in the dataset.</p>
<p>For further analysis, we focused on the stability and the sensitivity reasoning of this study with respect to sample size and uneven distribution of samples to different groups. The sensitivity and stability analyses were done on EEG2 with SVM classifier. With the purpose of measuring the effect of unevenly distributed sets, we compared the classification results of balanced and unbalanced training sets. An unbalanced validation data were causing results to fluctuate wildly, hence harming the model optimization process. Similarly, with an unbalanced test data, numerical results were misleading about the quality of the model. Hence, the numerical analysis was performed always with balanced test and validation datasets.</p>
<p>The size of training and validation sets are the important aspects. By optimizing these values, the required sample size would be determined, and the computational studies can be optimized for the targeted tasks. To see the effect of the size of validation and training datasets, we repeated classification analysis with different dataset sizes and reported the results in the Results section.</p>
<p>During the classification analysis, we applied for the leave-one-out cross-validation since the dataset size was limited. For that purpose, we chose one patient as a test sample, and the remaining samples were used to form a balanced train and validation sets (Hastie et al., <xref ref-type="bibr" rid="B26">2009</xref>). From the remaining 13 patients, over 30 random selections, we selected a maximal balanced set. The balanced dataset was divided into two sets such as training and validation sets.</p>
<p>After observing satisfying results by the leave-one-out validation, we continued our analysis with direct classification despite the small dataset size. However, during the data classification process, due to a small sample number, we could not use a validation data. We split the dataset into training and test sets. This model may not be generalizable due to the limited sample size, but the results can be considered as an indication of the feasibility of the task.</p>
<sec>
<title>Optimization of SVM Parameters</title>
<p>The selection of the SVM parameters is very critical for the performance of the method. SVM works on high-dimensional spaces and may tend to overfit training data. In the literature, the grid search is the most widely used technique for optimizing SVM parameters. The grid search is based on generating a grid by all possible combinations of parameters. In the generated grid, all values are examined and the combination which yields the maximum classification result is determined as the optimum set of parameters. Since all possible parameters are checked, theoretically, the most optimum results will be determined by the grid search. However, for our case, since we applied Monte Carlo technique, we had to perform parameter optimization several times which is computationally expensive and time-consuming. Hence, in our study, we used an evolutionary algorithm to optimize the parameters (Syarif et al., <xref ref-type="bibr" rid="B49">2016</xref>). Our parameter optimization algorithm is an iterative method, starting with user-determined step size and minimum and maximum values for each parameter. In every iteration, it updates the search interval accordingly and reduces the step size to find the optimum parameters. Once the accuracy values start decreasing, it is considered as a stopping criterion. In contrast to grid search, this method does not check all possible combinations. However, the computational complexity is significantly reduced. In the literature, such genetic algorithms were shown to estimate the optimum values by grid search with a minimal error (Syarif et al., <xref ref-type="bibr" rid="B49">2016</xref>). This process was repeated for each random trial, so for each run in Monte Carlo, we used the optimal parameters for the SVM classifier.</p></sec></sec></sec></sec>
<sec sec-type="results" id="s3">
<title>Results</title>
<p>All patients had mild to moderate AD symptoms (CDR = 1&#x02013;2). The mean cognitive scores and demographic features of the patients are shown in <xref ref-type="table" rid="T2">Table 2</xref>. The change in MMSE scores before and after the TMS application in TMS&#x0002B; and TMS&#x02013; patients can be seen in <xref ref-type="fig" rid="F4">Figure 4</xref>. Between two groups (MMSE pre-TMS and MMSE post-TMS), one-way ANOVA was applied, and statistically significant results were observed in MMSE pre-TMS group (<italic>F</italic> = 5.153, <italic>p</italic> &#x0003C; 0.05). According to <italic>post hoc</italic> analysis, statistical difference has been obtained between TMS&#x0002B; and TMS&#x02013; (<italic>t</italic> = &#x02212;2.27, <italic>p</italic> &#x0003C; 0.05).</p>
<table-wrap position="float" id="T2">
<label>Table 2</label>
<caption><p>Baseline demographic characteristics of the patients.</p></caption>
<table frame="hsides" rules="groups">
<tbody>
<tr>
<td valign="top" align="left">N</td>
<td valign="top" align="center">14</td>
</tr>
<tr>
<td valign="top" align="left">Age</td>
<td valign="top" align="center">69.86 &#x000B1; 8.23</td>
</tr>
<tr>
<td valign="top" align="left">Sex (male/female)</td>
<td valign="top" align="center">5/9</td>
</tr>
<tr>
<td valign="top" align="left">MMSE score (pretreatment)</td>
<td valign="top" align="center">18.09 &#x000B1; 4.74</td>
</tr>
</tbody>
</table>
</table-wrap>
<fig id="F4" position="float">
<label>Figure 4</label>
<caption><p>The change in MMSE scores of TMS&#x0002B; and TMS- patients with the TMS application. As it is seen in the figure, these two groups can be separated from each other with the red dashed decision line. The patients with the same MMSE score post- and pre-TMS which are labeled as TMS&#x0002B; are represented as red dots inside blue.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fncel-16-845832-g0004.tif"/>
</fig>
<p>For the feature selection process, we applied PCA, wavelet transform, and SWLDA, and by the lead of SVM and SWLDA weights, we used only theta band features. We compared the classification accuracies with these different feature selection techniques (<xref ref-type="table" rid="T3">Table 3</xref>) where the classification accuracy is calculated as the percentage of the number of correctly classified samples to the total number of samples. For the classification analysis, we applied SVM, EEGNet, Shallow ConvNet, and SWLDA and compared their performances (refer to <xref ref-type="table" rid="T4">Table 4</xref>; <xref ref-type="fig" rid="F5">Figure 5</xref>).</p>
<table-wrap position="float" id="T3">
<label>Table 3</label>
<caption><p>Comparison of performance of feature selection methods with SVM leave-one-out validation of EEG2.</p></caption>
<table frame="hsides" rules="groups">
<thead><tr>
<th/>
<th valign="top" align="center"><bold>No feature selection method</bold></th>
<th valign="top" align="center"><bold>PCA</bold></th>
<th valign="top" align="center"><bold>Wavelet transform</bold></th>
<th valign="top" align="center"><bold>SWLDA</bold></th>
<th valign="top" align="center"><bold>Theta band features</bold></th>
</tr>
</thead>
<tbody>
<tr>
<td valign="top" align="left">EEG2</td>
<td valign="top" align="center">78%</td>
<td valign="top" align="center">62%</td>
<td valign="top" align="center">67%</td>
<td valign="top" align="center">58%</td>
<td valign="top" align="center"><bold>93</bold>%</td>
</tr>
</tbody>
</table>
<table-wrap-foot>
<p><italic>As it is seen, whole theta band features give significantly higher accuracy than other methods. The maximum accuracy is in bold</italic>.</p>
</table-wrap-foot>
</table-wrap>
<table-wrap position="float" id="T4">
<label>Table 4</label>
<caption><p>Comparison of performance of classification methods with power spectra features of EEG2.</p></caption>
<table frame="hsides" rules="groups">
<thead><tr>
<th/>
<th valign="top" align="center"><bold>SVM</bold></th>
<th valign="top" align="center"><bold>ShallowNet</bold></th>
<th valign="top" align="center"><bold>EEGNet</bold></th>
<th valign="top" align="center"><bold>SWLDA</bold></th>
</tr>
</thead>
<tbody>
<tr>
<td valign="top" align="left">Leave-one-out validation</td>
<td valign="top" align="center"><bold>78</bold>%</td>
<td valign="top" align="center"><bold>71</bold>%</td>
<td valign="top" align="center">43%</td>
<td valign="top" align="center">36%</td>
</tr>
</tbody>
</table>
<table-wrap-foot>
<p><italic>The maximum accuracies for each category are in bold</italic>.</p>
</table-wrap-foot>
</table-wrap>
<fig id="F5" position="float">
<label>Figure 5</label>
<caption><p>The comparison of different classification methods with leave-one-out validation of EEG2 signals of each patient in our dataset. <bold>(A)</bold> SVM classifier (78%), <bold>(B)</bold> SWLDA classifier (36%), <bold>(C)</bold> EEGNet (43%), <bold>(D)</bold> Shallow ConvNet (71%). As it is seen, there is a large variance among patients for all classification methods. However, most of the patients form a coherent team whereas two patients&#x00027; accuracies are around 60%, and one has an accuracy about 10%. The best two performances can be seen by SVM and Shallow ConvNet.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fncel-16-845832-g0005.tif"/>
</fig>
<p>In the PCA, the decay of eigenvalues showed us that most of the information was stored in 20 features only. Using the most informative 20 features selected by PCA, we got an average of 62% accuracy on leave-one-out validation of EEG2. In the wavelet transform application for the feature selection purpose, by optimizing the numerical analysis, we decided to keep the largest 10% of the wavelet coefficients. Through that, we got 67% on the leave-one-out validation of EEG2. In the SVM and SWLDA, we observed that the average weights of theta wavelength were remarkably higher than the others (<xref ref-type="fig" rid="F6">Figure 6</xref>). We used this idea as a lead and measured the performance of the data classification task using features collected only from theta wavelength which yield the best classification performance. In EEG1 analysis, we observed that none of the wavelengths nor the locations were given significantly more information than others.</p>
<fig id="F6" position="float">
<label>Figure 6</label>
<caption><p>The cumulated weights assigned by SWLDA to each wavelength features. SWLDA mostly selected theta band features as informative and assigned the largest weights to the theta band features.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fncel-16-845832-g0006.tif"/>
</fig>
<p>To measure the effect of using the multi-source form of EEG signals, we analyzed the dataset with the power of each wavelength. As a result, we got 42% of accuracy overall for leave-one-out validation of EEG2. Hence, keeping the signals collected from each location gives significant information for classification purposes.</p>
<p>We observed the highest classification performance as 93% with SVM on theta band features. As it is seen in <xref ref-type="table" rid="T5">Table 5</xref>, regardless of the classification or feature selection technique, we always got better performance with EEG2 than EEG1. The average performance of the leave-one-out validation of EEG2, without any further feature selection or dimension reduction method, was recorded as 78%. In this analysis, we have seen that most of the patients always give a classification accuracy &#x0003E;50%. Always, the same patients had poor classification accuracies. For example, one of the patient&#x00027;s average EEG2 classification rate was about 10% (<xref ref-type="fig" rid="F5">Figure 5A</xref>), which means that this patient information does not fit on the general of the other patients. However, we observed that the majority of the patients form a coherent team and give more than 90% of accuracy each. On the other hand, in the EEG1 analysis, we recorded 68% as the average accuracy and the coherence between the samples was lower. Furthermore, the gain by the feature selection and dimension reduction methods was higher in EEG2 analysis. Through the feature selection method application, the average performance of the leave-one-out validation of EEG2 increased to 93%, while we obtained 79% of accuracy on validation of EEG1.</p>
<table-wrap position="float" id="T5">
<label>Table 5</label>
<caption><p>All given accuracies are the result of the Monte Carlo Technique.</p></caption>
<table frame="hsides" rules="groups">
<thead><tr>
<th valign="top" align="left" style="border-bottom: thin solid #000000;"><bold>Feature vector</bold></th>
<th valign="top" align="center" colspan="2" style="border-bottom: thin solid #000000;"><bold>Power spectra features</bold></th>
<th valign="top" align="center" colspan="2" style="border-bottom: thin solid #000000;"><bold>Theta band features</bold></th>
</tr>
<tr>
<th valign="top" align="left"><bold>Validation technique</bold></th>
<th valign="top" align="center"><bold>Leave-one-out</bold></th>
<th valign="top" align="center"><bold>Direct classification</bold></th>
<th valign="top" align="center"><bold>Leave-one-out</bold></th>
<th valign="top" align="center"><bold>Direct classification</bold></th>
</tr>
</thead>
<tbody>
<tr>
<td valign="top" align="left">EEG1</td>
<td valign="top" align="center">68%</td>
<td valign="top" align="center"><bold>64</bold>%</td>
<td valign="top" align="center"><bold>79</bold>%</td>
<td valign="top" align="center">63%</td>
</tr>
<tr>
<td valign="top" align="left">EEG2</td>
<td valign="top" align="center">78%</td>
<td valign="top" align="center">77%</td>
<td valign="top" align="center"><bold>93</bold>%</td>
<td valign="top" align="center"><bold>82. 5</bold>%</td>
</tr>
</tbody>
</table>
<table-wrap-foot>
<p><italic>In the table, it is seen that EEG2 gives better results than EEG1 for all cases. Feature selection technique also improves the results. The maximum accuracies for each category are in bold</italic>.</p>
</table-wrap-foot>
</table-wrap>
<p>After observing satisfactory results in the leave-one-out validation, we continued with the classification model generation. We obtained 82.5% of accuracy on the EEG2 test data as an average by the Monte Carlo technique, and 64% of accuracy on the classification of EEG1, by the same procedure. During our classification model generation, we observed that the optimum training sample ratio was about 65% (9 samples out of 14). As the mean of 50 random trials, we got 82.5% of accuracy while the standard deviation was 12.5%. So, the minimum accuracy was about 75%, which means that we got 3 correct predictions out of 4 on average by EEG2 analysis.</p></sec>
<sec sec-type="discussion" id="s4">
<title>Discussion</title>
<p>As a conclusion of our analysis, we observed that it is possible to predict the response of patients with AD to the TMS treatment by both EEG1 and EEG2 analyses. However, EEG2 is more informative than EEG1 with an average accuracy of 93 % while we obtained 79% on EEG1. Besides, due to our limited sample number, we focused on the data analysis and investigated the challenges of the EEG data analysis on patients with AD. We measured the feasibility of generating a personal TMS treatment route for the patients with AD and showed the possibility is higher if we use the lead of EEG signals after rTMS application. In addition to that, this type of algorithm, which can predict which patient will benefit from the TMS application by using the EEG data, will not only prevent patients from losing time with TMS treatment, but also will save on health expenditures. Thus, it will be possible to use TMS more effectively in neurodegenerative diseases such as Alzheimer&#x00027;s disease.</p>
<p>Due to its non-invasive nature, relative ease of administration, and low cost compared to other neuroimaging methods, EEG represents a promising tool for developing a personalized medicine approach for rTMS treatment (Garnaat et al., <xref ref-type="bibr" rid="B20">2019</xref>). However, there is still a very limited number of studies in this area.</p>
<p>One of the previous studies that tried to distinguish responders from non-responders to rTMS treatment in depression using resting-state EEG analysis differentiated these groups and retrospectively identified several pretreatment variables such as higher anterior individual alpha frequency values, lower power in the frontocentral theta frequency band, and increased prefrontal delta and beta values (Arns et al., <xref ref-type="bibr" rid="B3">2012</xref>). However, a repeat study to confirm these promising results found no significant differences between responders and non-responders in terms of individual alpha frequency, frontal theta, even after controlling for sex or age (Krepel et al., <xref ref-type="bibr" rid="B35">2018</xref>; Garnaat et al., <xref ref-type="bibr" rid="B20">2019</xref>).</p>
<p>There are only a few studies on machine-learning analysis of EEG in predicting the effectiveness of rTMS. All these attempts focus on depressive disorders. Although sophisticated methods are introduced, since the TMS therapy for patients with AD have more factors to be considered than depressive disorders, the given methods are weak to handle the current task.</p>
<p>In prospective studies, there were only a few possible predictors of the outcome for rTMS treatment for depression patients, and further research will be required to translate any of the findings described above into guidance for the implementation of treatment (Garnaat et al., <xref ref-type="bibr" rid="B20">2019</xref>).</p>
<p>Our findings are particularly parallel to the findings by Bailey et al., He also reported that alpha and theta bands are useful to predict benefiting from rTMS therapy for depression (Bailey et al., <xref ref-type="bibr" rid="B4">2019</xref>). However, in another study, they conducted with a larger and different sample group, and they could not replicate these findings for alpha and theta (Bailey et al., <xref ref-type="bibr" rid="B5">2021</xref>). As it is known, the main element of the changes in EEG in Alzheimer&#x00027;s disease is the changes in alpha and theta (Bhattacharya et al., <xref ref-type="bibr" rid="B8">2011</xref>; G&#x000FC;ntekin et al., <xref ref-type="bibr" rid="B22">2020</xref>). The results of our study are compatible with this general framework. There are also some findings showing that EEG event-related theta responses may be sensitive to neuromodulation interventions (Cespon et al., <xref ref-type="bibr" rid="B14">2019</xref>; G&#x000FC;ntekin et al., <xref ref-type="bibr" rid="B22">2020</xref>). However, the impaired dynamics associated with EEG in AD are not unique only to theta and alpha. Again, it is clear that the localization of rTMS applications can cause responses at different frequencies by many different mechanisms. Indeed, Koch et al., showed that rTMS intervention on the precuneus for 2 weeks modulated only beta responses in AD (Koch et al., <xref ref-type="bibr" rid="B33">2018</xref>). For this reason, there is a need for larger sample groups and further research on patients with other frequencies.</p>
<sec>
<title>Justification of Sample Size for the Data Analysis</title>
<p>For the aimed data analysis task, considering the small sample number with the large feature vectors, getting a unique solution was not possible. Instead, we got statistical confidence by applying the Monte Carlo technique. For that purpose, we repeated the analysis with multiple randomly selected datasets with varying sizes and combinations of samples. Through that, we obtained the best estimation of the actual performance of our model despite the limited sample size. Still, to have theoretically justified results, we repeated the analysis with the low number of features. For that purpose, we applied different dimension reduction techniques. However, we observed that a low feature number does not satisfy a qualified data classification model for the current dataset.</p>
<p>We observed that SVM works better than EEGNet, Shallow ConvNet, and SWLDA for the classification of our dataset. Since the task is challenging, the low feature number would not store the required information. As a result, SWLDA failed in the analysis. The ANN architectures such as Shallow ConvNet and EEGNet tend to overfit for our dataset with a small sample number. However, Shallow ConvNet showed a close performance to SVM classifier. This would lead that with a larger dataset, a shallow convolutional neural network would give promising classification results.</p>
<p>During the data analysis, we measured that the intra-class variance is large. As a result, the selection of the training samples was significantly changing the classification accuracy. We have seen that in a class (TMS&#x0002B; or TMS&#x02013;), some patients&#x00027; information conflicts with each other. So, the inclusion of one of these patients in the training dataset was leading to a model which would mislabel the other patient. This information supports the large variance in the dataset. Besides, as it is seen in <xref ref-type="fig" rid="F7">Figure 7</xref>, with increasing training dataset size, the classification accuracy increases and has not stabilized with the maximal training set size. This suggests that we have not reached the required sample number yet. Hence, the data analysis indicates that more patients are required to generate a more generalizable model.</p>
<fig id="F7" position="float">
<label>Figure 7</label>
<caption><p>The leave-one-out classification result for varying training and validation dataset sizes. The orange plot: for different training dataset sizes, the validation dataset is always formed by two samples. The blue plot: for different training dataset sizes, the validation dataset is formed as the remaining samples after the test and training sets are generated.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fncel-16-845832-g0007.tif"/>
</fig>
<p>We observed that the increasing validation data size may have a negative effect on the classification performance in some cases (<xref ref-type="fig" rid="F8">Figure 8</xref>). The reason for that may be the fact that some patients&#x00027; information totally conflicts with others, and hence, the inclusion of certain patients in the validation data would cause a decrease in classification accuracy. When we use smaller validation data, the effect of these certain patients would decrease. Hence, smaller validation data with a large training dataset gave the largest accuracy during our analysis.</p>
<fig id="F8" position="float">
<label>Figure 8</label>
<caption><p>The leave-one-out classification result for increasing validation dataset sizes while training size is fixed. There is no visible increasing pattern in the graph. So, increasing validation size does not have a significant effect on the classification process.</p></caption>
<graphic mimetype="image" mime-subtype="tiff" xlink:href="fncel-16-845832-g0008.tif"/>
</fig>
<p>By considering the classification analysis results (<xref ref-type="table" rid="T5">Table 5</xref>), we concluded that the EEG2 signals are more informative than the EEG1 to recognize TMS&#x0002B; and TMS- patients. Similarly, EEG2 has more capacity to be improved by feature selection processes. Hence, EEG signal collected after a TMS therapy would lead the more efficient prediction models for the personalized TMS treatments for patients with AD.</p>
<p>For the future direction, this analysis must be repeated with a larger dataset. This will increase the validity of the data analysis results. We also observed that common data augmentation techniques for signals did not supply any improvement in the classification performance. In the literature, it is seen that for different neurological tasks, different types of data augmentation techniques must be applied (Lashgari et al., <xref ref-type="bibr" rid="B37">2020</xref>). Hence, for future studies, the best augmentation techniques for the Alzheimer&#x00027;s disease must be determined, and those techniques must be applied to the present data. Through that, a more reliable and consistent classification model could be presented. Similarly, using a multi-kernel technique such as MKSVM instead would produce a better result due to the multi-source nature of our dataset. Besides that, a shallow artificial neural network would increase efficiency with its capability to handle challenging tasks.</p>
<p>In this study, due to the limited sample number and observed biological variance, we mostly focused on the investigation of the data from various perspectives. We measured the feasibility of generating a personal treatment route for the TMS treatments for patients with AD and showed that the possibility is high. However, for getting a reliable and less risky decision model, we need to have a better observation of the biological variance, which requires more data.</p></sec></sec>
<sec sec-type="data-availability" id="s5">
<title>Data Availability Statement</title>
<p>The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.</p></sec>
<sec id="s6">
<title>Ethics Statement</title>
<p>The studies involving human participants were reviewed and approved by the Ethical Committee of Istanbul Medipol University the number of Ethical Report: 10840098&#x02013;604.01.01&#x02013;E.45425. The patients/participants provided their written informed consent to participate in this study.</p></sec>
<sec id="s7">
<title>Author Contributions</title>
<p>LH designed and supervised all the experiments. HV conceived the experiments. CK designed the figures and analyzed and performed the machine-learning analysis. All authors performed the experiments and/or analyzed the data. All authors contributed to the interpretation of the results, writing the article, and approved the submitted version.</p></sec>
<sec sec-type="funding-information" id="s8">
<title>Funding</title>
<p>Istanbul Medipol University support for researchers for submission expenses.</p></sec>
<sec sec-type="COI-statement" id="conf1">
<title>Conflict of Interest</title>
<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.</p></sec>
<sec sec-type="disclaimer" id="s9">
<title>Publisher&#x00027;s Note</title>
<p>All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.</p></sec></body>
<back>
<ack><p>The authors would like to thank Tuba Akturk for her valuable advice and support for the EEG signal analysis.</p>
</ack>
<ref-list>
<title>References</title>
<ref id="B1">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>al-Qerem</surname> <given-names>A.</given-names></name> <name><surname>Kharbat</surname> <given-names>F.</given-names></name> <name><surname>Nashwan</surname> <given-names>S.</given-names></name> <name><surname>Ashraf</surname> <given-names>S.</given-names></name> <name><surname>Blaou</surname> <given-names>K.</given-names></name></person-group> (<year>2020</year>). <article-title>General model for best feature extraction of EEG using discrete wavelet transform wavelet family and differential evolution</article-title>. <source>Int. J. Distrib. Sens. Netw.</source> <volume>16</volume>, <fpage>1550147720911009</fpage>. <pub-id pub-id-type="doi">10.1177/1550147720911009</pub-id></citation>
</ref>
<ref id="B2">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Anuragi</surname> <given-names>A.</given-names></name> <name><surname>Sisodia</surname> <given-names>D. S.</given-names></name></person-group> (<year>2020</year>). <article-title>Empirical wavelet transform based automated alcoholism detecting using EEG signal features</article-title>. <source>Biomed. Sig. Process. Control</source> <volume>57</volume>, <fpage>101777</fpage>. <pub-id pub-id-type="doi">10.1016/j.bspc.2019.101777</pub-id></citation>
</ref>
<ref id="B3">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Arns</surname> <given-names>M.</given-names></name> <name><surname>Drinkenburg</surname> <given-names>W. H.</given-names></name> <name><surname>Fitzgerald</surname> <given-names>P. B.</given-names></name> <name><surname>Kenemans</surname> <given-names>J. L.</given-names></name></person-group> (<year>2012</year>). <article-title>Neurophysiological predictors of non-response to rTMS in depression</article-title>. <source>Brain Stimul</source>. <volume>5</volume>, <fpage>569</fpage>&#x02013;<lpage>576</lpage>. <pub-id pub-id-type="doi">10.1016/j.brs.2011.12.003</pub-id><pub-id pub-id-type="pmid">29433985</pub-id></citation></ref>
<ref id="B4">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Bailey</surname> <given-names>N. W.</given-names></name> <name><surname>Hoy</surname> <given-names>K. E.</given-names></name> <name><surname>Rogasch</surname> <given-names>N. C.</given-names></name> <name><surname>Thomson</surname> <given-names>R. H.</given-names></name> <name><surname>McQueen</surname> <given-names>S.</given-names></name> <name><surname>Elliot</surname> <given-names>D.</given-names></name> <etal/></person-group>. (<year>2019</year>). <article-title>Differentiating responders and non-responders to rTMS treatment for depression after one week using resting EEG connectivity measures</article-title>. <source>J. Affect. Disord.</source> <volume>242</volume>, <fpage>68</fpage>&#x02013;<lpage>79</lpage>. <pub-id pub-id-type="doi">10.1016/j.jad.2018.08.058</pub-id><pub-id pub-id-type="pmid">30172227</pub-id></citation></ref>
<ref id="B5">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Bailey</surname> <given-names>N. W.</given-names></name> <name><surname>Krepel</surname> <given-names>N.</given-names></name> <name><surname>van Dijk</surname> <given-names>H.</given-names></name> <name><surname>Leuchter</surname> <given-names>A. F.</given-names></name> <name><surname>Vila-Rodriguez</surname> <given-names>F.</given-names></name> <name><surname>Blumberger</surname> <given-names>D. M.</given-names></name> <etal/></person-group>. (<year>2021</year>). <article-title>Resting EEG theta connectivity and alpha power to predict repetitive transcranial magnetic stimulation response in depression: A non-replication from the ICON-DB consortium</article-title>. <source>Clin. Neurophysiol</source>. <volume>132</volume>, <fpage>650</fpage>&#x02013;<lpage>659</lpage>. <pub-id pub-id-type="doi">10.1016/j.clinph.2020.10.018</pub-id><pub-id pub-id-type="pmid">33223495</pub-id></citation></ref>
<ref id="B6">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Bauer</surname> <given-names>W. F.</given-names></name></person-group> (<year>1958</year>). <article-title>The monte carlo method</article-title>. <source>J. Soc. Ind. Appl. Math.</source> <volume>6</volume>, <fpage>438</fpage>&#x02013;<lpage>451</lpage>. <pub-id pub-id-type="doi">10.1137/0106028</pub-id></citation>
</ref>
<ref id="B7">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Begemann</surname> <given-names>M. J.</given-names></name> <name><surname>Brand</surname> <given-names>B. A.</given-names></name> <name><surname>&#x00106;ur&#x0010D;i&#x00107;-Blake</surname> <given-names>B.</given-names></name> <name><surname>Aleman</surname> <given-names>A.</given-names></name> <name><surname>Sommer</surname> <given-names>I. E.</given-names></name></person-group> (<year>2020</year>). <article-title>Efficacy of non-invasive brain stimulation on cognitive functioning in brain disorders: A meta-analysis</article-title>. <source>Psychol. Med.</source> <volume>50</volume>, <fpage>2465</fpage>&#x02013;<lpage>2486</lpage>. <pub-id pub-id-type="doi">10.1017/S0033291720003670</pub-id><pub-id pub-id-type="pmid">33070785</pub-id></citation></ref>
<ref id="B8">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Bhattacharya</surname> <given-names>B. S.</given-names></name> <name><surname>Coyle</surname> <given-names>D.</given-names></name> <name><surname>Maguire</surname> <given-names>L. P.</given-names></name></person-group> (<year>2011</year>). <article-title>Alpha and theta rhythm abnormality in Alzheimer&#x00027;s disease: a study using a computational model</article-title>. <source>Adv. Exp. Med. Biol</source>. <volume>718</volume>, <fpage>57</fpage>&#x02013;<lpage>73</lpage>. <pub-id pub-id-type="doi">10.1007/978-1-4614-0164-3_6</pub-id><pub-id pub-id-type="pmid">21744210</pub-id></citation></ref>
<ref id="B9">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Bishop</surname> <given-names>C. M.</given-names></name> <name><surname>Nasrabadi</surname> <given-names>N. M.</given-names></name></person-group> (<year>2006</year>). <source>Pattern Recognition and Machine Learning, Vol. 4</source>. <publisher-loc>New York, NY</publisher-loc>: <publisher-name>Springer</publisher-name>.</citation>
</ref>
<ref id="B10">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Blankertz</surname> <given-names>B.</given-names></name> <name><surname>Acqualagna</surname> <given-names>L.</given-names></name> <name><surname>D&#x000E4;hne</surname> <given-names>S.</given-names></name> <name><surname>Haufe</surname> <given-names>S.</given-names></name> <name><surname>Schultze-Kraft</surname> <given-names>M.</given-names></name> <name><surname>Sturm</surname> <given-names>I.</given-names></name> <etal/></person-group>. (<year>2016</year>). <article-title>The Berlin Brain-Computer Interface: Progress Beyond Communication and Control</article-title>. <source>Front. Neurosci</source>. <volume>10</volume>, <fpage>530</fpage>. <pub-id pub-id-type="doi">10.3389/fnins.2016.00530</pub-id><pub-id pub-id-type="pmid">27917107</pub-id></citation></ref>
<ref id="B11">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Bloom</surname> <given-names>G. S.</given-names></name></person-group> (<year>2014</year>). <article-title>Amyloid-&#x003B2; and tau: the trigger and bullet in Alzheimer disease pathogenesis</article-title>. <source>JAMA Neurol</source>. <volume>71</volume>, <fpage>505</fpage>&#x02013;<lpage>508</lpage>. <pub-id pub-id-type="doi">10.1001/jamaneurol.2013.5847</pub-id><pub-id pub-id-type="pmid">24493463</pub-id></citation></ref>
<ref id="B12">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Brown</surname> <given-names>K. E.</given-names></name> <name><surname>Neva</surname> <given-names>J. L.</given-names></name> <name><surname>Ledwell</surname> <given-names>N. M.</given-names></name> <name><surname>Boyd</surname> <given-names>L. A.</given-names></name></person-group> (<year>2014</year>). <article-title>Use of transcranial magnetic stimulation in the treatment of selected movement disorders. Degener. Neurol. Neuromuscul. Dis</article-title>. <volume>4</volume>, <fpage>133</fpage>&#x02013;<lpage>151</lpage>. <pub-id pub-id-type="doi">10.2147/DNND.S70079</pub-id><pub-id pub-id-type="pmid">32669907</pub-id></citation></ref>
<ref id="B13">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Burges</surname> <given-names>C. J. C.</given-names></name></person-group> (<year>1998</year>). <article-title>A tutorial on support vector machines for pattern recognition</article-title>. <source>Data Mining Knowl. Discov.</source> <volume>2</volume>, <fpage>121</fpage>&#x02013;<lpage>167</lpage>. <pub-id pub-id-type="doi">10.1023/A:1009715923555</pub-id></citation>
</ref>
<ref id="B14">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Cespon</surname> <given-names>J.</given-names></name> <name><surname>Rodella</surname> <given-names>C.</given-names></name> <name><surname>Miniussi</surname> <given-names>C.</given-names></name> <name><surname>Pellicciari</surname> <given-names>M. C.</given-names></name></person-group> (<year>2019</year>). <article-title>Behavioural and electrophysiological modulations induced by transcranial direct current stimulation in healthy elderly and Alzheimer&#x00027;s disease patients: a pilot study</article-title>. <source>Clin. Neurophysiol.</source> <volume>130</volume>, <fpage>2038</fpage>&#x02013;<lpage>2052</lpage>. <pub-id pub-id-type="doi">10.1016/j.clinph.2019.08.016</pub-id><pub-id pub-id-type="pmid">31541981</pub-id></citation></ref>
<ref id="B15">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Chekroud</surname> <given-names>A. M.</given-names></name> <name><surname>Zotti</surname> <given-names>R. J.</given-names></name> <name><surname>Shehzad</surname> <given-names>Z.</given-names></name> <name><surname>Gueorguieva</surname> <given-names>R.</given-names></name> <name><surname>Johnson</surname> <given-names>M. K.</given-names></name> <name><surname>Trivedi</surname> <given-names>M. H.</given-names></name> <etal/></person-group>. (<year>2016</year>). <article-title>Cross-trial prediction of treatment outcome in depression: a machine learning approach</article-title>. <source>Lancet Psychiatry</source> <volume>3</volume>, <fpage>243</fpage>&#x02013;<lpage>250</lpage>. <pub-id pub-id-type="doi">10.1016/S2215-0366(15)00471-X</pub-id><pub-id pub-id-type="pmid">26803397</pub-id></citation></ref>
<ref id="B16">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Corlier</surname> <given-names>J.</given-names></name> <name><surname>Wilson</surname> <given-names>A.</given-names></name> <name><surname>Hunter</surname> <given-names>A. M.</given-names></name> <name><surname>Vince-Cruz</surname> <given-names>N.</given-names></name> <name><surname>Krantz</surname> <given-names>D.</given-names></name> <name><surname>Levitt</surname> <given-names>J.</given-names></name> <etal/></person-group>. (<year>2019</year>). <article-title>Changes in functional connectivity predict outcome of repetitive transcranial magnetic stimulation treatment of major depressive disorder</article-title>. <source>Cereb. Cortex</source> <volume>29</volume>, <fpage>4958</fpage>&#x02013;<lpage>4967</lpage>. <pub-id pub-id-type="doi">10.1093/cercor/bhz035</pub-id><pub-id pub-id-type="pmid">30953441</pub-id></citation></ref>
<ref id="B17">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Cummings</surname> <given-names>J. L.</given-names></name> <name><surname>Tong</surname> <given-names>G.</given-names></name> <name><surname>Ballard</surname> <given-names>C.</given-names></name></person-group> (<year>2019</year>). <article-title>Treatment combinations for Alzheimer&#x00027;s disease: current and future pharmacotherapy options</article-title>. <source>J. Alzheimers Dis</source>. <volume>67</volume>, <fpage>779</fpage>&#x02013;<lpage>794</lpage>. <pub-id pub-id-type="doi">10.3233/JAD-180766</pub-id><pub-id pub-id-type="pmid">30689575</pub-id></citation></ref>
<ref id="B18">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Dunlop</surname> <given-names>K.</given-names></name> <name><surname>Woodside</surname> <given-names>B.</given-names></name> <name><surname>Olmsted</surname> <given-names>M.</given-names></name> <name><surname>Colton</surname> <given-names>P.</given-names></name> <name><surname>Giacobbe</surname> <given-names>P.</given-names></name> <name><surname>Downar</surname> <given-names>J.</given-names></name></person-group> (<year>2016</year>). <article-title>Reductions in cortico-striatal hyperconnectivity accompany successful treatment of obsessive-compulsive disorder with dorsomedial prefrontal rTMS</article-title>. <source>Neuropsychopharmacology</source> <volume>41</volume>, <fpage>1395</fpage>&#x02013;<lpage>1403</lpage>. <pub-id pub-id-type="doi">10.1038/npp.2015.292</pub-id><pub-id pub-id-type="pmid">26440813</pub-id></citation></ref>
<ref id="B19">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Freitas</surname> <given-names>C.</given-names></name> <name><surname>Mondrag&#x000F3;n-Llorca</surname> <given-names>H.</given-names></name> <name><surname>Pascual-Leone</surname> <given-names>A.</given-names></name></person-group> (<year>2011</year>). <article-title>Noninvasive brain stimulation in Alzheimer&#x00027;s disease: systematic review and perspectives for the future</article-title>. <source>Exp. Gerontol.</source> <volume>46</volume>, <fpage>611</fpage>&#x02013;<lpage>627</lpage>. <pub-id pub-id-type="doi">10.1016/j.exger.2011.04.001</pub-id><pub-id pub-id-type="pmid">21511025</pub-id></citation></ref>
<ref id="B20">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Garnaat</surname> <given-names>S. L.</given-names></name> <name><surname>Fukuda</surname> <given-names>A. M.</given-names></name> <name><surname>Yuan</surname> <given-names>S.</given-names></name> <name><surname>Carpenter</surname> <given-names>L. L.</given-names></name></person-group> (<year>2019</year>). <article-title>Identification of clinical features and biomarkers that may inform a personalized approach to rTMS for depression</article-title>. <source>Pers. Med. Psychiatry</source> 17&#x02013;<volume>18</volume>, <fpage>4</fpage>&#x02013;<lpage>16</lpage>. <pub-id pub-id-type="doi">10.1016/j.pmip.2019.09.001</pub-id><pub-id pub-id-type="pmid">33954269</pub-id></citation></ref>
<ref id="B21">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Guarino</surname> <given-names>A.</given-names></name> <name><surname>Favieri</surname> <given-names>F.</given-names></name> <name><surname>Boncompagni</surname> <given-names>I.</given-names></name> <name><surname>Agostini</surname> <given-names>F.</given-names></name> <name><surname>Cantone</surname> <given-names>M.</given-names></name> <name><surname>Casagrande</surname> <given-names>M.</given-names></name></person-group> (<year>2019</year>). <article-title>Executive functions in Alzheimer disease: a systematic review</article-title>. <source>Front. Aging Neurosci.</source> <volume>10</volume>, <fpage>437</fpage>. <pub-id pub-id-type="doi">10.3389/fnagi.2018.00437</pub-id><pub-id pub-id-type="pmid">30697157</pub-id></citation></ref>
<ref id="B22">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>G&#x000FC;ntekin</surname> <given-names>B.</given-names></name> <name><surname>Uzunlar</surname> <given-names>H.</given-names></name> <name><surname>&#x000C7;ali&#x0015F;oglu</surname> <given-names>P.</given-names></name> <name><surname>Eroglu-Ada</surname> <given-names>F.</given-names></name> <name><surname>Yildirim</surname> <given-names>E.</given-names></name> <name><surname>Akt&#x000FC;rk</surname> <given-names>T.</given-names></name> <etal/></person-group>. (<year>2020</year>). <article-title>Theta and alpha oscillatory responses differentiate between six-to seven-year-old children and adults during successful visual and auditory memory encoding</article-title>. <source>Brain Res.</source> <volume>1747</volume>, <fpage>147042</fpage>. <pub-id pub-id-type="doi">10.1016/j.brainres.2020.147042</pub-id><pub-id pub-id-type="pmid">32758480</pub-id></citation></ref>
<ref id="B23">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hamid</surname> <given-names>P.</given-names></name> <name><surname>Malik</surname> <given-names>B. H.</given-names></name> <name><surname>Hussain</surname> <given-names>M. L.</given-names></name></person-group> (<year>2019</year>). <article-title>Noninvasive transcranial magnetic stimulation (TMS) in chronic refractory pain: a systematic review</article-title>. <source>Cureus</source> <volume>11</volume>, <fpage>e6019</fpage>. <pub-id pub-id-type="doi">10.7759/cureus.6019</pub-id><pub-id pub-id-type="pmid">31824787</pub-id></citation></ref>
<ref id="B24">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hanoglu</surname> <given-names>L.</given-names></name> <name><surname>Velioglu</surname> <given-names>H. A.</given-names></name> <name><surname>Hanoglu</surname> <given-names>T.</given-names></name> <name><surname>Yulug</surname> <given-names>B.</given-names></name></person-group> (<year>2021</year>). <article-title>Neuroimaging-guided transcranial magnetic and direct current stimulation in MCI: toward an individual, effective and disease-modifying treatment</article-title>. <source>Clin. EEG Neurosci.</source> <pub-id pub-id-type="doi">10.1177/15500594211052815</pub-id><pub-id pub-id-type="pmid">34751037</pub-id></citation></ref>
<ref id="B25">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hasanzadeh</surname> <given-names>F.</given-names></name> <name><surname>Mohebbi</surname> <given-names>M.</given-names></name> <name><surname>Rostami</surname> <given-names>R.</given-names></name></person-group> (<year>2019</year>). <article-title>Prediction of rTMS treatment response in major depressive disorder using machine learning techniques and nonlinear features of EEG signal</article-title>. <source>J. Affect. Disord.</source> <volume>256</volume>, <fpage>132</fpage>&#x02013;<lpage>142</lpage>. <pub-id pub-id-type="doi">10.1016/j.jad.2019.05.070</pub-id><pub-id pub-id-type="pmid">31176185</pub-id></citation></ref>
<ref id="B26">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Hastie</surname> <given-names>T.</given-names></name> <name><surname>Tibshirani</surname> <given-names>R.</given-names></name> <name><surname>Friedman</surname> <given-names>J. H.</given-names></name> <name><surname>Friedman</surname> <given-names>J. H.</given-names></name></person-group> (<year>2009</year>). <source>The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Vol. 2</source>. <publisher-loc>New York</publisher-loc>: <publisher-name>springer</publisher-name>. <pub-id pub-id-type="doi">10.1007/978-0-387-84858-7</pub-id></citation>
</ref>
<ref id="B27">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hazarika</surname> <given-names>N.</given-names></name> <name><surname>Chen</surname> <given-names>J. Z.</given-names></name> <name><surname>Tsoi</surname> <given-names>A. C.</given-names></name> <name><surname>Sergejew</surname> <given-names>A.</given-names></name></person-group> (<year>1997</year>). <article-title>Classification of EEG signals using the wavelet transform</article-title>. <source>Signal Process.</source> <volume>59</volume>, <fpage>61</fpage>&#x02013;<lpage>72</lpage>. <pub-id pub-id-type="doi">10.1016/S0165-1684(97)00038-8</pub-id></citation>
</ref>
<ref id="B28">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Heil</surname> <given-names>C.</given-names></name></person-group> (<year>2010</year>). <source>A Basis Theory Primer: Expanded Edition</source>. <publisher-loc>Berlin</publisher-loc>: <publisher-name>Springer Science &#x00026; Business Media</publisher-name>. <pub-id pub-id-type="doi">10.1007/978-0-8176-4687-5</pub-id></citation>
</ref>
<ref id="B29">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Hermiller</surname> <given-names>M. S.</given-names></name> <name><surname>Karp</surname> <given-names>E.</given-names></name> <name><surname>Nilakantan</surname> <given-names>A. S.</given-names></name> <name><surname>Voss</surname> <given-names>J. L.</given-names></name></person-group> (<year>2019</year>). <article-title>Episodic memory improvements due to noninvasive stimulation targeting the cortical&#x02013;hippocampal network: a replication and extension experiment</article-title>. <source>Brain Behav.</source> <volume>9</volume>, <fpage>e01393</fpage>. <pub-id pub-id-type="doi">10.1002/brb3.1393</pub-id><pub-id pub-id-type="pmid">31568683</pub-id></citation></ref>
<ref id="B30">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Herwig</surname> <given-names>U.</given-names></name> <name><surname>Satrapi</surname> <given-names>P.</given-names></name> <name><surname>Sch&#x000F6;nfeldt-Lecuona</surname> <given-names>C.</given-names></name></person-group> (<year>2003</year>). <article-title>Using the international 10-20 EEG system for positioning of transcranial magnetic stimulation</article-title>. <source>Brain Topogr.</source> <volume>16</volume>, <fpage>95</fpage>&#x02013;<lpage>99</lpage>. <pub-id pub-id-type="doi">10.1023/B:BRAT.0000006333.93597.9d</pub-id><pub-id pub-id-type="pmid">14977202</pub-id></citation></ref>
<ref id="B31">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Iimori</surname> <given-names>T.</given-names></name> <name><surname>Nakajima</surname> <given-names>S.</given-names></name> <name><surname>Miyazaki</surname> <given-names>T.</given-names></name> <name><surname>Tarumi</surname> <given-names>R.</given-names></name> <name><surname>Ogyu</surname> <given-names>K.</given-names></name> <name><surname>Wada</surname> <given-names>M.</given-names></name> <etal/></person-group>. (<year>2019</year>). <article-title>Effectiveness of the prefrontal repetitive transcranial magnetic stimulation on cognitive profiles in depression, schizophrenia, and Alzheimer&#x00027;s disease: a systematic review</article-title>. <source>Prog. Neuropsychopharmacol. Biol. Psychiatry</source> <volume>88</volume>, <fpage>31</fpage>&#x02013;<lpage>40</lpage>. <pub-id pub-id-type="doi">10.1016/j.pnpbp.2018.06.014</pub-id><pub-id pub-id-type="pmid">29953934</pub-id></citation></ref>
<ref id="B32">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Jolliffe</surname> <given-names>I.</given-names></name></person-group> (<year>2014</year>). <article-title>Principal component analysis,</article-title> in <source>Wiley StatsRef: Statistics Reference Online</source>, eds <person-group person-group-type="editor"><name><surname>Balakrishnan</surname> <given-names>N.</given-names></name> <name><surname>Colton</surname> <given-names>T.</given-names></name> <name><surname>Everitt</surname> <given-names>B.</given-names></name> <name><surname>Piegorsch</surname> <given-names>W.</given-names></name> <name><surname>Ruggeri</surname> <given-names>F.</given-names></name> <name><surname>Teugels</surname> <given-names>J. L.</given-names></name></person-group> (<publisher-name>John Wiley and Sons</publisher-name>). <pub-id pub-id-type="doi">10.1002/9781118445112.stat06472</pub-id><pub-id pub-id-type="pmid">25855820</pub-id></citation></ref>
<ref id="B33">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Koch</surname> <given-names>G.</given-names></name> <name><surname>Bonn,&#x000EC;</surname> <given-names>S.</given-names></name> <name><surname>Pellicciari</surname> <given-names>M. C.</given-names></name> <name><surname>Casula</surname> <given-names>E. P.</given-names></name> <name><surname>Mancini</surname> <given-names>M.</given-names></name> <name><surname>Esposito</surname> <given-names>R.</given-names></name> <etal/></person-group>. (<year>2018</year>). <article-title>Transcranial magnetic stimulation of the precuneus enhances memory and neural activity in prodromal Alzheimer&#x00027;s disease</article-title>. <source>NeuroImage</source> <volume>169</volume>, <fpage>302</fpage>&#x02013;<lpage>311</lpage>. <pub-id pub-id-type="doi">10.1016/j.neuroimage.2017.12.048</pub-id><pub-id pub-id-type="pmid">29277405</pub-id></citation></ref>
<ref id="B34">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Koutsouleris</surname> <given-names>N.</given-names></name> <name><surname>Wobrock</surname> <given-names>T.</given-names></name> <name><surname>Guse</surname> <given-names>B.</given-names></name> <name><surname>Langguth</surname> <given-names>B.</given-names></name> <name><surname>Landgrebe</surname> <given-names>M.</given-names></name> <name><surname>Eichhammer</surname> <given-names>P.</given-names></name> <etal/></person-group>. (<year>2018</year>). <article-title>Predicting response to repetitive transcranial magnetic stimulation in patients with schizophrenia using structural magnetic resonance imaging: a multisite machine learning analysis</article-title>. <source>Schizophr. Bull.</source> <volume>44</volume>, <fpage>1021</fpage>&#x02013;<lpage>1034</lpage>. <pub-id pub-id-type="doi">10.1093/schbul/sbx114</pub-id><pub-id pub-id-type="pmid">28981875</pub-id></citation></ref>
<ref id="B35">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Krepel</surname> <given-names>N.</given-names></name> <name><surname>Sack</surname> <given-names>A. T.</given-names></name> <name><surname>Kenemans</surname> <given-names>J. L.</given-names></name> <name><surname>Fitzgerald</surname> <given-names>P. B.</given-names></name> <name><surname>Drinkenburg</surname> <given-names>W. H.</given-names></name> <name><surname>Arns</surname> <given-names>M.</given-names></name></person-group> (<year>2018</year>). <article-title>Non-replication of neurophysiological predictors of non-response to rTMS in depression</article-title>., <source>Brain Stimul</source>. <volume>11</volume>, <fpage>639</fpage>&#x02013;<lpage>641</lpage>. <pub-id pub-id-type="doi">10.1016/j.brs.2018.01.032</pub-id><pub-id pub-id-type="pmid">29433985</pub-id></citation></ref>
<ref id="B36">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Lane</surname> <given-names>C. A.</given-names></name> <name><surname>Hardy</surname> <given-names>J.</given-names></name> <name><surname>Schott</surname> <given-names>J. M.</given-names></name></person-group> (<year>2018</year>). <article-title>Alzheimer&#x00027;s disease</article-title>. <source>Eur. J. Neurol</source>. <volume>25</volume>, <fpage>59</fpage>&#x02013;<lpage>70</lpage>. <pub-id pub-id-type="doi">10.1111/ene.13439</pub-id><pub-id pub-id-type="pmid">28872215</pub-id></citation></ref>
<ref id="B37">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Lashgari</surname> <given-names>E.</given-names></name> <name><surname>Liang</surname> <given-names>D.</given-names></name> <name><surname>Maoz</surname> <given-names>U.</given-names></name></person-group> (<year>2020</year>). <article-title>Data augmentation for deep-learning-based electroencephalography</article-title>. <source>J. Neurosci. Methods</source> <volume>346</volume>, <fpage>108885</fpage>. <pub-id pub-id-type="doi">10.1016/j.jneumeth.2020.108885</pub-id><pub-id pub-id-type="pmid">32745492</pub-id></citation></ref>
<ref id="B38">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Lawhern</surname> <given-names>V. J.</given-names></name> <name><surname>Solon</surname> <given-names>A. J.</given-names></name> <name><surname>Waytowich</surname> <given-names>N. R.</given-names></name> <name><surname>Gordon</surname> <given-names>S. M.</given-names></name> <name><surname>Hung</surname> <given-names>C. P.</given-names></name> <name><surname>Lance</surname> <given-names>B. J.</given-names></name></person-group> (<year>2018</year>). <article-title>EEGNet: a compact convolutional neural network for EEG-based brain&#x02013;computer interfaces</article-title>. <source>J. Neural Eng.</source> <volume>15</volume>, <fpage>056013</fpage>. <pub-id pub-id-type="doi">10.1088/1741-2552/aace8c</pub-id><pub-id pub-id-type="pmid">29932424</pub-id></citation></ref>
<ref id="B39">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Manly</surname> <given-names>B. F. J.</given-names></name></person-group> (<year>2018</year>). <source>Randomization, Bootstrap and Monte Carlo Methods in Biology</source>. <publisher-loc>London</publisher-loc>: <publisher-name>Chapman and Hall/CRC</publisher-name>. <pub-id pub-id-type="doi">10.1201/9781315273075</pub-id><pub-id pub-id-type="pmid">21118243</pub-id></citation></ref>
<ref id="B40">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>McKhann</surname> <given-names>G. M.</given-names></name> <name><surname>Knopman</surname> <given-names>D. S.</given-names></name> <name><surname>Chertkow</surname> <given-names>H.</given-names></name> <name><surname>Hyman</surname> <given-names>B. T.</given-names></name> <name><surname>Jack</surname> <given-names>C. R.</given-names> <suffix>Jr</suffix></name> <name><surname>Kawas</surname> <given-names>C. H.</given-names></name> <name><surname>Klunk</surname> <given-names>W. E.</given-names></name> <etal/></person-group>. (<year>2011</year>). <article-title>The diagnosis of dementia due to Alzheimer&#x00027;s disease: recommendations from the National Institute on Aging-Alzheimer&#x00027;s Association workgroups on diagnostic guidelines for Alzheimer&#x00027;s disease</article-title>. <source>Alzheimer&#x00027;s Dement.</source> <volume>7</volume>, <fpage>263</fpage>&#x02013;<lpage>269</lpage>. <pub-id pub-id-type="doi">10.1016/j.jalz.2011.03.005</pub-id><pub-id pub-id-type="pmid">21514250</pub-id></citation></ref>
<ref id="B41">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Mimura</surname> <given-names>Y.</given-names></name> <name><surname>Nishida</surname> <given-names>H.</given-names></name> <name><surname>Nakajima</surname> <given-names>S.</given-names></name> <name><surname>Tsugawa</surname> <given-names>S.</given-names></name> <name><surname>Morita</surname> <given-names>S.</given-names></name> <name><surname>Yoshida</surname> <given-names>K.</given-names></name> <etal/></person-group>. (<year>2021</year>). <article-title>Neurophysiological biomarkers using transcranial magnetic stimulation in Alzheimer&#x00027;s disease and mild cognitive impairment: A systematic review and meta-analysis</article-title>. <source>Neurosci. Biobehav. Rev.</source> <volume>121</volume>, <fpage>47</fpage>&#x02013;<lpage>59</lpage>. <pub-id pub-id-type="doi">10.1016/j.neubiorev.2020.12.003</pub-id><pub-id pub-id-type="pmid">33307047</pub-id></citation></ref>
<ref id="B42">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Qu</surname> <given-names>Y.</given-names></name> <name><surname>Adam</surname> <given-names>B. L.</given-names></name> <name><surname>Thornquist</surname> <given-names>M.</given-names></name> <name><surname>Potter</surname> <given-names>J. D.</given-names></name> <name><surname>Thompson</surname> <given-names>M. L.</given-names></name> <name><surname>Yasui</surname> <given-names>Y.</given-names></name> <etal/></person-group>. (<year>2003</year>). <article-title>Data reduction using a discrete wavelet transform in discriminant analysis of very high dimensionality data</article-title>. <source>Biometrics</source> <volume>59</volume>, <fpage>143</fpage>&#x02013;<lpage>151</lpage>. <pub-id pub-id-type="doi">10.1111/1541-0420.00017</pub-id><pub-id pub-id-type="pmid">12762451</pub-id></citation></ref>
<ref id="B43">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Roy</surname> <given-names>G.</given-names></name> <name><surname>Bhoi</surname> <given-names>A. K.</given-names></name> <name><surname>Bhaumik</surname> <given-names>S.</given-names></name></person-group> (<year>2021</year>). <article-title>A comparative approach for MI-based EEG signals classification using energy, power and entropy</article-title>. <source>IRBM</source> (In Press). <pub-id pub-id-type="doi">10.1016/J.IRBM.2021.02.008</pub-id></citation>
</ref>
<ref id="B44">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Roy</surname> <given-names>Y.</given-names></name> <name><surname>Banville</surname> <given-names>H.</given-names></name> <name><surname>Albuquerque</surname> <given-names>I.</given-names></name> <name><surname>Gramfort</surname> <given-names>A.</given-names></name> <name><surname>Falk</surname> <given-names>T. H.</given-names></name> <name><surname>Faubert</surname> <given-names>J.</given-names></name></person-group> (<year>2019</year>). <article-title>Deep learning-based electroencephalography analysis: a systematic review</article-title>. <source>J. Neural Eng.</source> <volume>16</volume>, <fpage>051001</fpage>. <pub-id pub-id-type="doi">10.1088/1741-2552/ab260c</pub-id><pub-id pub-id-type="pmid">31151119</pub-id></citation></ref>
<ref id="B45">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Sale</surname> <given-names>M. V.</given-names></name> <name><surname>Mattingley</surname> <given-names>J. B.</given-names></name> <name><surname>Zalesky</surname> <given-names>A.</given-names></name> <name><surname>Cocchi</surname> <given-names>L.</given-names></name></person-group> (<year>2015</year>). <article-title>Imaging human brain networks to improve the clinical efficacy of non-invasive brain stimulation</article-title>. <source>Neurosci. Biobehav. Rev.</source> <volume>57</volume>, <fpage>187</fpage>&#x02013;<lpage>198</lpage>. <pub-id pub-id-type="doi">10.1016/j.neubiorev.2015.09.010</pub-id><pub-id pub-id-type="pmid">26409343</pub-id></citation></ref>
<ref id="B46">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Schirrmeister</surname> <given-names>R. T.</given-names></name> <name><surname>Springenberg</surname> <given-names>J. T.</given-names></name> <name><surname>Fiederer</surname> <given-names>L. D. J.</given-names></name> <name><surname>Glasstetter</surname> <given-names>M.</given-names></name> <name><surname>Eggensperger</surname> <given-names>K.</given-names></name> <name><surname>Tangermann</surname> <given-names>M.</given-names></name> <etal/></person-group>. (<year>2017</year>). <article-title>Deep learning with convolutional neural networks for EEG decoding and visualization</article-title>. <source>Hum. Brain Map.</source> <volume>38</volume>, <fpage>5391</fpage>&#x02013;<lpage>5420</lpage>. <pub-id pub-id-type="doi">10.1002/hbm.23730</pub-id><pub-id pub-id-type="pmid">28782865</pub-id></citation></ref>
<ref id="B47">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Sharma</surname> <given-names>K.</given-names></name></person-group> (<year>2019</year>). <article-title>Cholinesterase inhibitors as Alzheimer&#x00027;s therapeutics (Review)</article-title>. <source>Mol. Med. Rep</source>. <volume>20</volume>, <fpage>1479</fpage>&#x02013;<lpage>1487</lpage>. <pub-id pub-id-type="doi">10.3892/mmr.2019.10374</pub-id><pub-id pub-id-type="pmid">31257471</pub-id></citation></ref>
<ref id="B48">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Sonmez</surname> <given-names>A. I.</given-names></name> <name><surname>Camsari</surname> <given-names>D. D.</given-names></name> <name><surname>Nandakumar</surname> <given-names>A. L.</given-names></name> <name><surname>Voort</surname> <given-names>J. L. V.</given-names></name> <name><surname>Kung</surname> <given-names>S.</given-names></name> <name><surname>Lewis</surname> <given-names>C. P.</given-names></name> <etal/></person-group>. (<year>2019</year>). <article-title>Accelerated TMS for depression: a systematic review and meta-analysis</article-title>. <source>Psychiatry Res</source>. <volume>273</volume>:<fpage>770</fpage>&#x02013;<lpage>781</lpage>. <pub-id pub-id-type="doi">10.1016/j.psychres.2018.12.041</pub-id><pub-id pub-id-type="pmid">31207865</pub-id></citation></ref>
<ref id="B49">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Syarif</surname> <given-names>I.</given-names></name> <name><surname>Prugel-Bennett</surname> <given-names>A.</given-names></name> <name><surname>Wills</surname> <given-names>G.</given-names></name></person-group> (<year>2016</year>). <article-title>SVM parameter optimization using grid search and genetic algorithm to improve classification performance</article-title>. <source>Telkomnika</source> <volume>14</volume>, <fpage>1502</fpage>. <pub-id pub-id-type="doi">10.12928/telkomnika.v14i4.3956</pub-id></citation>
</ref>
<ref id="B50">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Wang</surname> <given-names>J. X.</given-names></name> <name><surname>Rogers</surname> <given-names>L. M.</given-names></name> <name><surname>Gross</surname> <given-names>E. Z.</given-names></name> <name><surname>Ryals</surname> <given-names>A. J.</given-names></name> <name><surname>Dokucu</surname> <given-names>M. E.</given-names></name> <name><surname>Brandstatt</surname> <given-names>K. L.</given-names></name> <etal/></person-group>. (<year>2014</year>). <article-title>Targeted enhancement of cortical-hippocampal brain networks and associative memory</article-title>. <source>Science</source> <volume>345</volume>, <fpage>1054</fpage>&#x02013;<lpage>1057</lpage>. <pub-id pub-id-type="doi">10.1126/science.1252900</pub-id><pub-id pub-id-type="pmid">25170153</pub-id></citation></ref>
<ref id="B51">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Wang</surname> <given-names>X.</given-names></name> <name><surname>Gong</surname> <given-names>G.</given-names></name> <name><surname>Li</surname> <given-names>N.</given-names></name> <name><surname>Qiu</surname> <given-names>S.</given-names></name></person-group> (<year>2019</year>). <article-title>Detection analysis of epileptic EEG using a novel random forest model combined with grid search optimization</article-title>. <source>Front. Hum. Neurosci.</source> <volume>13</volume>, <fpage>52</fpage>. <pub-id pub-id-type="doi">10.3389/fnhum.2019.00052</pub-id><pub-id pub-id-type="pmid">30846934</pub-id></citation></ref>
<ref id="B52">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Weiler</surname> <given-names>M.</given-names></name> <name><surname>Stieger</surname> <given-names>K. C.</given-names></name> <name><surname>Long</surname> <given-names>J. M.</given-names></name> <name><surname>Rapp</surname> <given-names>P. R.</given-names></name></person-group> (<year>2020</year>). <article-title>Transcranial magnetic stimulation in Alzheimer&#x00027;s disease: are we ready?</article-title> <source>eNeuro</source> <volume>7</volume>. <pub-id pub-id-type="doi">10.1523/ENEURO.0235-19.2019</pub-id><pub-id pub-id-type="pmid">31848209</pub-id></citation></ref>
<ref id="B53">
<citation citation-type="book"><person-group person-group-type="author"><name><surname>Wojtaszczyk</surname> <given-names>P.</given-names></name></person-group> (<year>1997</year>). <source>A Mathematical Introduction to Wavelets</source> (No. 37). <publisher-loc>Cambridge</publisher-loc>: <publisher-name>Cambridge University Press</publisher-name>. <pub-id pub-id-type="doi">10.1017/CBO9780511623790</pub-id></citation>
</ref>
<ref id="B54">
<citation citation-type="journal"><person-group person-group-type="author"><name><surname>Zou</surname> <given-names>H.</given-names></name> <name><surname>Hastie</surname> <given-names>T.</given-names></name></person-group> (<year>2005</year>). <article-title>Regularization and variable selection via the elastic net</article-title>. <source>J. R. Stat. Soc. Ser. B.</source> <volume>67</volume>, <fpage>301</fpage>&#x02013;<lpage>320</lpage>. <pub-id pub-id-type="doi">10.1111/j.1467-9868.2005.00503.x</pub-id></citation>
</ref>
</ref-list> 
</back>
</article>
