A novel 4-channel EEG device was developed as part of this project, offering a significant advantage in terms of usability compared to conventional EEG systems. The headband (Figure 2), designed with dry electrodes, is user-friendly and easy to apply, making it accessible to many users. It records four EEG channels from the occipital region, with an additional electrode placed just above the right ear as the reference and ground. The device captures EEG with 24-bit resolution at a sampling rate of 250 Hz. It transmits the data wirelessly via Bluetooth Low Energy (BLE), ensuring efficient and reliable data transmission with minimal latency.

The dress components were designed using PTC’s Onshape cloud-native product development platform (PTC Inc., Boston, MA, USA) and 3D-printed by HP Inc. (HP Inc., Palo Alto, CA, USA) using their Jet Fusion 3D Printing Solution. The dress was fabricated with HP’s HPMulti Jet Fusion 5,420 W printer and HR 3D PA12 W material. Interactive elements of the dress include Hyperpixel 2.1 round displays (Pimoroni Ltd., Sheffield, UK), controlled by a Raspberry Pi Zero 2 W (Raspberry Pi Foundation, Cambridge, UK), featuring a 1GHz quad-core 64-bit Arm Cortex-A53 CPU, 512 MB SDRAM, and 2.4GHz wireless LAN, meeting the system’s visualization requirements. The 3D eyes, designed in Unity (Unity Technologies, San Francisco, CA, USA), were connected to a UDP receiver socket, enabling real-time control of eye dilation and movement based on data received via the network path.

Connecting BCI and the dress allows the dress to react to the wearer’s brain activity in real-time. First, a machine learning algorithm was trained by acquiring data from the specific user in different mental states. Afterward, new data was fed through the BCI. This information was then used in real-time to calculate the level of engagement and visualize it by adapting eye movements on the screens, such as dilation, speed, etc. Data from one representative participant was analyzed and presented in this paper to demonstrate the system’s functionality and performance. The analysis was based on data collected from a 37-year-old healthy female participant during the exhibition settings.

As detailed in Natalizio et al. (2024), a specialized application was developed to present stimuli, acquire EEG, and process real-time data. This application processes EEG data to estimate user engagement levels and provides real-time feedback, such as current engagement estimates. For example, in the study by Natalizio et al. (2024), the application involved estimating engagement during gameplay by monitoring participants’ interaction with Tetris at varying speed levels and assessing engagement while watching different video content. Before initiating measurements, the system assesses signal quality, continuously monitoring and reporting noisy channels. Users can also select from various interaction paradigms. Specifically, the d2 test paradigm (Figure 3) was employed to train the classification model, including the d2 test and a fixation cross. The application calculates reliable performance scores based on the d2 test paradigm. After the initial test, the system evaluates the model’s accuracy in distinguishing between engagement (d2 test) and rest (fixation cross) states. The model can be retrained by the user as needed. The calibration phase, including electrode preparation, d2 test execution, and model training, is completed within approximately 5 min. Once trained, the model enables real-time tracking of user engagement in any external task, which, in this project, controls the movement and dilation of 3D-rendered eyes on six hyperpixel screens integrated into the Screen Dress.

The proposed classification model is optimized for low computational cost to support real-time BCI experiments. A filter bank common spatial patterns (CSP) approach was applied to estimate engagement, with models trained individually for each user. Raw EEG signals were notch-filtered at 50 Hz and further processed with bandpass filters (4–8 Hz, 6–10 Hz, and 8–12 Hz) to exclude higher frequency components that could be affected by muscle artifacts from task-related tension. EEG data were segmented into 1-s windows, and CSP was used to extract features that maximize the variance between engagement and resting states. These features were then used to train a linear discriminant analysis (LDA) model, which outputs a binary classification label and a continuous score to estimate user engagement (Natalizio et al., 2024).

Figure 4 provides a schematic overview of the BCI control system. The process begins with extracting raw EEG data from the four sensors transmitted via Bluetooth Low Energy (BLE) to the control PC. The PC performs signal processing as described earlier and trains the classifier. After the classifier is trained, the system calculates the scores in real-time and transmits them via Wi-Fi to the Hyperpixel unit. The RPi at the Hyperpixel unit hosts a Unity application that reacts to the values received via UDP and employs them to control eye movements. The scores are divided into two groups: positive and negative. Positive scores, indicating a greater likelihood of class 1 (d2 test), cause the eyes to perform rapid horizontal movements and increase pupil diameter. Negative scores, corresponding to class 2 (resting state), reduce eye movement and pupil constriction. The left and right eye displays mirror each other, following the same movement and dilation patterns.

Figure 1B depicts the Pangolin Dress worn by a model. All the dress parts are 3D printed via selective laser sintering (SLS) from PA-11 and PA-12 (nylon) at Shapeways (Shapeways Inc., New York, NY, USA). This approach led to a very lightweight dress with custom housings for actuators and LEDs mounted on the scales. It also made it possible to design the dress to form something akin to an exoskeleton around the body. With the pangolin as an inspiration, the designer created diamond-shaped scales covering the model’s body. Similar to the ones of the animal that protect the pangolin animal against predators. The housings of the servomotors that move the scales are shaped like eggs. The drive axis led out of the housing so the scales could be mounted. On top of the scales, multicolor LEDs were fixed in a recess.

We used an Arduino nano microcontroller to move the scales and interact with the multicolor LEDs (see Figure 7A1). We chose this board due to its small size, output pin structure, and easy programming interface. Digital metal gear servomotors (Corona DS-939MG) were employed for moving the scales. A total of 32 servomotors were installed into the 3D-printed dress. The motors were placed inside the 3D-printed structure and closed via a cover to look like eggs (see Figure 7B). Two (PCA9685) motor driver boards (Adafruit Industries LLC, USA), one of which can drive 16 motors, were adopted for control purposes (see Figure 7A2). Pulse width modulation (PWM) was chosen for interaction with the motors and positioning of the axle. The scale was mounted onto the axle using a distance part for optimal freedom of movement. The entire device was powered by a single battery pack (see Figure 7A3), with data transmitted via USB serial communication (see Figure 7A4).

Data from one representative participant was analyzed and presented in this paper to demonstrate the system’s functionality and performance. For this project, the analysis focused on data collected from a 31-year-old healthy female participant during the exhibition settings.

The BCI user in this project was designated as being in an idle state or one of three other states:

Theta waves (Θ), which range from 4 to 8 Hz, are the slowest frequencies used for control in this BCI system. These waves are commonly associated with states of deep relaxation and inward focus, as well as early sleep stages (Vyazovskiy and Tobler, 2005). Additionally, theta activity in the prefrontal cortex has been connected to the “flow state,” which is characterized by enhanced creativity and cognitive engagement (Katahira et al., 2018). In the system context, the dress color purple represents the presence of theta waves, symbolizing creativity and meditation.

Alpha waves (α), ranging between 8 and 12 Hz, are typically observed when users are awake but relaxed, especially with closed eyes. In motor imagery-based BCI research, the alpha band is also called the mu band (Pfurtscheller et al., 2006). Alpha rhythms are most prominent over the occipital cortex at the back of the head and have been linked to various aspects of human life, including sensorimotor functions (Neuper and Pfurtscheller, 2001; Hanslmayr et al., 2005), psycho-emotional markers (Cacioppo, 2004; Allen et al., 2018), and physiological research (Cooray et al., 2011; Halgren et al., 2019). In this system, the dress color blue represents alpha waves, reflecting a state of relaxed wakefulness.

Beta waves (β), which occupy the frequency range of 12–35 Hz, are associated with alertness, attention, and stress. These waves are most prominent when individuals are focused or experiencing stress, making them a key marker of heightened cognitive engagement (Schreiner et al., 2021). Due to the wide range of frequencies, the beta band is often divided into subbands, although precise definitions of these subbands vary in the literature. Frequencies above 35 Hz are classified as gamma waves. They are often discussed in EEG research related to human emotions (Yang et al., 2020) and motor-related functions, mainly through invasive methods such as ECoG (Kapeller et al., 2014; Gruenwald et al., 2017). In the BCI system, white represents beta waves, which signal alertness or stress.

This framework of brainwave frequencies forms the basis of the BCI control system, where changes in cognitive states are mapped to specific visual feedback, as seen through the dynamic color changes of the dress.

The BCI system must classify features calculated from the measured EEG to enter one of the above states. Data acquisition and online signal processing were performed using g.HIsys Professional (g.tec medical engineering GmbH, Austria) running under MATLAB/Simulink (The MathWorks, Inc., USA). Figure 8 depicts the BCI system’s preprocessing, feature extraction, and class decision steps.

The data were notch-filtered at 50 Hz using a 4th-order Butterworth filter. Next, bad channels were detected using the signal quality scope from the g.HIsys online processing platform. A common average reference (CAR) was used to reference all channels, excluding those classified as having bad signal quality. Finally, a 0.5–40 Hz bandpass filter was selected to pre-filter the frequency band of interest.

For feature extraction, the band powers for the specified frequency ranges were estimated continuously and used for further processing. First, the EEG data were bandpass filtered for the respective frequency band (Θ, α, β). For power estimation, a moving average with a buffer of 256 samples ( N buffer ) and an overlap of 128 samples ( N overlap ) was selected. The sampling rate of the EEG amplifier was 256 Hz ( f system ),which then resulted in an update rate of 2 Hz for the band power features ( f band ) (see Equation 1) To improve Gaussianity, band power features were log-transformed since they are commonly Chi-squared distributed otherwise. The power from each channel of the uHD EEG systems was calculated online in real-time.

For each channel, the respective frequency band features were calculated (Θ, α, β). This information was then used in real-time to calculate the dress states’ class decisions.

To enhance the sensitivity of the BCI system to cognitive states, additional weighting was assigned to the features extracted from electrode grids based on their neuroanatomical locations. These locations were grouped into three key brain regions, as illustrated in Figure 9. Electrode grids positioned over the frontal lobe (green, Figure 9B) were weighted higher for theta wave (Θ) detection, reflecting the region’s role in creative processes and the flow state. The frontal electrodes were specifically assigned a weight of 2 for the theta band, while the weights for the alpha and beta bands remained at 1. The occipital cortex, crucial for visual processing, strongly influences alpha activity. Therefore, electrodes placed over the occipital region (dark blue, Figure 9B) were given a higher weight for alpha waves (α). The occipital region’s weighting for the alpha band was 2, while the beta band remained weighted at 1. Electrodes placed over the pre-and postcentral motor cortices (blue and red, Figure 9B), which are associated with motor control and alertness, were weighted more heavily for beta wave (β) detection. These regions were specifically assigned a weight of 2 for the beta band, while the weights for the theta and alpha bands remained at 1.

This weighting strategy ensures that the brain regions most relevant to each state have a stronger influence on the BCI system’s performance.

To assess the system’s performance, we designed specific tasks for each cognitive state (Theta, Alpha, Beta). Each task was performed for 20 trials for 1 min each, providing sufficient data for analysis. The tasks were chosen to elicit targeted brainwave activity associated with the respective cognitive states:

Alpha (α): Relaxed wakefulness

Beta (β): Alertness and stress

The BCI system requires calibration for each user. To achieve this, the participant undergoes a training procedure in front of a computer screen. The training protocol consists of two rounds of the d2 test and two resting periods, each lasting 1 min, during which a fixation cross is displayed on the screen. This results in a total training time of 4 min. Following the acquisition of training data, the classifier is trained. A within-subject classification model was developed for the representative subject to distinguish between engaging and resting states using EEG data recorded during a d2 test-based paradigm. The EEG was captured from four electrodes, and the model was trained utilizing filter-bank common spatial patterns and linear discriminant analysis.

After training, an evaluation run is conducted to assess the classifier’s performance. The calculated scores over time from the evaluation run for the representative subject are shown in Figure 10A. The d2 test, marked in red, is assigned a label of +1, while the resting condition, marked in red, is assigned a label of −1. It is evident that the score values consistently align with the corresponding class for the task being performed. Based on the correct and incorrect estimates during the evaluation run, the classification accuracy reached 97.1%. In contrast, the chance level would be 50% as this is a two-class problem. Furthermore, a significance test using permutation statistics was performed to estimate the probability that the observed performance of 97.1% was obtained by chance (Ojala and Garriga, 2009). Labels and corresponding score values are available in non-overlapping 1-s segments, resulting in 240 segments in total (Figure 10A, 120 s of engaging and 120 s of resting state). The following procedure was performed for B = 10,000 times: Randomly permuting (i.e., shuffling) the labels breaks the relationship between the state and the estimates scores and a permutation accuracy can be calculated, which is based on the assumption that there is no relationship between the labels and the scores. As this procedure was performed 10,000 times, one obtains 10,000 permutation accuracies and then an empirical p-value can be computed based on:

With A c c perm being the 10,000 permutation accuracies, A c c o b s being the observed accuracy of 97.1% and B being 10,000 (see Equation 2). In other words, one calculates how often the permutation accuracy was greater or equal to the observed accuracy. Here, we obtained a p-value of 9.99E-5 indicating a highly significant model performance.

Figure 10B shows the Score distribution reflects the probability density estimated for the scores during the Rest and Engaging condition, respectively. A non-parametric kernel with a width of 1 was used to fit the distribution.

The trained classifier was applied to new incoming data after calibrating the system and confirming its satisfactory performance. EEG features were extracted in real-time, and corresponding score values were continuously computed and output in real-time. This allowed for constant monitoring and control of the score values. The real-time score was then used to modulate the dilation and movement of the pupil in the animated 3D eye model (see Figure 10C).

To represent the pupil dilation and movement as a function of the score value (ranging from −10 to +10), we can define a linear equation that maps this range to the desired changes in dilation and movement. Let us assume: s is the score value (ranging from −10 to +10), P_dilation(s) represents the pupil dilation, where a positive score increases dilation, and a negative score decreases it, and P_movement(s) represents the pupil movement, which could be proportional to the score value.

We can define the dilation and movement equations as follows:

Where P_0 is the baseline pupil dilation (when the score is 0) (see Equation 3), M_0 is the baseline position of the pupil (when the score is 0) (see Equation 4), α and β scaling factors determine how much pupil dilation and movement change in response to the score. This model assumes a linear relationship for simplicity, but nonlinear models could also be used depending on the desired dynamic behavior.

The Screen Dress project was showcased at the ARS Electronica Festival 2023¹. The Ars Electronica Center is a major public science museum in Linz, Austria. Multiple wearers were selected to interact with the Screen Dress during the exhibition using the BCI system. For each wearer, a new classifier was trained and applied to the incoming EEG data streams in real-time. We conversed and interacted with other art exhibits throughout the festival to observe the Screen Dress’s responses. Attendees were notably impressed by the rapid reactions of the digital eyes, which provided subconscious, real-time feedback about the wearer’s engagement. This added an interaction layer, creating a unique experience for bystanders. Videos of the dress, the exhibition, and testing can be found online² (see Supplementary material for corresponding links).

For each of the 64 electrode grids, the respective frequency band features were calculated in real-time, focusing on theta (Θ), alpha (α), and beta (β) bands. These frequency features were subsequently utilized to determine the state of the animatronic dress. Additional weights were applied to the features based on their corresponding neuroanatomical locations, as shown in Figure 9. The electrode grids were grouped into three distinct neuroanatomical regions for analysis.

Once the frequency features were weighted according to their region, the system performed online classification to decide the dress’s state. The dress could distinguish between three mental states and an idle state. During the idle state, the dress’s motors returned to their default positions, and all LEDs were deactivated.

The system’s performance in detecting cognitive states during task-specific evaluation is summarized in the confusion matrix below. Each cell represents the percentage of trials classified as the predicted state for a given true state. Labels and corresponding accuracy values are computed for non-overlapping 1-s segments. The proportion of total seconds during which power values from specific frequency bands correctly corresponded to the intended cognitive state provides further insight into the system’s accuracy (see Table 1).

For each state, the observed accuracy (proportion of trials correctly associated with the intended state: Θ: 64%, α: 91%, β: 78%) was compared against a null distribution generated through 10,000 random permutations of the labels (see Equation 1 in BCI Interaction Screen Dress). This approach breaks the relationship between the task and the frequency band power, creating a distribution of accuracies under the assumption of no association. The mean permutation accuracy was 33% confirming the expected chance level under random guessing for a three-state system. The observed accuracies for all three states were significantly higher than the permutation accuracy distribution (p < 0.001p).

A minimum interaction time was implemented to ensure a smooth transition between states. The dress was maintained in each state for at least 6 s to avoid rapid changes. Additionally, a threshold for minimum band power was defined to ensure that the BCI system only reacted when the power values in at least one of the frequency bands exceeded this threshold. Once this condition was met, the dress executed the pattern corresponding to the state with the highest power.

After performing the pattern, the system reassessed the frequency power values and, if applicable, updated the dress’s state and corresponding movement and lighting animations. This process allowed the dress to operate autonomously, driven entirely by real-time BCI decisions. However, manual control was also available for demonstration purposes.

The dress looks angelic when it is turned off (idle). In action, it becomes a flowing canvas of color, express and admirable in form and movement. The dress scales move up and light up with animated color patterns depending on the wearer’s brain state. The dress remained in one of the four states until it changed to a new state or was turned off. The following patterns were programmed:

We presented the pangolin scales project at the ARS Electronica Festival 2020³. A model wore both the BCI and the dress for this presentation. The live presentation included preparing the BCI system (electrode preparation, mounting, data acquisition procedure) and the actuation of the dress. Finally, both components were linked, and the dress performed its animations according to the model’s brain state determined by the BCI. The positioning of all 64 electrode grids (preparing them and attaching them to the scalp) took about 2 h (see Figure 12). We asked the model wearing the Pangolin dress to actively engage and interact with the venue, allowing the dress to respond in real-time to the model’s cognitive states. This dynamic feedback provided an additional interactive layer, offering bystanders a unique and immersive experience. Videos documenting the dress in action and footage from the exhibition and testing are available online⁴ (see Supplementary material for corresponding links).