Perceived Mental Workload Classification Using Intermediate Fusion Multimodal Deep Learning - Dolmans, Tenzing C.; Poel, Mannes; van ’t Klooster, Jan-Willem J. R.; Veldkamp, Bernard P. (Frontiers in Human Neuroscience, 2021)
A lot of research has been done on detection of mental workload using various bio-signals. Recently, deep learning has allowed for novel methods and results. A plethora of modalities have proven to be valuable in this task, yet studies currently often only use a single modality to classify mental workload. The goal of this research was to classify perceived mental workload using a deep neural network that flexibly makes use of multiple modalities, in order to allow for feature sharing between modalities. To achieve this goal, an experiment was conducted in which mental workload was simulated with the help of verbal logic puzzles. The puzzles came in five levels of difficulty and were presented in a random order. Participants had one hour to solve as many puzzles as they could. Between puzzles they gave a difficulty rating between one and seven, seven being the highest difficulty. Galvanic skin response, photoplethysmograms, functional near-infrared spectrograms, and eye movements were collected simultaneously using LabStreamingLayer. Marker information from the puzzles was also streamed on LabStreamingLayer. We designed and evaluated a novel intermediate fusion multimodal deep neural network for the classification of perceived mental workload using the aforementioned four modalities. Two main criteria that guided the design and implementation of our deep neural network are modularity and generalisability. We were able to classify perceived mental workload within-level accurate (0.985 levels) on a seven-level workload scale using the aforementioned modalities. The model architecture allows for easy addition and removal of modalities without major structural implications because of the modular nature of the design. Furthermore, we showed that our neural network performed better when using multiple modalities, as opposed to a single modality. The dataset and code used in this paper are openly available.