Investigation of Multimodal Biometric-Based Methods for the Development of an Emotion Detection System

Abstract

Human emotion recognition is significant application for intelligent human–computer interaction, healthcare monitoring, adaptive learning, security systems, etc. Single-modal emotion detection systems usually do not perform well in terms of accuracy and robustness in real time. This paper introduces the concept a Multimodal Biometric Emotion Recognition System (MBERS) which combines Electroencephalography (EEG) signals and facial expressions in the field of emotional state recognition. It uses median filtering and normalizing techniques during the pre-processing phase and feature extraction is done using Gabor Filters (GF), Principal Component Analysis (PCA), and Short-Time Fourier Transform (STFT). The classification is done using a tool called a Neural Network (NN) and Support Vector Machine (SVM). It captures multimodal outputs and fuses them effectively at the decision level by adopting sum and product rules in the framework. In the experimental analysis, emotional data from 72 individuals were taken from emotional movies shown consisting of various emotions such as happy, sad, anger, fear, surprised, disgusted and neutral. The performance was measured by the different accuracy and evaluation parameters including accuracy, precision, recall, F1-Score, TNR and FDR. The proposed MBERS achieved the better emotion recognition performance, which reflected an overall accuracy of 97.1%, it demonstrated that this approach is more robust, reliable and effective than the existing emotion recognition approaches.