Affective analysis of visual scenes using face pareidolia and scene-context

With recent advancements in affective computing, computers have become better at describing human emotional responses using facial expression and scene-context analysis. However, the topic of face pareidolia, which is the imagination of non-existent faces in random patterns, its computational modelling and application in product design is recently gaining increased attention in the research community. This thesis aims to answer an interdisciplinary research question, "Can we simulate the face pareidolia ability of humans and predict associated emotional responses using deep learning?" This interdisciplinary thesis question was based on research literature published in the disciplines of Cognitive Science and Psychology. Findings from these research disciplines suggest that abstract and minimal face-like patterns can subconsciously activate face-selective regions in the brain producing face pareidolia. Moreover, face pareidolia, along with various other contextual variables present in the visual scene, can trigger an emotional response in our neurobiological systems without conscious awareness. The first part of this thesis investigates a group emotion recognition task by using an ensemble of convolutional neural networks. The findings from this part are consistent with the literature in cognitive science and psychology that an individual's face and associated facial expressions are always perceived and evaluated within a surrounding context. The second part of this thesis presents two research studies addressing the poor generalisation capacity of convolutional neural networks, as poor generalisation capacity limits the direct application of deep learning techniques to simulate face pareidolia. In the first study, the impact of image resolution on the generalisation capacity of convolutional neural networks trained for facial emotion recognition was analysed. The second study was inspired by face perception in the brain and improves the generalisation capacity of a convolutional neural network, trained for face recognition, to simulate the facial pareidolia capability of the human visual system. The third part of this thesis consolidates previous findings and presents a novel deep learning-based cross-domain weakly supervised three-step progressive domain adaptation framework. The proposed framework can simulate face pareidolia and predict associated emotional responses in two-dimensional valence and arousal space for various product designs. The proposed framework was evaluated on a new image dataset. Both quantitative and qualitative experimental results show that our approach can outperform other state-of-the-art methods.