Film Rating using Emotion Recognition
Module
IoT
I developed a film-rating system based on the emotion and movement of the viewer. It consists of a PC media player app that starts the film and initiates data collection, stores the data on a remote Mongo DB server and a website powered by a Python back-end that displays insights from the collected data.
Subconscious Reaction to a Film
A film review is usually a combination of objective analysis of aspects of a film, like directing, plot, cinematography and acting, with subjective evaluation of aspects such as emotional impact and authenticity. Both the objective and subjective sides of a film review involve conscious analysis. This project aims to bring the viewer information on their subconscious, visceral reaction to a film. It is not in any way meant to be a standalone review system. The purpose is to provide a source of nuanced, previously unknown information that a film viewer can take into consideration when forming an opinion.
Development
Firstly, a neural network was created and trained on a dataset of faces. It can recognise five emotions - happy, sad, neutral, surprised, angry.
A PC app was built for data collection. It guides the user to choose a film to watch and starts the playback using a VLC API. While the film is played, the app collects image data from the webcam, processes it and feeds it into the neural network. Emotions are recorded and stored for every second of the film playback. The camera data is also used to calculate the face movement of the user, which is used together with the facial expression data to infer the intensity of the recorded emotions.
The appropriate subtitle file is processed and correlated with playback time from the VLC API to record what was said during each second of the film.
A PC app was built for data collection. It guides the user to choose a film to watch and starts the playback using a VLC API. While the film is played, the app collects image data from the webcam, processes it and feeds it into the neural network. Emotions are recorded and stored for every second of the film playback. The camera data is also used to calculate the face movement of the user, which is used together with the facial expression data to infer the intensity of the recorded emotions.
The appropriate subtitle file is processed and correlated with playback time from the VLC API to record what was said during each second of the film.
Data Storage
Sixty seconds of this data - emotions, movement, playback time, lines from the script - is stored and transferred every minute to a MongoDB Atlas Server. A new collection is created for each film. Each minute of data is stored as a single document inside this collection.
Web App
A web app was developed and hosted locally. It pulls data from the Mongo Server and visualises it in a dashboard using the Dash module. The data is processed on load, directly on the client’s device using Python. Additional metrics like film rating, a favourite section of film, distracted time and predominant emotions are inferred from the data. The Plot.ly library was used to display interactive graphs to visualise how the user’s movement and emotions developed throughout the film. An IMDb rating of the film, director and cast are also displayed using the IMDb Python API.
