How does it work?
Watch a film through a desktop client, check your subconscious rating based on emotion recognition in a web app.
1.1
Web app demo
Video
Subconscious Reaction to a Film
A film review is usually a combination of objective analysis of aspects of a film, like cinematography and acting, with subjective evaluation of aspects such as emotional impact and authenticity. Both the objective and subjective sides of a film review involve conscious analysis. This project aims to bring the viewer information on their subconscious, visceral reaction to a film. It is not in any way meant to be a standalone review system. The purpose is to provide a source of previously unknown information that a film viewer can take into consideration when forming an opinion.
Narrated demo & explanation:
1.2
Explanation
Video
Development
Emotion classification
Firstly, a neural network was created and trained on a dataset of faces. It can recognise five emotions - happy, sad, neutral, surprised, angry.
Desktop client app
A PC app was built for data collection. It guides the user to choose a film to watch and starts the playback using a VLC API. While the film is played, the app collects image data from the webcam, processes it and feeds it into the neural network. Emotions are recorded and stored for every second of the film playback. The camera data is also used to calculate the face movement of the user, which is used together with the facial expression data to infer the intensity of the recorded emotions.
Subtitles
The appropriate subtitle file is processed and correlated with playback time from the VLC API to record what was said during each second of the film.
2.1
System diagram
Image
2.2
Emotion recognition
Image
2.3
Emotion ecognition
Image
2.4
Emotion ecognition
Image
2.5
Client app - playback starting
Image
Data storage
MongoDB
Sixty seconds of this data - emotions, movement, playback time, lines from the script - is stored and transferred every minute to a MongoDB Atlas Server. A new collection is created for each film. Each minute of data is stored as a single document inside this collection.
3.1
Python data packet
Image
3.2
MongoDB dictionary
Image
Web app
Dash + Python
Client-side processing
A web app was developed and hosted locally. It pulls data from the Mongo Server and visualises it in a dashboard using the Dash module. The data is processed on load, directly on the client’s device using Python. Additional metrics like film rating, a favourite section of film, distracted time and predominant emotions are inferred from the data. The Plot.ly library was used to display interactive graphs to visualise how the user’s movement and emotions developed throughout the film. An IMDb rating of the film, director and cast are also displayed using the IMDb Python API.
4.1
Film collection with ratings
Video