Emotions matter. They are at the core of human experience, shape our lives in the profoundest of ways and trigger our decision-making and social behaviour. Evolutionary theories indicated that emotions have an important survival functions and helped us solve certain challenges during our evolution. In social and business sciences, emotions are associated with the idea of human irrationality and non-logical behaviour. Affective computing deals with the study and development of digital systems and devices that can recognize, interpret, process, and simulate human affects. The new interplay of research findings from consumer behaviour, psychology, neuroscience and computer science opens up a new research universe and is attracting more and more scientists. Measuring human emotions with cutting-edge and AI-based technology support researchers on their mission to further explore these new worlds and set up experimental projects.
Integrate AI-based emotion analytics into your research
- Fresh up your scientific research with new ways of data collection and emotion analytics beyond the traditional self-reporting, observation and interview techniques. These are used for decades in research, now it is time for digital enhancements.
- You only need to integrate video recordings in the experimental setting and data collection process as the basis for cutting-edge emotion analysis based on AI-algorithms.
- You can access all raw data as .csv for further processing and statistical analysis in your tool of choice, e.g. R, SPSS, Python etc.
- Go beyond self-reporting surveys. With Emotion AI it is easy to create alternative experiments to measure emotional responses without the known bias of self-assessment techniques. Be able to move out of the labs into the wild and natural usage scenarios for data collection.
- Go beyond interviews and observations. The science of facial coding, computer vision and machine learning to understand human affective states is getting so precise that it rivals psychologists and trained human observers.
Some selected examples for research projects with TAWNY technology:
Effect of nudging in
online-shopping environments
regarding the consumption of fast food
Nudges are nowadays commonly used by companies and governments to prompt consumer choices in online and offline settings. However, the reasons for the effectiveness of specific characteristics in nudges is still scarce. The aim of the research is to understand the effectiveness and the impact of nudges to reduce caloric intake. The avatar-based nudges displayed either a face of dissatisfaction when the participant chooses a meal over the calorie intake goal, or a facial expression of satisfaction when the calories stay within the recommended goal. During the whole experiment, the participants' facial expressions were recorded and the triggered user emotions were measured with TAWNY technology to answer the question of what made the avatar more effective than the other nudge options.
Deep Flow: Detecting Optimal User Experience From Physiological Data
Using Deep Neural Networks
The affective state called flow is described as a state of optimal experience, total immersion and high productivity. As an important metric for various scenarios ranging from (professional) sports to work environments to user experience evaluations, it is extensively studied using traditional questionnaires. In order to make flow measurement accessible for online, real-time environments TAWNY technology was used in an experimental setting to automatically estimate a user’s flow state based on physiological signals measured with a wearable device. We conducted a study of subjects playing the game Tetris in varying difficulty levels, leading to boredom, stress, and flow. Using a convolutional neural network, we achieve an accuracy of 70% in recognizing flow-inducing levels. In the future, we expect flow to be a potential reward signal for human-in-the-loop reinforcement learning systems.
How to Stay in Flow at Work –
An Experimental Approach
To investigate the occurrence of flow during work as well as its connection to performance and physiological parameters, 59 subjects performed a simulated work task in an experiment. The participants had to retyped different texts on a laptop. These texts had three different difficulty levels: simple (children’s stories), medium and hard (chemistry science text), with the goal to induce three different performance and flow levels as well as physiological activations. Next to video data of the test persons the heart rate, R-R intervals and skin conductance were measured with the TAWNY setup.
The Impact of Ad Breaks -
Tested on an Online Video Platform
A test and control group (n=33) was shown a trailer with/without commercial breaks. Arousal and valence of the test persons were measured with the Tawny emotion scan. Significant interaction effects between the ad and the environment were found. This brings the discussion about the advertising environment back to the foreground, which has been neglected for a long time but is gaining relevance through the common practice of programmatic buying for the automated placement of advertising media in online environments.
Researching mystery deals’ impact on
consumer purchase motivation & loyalty
Researchers from the University of Passau used the TAWNY Emotion Analytics platform in an experiment to analyze people’s facial expressions when they receive and unbox a mystery deal, in addition to a verbal interview. The task was to understand which affective reactions are triggered by Mystery Deals when consumers first encounter the deal (pre-purchase phase) and, especially, after the mystery is revealed (post-purchase phase). The research also investigated how affective reactions drive purchase motivation and brand loyalty.
Emotion AI:
New Forms of AI-powered Emotion Analytics
In this paper, we built on existing emotion theories and explained the concept of emotional artificial intelligence. In particular, the focus of this paper was to compare traditional and modern AI-based methods for emotion recognition. For this purpose, using an experiment in online shopping, we compared a traditional self-report method with our automated and AI-based emotion recognition software, which were equally used to measure user experience. We also showed how, especially in market and consumer research, there are numerous possible applications for Emotion AI technology.
EROC:
Emotional Reactions on Creatives
The EROC test sets new standards in the evaluation of out-of-home advertising media. Through the novel and unique combination of classic advertising media KPIs, such as enforcement and likeability, and the implicit emotion measurement - powered by TAWNY Emotion AI - EROC enables a holistic assessment of the impact of the advertising media. Thus, EROC captures the conscious and unconscious reactions of consumers to the creative.
What are you working on?
Reach out if you're interested in using TAWNY's
Emotion Analytics Platform for your research project.
"There are a number of complex lab settings on the market which I used for my research on affective computing and digital empathy. TAWNY now offers a very convenient, scalable, cost-efficient, valid and reliable new research tool. I use the SaaS platform to organize all my experiments and directly share and discuss it with my research fellows. I also offered it for project work to my students and they love it. Finally, I also use it in applied research projects with partners from industry in academia."
Prof. Dr. Alexander Hahn
(Digital Marketing & Affective Computing)
TAWNY opens up a whole new universe for research projects and proposals in the area of consumer behaviour and affective computing!
Publications
Bartl, Füller (2020): The Rise of Emotion AI: Decoding Flow Experiences in Sports, In: 21st Century Sports: How New Technologies Change Sports in the Digital Age, Springer.
Bartl M. (2019): The Missing Piece in New Work: Decoding Human Flow at the Workplace, In: The Making-of Innovation, November 2020. https://michaelbartl.com/wp-content/uploads/2019_MoI_New-work-Flow_eng.pdf
Bartl M. (2019): The Rise of Emotion AI, In: Handbuch Künstliche Intelligenz.
Bartl M. (2018): Von der künstlichen zur emotionalen Intelligenz und was das für die Marktforschung bedeutet, In: marktforschung.de 27.06.2018
Baur M. (2021): Increasing Robustness of Facial Expression Recognition against Speech. (Master‘s Thesis | LMU)
Binder K., Goliasch A., Kolmann A., Ruhl M, Tiemeier T. (2021) AI-Powered Emotion Analytics in the Context of Billboard Marketing (Projektarbeit)
Eick E., Dierks S., Garrecht G., Tiemeier T. (2021): Die Auswirkung von Werbeunterbrechung im Action-Trailer auf Arousal, Valenz und unterschiedliche Emotionen unter Einbezug des TAWNY-Emotionserkennungstools
Hahn A., Bartl M., Klug K. (2020): Digital Empathy: Wie Künstliche Intelligenz und Affective Computing die Marktforschung verändern, In: Dem Konsumenten auf der Spur: Erfolgreiches Marketing durch zeitgemäße Marktforschung. Pusler M. (Hrsg.) Freiburg: Haufe.
Hahn A., Maier M. (2018): Affective Computing – Potential für empathisches digitales Marketing. In: Marketing Review. St. Gallen
Maier, M., Elsner, D., Marouane, C., Zehnle, M., and Fuchs, C. (2019): DeepFlow: Detecting Optimal User Experience From Physiological Data Using Deep Neural Networks, In: International Joint Conference on Artificial Intelligence (IJCAI 2019).
Marichalar Quezada R (2020): Effect of nudging in online-shopping environments regarding the consumption of fast food (Master’s Thesis | TUM)
Marichalar Quezada R., Bartl M., Garrecht G. (2022) Emotion AI: Neue Formen der Emotionsmessung durch Künstliche Intelligenz, In: Künstliche Intelligenz erfolgreich umsetzen – Praxisbeispiele für integrierte Intelligenz. Lichtenthaler U. (Hrsg.) Wiesbaden: Springer Gabler.
Nataraj A., Chellew C., Balaji D., Bilbao I., Frei J., Peponnet S. (2021): An Analysis of how the matching of emotions between online video content and online ads can mitigate the negative effects of ad interruptions, (Student Project)
Richter D. and Bartl M. (2018): Affective Computing Applied as a Recipe Recommender System, In: Computational Social Science in the Age of Big Data. Herbert von Halem Verlag Cologne.
Schmidt J. (2022): Using Emotion Analytics to Evaluate Nudges in Behavioural Design (Master's Thesis | TUM).
TV Feature
ARTE TV (2019): Helena. Die Künstliche Intelligenz – Gefühle. https://www.arte.tv/de/videos/086131-002-A/helena-die-kuenstliche-intelligenz/
ARD TV (2020): Arbeiten im Flow. https://www.daserste.de/information/wissen-kultur/w-wie-wissen/arbeitswelt-110.html
KIKA (2021): Checker Tobi: Der Künstliche-Intelligenz Check https://www.kika.de/checker-tobi/sendungen/der-kuenstliche-intelligenz-check-106.html
