1. Introduction

Mental health is a serious issue in today’s society. “A mental disorder is characterized by a clinically significant disturbance in an individual’s cognition, emotional regulation, or behaviour”^[1]. When asked, people report having only limited intimate contact with individuals who have a mental disorder^[2], although statistics range from nearly every third^[3] to every seventh^[1] individual worldwide suffering from a mental disorder. This perceptional gap is explained by the large number of affected individuals hiding their illness not only from the general public^[2], but also from their friends and families^[4]. Studies show that the prevalence of mental health disorders is constantly increasing^[3,5], so “that almost the whole population will at some time have direct experience of such a disorder, either in themselves or in someone close”^[2,6]. There are many types of mental disorders, with the most prevalent ones being anxiety disorders, depressive disorders, and stress^[5,7]. While the direct outcomes of mental disorders, such as distress, impairments of functioning in important areas, and disabilities, already hinder affected individuals in pursuing their goals^[1,8,9], mental illness is also associated with a secondary burden: stigma^[8].

“Stigma is a social phenomenon”^[10] that is characterized by labeling, stereotyping by association with negative attributes, separation, status loss, and discrimination^[10,11]. In doing so, mental illness is often seen differently than physical illness^[4], including being taken less seriously^[5] and judged medically unwarranted^[12]. Although the effectiveness of medication indicates a solvable problem with brain chemistry, mental illness is often seen as the ‘fault’ of affected individuals themselves^[4]. This results in not only the attribution of responsibility, but also perceived incompetence and unpredictable dangerousness^[9], prejudices that build the core of stigma^[13], which lead to discriminating behavior such as avoidance, withdrawal, and segregation^[9,13]. In consequence, affected individuals do not declare mental disorders publicly for fear of, for instance, not getting hired^[4]. “Indeed, many people describe stigma as being worse than the condition itself”^[14].

While the reduction of stigma related to mental health cannot directly reduce symptoms associated with a specific disorder, it can have a positive effect on affected individuals’ well-being^[15] by improving social standing and reducing social distance, thereby supporting their overall well-being. To achieve this, attitudes and beliefs, in which stereotyping and distancing are rooted, have to be changed^[11]. An approach to correcting myths about mental illness is improving knowledge, for instance by enhancing mental health literacy^[6], while attitudes, particularly perceived dangerousness, can be improved by increasing familiarity^[2]. Classical interventions aim to increase familiarity through contact^[16], often performed through individual interventions in which affected people tell their personal story^[9]. While contact reduces stigma through increased knowledge, its effects are driven primarily by reductions in anxiety and increased empathy and perspective-taking^[16]. Such contact can be simulated by new technological tools such as chatbots, which are particularly effective at reducing fear and social distance stigma when using a first-person perspective^[15]. Alike, communication with virtually embodied agents can approximate this contact, resulting in stigma reduction comparable to traditional video interventions^[17]. Furthermore, virtual reality (VR) technology offers great advantages for stigma reduction interventions. “By allowing users to ‘walk in another’s shoes’ in simulated environments, VR is thought to evoke empathy and understanding more effectively than traditional media”^[18]. This positive effect of technology-based contact simulation was reported within multiple studies across diverse stigmatized groups^[18-20], including people with mental disorders^[21-23]; for an overview see^[24,25].

A relatively new medium for technological persuasive interventions is the social robot. Social robots are physically embodied agents^[26] capable of verbal and non-verbal communication, enabling natural and interpersonal interaction with humans^[27] and supporting a socially interactive focus^[28]. While social robots are heavily researched in the persuasive context of education^[29-32] and their integration as health technology is also gaining ground^[33-35], including their use in mental health care in the role of, i.e., therapy assistants, social companions, peers, playmates, social mediators, or instructors^[36], the deployment of robots in interventions aiming to reduce stigma and biases is still in its infancy. Although studies show that social robots are capable of improving attitudes towards and provoking behavior changes within important societal issues such as environmental sustainability^[32,37,38] or personal issues such as healthy nutrition^[30,39], there are only a few attempts to utilize this technology to benefit stigmatized groups. One approach to directly promoting equality among individuals belonging to a stigmatized minority using social robots is the concept of Fair Proxy Communication^[40,41]. Defined as “fairness-enhancing communication with robotic proxies”^[41], this approach uses social robots as avatars in telepresent social interactions such as job interviews to represent individuals that may otherwise be vulnerable to bias-driven decisions based on their visual cues on, for instance, gender or cultural ethnicity^[40,41]. While this approach may mitigate the consequences of stigma and bias, it leaves the underlying problem itself largely unaddressed.

To the best of our knowledge, yet only one single study had been carried out that tried to reduce stigma using a social robot: in their study Korner et al.^[42] implemented an interactive presentation on stuttering that included educational information, storytelling, practical tips, and interactive components in a social robot. The authors reported a significant improvement in participants’ attitude towards stuttering individuals from pre- to post-measurement, showing that social robots can help reduce stigma^[42]. Given the great societal importance of mental health stigma outlined above, it is crucial to examine whether this effect translates to other stigmatized groups, especially those diagnosed with mental disorders. Storytelling robots, in particular, have been shown to be persuasive^[43] and capable of evoking empathy^[44,45], a key emotion in classical stigma-reduction interventions^[16], which already has been shown to be able to foster prosocial behavior in interaction with social agents such as robots (for an overview, see^[46]). Their physical embodiment may provide an advantage over storytelling technologies previously used in the classical stigma reduction interventions described above, such as video interventions^[47,48]. Moreover, compared to technological interventions delivered by, for example, chatbots^[15], social robots are capable of multimodal communication, including not only human-like modalities such as speech and body language, but also non-biomimetic modalities such as the integration of images^[49,50], colored lights^[51,52], and non-speech sounds, i.e., sound effects and background music^[52,53]. As music in particular is known to enhance the emotional perception in other storytelling media such as film^[54-56], while sound effects can increase recipients’ attention, ease understanding^[57], and help to create mental imagery^[58-60], these positive effects could further benefit stigma reduction through robotic storytelling.

Accordingly, we examined the potential of robotic storytelling to influence stigma and prejudice toward individuals with mental disorders, with a particular focus on the added value of sound effects and background music. To this end, we compared four versions of a robotic storytelling about a person with a mental disorder: (1) the story told by the social robot Pepper using speech and emotional body language; (2) with added sound effects; (3) with added background music; and (4) with both sound effects and background music integrated. Given that anxiety disorders are among the most prevalent mental disorders^[5], we focused on anxiety disorders, specifically obsessive-compulsive disorder (OCD), characterized by obsessional intrusive thoughts^[61], whose symptoms can be effectively portrayed in a short story depicting a brief time frame.

2. Related Work

In the following, theoretical background and related works on stigma and prejudice as well as interventions on their reduction across diverse media including social robots are presented.

2.1 Interventions reducing stigma

Stigma is a social process based on social judgment^[12] and linked value^[62]. When encountering a stranger, we immediately form an impression of who this person is. In doing so, we construct what Goffman^[63] terms a virtual social identity, that is, an assumed identity based on available social cues and category memberships. This constructed identity does not necessarily correspond to the person’s actual social identity, which comprises their real attributes and characteristics. If an attribute becomes apparent that deviates negatively from what is expected for the social category in question, the individual may be redefined in light of this attribute. In extreme cases, this can result in a reduction of that person from a usual to a discounted one. This attribute constituting a discrepancy between the virtual and actual social identity is called a stigma^[63]. Hence, stigma is based on the labeling of distinguishing differences that are associated with negative attributes, resulting in a separation between “us” and “them”^[11], ranging up to “dehumanization, threat, aversion, and sometimes the depersonalization of others into stereotypic caricatures”^[62]. Goffman describes three types of such stigma based on the type of the distinguishing attribute: 1) “abominations of the body, the various physical deformities”, 2) “blemishes of individual character”, including, among others, mental disorders but also homosexuality or unemployment, and 3) tribal identities such as “race, nation, and religion”^[63]. According to this categorization, stigma against mental illnesses is a stigma against one’s personality or character, which is perceived as weak^[63], even though “the judgment is in some essential way medically unwarranted”^[12]. This interpersonal or public stigma includes prejudice, stereotypes, and discrimination, i.e., negative behavior^[9,14,64]. Prejudice refers to negative attitudes towards outgroup members^[13,62], respectively members of a particular outgroup with a specific distinguishing attribute^[65], or, in the case of prejudice towards people with mental illness, people with one or more mental disorders^[66]. Etymologically, the word stems from the Latin term praejudicium, meaning “prior judgment”^[13], describing the judgment based on group membership rather than personal attributes^[67]. Bizumin and Sheppard^[13] postulate four dimensions of prejudice towards people with mental illness, including 1) fear of affected people and thus a preference for social distance, 2) a perceived unpredictability of their behavior, 3) a perceived need for control over people with mental illness and restriction of their lives, and 4) malevolent beliefs that affected people are “physically, genetically, and psychologically inferior to people without mental illness” and should not receive help from societal resources^[68]. While in the past social scientists suggested prejudice being rooted in problematic personality traits, “social psychologists now consider stereotyping to be a normal (if undesirable) consequence of people’s cognitive abilities and limitations, and of the social information and experiences to which they are exposed”^[62]. A stereotype is the cognitive element of prejudice^[13,69], describing rather generalized beliefs^[13,69] than attitudes. They “become prejudices when people develop negative emotions and evaluations” against stigmatized individuals^[9]. Finally, prejudice can result in discrimination^[9], an unjustified negative or harmful behavior toward an individual solely based on their membership in a stigmatized group^[13,69]. Taken together, stereotypes form the cognitive aspect of stigma^[13], prejudice describes its affective, attitudinal, and evaluative facets^[13,70], that drive behavior in the form of discrimination^[13]. Stigma includes prejudice “but extends to more general attributions about character and identity”^[62], although it is suggested that stigma and prejudice “describe a single animal”^[70].

Stigma against mental illness, particularly in terms of stereotypes of perceived dangerousness and discrimination through avoidance, is negatively correlated with familiarity, namely the knowledge about and experience with mental disorders^[2]. Providing information on mental illness and promoting contact with affected persons can challenge stereotypical expectations by disconfirming them^[71], for an overview see^[72]. Therefore, contact is a widespread and well-researched method for reducing prejudice via three routes: 1) increasing knowledge, 2) reducing anxiety, and 3) enhancing both empathy, hence, emotional responses^[73], and perspective taking, the attempt to see the world from another person’s point of view^[74], with anxiety reduction and empathy enhancement being the strongest mediators^[16]. For instance, a field study examining contact with prisoners, in more detail murderers or convicted sex offenders with personality disorders, revealed an increase in empathy as well as decreased prejudice after five hours of contact^[75]. This stigma reduction can also be achieved by simulating contact through fictional stories such as in vignette studies^[15]. “Stories engage our minds and change the way we see the world”^[76]. In doing so, traditional storytelling, “characterized by a single teller addressing an audience using speech, physical gestures, possibly props, and non-speech sounds”^[77], has already been shown powerful within informal learning settings, such as with social stories for children fostering positive social behavior and decreasing aggressive behavior^[78]. In fighting prejudice, storytelling disconfirming stereotypes about African Americans in the form of a radio show was reported to improve perspective taking and decrease stereotypic perceptions^[72]. Moreover, animated storytelling videos were shown to decrease addiction stigma^[79] as well as transphobia^[80] in the short-term. Animations presented in VR were reported to successfully reduce stigma against people with anxiety and depressive disorder from pre- to post-intervention, with the effect sustaining to the one-week follow-up^[23]. Lastly, traditional oral storytelling was shown to effectively reduce stigma against people with a human immunodeficiency virus diagnosis directly after the intervention as well as six months later^[81]. These results reveal the great potential of storytelling as a substitute for direct contact with outgroup members through diverse media, reflecting findings on the persuasiveness of stories reported in general health communication^[82].

Next to empathy, transportation is an important variable in storytelling that positively influences changes in attitude. Transportation is defined as the recipients’ absorption into the story, including an “integrative melding of attention, imagery, and feelings”^[76], and can occur across various media^[76]. Transportation can benefit persuasion and suppress counterarguments (for an overview see^[83]), leading to more story-consistent beliefs and less disbelief^[76]. With regard to story characters, transportation increases liking for them and subsequently reinforces beliefs that align with their experiences and perspectives as presented in the story^[76]. Finally, transportation has been shown to correlate positively with empathy, to promote prosocial behavior consistent with the narrative^[84], and to be negatively related to stigma^[23]. However, in the context of persuasive health messages, instrumental arguments can reduce transportation and increase psychological reactance. Thus, affective arguments are recommended^[85]. While multiple meta-analyses^[82,86] recommend the first-person perspective for written persuasive storytelling, Christy^[87] reported a beneficial effect of third- and second-person perspective compared to first-person perspective in written storytelling for improving attitudes towards gay people, although no differences in transportation or identification were obtained. For audiovisual media, the effect of perspective seems to be even less clear^[82]. Regarding the presentation medium, Zhuang and Guidry^[86] found no significant differences between written texts and videos or their combination in their stigma-reducing effects, suggesting the transferability of the approach between different media types^[86]. Nonetheless, adding pictures to text-based health messages can improve comprehension, recall, and adherence through increased attention^[88,89]. In stigma reduction, particularly emotion-conveying visual material may benefit intervention outcomes, given that the correct understanding of another person’s distress cues, such as a fearful facial expression, enhances empathy and increases prosocial behavior^[90]. Also, other additional modalities could further benefit intervention effects. For example, music has been shown to influence prosocial behavior. Wu and Wang found that presenting sad music while participants read a story about a person in need increased their willingness to help compared to happy music or white noise, probably due to enhanced empathy^[91]. Within anti-stigma interventions, emotionally moving background music was reported to be a strong predictor for both empathy and reflective thoughts, although no significant effect on attitudes was found^[92]. In contrast, advantages of increased attention due to multimodality in stigma-reduction interventions have yet not been investigated. Taken together, storytelling across various media, whether written, animated, or traditionally orally delivered, holds substantial potential for anti-stigma interventions, with additional modalities such as music offering further opportunities to amplify their impact.

2.2 Non-speech sounds in storytelling media

The perception and interpretation of sound have played a vital role throughout human history, from early survival, when identifying sounds as potential prey or predators was essential, to modern contexts such as navigating traffic^[93,94]. Similarly, music is omnipresent in daily life, used to influence consumer behavior in retail settings and to support both physical and mental health treatments^[95]. Unsurprisingly, music and sound effects have long been intertwined with storytelling. Early oral storytelling incorporated musical elements^[96], which were later adapted for radio plays and films in the 1920s^[97,98]. Sound effects, originating in the theater^[60], were subsequently adapted in film and later in audiobooks in the 2000s^[97]. Nowadays, storytelling media soundtracks, such as those of films, “typically consist of music, speech, and sound effects”^[99].

Sound effects serve both orienting and descriptive functions, helping to create and sustain imagined environments^[60,94] by reconstructing reality through explicit sounds that represent, for example, objects or weather conditions^[100]. These sound effects can appear as short and specifically placed sounds, such as a barking dog, or as layered background ambience, such as echoing conversations and footsteps in a large city hall. Both approaches can also be mixed^[101]. Sound effects have been shown to improve immersion and focus on a story, making storytelling more enjoyable by strengthening mental imagery^[58,100]. When applied in persuasive communication, these effects can further enhance attitudes and memory performance^[59]. Beyond their merely diegetic functions, sound effects can also evoke emotional responses^[98,101], such as the tension created by the slow creak of an opening door^[101].

“Unlike speech and sound effects, music plays no part in the story as music per se”^[99], but functions as emotional guidance^[102]. Music, in general, has the high potential to express and evoke emotions^[103]. In doing so, it can influence human perception and information processing when combined with visual stimuli^[104,105]. For example, Jeong et al.^[106] found that happy faces presented alongside sad music were rated less happy, whereas the same faces paired with happy music were rated as happier, and vice versa. Alike, interpretations of social contexts were shown to be affected by music in terms of, inter alia, perceived valence and pleasantness^[104]. This cross-modal effect also extends to storytelling media: especially in film, emotionally charged background music has been shown to bias interpretation and memory in an emotionally congruent direction^[107-110], even affecting recipients’ physiology^[95]. Part of this effect is explained by music’s ability to evoke emotionally congruent thoughts, although the strength of this effect depends on familiarity with and liking of the musical piece^[111]. A widely accepted framework for understanding these phenomena is the Congruence-Association Model (CAM)^[112]. The CAM proposes that information congruent across modalities, such as the structural overlap of video and music (congruence), is processed with priority. This facilitates the aggregation of meanings from both channels (association), allowing “associations from the music [to] be ascribed to that focus of attention”^[112], ultimately resulting in an emotionally congruent interpretation of the combined audiovisual experience. The structural accordance further improves the perceived presence and transportation of recipients^[55,113], whereas visual storytelling without music was shown to decrease transportation into the story, causing less identification with the protagonist^[113]. Investigating the effects of sound effects, music, and their combination, or omission of both, on movie reception, Kock and Louven^[114] found that music alone reduced the immersion range, while sound effects alone diminished suspense, compared to presenting both together. Based on these findings, they recommended using a combination of music and sound effects for audiovisual storytelling.

2.3 Multimodal robotic storytelling as an intervention tool

Social robots are a rather new but highly promising medium for storytelling. Being physically embodied^[26], these agents are designed for natural and interpersonal interaction with their human users, utilizing speech but also non-verbal cues such as body language or facial expressions^[27]. In contrast to service or industrial robots, social robots have a social interface^[115], so that they can support humans with a socially-interactive focus^[27,28], for instance, in educational contexts^[31,116] or aging^[117]. In doing so, social robots bear advantages over virtual agents due to their physical embodiment, offering richer interaction and sharing the user’s interaction space^[118]. Based on their multimodal communicative abilities, robots can serve as storytellers, creating narrative experiences that are comparable to traditional audiobooks^[119]. Furthermore, both social robots in general^[46] and robotic storytellers in particular^[120] have been shown to be able to elicit empathy in their recipients. Initial intervention studies further suggest their potential effectiveness: for example, a robotic storytelling intervention against bullying improved children’s empathy compared to a loudspeaker-based storytelling^[45]. To the best of our knowledge, however, robotic storytelling has not yet been examined in the context of mental illness stigma-reduction interventions. These early findings provide a promising foundation for future research exploring the potential of robotic storytellers in this domain.

Moreover, social robots offer not only human-like modalities such as voice and body language, but are also capable of integrating non-biomimetic modalities, including music^[121,122], which could further benefit anti-stigma interventions (see Section 2.1). Although research on additional sound integration in human-robot interaction is still in its infancy compared to, e.g., speech^[123], music has been shown to improve enjoyment of and transportation into robotic storytelling^[53], with the latter being an important key lever in stigma reduction (see Section 2.1). Moreover, music has been shown to facilitate emotion recognition from robotic expressions^[124]. As accurate recognition of negative emotions is a reliable predictor of prosocial behavior^[90], music accompanying a stigma-reducing storytelling intervention might amplify empathy and thus stigma changes. Given that robots can not only automatically integrate music into their storytelling^[125] without the need for additional devices (as compared to humans, see^[77]), but can also provide consistent telling over and over again (as compared to humans, see^[126]), they are a promising tool for providing standardized anti-stigma interventions and need further investigation.

3. Contribution

Facing the need for mental health awareness in today’s society, we investigated the potential of multimodality integrating both human-like and non-biomimetic modalities in robotic storytelling to raise this awareness in recipients’ and to improve their attitudes toward concerned persons. Therefore, we implemented a social robot telling a story of a person experiencing a panic attack accompanied by intrusive thoughts. To examine the efficiency of non-speech sounds in improving the influences on awareness and attitudes, we tested four variants of our storytelling: (1) using both sound effects and background music, (2) using only sound effects, (3) using only background music, or (4) omission of non-speech sounds. Both background music^[127] and sound effects^[100] can increase recipients’ attention to storytelling media. Moreover, background music can have a positive effect on recipients’ transportation into the story^[113] as well as the related concepts of immersion^[55] and cognitive absorption^[128]. However, as reported by Kock and Louven^[114], sound effects as well as the combination of background music and sound effects outperform the addition of music only. Therefore, we postulate the following hypotheses:

H1a: Attention is higher during the storytelling with non-speech sounds compared to that without.

H1b: Attention is higher during the storytelling including both sound effects and background music compared to those integrating only one type of non-speech sound.

H2a: Transportation is higher for storytelling with non-speech sounds compared to that without.

H2b: Transportation is higher for storytelling including both sound effects and background music compared to those integrating only one type of non-speech sound.

Both the increased transportation^[84] as well as the improved recipients’ perception of the protagonist’s emotions^[73], might increase their empathy for them. In addition, especially sound effects can increase attention during story reception^[57]. On this basis, we postulate the following hypothesis and research question:

H3: Empathy is higher during the storytelling with non-speech sounds compared to that without.

RQ1: Does the combination of sound effects and background music improve the positive effects of the individual non-speech sound types on empathy?

Combining modalities, such as audio and visuals, improves persuasive effects^[129], especially for narrative persuasive messages^[130]. Also for robotic storytellers, improved persuasiveness was obtained when adding emotionally matched facial expressions to the telling^[43]. Furthermore, multimodality was shown to improve sympathy and prosocial intentions while decreasing prejudice toward the target of hate speech^[131]. Although the addition of non-speech sounds has yet not been tested in terms of enhanced persuasion, based on the related works on multimodality in persuasive communication, we expect lower prejudice in recipients of the storytellings with added non-speech sounds.

H4a: Prejudice is lower after receiving the story with non-speech sounds compared to that without.

H4b: Prejudice is lower after receiving the storytelling including both sound effects and background music compared to storytelling integrating only one type of non-speech sounds.

Lastly, transportation was shown to improve the liking of story characters^[76] and empathy building^[84] as well as to reduce negative cognitive activities such as disbelief^[76], paving the way for the reduction of stigma. According to our focus on storytelling, we exploratorily analyzed whether transportation mediated our intervention’s effects on both empathy and stigma reduction.

H5a: Transportation mediates the sound conditions’ effect on empathy.

H5b: Transportation mediates the sound conditions’ effect on prejudice.

4. Materials

A story about a person suffering from OCD was specifically written for this study. After pretesting the story in an online survey, it was implemented using the Pepper robot to realize four versions of robotic storytelling: (1) using both sound effects and background music, (2) using only sound effects, (3) using only background music, or (4) omission of non-speech sounds.

4.1 Story concept

We chose a female student as protagonist, named “Cora”, for our story to facilitate identification with our predominantly young and female sample. The story (The whole story can be found at https://go.uniwue.de/mentalhealth-material) describes an everyday situation of a university student who suffers from OCD. The disorder manifests itself in obsessive thoughts that trigger states of anxiety and panic attacks. The protagonist has been in therapy for many years and is therefore aware of coping mechanisms that help her handle her condition.

The story was divided into seven paragraphs that describe Cora’s journey to university on a warm summer’s day. It contains 1,350 words, which translates to a duration of approximately ten minutes when told. The plot of the story is summarized as follows.

On her way to the bus stop, the protagonist accidentally collides with a child who runs off laughing. Shortly afterward, she is struck by an intrusive thought, “What if I was a pedophile?”, which triggers intense anxiety despite her therapeutic knowledge that such thoughts are automatic and unfounded. The pedophilia-themed intrusive thought was selected because OCD-related intrusive thoughts frequently involve “deviant or non-normative sexual thoughts, as defined by Western cultural standards, such as the fear of developing an attraction to children”^[132]. In contrast to more commonly recognized OCD themes, such as contamination or checking behaviors, this theme was chosen to portray a particularly stigmatized and distressing form of obsessive thought^[133]. The protagonist’s fear escalates when an ambulance passes by, leading her to worry that the child was hurt. While rushing to the bus, she feels a panic attack building, but once on board, after a boy asks whether she is okay, she applies coping strategies learned in therapy and gradually regains control. Arriving at the university, the protagonist is proud of her progress and her ability to manage her chronic condition. The story highlights typical thinking processes and emotional states that someone suffering from OCD might experience during an everyday situation. The intrusive obsessive thought was specifically chosen to be shocking to the recipients since these kinds of thoughts are often disturbing and violent in nature, causing high levels of anxiety and discomfort in affected parties^[134]. The story had a linear structure and offered no points of interaction for the recipient. The story was proofread by three persons. Moreover, to ensure the accuracy and appropriateness of the mental health content, the stimulus material was reviewed by a staff member of the Counseling and Information Service (KIS: https://www.uni-wuerzburg.de/en/equity/kis/) at the University of Würzburg.

4.2 Pretesting the story

In order to test the suitability of the story in terms of transportation and empathy elicitation, an online prestudy was carried out.

4.2.1 Measures

Transportation was assessed using the Transportation Scale - Short Form (TS-SF)^[135] comprising six items such as “I wanted to learn how the narrative ended” answered on a seven-point Likert-scale anchored by 1 - “not at all” and 7 - “very much”. Appel et al.^[135] reported alpha reliability of .80 to .87, alpha for the current sample was .89.

Empathy was measured using the State Empathy scale by Shen^[136]. The scale consists of three dimensions comprising four items each: (1) Affective Empathy, sharing the protagonist’s feelings (e.g., “The character’s emotions are genuine.”), (2) Cognitive Empathy, understanding the protagonist’s motivations and perceptions (e.g., “I see the character’s point of view.”), and (3) Associative Empathy, identification with the protagonist (e.g., “When watching the message, I was fully absorbed.”). We adapted the third scale to better align with the context of robotic storytelling by replacing the term “the message” with “the story” and the term “the character” with the protagonist’s name (“Cora”). All items were measured on a 5-point Likert scale anchored by 0 = “not at all” and 4 = “completely”. Alpha reliabilities of the original scale reported by Shen^[136] were .83 to .91 for Affective Empathy, .86 to .91 for Cognitive Empathy, .82 to .92 for Associative Empathy, and .92 to .93 for the whole scale. For the current sample, alphas of .84 for Affective Empathy, .77 for Cognitive Empathy, .76 for Associative Empathy, and .91 for the whole scale were computed.

Personal relevance of the OCD topic was measured using the Personal Relevance Scale^[137]. The scale comprises four items such as “To what extent (if any) does the treated topic make you think of current situations or events in your own life?”, which are answered on a five-point Likert scale anchored by 1 = “not at all” and 5 = “very much”. Ellard et al.^[137] does not provide reliability measures, and as the scale was only used to exclude participants in our study but not for further analyses, we did not compute them either.

4.2.2 Procedure and exclusion criteria

Participants were acquired using the university’s participant pool. In the study advertisement, a trigger warning concerning the topic of mental illness and OCD was provided. When entering the survey hosted via LimeSurvey, participants again received a trigger warning before they provided informed consent. Furthermore, we provided a list of contact points such as the crisis hotline or the university’s internal psychotherapeutic counseling.

For ethical reasons, participants with current or past experiences with OCD, either in themselves or in a family member or friend, were excluded from the study to avoid triggering distress or exacerbating symptoms during the storytelling intervention. This exclusion was implemented through an aptitude screening conducted prior to the actual survey. Participants were asked whether they had ever received an OCD diagnosis (yes/no/prefer not to disclose), completed the personal relevance scale, indicated whether they met the minimum age requirement (yes/no), and confirmed their willingness to participate after reading the trigger warning (yes/no). Participants were excluded if they reported an OCD diagnosis, scored above the midpoint on the personal relevance scale, or chose not to disclose diagnostic information. In this case, they were informed about their exclusion and thanked for their willingness to participate. In doing so, individuals with high familiarity with OCD were excluded from the study. Data collected in the aptitude test were automatically analyzed during the participation and deleted prior to the main analyses.

Proceeding to the actual survey, participants first read the text version of the story described in Section 4.1. Afterwards, they filled in the questionnaires on transportation and empathy, provided demographic information (age, gender), and were invited to provide comments. Last, the list of contact points was presented again.

The prestudy took about fifteen minutes to complete.

4.2.3 Participants

A total of 68 individuals participated in the pretest. Based on the screening results, 24 participants were excluded due to prior experience with mental illness (n = 22) or not meeting the minimum age requirement (n = 2). Therefore, 44 datasets were analyzed. The mean age of the participants was 20.86 years (SD = 2.11). The majority of 43 individuals self-indicated being female (age: M = 20.86, SD = 2.13), whereas only one person self-identified as male (age: 21). No one self-identified as a diverse gender.

4.2.4 Results and discussion

The average level of transportation into the story was 4.91 (SD = 1.34). This value is comparable to those achieved in related studies investigating transportation in written texts^{[135,138,139]}, suggesting that our generated story elicited a sufficient degree of transportation.

For the three dimensions of state empathy, mean values of 2.00 (SD = 0.86) for Affective Empathy, 2.18 (SD = 0.85) for Cognitive Empathy, and 1.92 (SD = 0.87) for Associative Empathy were computed. As the scale has yet mainly used in combination with written news articles, video stimuli^[140,141], or immersive technologies^[142], but not written narratives, no data for direct comparisons are available. However, as the values range around the scale’s center and are descriptively comparable to related works^[140,142], they were interpreted as sufficient.

Only one person provided qualitative feedback in the comment section, noting that the story was “very well-written” and that “it was easy to imagine oneself in the scene and empathize with the events”, reinforcing the suitability of our story. Based on the overall feedback, we utilized the story without further adjustments.

4.3 Implementation

We implemented the story using the Pepper robot, generating four versions with either one type, both types, or no type of non-speech sounds added to the storytelling, resulting in four versions in total.

4.3.1 Story annotation

The story was annotated by a single human rater, the story author, using a flexible tokenization. This approach was chosen to allow for emphasizing the underlying intention with the annotation. As possible labels, we utilized the eight inner emotions of Plutchik’s Wheel of Emotions^[143], namely ecstasy, admiration, terror, amazement, grief, loathing, rage, and vigilance, plus a neutral label. In doing so, terror was assigned six times, whereas admiration, ecstasy, and grief were assigned three times each. The rest of the text was annotated as neutral.

4.3.2 Robot behavior

The story was implemented using Choregraphe version 2.5.10.7 (http://doc.aldebaran.com/2-5/software/choregraphe/index.html). The Pepper robot was chosen due to its popularity and widespread use^[144], its relatively low intensity of consequential sounds^[145], and its extensive use in previous robotic storytelling research^{[50,51,53,125,146,147]}.

Concerning body language, emotional expressions matching Plutchik’s emotions retrieved from related work^[148] were matched to the annotated labels (Figure 1a). To improve the naturalness of the storytelling, further beat gestures and iconic gestures were implemented. Examples are shown in Figure 1b.

Display Full Size

Download

Figure 1. Robot’s body language. (a) Emotional expressions for (1) ecstasy, (2) admiration, (3) terror; (b) Narrative behavior using the beat-gestures Explain-1 (1), and (4) grief. Explain-2 (2), Explain-4 (3), Explain-11 (4), Explain-5 (5), and iconic gestures No-1 (6) and YouKnowWhat-1 (7).

The auditory implementation of the story made use of the robot’s preinstalled synthetic voice. The story alternates between the four perspectives of the storyteller, the protagonist’s inner monologue, her spoken words, and the spoken words of a character met on the bus. The different perspectives were realized via differing pitch and speed of the robot’s voice. The standard storyteller voice used a pitch of 85 percent and a speed of the standard 100 percent. The protagonist’s inner monologues and spoken words used a pitch of 120 percent, while speed alternated between 100 and 115 percent based on her emotional state. For example, a fast and loud voice mirrored fear as it expresses a high state of emotional arousal^[149]. In doing so, the inner monologue was adjusted to represent emotional duress during the panic attack. Lastly, the voice of the boy on the bus was implemented with a pitch of 60 percent and the standard speed of 100 percent.

4.3.3 Music and sound effects

In order to underline the protagonist’s emotional states, emotionally charged music was added to the story while sound effects were added to illustrate environmental cues.

The music was attached paragraph-wise based on the initial emotional annotation of the story. In accordance with Steinhaeusser et al.^[53], four musical pieces were used, representing the emotions of fear–matched to terror, joy–matched to ecstasy, sadness–matched to grief, as well as heroic emotions, which were matched to admiration. The utilized pieces were “Torture Chamber” (https://www.fesliyanstudios.com/royalty-free-music/download/torture-chamber/2856) by David Robson, “Beautiful Memories” (https://www.fesliyanstudios.com/royalty-free-music/download/beautiful-memories/234), “A Sad Meme” (https://www.fesliyanstudios.com/royalty-free-music/download/a-sad-meme/2479) and “Inspirational Advertising” (https://www.fesliyanstudios.com/royalty-free-music/download/inspirational-advertising-1/260) by David Fesliyan, respectively.

The sound effects were individually assigned to the story text by the human annotator. They were retrieved from the platforms Fesliyan Studios (https://www.fesliyanstudios.com/de/) and Freesound (https://freesound.org/). Examples of sound effects are birds chirping, sirens of the rescue vehicle, or sounds representing the panic attack, such as a fast heartbeat. Sound effects representing the protagonist’s breathing and laughing were recorded especially for this study by a young female voice actor. Some of the sound effects were combined using the tool Audacity (https://www.audacityteam.org/). For instance, to symbolize the protagonist’s panic attack, the sound of a beating heart was combined with a high-pitched noise representing her ears ringing.

The sound effects used the standard pitch of 100 percent while the music was played at 50 percent in the background so as to not distract from the plot while still setting the scene.

In doing so, four conditions of the storytelling were created, either omitting (Basic condition) or integrating (BothAdds condition) both types of non-speech sounds, or integrating only background music (Music condition) or sound effects (Effects condition).

5. Methods

To evaluate the integration of non-speech sounds, namely background music and sound effects, into a robotic storytelling intervention to lower prejudice against people suffering from mental illnesses, a multi-methods laboratory study was conducted.

5.1 Study design

A 2 (integration or omission of music) × 2 (integration or omission of sound effects) between-subjects was applied, utilizing four versions of the robotic storytelling intervention as conditions: (1) storytelling integrating both background music and sound effects (BothAdds), storytelling integrating either (2) background music (Music) or (3) sound effects (Effects), and (4) a storytelling without any additional non-speech sounds (Basic).

5.2 Measures

We applied a multi-methods approach using both subjective self-reports of the participants as well as objective observations.

5.2.1 Observation of blinking behavior

Blinking behavior is strongly connected to the task that is currently performed^[150]. Since closing one’s eyes, apparently, hinders information intake^[151], blinking is suppressed when acquiring visual information^[152] within, for example, difficult^[150,153] and attention-demanding tasks^[150]. Therefore, the blink rate adapts inversely to the attention spent on a task. This holds also true for non-visual tasks^[151] such as music listening^[154]. Based on this relation, we analyzed participants’ blinking behavior as an objective measure to assess their Attention during the storytelling.

To measure blinking behavior, we video-recorded participants’ faces during the storytelling intervention. The videos were then analyzed using a Python-implemented eye blink counter. As a base, we used the face mesh module and plot module from CVZone (https://www.computervision.zone/courses/eye-blink-counter/). We adapted the program to count blinks more accurately following the procedure described by Steinhaeusser et al.^[53].

5.2.2 Self-reports via questionnaires

Recipients’ storytelling experience was operationalized with transportation, using the same questionnaire as in the prestudy (TS-SF; see Section 4.2.1). Reliability computed for the current sample was .81.

Recipients’ empathy with the story’s protagonist was again assessed using the State Empathy Scale by Shen^[136] described in Section 4.2.1. Cronbach’s alpha for the current sample was .81 for Affective Empathy, .76 for Cognitive Empathy, .68 for Associative Empathy, and .88 for the whole scale.

Last, recipients’ prejudice toward people with mental illness was measured using the Prejudice Towards People with Mental Illness scale by Kenny et al.^[66]. The scale comprises three subscales: (1) the Fear/Avoidance subscale including items on social distance and interaction difficulties (8 items, e.g., “It is best to avoid people who have mental illness.”), (2) the Malevolence subscale with items on inferiority and unsympathetic attitudes (8 items, e.g., “People who develop mental illness are genetically inferior to other people.”), (3) the Authoritarianism subscale on needs for control over affected people (6 items, e.g., “People who are mentally ill need to be controlled by any means necessary.”), and (4) the Unpredictability subscale about unforeseeable behavior of affected persons (6 items, e.g., “People with mental illness often do unexpected things.”). To adjust the answer format to the other questionnaires in the study, the original nine-point scale was adapted to a five-point scale, anchored by 1 - “Strongly disagree” and 5 - “Strongly agree”. Kenny et al. report reliabilities of .91 for Fear/Avoidance, .80 for Malevolence, .79 for Authoritarianism, and .82 for Unpredictability. Cronbach’s alpha for the current sample was .71 for Fear/Avoidance, .34 for Malevolence, .61 for Authoritarianism, and .72 for Unpredictability. Due to their low reliabilities, both the Malevolence and Authoritarianism subscales should be treated with caution.

5.3 Procedure and exclusion criteria

Once more, participants were acquired via the university’s participant pool. In doing so, participants from the prestudy were excluded from participation in the main study. Similar to the prestudy, we implemented trigger warnings in the study advertisement. When arriving at the laboratory, participants first provided written informed consent to participate in the study.

Again, we integrated an aptitude test into our study protocol to exclude individuals with personal experience with OCD from the actual experiment. For this screening, again the personal relevance scale described in Section 4.2.1 was utilized together with a question on prior OCD diagnosis, the minimum age requirement, and willingness to participate after reading the trigger warning again. As the experimenter was aware of the test result in terms of inclusion or exclusion, we added an additional answer to the last question (not interested in the study), as well as an additional question on hearing impairment to make participants feel more comfortable if they get an exclusion notice in the presence of the experimenter. However, the experimenter had no access to the data collected in the aptitude test and was only informed about passing or failing the test.

After the aptitude test, the actual experiment started. Firstly, the participants received the story version to which they were randomly assigned. During the reception, their face was videotaped to allow for analyses of blinking behavior afterwards. Then, they filled in the questionnaires on transportation, state empathy, and prejudice against people suffering from mental illnesses. Next, participants provided demographic data (age, gender, prior experience with the robot) and were invited to leave a comment. Lastly, the participants were thanked, debriefed about the study goal, and handed a list of contact points such as the crisis hotline or the university’s psychotherapeutic counseling.

5.4 Participants

In total, 131 individuals started the experiment, however, ten of them were excluded due to failing the aptitude screening. In consequence, 121 individuals with a mean age of 21.66 (SD = 3.14) took part in the study. The majority of 94 participants self-identified as female (age: M = 21.38, SD = 3.19), whereas 27 of them self-identified as male (age: M = 22.63, SD = 2.80). No one self-identified as a diverse gender.

Being randomly assigned to the conditions, 30 participants each received the Music (23 female, 7 male, 0 diverse; age: M = 21.27, SD = 2.35), Effects (22 female, 8 male, 0 diverse; age: M = 21.17, SD = 1.88), respectively Basic condition (24 female, 6 male, 0 diverse; age: M = 22.50, SD = 4.65), whereas 31 individuals received the BothAdds condition (25 female, 6 male, 0 diverse; age: M = 21.71, SD = 2.95).

6. Results

All analyses were carried out using JASP version 0.19.1.0^[155] and a significance level of .05. Descriptive values are displayed in Table 1.

6.1 Blink rates

Each of the 121 trials was video-recorded. Unfortunately, one video of the BothAdds condition was corrupted and thus excluded from the analysis, resulting in 30 videos per condition. Out of the 120 valid video recordings, outliers as well as randomly chosen ones were cross-coded by a human rater for quality assurance, resulting in cross-coding of 13 videos. For these videos, the human rater’s counts were adopted. The interrater reliability between the program and both human raters was Krippendorff’s α = .63, calculated using the k-alpha calculator^[156]. Blink rates per condition are illustrated in Figure 2.

Display Full Size

Download

Figure 2. Blink count per condition. Error bars represent standard errors.

Shapiro Wilk tests indicated a normal distribution (ps >.5). The calculated Levene’s test validated equality of variances (p = .592). Again, a 2 (integration or omission of music) × 2 (integration or omission of sound effects) analysis of variance (ANOVA) was computed to investigate differences in blinking behavior between the conditions. Results indicated no significant main effect of music or sound effects, nor a significant interaction effect. A planned contrast comparing the Basic condition to the conditions integrating non-speech sounds yielded no significant difference (t(116) = -0.65, p = .515, d = .41). Similarly, the planned contrast comparing the BothAdds condition to the ones integrating only one type of non-speech sound indicated no significant difference (t(116) = -0.21, p = .836, d = .09).

6.2 Questionnaire data

For all variables, 2 (integration or omission of music) × 2 (integration or omission of sound effects) comparisons were computed. Their results are presented in Table 2. If applicable, Bonferroni-Holm corrected post hoc pairwise comparisons were carried out. To gain deeper insights into the added value of non-speech sounds in general compared to their omission as well as the combination of non-speech sounds compared to using single types only, contrasts were specified (Basic vs. rest; BothAdds vs. Music and Effects).

6.2.1 Storytelling experience

For Transportation, homogeneity of variances was suggested by Levene’s test (p = .179). Shapiro-Wilk tests indicated a violation of the normality assumption for the Music condition (p < .001). However, given the reported robustness of ANOVA against violations of normality^[157], two-way ANOVAs were performed to examine the main effects and interactions of non-speech sound additions to the robotic storytelling. No significant main effects of music nor sound effects were obtained. Similarly, no significant interaction was found. Planned contrasts indicated no significant difference between the conditions with non-speech sounds added compared to the Basic condition (t(117) = -0.99, p = .324, d = .63) but a significant difference between the conditions with only one type of non-speech sound compared to the BothAdds condition (t(117) = -3.32, p = .001, d = 1.47) with higher values for the combined non-speech sounds condition.

6.2.2 Empathy

As state empathy was assessed on three dimensions, a multivariate ANOVA (MANOVA) was computed. A Shapiro-Wilk test indicated multivariate normality (p = .168). Levene’s tests indicated homogeneity of variances for all dimensions (ps >.05). Correlations between dependent variables were low (rs < .73), indicating that multicollinearity was not a confounding factor in the analysis. The calculated MANOVA did not show statistically significant main effects of music (Pillai’s Trace = .03, F(3,115) = 0.97, p = .409) or sound effects (Pillai’s Trace = .02, F(3,115) = 0.95, p = .421) on the combined dependent variables, but a significant interaction of both (Pillai’s Trace = .12, F(3,115) = 5.19, p = .002). Post-hoc univariate ANOVAs were conducted for every empathy dimension.

Regarding Affective Empathy, the follow-up univariate ANOVA yielded no significant main effect of music or sound effects. Similarly, the interaction effect was not significant. The planned contrasts comparing the Basic to the three other conditions (t(117) = 0.04, p = .968, d = .03) as well as the one contrasting the BothAdds to the Music and Effects conditions (t(117) = -1.94, p = .055, d = .86) also revealed no significant differences.

Likewise, no significant main effect of music or sound effects was found for Cognitive Empathy. Moreover, the interaction effect was not significant. Neither the planned contrast comparing the Basic condition to the ones with additional non-speech sounds (t(117) = -1.13, p = .263, d = .71) nor the contrast between the conditions with only one additional sound type and the BothAdds condition (t(117) = -0.76, p = .450, d = .34) revealed significant differences.

Lastly, for Associative Empathy, again no significant main effects of music or sound effects were found. However, a significant interaction effect was obtained. Pairwise post-hoc comparisons indicated significantly lower values in the Music compared to the Basic (p = .028, d = .75) as well as compared to the BothAdds condition (p = .005, d = .88). While the planned contrast between the Basic and the three other conditions yielded no significant difference (t(117) = -1.56, p = .121, d = .99), the difference in Associative Empathy between the BothAdds condition and the conditions integrating only one additional sound type indicated significantly higher values in the BothAdds condition (t(117) = -3.12, p = .002, d = 1.38).

6.2.3 Prejudice

Since prejudice was measured on four dimensions, a MANOVA was computed. A Shapiro-Wilk test indicated lack of multivariate normality (p < .001). However, Finch^[158] reported robustness of the MANOVA against violation of the normality assumption. Levene’s tests indicated homogeneity of variances for all dimensions (ps >.05). Correlations between dependent variables were low (rs < .33), indicating that multicollinearity was not a confounding factor in the analysis. The calculated MANOVA did not show statistically significant main effects of music (Pillai’s Trace = .02, F(4,114) = 0.44, p = .777) or sound effects (Pillai’s Trace = .01, F(4,114) = 0.21, p = .933) on the combined dependent variables. In contrast, a significant interaction of both (Pillai’s Trace = .0.09, F(4,114) = 2.69, p = .035) was indicated. Therefore, post-hoc univariate ANOVAs were conducted for every prejudice dimension. However, due to the low reliability of the subscales Malevolence and Authoritarianism reported in Section 5.2.2, the respective results should be treated with caution and are presented separately.

Regarding self-reported Fear/Avoidance behavior, no significant main effect of music or sound effects, nor a significant interaction effect was obtained. Similarly, planned contrasts did not indicate significant differences between conditions with non-speech sounds and the Basic condition (t(117) = -0.33, p = .742, d = .21), nor between the BothAdds condition and those integrating only one type of non-speech sounds (t(117) = 0.16, p = .877, d = .07).

Concerning Unpredictability, again no significant main effects, neither of music nor of sound effects, or an interaction effect were obtained. Similarly, planned contrasts indicated no significant difference between the Basic and the other conditions (t(117) = -1.55, p = .123, d = .98), as well as contrasting the BothAdds condition to the conditions integrating either music or sound effects (t(117) = -0.83, p = .409, d = .37).

Although results should be treated with caution due to low reliability, we computed the same tests for the subscales Malevolence and Authoritarianism. This again resulted in insignificant group differences, but a significant interaction effect for Authoritarianism. However, pairwise post hoc comparisons yielded no significant differences between the individual conditions. Regarding planned contrasts, for Malevolence, again no significant difference between the Basic and the other conditions (t(117) = 0.24, p = .810, d = .15), as well as contrasting the BothAdds condition to the conditions integrating either music or sound effects (t(117) = -0.87, p = .385, d = .39) were indicated. For Authoritarianism, the planned contrasts also revealed no significant differences between the Basic and the three other conditions (t(117) = 1.66, p = .099, d = 1.05) and the BothAdds compared to both the Music and Effects (t(117) = 1.74, p = .084, d = .77) conditions.

6.2.4 Transportation as a mediator

To analyze the mediating effect of participants’ transportation into the story on their empathy toward the story protagonist, we computed a mediation analysis with bootstrapping with 5,000 bootstrap samples. Music and sound effects (present vs. absent) served as predictors, transportation as mediator, and affective, cognitive, and associative empathy as outcome variables. The estimated model is depicted in Figure 3. As indicated by R² values, the model explained only 1.0% of the variance in Transportation, but 46.5% of the variance of Affective, 22.3% of the variance of Cognitive, and 43.0% of the variance of Associative Empathy. We did not observe any significant total effects (ps >.05). Moreover, no significant indirect effects of music or sound effects on any empathy dimension via Transportation were revealed (ps >.05), indicating no mediating role of transportation. In contrast, a significant direct effect of music on Cognitive Empathy (p = .044) was indicated, whereas no direct effects emerged for music on Affective Empathy (p = .533) or Associative Empathy (p = .138). For sound effects, no direct effect was found for Cognitive (p = .658), Affective (p = .188) or Associative Empathy (p = .269). Regarding the path coefficients, neither music (p = .441) nor sound effects (p = .441) predicted Transportation significantly, while Transportation significantly predicted Affective (p < .001), Cognitive (p < .001), and Associative Empathy (p < .001).

Display Full Size

Download

Figure 3. Mediation model predicting empathy.

To investigate the mediating effect of participants’ transportation on their prejudice toward individuals affected by mental disorders measured after the intervention, we computed a mediation analysis using the presence or absence of music and sound effects as predictors, transportation as mediator, and the prejudice dimensions Fear/Avoidance, Malevolence, Authoritarianism and Unpredictability as outcome. Again, 5,000 bootstrap samples were used. The estimated model is depicted in Figure 4. As indicated by R² values, the model explained only small proportions of variance in prejudice, namely 1% of the variance of Unpredictability, 3.5% of the variance for Authoritarianism, 4.6% of the variance for Fear/Avoidance, and 7.3% of the variance for Malevolence. We did not observe any significant total effects (ps >.05). Furthermore, no significant indirect effects of music or sound effects on any prejudice subscale through Transportation were revealed (ps >.05), indicating that there was no mediation. Also, no significant direct effects were observed (ps >.05). Regarding the path coefficients, neither music (p = .447) nor sound effects (p = .444) significantly predicted Transportation. In contrast, Transportation significantly predicted the prejudice dimensions Fear/Avoidance (p = .034), Authoritarianism (p = .042), and Malevolence (p = .005) but not Unpredictability (p = .769). Again, reliabilities for the Malevolence and Authoritarianism subscales were low, thus interpretability is limited.

Display Full Size

Download

Figure 4. Mediation model predicting prejudice.

6.3 Qualitative data

Only 19 participants provided qualitative feedback using the comments section. Seven comments were entirely positive (n_BothAdds = 3, n_Music = 2, n_Effects = 1, n_Basic = 1), while ten comments were of negative valence (n_BothAdds = 1, n_Music = 5, n_Effects = 3, n_Basic = 1). Lastly, two comments were neutral (n_Effects = 1) or mixed in their valence (n_BothAdds = 1). Concerning positive feedback, participants liked the overall study idea of a robotic storytelling on mental health issues (n_BothAdds = 2, n_Music = 2, n_Effects = 1), e.g., “The story was very interesting to hear and really moved me. I would be delighted to participate in further studies with Pepper and mental illness in the future!”, and enjoyed listening to the robot (n_Music = 1, n_Basic = 1). One person in the BothAdds condition specifically referred to the music and sound effects as “effective”, whereas another person from this group expressed specific liking for the robot’s body language. In contrast, other participants claimed the body language to be distracting (n_Music = 2) and the robot’s voice to be hard to understand (n_Music = 1, n_Effects = 1, n_Basic = 1), e.g., “The robot spoke quite quickly and somewhat unclearly, and I didn’t understand some of the important words”, also due to the sound effects (n_Effects = 1). One person found the music partly inappropriate (n_Music = 1). Lastly, three participants found it difficult to answer the questionnaires (n_Music = 1, n_Effects = 2).

6.4 Comparison to written story

As an exploratory analysis, we compared the storytelling experience between participants reading the story in the prestudy and participants receiving the story from the robot in the main study. Prior to statistical testing, assumptions were checked. Levene’s tests indicated equality of variances for all variables, but Shapiro-Wilk tests revealed a violation of the normality assumption for all variables (ps < .05). Thus, non-parametric Mann-Whitney U-tests were conducted comparing Transportation as well as Empathy between the prestudy and the main study regardless of the sound conditions.

As shown in Table 3, no significant differences between the text version and the robotic storytelling were obtained. Descriptively, Transportation was slightly higher in the main (M = 5.04, SD = 1.14) than in the prestudy (M = 4.91, SD = 1.34), whereas Affective, Cognitive and Associative Empathy scored slightly higher in the prestudy (M_Affective = 2.00, SD_Affective = 0.86; M_Cognitive = 2.18, SD_Cognitive = 0.85; M_Associative = 1.97, SD_Associtive = 0.87) compared to the main study (M_Affective = 1.91, SD_Affective = 0.81; M_Cognitive = 2.17, SD_Cognitive = 0.86; M_Associative = 1-76, SD_Associtive = 0.87).

7. Discussion

We conducted a laboratory study to investigate the beneficial effects of a robotic storyteller’s multimodality, focusing on its ability to seamlessly integrate music and sound effects, on its deployment as an anti-stigma intervention. In doing so, we compared storytelling experience, emotion induction, attention, and prejudice measured after the intervention. The high number of participants excluded from the experiment, i.e., due to their own experiences with mental illness, 24 in the pretest and 10 in the main study, underpin the importance of the topic.

As indicated by participants’ blink rates, no differences in attention during story reception were observed between the sound conditions. Thus, H1a and H1b are both rejected. Although listening to music has been shown to decrease blink rates compared to silence^[154], suggesting heightened attention, our results indicate that neither background music nor sound effects or their combination provided an additional attentional benefit beyond the attention elicited by the storytelling itself. This finding is quite surprising given the positive correlation reported by Santavirta et al. between audio intensity and blink rates in movie watching analyses^[159]. However, film audio is closely connected to both visuals and the narrative, and was not manipulated in their experiment. Comparing blink rates of participants viewing different versions of a movie with the same narrative and content, Andreu-Sanchez et al.^[160] found that editing style did not influence the timing of blinking, suggesting that rather narrative content than visual style is decisive for when to blink. The same may hold true for auditory style. Future work across different storytelling media is needed to shed more light on the relationship between non-speech sounds and attentional blinking in mediated storytelling, including robotic storytellers. Ideally, such studies should incorporate additional measures, such as questionnaires, to provide a more comprehensive understanding of these effects.

We found no significant differences between the conditions integrating additional sounds and the control condition concerning recipients’ transportation into the story, resulting in the rejection of H2a. This finding is in line with related research examining the integration of music and sound effects in robotic storytelling across different story genres^[53,128]. In contrast to related work recommending the use of music only^[53], we observed a positive effect of combined versus single sound additions, resulting in stronger narrative transportation for recipients in the BothAdds condition. Accordingly, H2b is accepted. The reason could be the utilized sound effects. While^[53] employed romantic short stories that focus on interactions between (to-be) couples and added sound effects primarily to illustrate the environment, our story centered on the experience of a single individual, using sound effects not only to depict the environment but also to convey the protagonist’s inner thoughts and emotions, for example through rapid breathing. Although we did not assess participants’ perceived appropriateness of the sounds, employing the full interpretive range of sound effects to communicate both descriptive^[60,94,100] and emotional information^[98,101], the stimuli integrating sound effects in our study, namely the conditions Effects and BothAdds, may have been better suited to provide insight into the transportative potential of sound effects, as reflected in the higher values compared to^[53]. Therefore, in terms of improving storytelling experience, we not only recommend the combination of sound effects and background music for robotic storytelling, but also the integration of sound effects to convey both environmental and emotional information. Furthermore, compared to related works employing relatively short sound effects, such as “the sound of something that is dropped on the floor, swallow and rustling paper, or opening a door”^[53], we used longer-lasting sound effects, such as breathing or heartbeats, facilitating the appropriate time-wise placement of the sound effects. This way, the timing might have been improved, highlighting the need for future research to systematically explore not only the type of sound effects integrated, but also their timing to achieve generalizability. Interestingly, comparing our transportation values to the same related work^[53] as well as further related studies^[128,161], we find generally improved transportation across all conditions in our study. This finding suggests the appropriateness of the story theme for robotic storytelling.

In contrast to our findings on transportation, combining both types of non-speech sounds did not enhance participants’ empathy towards the story protagonist, leading to the rejection of H3. However, while the integrated non-speech sounds did not influence participants’ Affective Empathy, their affective reactions to the protagonist’s emotions^[136], nor their Cognitive Empathy, their ability to adopt the protagonist’s perspective^[136], they did influence their Associative Empathy, reflecting participants’ sense of “social bonding” with the protagonist^[162]. We found a significant advantage of both the Basic and the BothAdds conditions over the Music condition for Associative Empathy as well as a significant difference contrasting the BothAdds condition to those integrating only one type of non-speech sounds (RQ1). Music seemed to increase the psychological distance between recipients and the story protagonist. Our mediation analysis sheds light on the underlying mechanisms. Using the presence or absence of music and sound effects as a predictors, we found that Transportation does not mediate the effect of the experimental manipulation on empathy. Nevertheless, Transportation appears to be a significant predictor of all three dimensions of empathy, while being only minimally influenced by the integration of non-speech sounds itself. The effect of music integration on Associative Empathy did not reach significance in the mediation model, likely due to opposing patterns in the Music and BothAdds conditions, which seem to cancel each other out. In contrast, a significant direct effect of music on Cognitive Empathy emerged, which only manifested when controlling for Transportation. This suggests that, although Transportation positively predicts Cognitive Empathy, the integration of music itself may have a negative influence, which is suppressed by the effect of Transportation. As a result, when controlling for Transportation, the integration of music was associated with slightly lower levels of Cognitive Empathy. As with Associative Empathy in the ANOVA analysis, music may have shifted attention away from cognitive perspective-taking processes. This result contradicts findings reported by McDonald et al.^[163], who observed that sad music accompanying videos of emotional distress in others improved empathy compared to neutral music. In our study, however, the story concluded with a happy ending, and emotionally-congruent heroic music was used to illustrate the protagonist’s pride in managing her chronic condition. The rapid change in emotional valence may have caused confusion, potentially negatively influencing the positive effect of music on empathy. Taken together, these mediation analysis results suggest that Transportation seems to be a competing predictor rather than a mediator of music integration in our study. Music only improved empathy when presented together with sound effects, strengthening the recommendation to combine both types of non-speech sounds in robotic storytelling.

Concerning prejudice toward individuals with mental illness, the manipulation of non-speech sounds did not yield any significant effects. While the results concerning Malevolence are not interpretable due to unacceptable internal consistency, none of the four dimensions, namely Fear/Avoidance behavior, Malevolence, Authoritarianism, or perceived Unpredictability, differed between the conditions, neither when contrasting non-speech sound integration with Basic nor when comparing single sound types to BothAdds. Accordingly, H4a and H4b were rejected. These null findings may partly be explained by floor effects, which also clarify the low reliabilities computed. For all dimensions except Unpredictability, mean scores were low and showed limited variance, leaving little room for between-condition differences. Mental health issues are particularly prevalent among university students compared to the general population of the same age^[164]. Although individuals with personal or close OCD experience were excluded, participants may still have had greater contact with affected individuals than the broader population, potentially contributing to generally low prejudice levels. Furthermore, particularly the Malevolence and Authoritarian subscales are referred to as “unfavourable attitudes”^[13], explaining the drop in internal consistency. Future evaluations should integrate measures on social desirability to account for such answer patterns. Moreover, given the generally low levels observed, incorporating pre-intervention assessments of prejudice or a placebo condition, such as a robotic storytelling of another genre, in future studies would allow for a more accurate evaluation of potential changes. Compared to the other dimensions, Unpredictability is more subtle^[13], presumably explaining the absence of floor effects in this dimension. In line, related studies report higher values on this compared to the other dimensions^[66,165]. One possible explanation for the absence of experimental effects is that Unpredictability represents a more cognitively entrenched dimension^[166,167]. Although our story conveyed self-efficacy of the protagonist and illustrated her control over the situation across all four conditions, our sound manipulation rather targeted emotional processing of the story. Thus, it may not have added persuasive impact beyond the information already conveyed in the story, resulting in comparable prejudice levels in terms of Unpredictability across conditions. A similar picture is drawn by our mediation results. Although Transportation did not mediate the sound conditions and prejudice dimensions, leading to the rejection of H5b, Transportation significantly predicted all prejudice dimensions except for Unpredictability. This finding strengthens the suggestion that this cognitive facet of stigma is less influenced by storytelling experience and its manipulations, than the more affective and evaluative aspects of prejudice are. However, the small proportion of explained variance in our model suggests that although Transportation affects prejudice, its influence is limited, indicating that other factors likely play a substantial role.

Comparing the results of the main study with those from the online prestudy revealed no significant differences in Transportation or empathy. This suggests that, despite participants in the main study reporting that the robot’s voice was difficult to understand, the overall processing fluency did not appear to hinder transportation^[168,169]. This finding is encouraging, as traditional stigma-reduction interventions frequently rely on written materials such as vignettes^[15,170], but do not bear the opportunity to add further modalities. Robotic storytelling may therefore represent an equally effective, or potentially even more effective, alternative. Nevertheless, future research should employ pre–post designs and/or placebo conditions to more precisely assess intervention-related changes.

8. Future Work

Although the sound manipulation did not yield the expected effects, our findings suggest that transportation may represent a key mechanism for enhancing the effectiveness of stigma-reducing robotic storytelling interventions. Strengthening transportation may therefore improve overall intervention outcomes. Robotic modalities that have been shown to foster transportation, such as facial expressions^[43,119], should be systematically examined in future research with regard to their potential to reduce prejudice. In line, the usage of different robot models, not only being capable of facial expressions but also differing in size or voice, should be taken into account in future studies to allow for greater generalizability of the findings. Moreover, Steinhaeusser et al.^[45] reported positive effects of interactivity in a robotic storytelling intervention against bullying for children, particularly in increasing bullying awareness. This suggests that interactivity may also be a promising component in stigma-reducing robotic storytelling and should be considered in future intervention designs. Finally, findings from chatbot research suggest that a first-person narrative perspective is more effective in simulating social contact and reducing mental health stigma than a third-person perspective^[15]. Accordingly, future research should also consider manipulations of the narrative itself, such as perspective, as potential mechanisms for enhancing stigma-reduction effects.

While we primarily investigated the role of transportation affecting intervention outcomes, namely empathy and prejudice, in line with our focus on storytelling, Birtel and Oldfield^[167] examined empathy itself as a mediating variable, reporting a significant negative relationship with stigma-relevant attitudes. In addition, they identified intergroup anxiety as an important mediator, which was not assessed in our study. Pettigrew and Tropp^[16] suggest that perhaps anxiety reduction is a prerequisite to increasing empathy and perspective taking for prejudice reduction. Consequently, the variable of intergroup anxiety should be taken into account in future studies to examine the full potential of robotic storytelling as an intervention tool to decrease mental health prejudice.

Prejudice also varies across specific mental disorders. For example, stigma tends to be stronger toward schizophrenia than toward depression^[68], at least in Western countries^[171]. To our knowledge, however, comparable research examining OCD in this context is scarce, although existing evidence indicates substantial stigma toward individuals with OCD^[172], particularly when symptoms involve taboo or unacceptable thoughts, such as those related to sexuality^[173], as depicted in our narrative. The specific form of OCD addressed in our study, namely pedophilia-themed OCD, may have introduced additional bias due to its association with a highly stigmatized and criminalized topic, potentially limiting our findings. The purpose of the story was to illustrate that OCD-related intrusive thoughts can be entirely inconsistent with an individual’s actual personality, values, and intentions. However, future studies on robotic interventions against stigma toward mental illness should examine different forms of mental illness and OCD symptom presentations in order to compare the effectiveness between them. Furthermore, the study was limited by its sample, which consisted predominantly of young female participants, as well as by its design, which assessed stigma only after the intervention. Consequently, changes in prejudice could not be evaluated relative to baseline levels. Therefore, future studies should not only integrate a placebo condition and a pre-measurement of prejudice to gain better insights into the stigma change, but also test the robotic storytelling intervention with a more diverse participant sample to improve the generalizability of the findings. Moreover, future research should adopt longitudinal designs with repeated interventions, given that the effects of single exposure interventions have been shown to fade as stereotypes re-emerge over time^[71]. In doing so, the suitability of social robots for stigma-reducing storytelling interventions suggested by our results could be further explored in the future.

9. Conclusion

Despite greater public awareness, stigma against people with mental illness is still a serious issue in today’s society. It impairs the lives of those affected by undermining their self-confidence and well-being. Storytelling-based anti-stigma interventions can help reduce prejudice by promoting education and substituting for direct contact. Social robots as physically embodied agents capable of multimodal communication offer additional advantages for such interventions. In particular, the integration of music and sound effects may benefit the outcomes. To this end, we evaluated a robotic storyteller as an intervention method, focusing on the influence of additional sound integration, namely background music and sound effects. While the integration of non-speech sounds did not directly reduce prejudice, their combined use enhanced transportation and associative empathy, highlighting the importance of carefully designed multimodal storytelling. In contrast, music presented in isolation yielded less favorable effects, suggesting that isolated auditory augmentation may not be sufficient and could interfere with certain empathic processes. Moreover, transportation emerged as a central mechanism, predicting empathy and, in part, prejudice-related outcomes. Although mediation effects were not supported, the results underscore transportation as a promising lever for strengthening the impact of robotic anti-stigma interventions. Enhancing recipients’ transportation into the story may therefore be more critical than simply increasing sensory richness used to convey emotions.

In summary, while our study provided valuable insight into the potential of robotic storytellers in mental health stigma interventions, and particularly into the transportation they can foster, future research should investigate how this transportation can be systematically stimulated through narrative perspective, interactivity, or additional multimodal cues. Moreover, future studies should employ longitudinal pre–post designs and placebo conditions to better capture changes in prejudice over time. Such work will be essential for advancing robotic storytelling as an evidence-based tool to reduce mental health stigma.

Supplementary materials

The supplementary material for this article is available at: Supplementary materials.

Acknowledgements

The authors would like to thank Sandra Molter from the Contact and Information Center for Students with Disabilities and Chronic Illnesses (KIS) at the University of Würzburg for her valuable advice on participant recruitment and in developing the story used in this study. The authors also thank Sophia Maier and Ohenewa Bediako Akuffo for their help in annotating the observational data.

Authors contribution

Steinhaeusser SC: Conceptualization, methodology, data curation, formal analysis, resources, visualization, writing-original draft, writing-review & editing.

Semineth M: Conceptualization, methodology, investigation, writing-original draft.

Lugrin B: Conceptualization, supervision, writing-review & editing.

Conflicts of interest

The authors declare no conflicts of interest.

Ethical approval

The presented study was approved by the ethics committee of the Human-Computer-Media Institute at the University of Würzburg (vote #240823).

Consent to participate

Written informed consent to participate in the study was obtained from all participants.

Consent for publication

Not applicable.

Availability of data and materials

Data supporting the findings of this study are available from supplementary materials and the corresponding authors upon reasonable request.

Funding

None.

Copyright

© The Author(s) 2026.