Aims: Multimodal emotion recognition (MER) can outperform unimodal approaches by integrating complementary information from multiple sources. However, real-world applications often involve incomplete or missing modalities, reducing the reliability of existing MER models. This study proposes a framework that remains robust under missing-modality conditions while preserving the advantages of multimodal integration.

Methods: To address this challenge, we propose a cross-modal latent regeneration and attention long short-term memory (CMLR-ALSTM) framework. The framework combines pretrained variational autoencoder encoders with residual projection networks, optimized with L2 loss, to achieve stable latent-space alignment across modalities. Regenerated and available latent representations are integrated using a cross-modal attention mechanism and processed by an LSTM to model temporal dependencies and improve multimodal fusion under incomplete data.

Results: The proposed framework was evaluated on three benchmark datasets under complete, partially missing, and completely missing modality scenarios. Experimental results demonstrate that CMLR-ALSTM achieves up to 17.22% improvement under missing-modality conditions while maintaining competitive performance under complete-modality settings. The results also show that the proposed latent regeneration strategy effectively preserves cross-modal relationships and robust latent representations.

Conclusion: The experimental results confirm the effectiveness of the proposed framework, particularly in realistic environments where data availability is inconsistent. By leveraging CMLR to reconstruct missing representations and modelling temporal dependencies through LSTM, the proposed approach provides a more robust and reliable MER framework for practical deployment. The results across different modality combinations demonstrate its ability to generalize across heterogeneous multimodal settings.

Aims: Mental illnesses affect millions worldwide. Despite greater public awareness, significant stigma remains. Anti-stigmatization interventions can help reduce prejudice by promoting education and meaningful contact with people with mental illness. Technology-based interventions may further support these efforts by simulating such contact through approaches such as storytelling. Social robots may offer additional advantages for storytelling-based interventions, especially through the integration of music and sound effects.

Methods: In multimodal robotic storytelling, non-speech sounds remain largely overlooked, despite their importance in related media. To address this gap, we compared a robotic storyteller narrating a story about a person experiencing a panic attack using only voice and bodily expression with versions integrating sound effects, background music, or both.

Results: While the addition of sounds did not affect recipients’ prejudice differently than storytelling without them, adding both sound effects and music increased transportation and improved associative empathy. In contrast, adding only music decreased associative empathy. Although mediation was not indicated, the results revealed transportation as a predictor for empathy and prejudice, while being only minimally influenced by sound integration itself.

Conclusion: The effects of sound integration in a robotic storytelling intervention on recipients’ prejudice were mixed, recommending either the combination of both sound types or complete omission. Transportation was indicated as a key lever for increasing empathy and decreasing prejudice that warrants further investigation. Future work is needed to gain deeper insights into robotic storytelling as an intervention tool for reducing prejudice, including work on transportation as a modifiable factor and the integration of pre-post-measurements.