Mozualization: Crafting Music and Visual Representation with Multimodal AI
Abstract:
In this work, we introduce Mozualization, a music generation and editing tool that creates multi-style embedded music by integrating diverse inputs, such as keywords, images, and sound clips (e.g., segments from various pieces of music or even a playful cat’s meow). Our work is inspired by the ways people express their emotions—writing mood-descriptive poems or articles, creating drawings with warm or cool tones, or listening to sad or uplifting music. Building on this concept, we developed a tool that transforms these emotional expressions into a cohesive and expressive song, allowing users to seamlessly incorporate their unique preferences and inspirations. To evaluate the tool and, more importantly, gather insights for its improvement, we conducted a user study involving nine music enthusiasts. The study assessed user experience, engagement, and the impact of interacting with and listening to the generated music.
Main reference:
Wanfang Xu, Lixiang Zhao, Haiwen Song, Xinheng Song, Zhaolin Lu, Yu Liu, Min Chen, Eng Gee Lim, and Lingyun Yu. 2025. Mozualization: Crafting Music and Visual Representation with Multimodal AI. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA '25). Association for Computing Machinery, New York, NY, USA, Article 407, 1–7. https://doi.org/10.1145/3706599.3719686
BibTeX:
@inproceedings{10.1145/3706599.3719686,
author = {Xu, Wanfang and Zhao, Lixiang and Song, Haiwen and Song, Xinheng and Lu, Zhaolin and Liu, Yu and Chen, Min and Lim, Eng Gee and Yu, Lingyun},
title = {Mozualization: Crafting Music and Visual Representation with Multimodal AI},
year = {2025},
isbn = {9798400713958},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3706599.3719686},
doi = {10.1145/3706599.3719686},
abstract = {In this work, we introduce Mozualization, a music generation and editing tool that creates multi-style embedded music by integrating diverse inputs, such as keywords, images, and sound clips (e.g., segments from various pieces of music or even a playful cat’s meow). Our work is inspired by the ways people express their emotions—writing mood-descriptive poems or articles, creating drawings with warm or cool tones, or listening to sad or uplifting music. Building on this concept, we developed a tool that transforms these emotional expressions into a cohesive and expressive song, allowing users to seamlessly incorporate their unique preferences and inspirations. To evaluate the tool and, more importantly, gather insights for its improvement, we conducted a user study involving nine music enthusiasts. The study assessed user experience, engagement, and the impact of interacting with and listening to the generated music.},
booktitle = {Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems},
articleno = {407},
numpages = {7},
keywords = {Multimodal Input, Music Editing, Music Visualization},
location = {
},
series = {CHI EA '25}
}