Sound Figures by Ernst Chladni (1787) is the origin of systematic attempt to visualize sounds as image. Inspired by Chladni’s acoustics research, thesis of Structural Analogy is that aesthetics of audiovisual transformation could be enriched by systematically integrating music information retrieval (MIR) and pattern formation techniques into creative processes. Sound visualization using short-term acoustic features is one of the effective strategies in engaging audience in live audiovisual performance. However, short-term acoustic features alone are not sufficient in engaging audience in art forms that require time to unfold narratives, such as in audiovisual composition and interactive installation. To examine this thesis, a custom computer graphics application that integrates the open-source Essentia MIR toolbox is developed and capable of visually modeling causality based on a source sound signal. This music video is created using the above software application, which enables examination of emergent connections between sound and visual at various spatial and temporal scales through computation and algorithms. Structural Analogy is an ongoing experiment that investigates techniques necessary to visualize narratives in source sound signals. Since 2016, Structural Analogy has been exhibited at international venues, including FOCA, SBCAST, DDP, WOCMAT-IRCAM Forum, and CURRENTS.
Yuan-Yi Fan studied computer music, media art, and multimedia engineering in Media Arts & Technology at University of California Santa Barbara. His research has evolved from new interfaces for musical expression, to audiovisual synthesis techniques, to creative music information retrieval applications. Currently he works at a Hollywood-based startup. He designs futuristic product UX by engineering systems that combine HCI/DSP/ML research with emerging media technologies. He made creative projects with natural user interfaces and connected devices at industrial research labs, including Nokia Research Center, Oblong Industries, and Nokia Advanced Design Studio.
His works have been presented and exhibited at festivals, museums, and conferences, including ACM Multimedia (ACM MM), ACM User Interface Software and Technology (ACM UIST), International Society for Music Information Retrieval (ISMIR), International Computer Music Conference (ICMC), New Interfaces for Musical Expression (NIME), International Community for Auditory Display (ICAD), Leonardo, Leonardo Music Journal, WOCMAT/IRCAM Forum (International Workshop on Computer Music and Audio Technology), The 1st L.A.S.T. Festival, International Symposium on Electronic Art (ISEA), IEEE VIS, Currents New Media Festival, ZERO1 Biennial, Dongdaemon Design Plaza (DDP), and ZKM Globale: inSONIC. Before UCSB, he built MEMS ultrasound array transducers at the Ultrasound Imaging Lab in Taiwan.