Deep Drawing

Project Overview:

Deep Drawing is an intermedia AI co-performer project, created in collaboration with Julie Zhu, John Granzow, and Zhiyu Zhang, as part of the FEAST (Faculty Engineering/Arts Student Teams) projects at ArtsEngine, University of Michigan, integrating drawing and music for live, interactive performances. The project explores AI’s role in intermedia performance by using machine learning models to transform live audio inputs from drawing or writing into visual outputs. This involves real-time interpretation of audio data captured by contact mics on a tabletop setup, generating projected digital art. The project addresses significant challenges in processing drawing sounds, characterized by high-variability noise, and the difficulty of leveraging time differences of sound arrival due to the rapid speed of sound. Achieving real-time performance adds substantial computational demands. A novel deep learning model is being developed to enhance speed and accuracy for seamless performances.

The Live Setup

Model 1.0

Keywords: Real-time Audio Processing, Deep Learning, Digital Art Generation, Algorithmic Creativity

Research Areas: Computational Creativity, Human-Computer Interaction (HCI), Artificial Intelligence, Interactive Systems Design, Digital Signal Processing

Key Details:

Machine Learning Innovations: Experimentation with deep learning architectures for improved real-time response.

Technical Challenges: Managing noise interpretation and real-time processing constraints.

Academic Recognition: Presented and performed at AI Music Creativity Conference (AIMC) 2024 at Oxford University.

Future Goals:

We aim to expand the Deep Drawing project by optimizing the newly developed deep learning model to further enhance speed and accuracy. Future work will explore additional interactive features to enrich the performance experience, such as more complex audio-visual mapping and adaptive learning for varied user input. Plans also include integrating new artistic modalities and scaling the system for broader artistic and educational applications, supporting larger performance spaces and diverse artistic collaboration.