Ripples: An Auditory Augmented Reality iOS Application for the Atlanta Botanical Garden

Master Project

Fall 2018 - Spring 2019

Abstract

“Ripples” is an iOS application for the Atlanta Botanical Garden that uses auditory augmented reality to provide an intuitive music guide by seamlessly integrating information about the garden into the visiting experience. For each point of interest nearby, “Ripples” generates music in real time, representing a location through data collected from users’ smartphones. The music is then overlaid onto the physical environment and binaural spatialization indicates real-world coordinates of their represented places. By taking advantage of the human auditory sense’s innate spatial sound source localization and source separation capabilities, “Ripples” makes navigation intuitive and information easy to understand.

Video Demo

Poster

Introduction

The Atlanta Botanical Garden provides paper maps to help visitors navigate their surroundings. However, the information on the map is not very legible. Many points of interest are close to one another, making the map hard to read. Moreover, constantly checking the map can distract from the experience. To provide a more intuitive guide and encourage users to explore new areas of the garden, we introduce “Ripples”, an auditory augmented reality (AR) iOS application that guides visitors through music.

The ideal of AR is the integration of virtual objects into the physical world. Accordingly, an auditory AR application integrates acoustic virtual objects into the real-world environment. Objects without intrinsically produced sound are sonified with a spatialized audio system that corresponds to real-world coordinates. In this way, auditory AR reveals additional information about the object and the environment.

“Ripples” employs auditory AR to enable the user to learn the environment through music. It takes environmental and user data collected by users’ smartphones to generate music that evolves during their visit. To better blend the generated music with the environment, “Ripples” also introduces pre-recorded, geo-tagged, environmental sound samples into the music when they revisit certain places. “Ripples” serves as a navigational tool through the binaural spatialization of generated music that corresponds to the two closest points of interest, creating an aural guide towards those locations. Finally, to help users attend to environmental sounds they might otherwise ignore, “Ripples” augments and emphasizes certain sounds by processing incoming ambient audio through location-based custom equalization.

Related Works

Previously, researchers and artists have developed several locative audio walk experiences. For example, The Rough Mile project and locative-aware album/iOS application The National Mall both feature geo-tagged audio/music segments that are triggered at specific locations. However, these projects use pre-recorded music that do not respond to environmental elements dynamically. As the environment constantly changes, these audio experiences might not fully bond with the environment. Other researchers have explored introducing environmental factors into audio/music generation. For example, the Sonic City project is a wearable system that collects environmental and user data to create electronic music in real time based on the user’s interactions with the urban environment. UrbanRemix is a collaborative locative sound project that allows users to record and share geo-tagged sounds through a smartphone application and later remix these sounds into compositions through a web interface. The iOS application RJDJ processes environmental sounds by listening to the surrounding environment through a microphone and harmonizing the audio to deliver an augmented listening experience. As geographic and environmental data become increasingly accessible through evolving technologies, the concept of auditory AR is being defined and built. For instance, Microsoft proposed an auditory AR system for the visually impaired that uses computer vision to identify high-level features of real-world objects and sonifies these objects at their real-world coordinates using spatialized 3D audio synthesis. “Ripples” further expands the locative-aware audio experience by combining auditory AR concepts with environment-based music generation techniques to create an artistic guide that enhances the experience of visiting the Atlanta Botanical Garden.

Design

The sound engine has four layers, ambient music, acoustic instrument, concrete, and ambient environmental sound (Figure 3). For the first two layers, environmental and user data are mapped onto parameters of the synthesizer and sampler and control them in real time. The output is then fed into a granular synthesis effect unit. Environmental and user data are also used to generate musical sequences. All mappings and sequence generating algorithms are carefully designed for each ripple to ensure the generated music is well integrated with the characteristics of the ripple, ensuring they are aurally distinct. The concrete layer, also known as the dejavu layer, introduces geo-tagged environmental sound samples we previously recorded in the garden. A sample is triggered by revisiting a certain place, creating a virtual entity that does not currently exist in the environment but intrinsically belongs to the space, thus naturally blending the virtual world with the real world. For instance, when revisiting the Children’s Garden, users will hear previously recorded children speaking and laughing. The ambient environmental sound layer also takes incoming ambient audio signals from the microphone and processes them through the equalizer. The equalizer’s parameters change according to environmental data, determining what environmental sounds are emphasized for the current location and time. For example, in the bird habitats, the system will emphasize the frequency bands of bird chirping. Finally, the user’s walking pace determines the overall BPM of music generation.

Conclusion and Future Works

This paper introduces “Ripples”, an auditory AR guide for the Atlanta Botanical Garden with real-time, spatialized, environmentally responsive music generation. To better integrate generated audio/music into the physical world, the next version will introduce a real-time ambience analysis feature, allowing the audio/music to be generated adaptively to environmental sounds so they will not mask one another, but instead integrate as one piece of music.

Acknowledgments

The authors appreciate the support and assistance of the Atlanta Botanical Garden and AudioKit development team, making this project possible. We particularly thank Wenyu Mao for UI/UX design assistance and consulting. We would also like to acknowledge the contributions of Yongliang He and Tejas Rode to the Tiani project, the foundation for “Ripples”.