LumaTouch Synesthesia is an interactive system for creating abstract artwork with electronic music by manipulating tangible objects. The system consists of a light table with five movable objects on the surface, four small cubes and a small cylinder. By manipulating these objects, the user can simultaneously create an abstract painting and compose electronic music to complement the painting. The phenomenon of synesthesia is experienced literally in the creation of color and sound.
The location and rotation of the four cubes are detected by a webcam in the light table. The custom software uses these inputs to choose up to eight of 16 reference images to be morphed and combined for the painting and up to eight of 16 music loops to be mixed for the song. The cylinder is used to change the color of the images and the musical key of the song.
While a viewer is interacting with the piece, it will show the digital painting projected on the wall behind the light table, and play music at a moderate volume level in the room. Headphones are also available to allow the operator to listen more closely. The user can retrieve his/her painting by sliding the “ROBGON” gadget to the center of the light table. The image will be uploaded to www.RobGon.com and instructions for retrieval will appear on the screen.
LumaTouch uses OpenFrameworks, an open-source C++ library for creative coding. The tracking used by LumaTouch is based on the concepts from TrackMate, an open-source tracking system. The reference paintings are from the Smithsonian Institute Website, www.si.edu, and are used in accordance with the Smithsonian's terms of use. Some of the music loops are original and some are from looperman.com and are used in accordance with their terms and conditions. I would like to thank Jennifer Lim for her help with this project. |