Turning lights into a piano

Turning crazy ideas into real-life concepts is what we try to do at Blue Hay. One of those crazy ideas from @EliDhae was to use led tubes as a piano keyboard. She even took the challenge one step further by suggesting to put the led tubes in a circle. That way one person could easily play a musical tune by pointing at the lights.

Once the concept was defined, we immediately began searching for a team to bring it to life. Next step, hardware and software. Thanks to our partner Vyvalite, we acquired 16 LED tubes to create our virtual piano.

The LED tubes were easily controlled through DMX interfaces, a standard in the lighting industry. However, to incorporate motion detection, we needed to connect them to a computer. To control the lights using motion detection, we employed Touch Designer. This software gathers all sensory information and sends out controls to the DMX interface.

Thanks to a large number of components in Touch Designer the link with Google’s MediaPipe solutions was easily made. This library of AI tools developed by Google provides a lot of ready algorithms like object detection, image classification, face recognition and of course the pose detection and tracking we needed. The input for the pose detection came from a GoPro camera that we tested by mounting it on the ceiling.

Of course, some math was needed to convert the x and y coordinates received from the pose detection into a number between 1 and 16. This number defined which segment of the circle the person in the middle was pointing at.

The final component to address was generating notes based on the position of the index finger pointing at an LED tube. At first we started by creating synthesizer sound within Touch Designer, but that didn’t give the right sound we needed for the experience. Instead we chose to use mp3 files for each of the sixteen notes we needed to create the virtual Piano. Why sixteen, you might ask. Well, to be able to play “Fur Elise” of course.

Each of the components were tested up front and worked on their own. The toughest parts was yet to come. Integrating everything.

We set up our project in a dark venue typically used for events and concerts. First, we fixed the 16 LED tubes to microphone stands. Then, we mounted a GoPro camera on the ceiling and connected it to a laptop running Touch Designer, using the camera as input. Finally, we linked a Bluetooth speaker to play the piano notes.

After a lot of try-outs and calibration of the camera we got everything working and were able to play the 16 notes, light the lights each on their own. Mission accomplished. And the overall feel? It was fun playing with it and we kept playing notes and light. The combination of these technologies opens up a world of possibilities. Looking to create a dazzling experience? Join our meetup or contact us.