Masaoka has developed compositions for interfaces for the computer and live koto; for computer and the live physiological response of plants and brainwave activity, and also for computer and infrared, laser and ultrasound sensors for use in both performance and installation.
As a composer/performer searching and developing new koto techniques for exciting and limitless timbral possibilities, the use of electronics for the purpose of transforming the koto sounds to an open-ended, virtual koto seems to be a reasonable solution.
From 1200 years ago in the Tang Dynasty, a significant psycho-acoustic act of extreme gesturalism has developed and survived in gagaku koto playing. After plucking a particular string, the right hand and arm gracefully rise, and with great ritualistic care, hold a hand position in the air for several seconds—fingers are curved torwards the inner wrist requiring a "limp wrist" articulation that resembles birdís beak and neck—reminiscent of a hand position in Tai Chi. With no obvious aural or sonic meaning, this simple yet characteristic motion in the air remains steadfastly part of traditional playing, and has survived when countless other koto techniques and repertoire have been lost.
This practice is what the author calls "aural gesturalism," a term for the physical act that is performed with a musical intention, yet makes no sound in and of itself. For a harp player, or a drummer, the motion might resemble an exaggeration of an arm movement after hitting the drum, or plucking a string, a follow-through of the arm that has just struck the instrument. In Japanese koto gagaku practices, the motion has been stylized in a way not seen in other musical traditions.
It is precisely this aural gesturalism, this "non-sonic" practice that is particularly intriguing, and has been the inspiration for the use of koto gesture control devices, as a kind of "bridge" between the tradition and the computer, and remains an effective metaphor for control.
Retaining a visceral and sensorimotor experience in control remains important in designing an interface that can be compatible and not incongruent to the physicality of koto playing. From an instrumentalistís perspective (as distinguished from, say a dancerís point of view) arms waving above the head in time and space must have meaning.
The koto, an acoustic wooden instrument, presents special acoustic design challenges for creating an effective method of recording, mapping, controlling and processing the samples. Using both pre-recorded koto samples (900 samples), and samples recorded in real time, captured koto sounds are then are layered, processed and triggered sometimes retaining distinctive koto-like sounds, and other times the samples are transformed beyond recognition. From the point of view of an instrumentalist, what is desired is an interface that can respond in real time and retain an improvisational and interactive quality that is new and unexpected.
Colleagues Jon Rose and Tom Cora were STEIM SensorLab users and fellow compatriots melding acoustic instruments to computative technology, and their expertise was invaluable.
Tom Zimmerman (Developer of the Body Glove, Scientific American, May, 1990) and I built an infrared sensor prototype for the koto. Multiple parameters of a single koto string could be tracked and differentiated: vertical, horizontal, circular movements, velocity, frequency and other parameters were successfully detected and analyzed, and the possibilities seemed endless.
Residency at STEIM (Studio for Electro Instrumental Music—unique in that it is dedicated to live performance) two prototypes were built, one that was triggered by gestural hand and arm movements, and the other by data from the koto strings. The SensorLab, an analog-to-MIDI converter, is a small, custom microcomputer that can convert incoming analog electrical information into a standard digital code, which can then be interpreted by a personal computer or to MIDI devices. Four ultrasound sensors were imbedded in two rings worn on each hand that trigger samples when gestured on an X and Y-axis. A series of six pedals, and a button switchboard attached to the koto could select several modes for digital signal processing, and the whether the pedals or the rings, or the pitch-to-MIDI converter were to be active.
Working in the unstable world of custom-built electronics, attempts at melding the acoustic real world to the digital world has proven to be a long voyage. Variants of "proto kotos" have been created, some with as many as 900 samples of original koto sounds—roughly equivalent to fifteen kotos—that can be accessed live and connected via computer to the Japanese traditional koto. These experiments, named "Koto Monster" and "Laser Koto" utilize multiple systems of hardware and software and were developed working with various technicians and institutions at STEIM, CNMAT; and with Donald Swearingen and Matt Wright. Ingenious inventor and colleague Swearingen, of sensorChip, has developed the Laser Harp and a microcomputer housed in acrylic that interfaces faders, lasers and a Basic Stamp analog-to-MIDI converter.