Highlighting JAPAN

An AI Accompanist Even Avid Musicians Can Appreciate

Yamaha’s Duet with YOO is an AI designed to ensure that anyone can play and enjoy a piano duet. This music technology incorporates an AI that analyzes the player’s performance in real time and provides expert accompaniment to spur its human partner’s musical and creative abilities.

Duet with YOO is a piano installation that accompanies people as they play the piano. Even “Twinkle Twinkle Little Star” played with one finger and one note at a time is transformed into a gorgeous piece thanks to this accompaniment. Duet with YOO will also slow down, play boldly or softly, and otherwise adjust to match the player’s tempo, power and tone.

The musical accompaniment is provided via a self-playing piano, Disklavier™, but it doesn’t sound automated. This is because the Duet with YOO AI “listens” to and instantly analyzes its human partner’s performance, predicting and adjusting its own playing based on that data. The AI Music Ensemble System Yamaha developed that YOO uses places greater importance on the musician’s enjoyment of the piece rather than simply supplying technical precision.

“There are already keyboards that light the keys up to guide you, but we wanted to create a more interactive experience,” says Minako Shintake, Yamaha’s marketing representative for the YOO project. “It’s more fun if you can get excited through performing with an accompanist that can turn things around even if you make a mistake. We wanted everyone to experience that joyful music making, even if they had never played the piano before. We developed a system that would intuit the player’s intentions and simulate very human fluctuations in the music rather than focusing on a technically precise system.”

Duet with YOO is made for beginners, but at the start of development the researchers did not have beginners in mind, according to Akira Maezawa, the technical development engineer who was in charge of creating the YOO AI. “My hobby is playing the violin, and I wanted an accompanist when practicing by myself at home,” he says. “The more advanced a player you are, the clearer your intentions for your performance are, so rather than something like a metronome which would only count out a rhythm, I wanted to create an accompanist that could match and follow the intent and inflection of my playing.”

Based on the many playing patterns it learned, the AI can create nuanced performances from fast and upbeat to slow and somber, allowing it to replicate a wide variety of music.

The engineers also wrestled with the issue of how close to the person’s playing the AI should get. “It would feel strange if it matched every note played, and the musician wouldn’t enjoy the experience of playing with an ensemble,” Maezawa explains. “In the end, we decided the AI should keep some degree of autonomy. The fun of playing in an ensemble is the give-and-take between performers, the feeling of ‘Listen to this!’ when you have a solo and ‘I’ve got your back’ when your partner has one. We set parameters that allow the AI to come forth and shine or hold back when appropriate, so the player can enjoy playing.”

Yamaha has taken its AI accompanist system—loaded with the performance data of Sviatoslav Richter, one of the twentieth-century’s greatest pianists—and connected it to a self-playing piano to give players the sensation that they’re in an ensemble with the Berlin Philharmonic. That means the technology enables players to perform virtually alongside their favorite musicians if the latter’s performance data is available.

“Some people are afraid that AI will take away human jobs, but AI systems like YOO can inspire us to find new awareness,” Shintake says enthusiastically. “We hope to keep providing people such experiences as this, which can lead to new means of expression and music.”