The Information Policy Council develops AI music understanding and smart concerto technology to turn AI into a real-time interactive virtual musician

Generative AI can already generate music quickly. When AI can understand performance changes in real time, it can move from a music generation tool to a "virtual musician" performing with real people. (Screen taken from the website of the Funding Policy Council)

[Reporter Qiu Qiaozhen/Taipei Report] Generative AI can already generate music quickly, but it is more difficult for AI to actually follow a person's performance, because a real person's performance will change the intensity, speed and rhythm according to the mood and style of the music. Traditional fixed concertos are mostly played according to the preset speed and style. It is often "human and machine", rather than the machine understanding the musician's performance. Therefore, the Institute of Science and Technology has developed AI music understanding and smart concerto technology, allowing AI to understand performance changes in real time, moving from "generated music" to "real-time concerto".

With the support of the Science and Technology Project of the Technology Department of the Ministry of Economic Affairs, the Institute focuses on "making AI understand music" and integrates the four key capabilities of "hearing clearly, understanding, simulating, and synchronizing the beat", allowing AI to go from generating accompaniment to a "virtual musician" that can sense, understand and respond to the musician's performance.

Please read on...

First, the system uses environmental noise reduction and separation of multiple sound sources to allow AI to "hear" the performance content. It then uses a music understanding model to identify rhythm, chords and genres to achieve "understanding". At the same time, it uses AI timbre modeling to reproduce the timbre of musical instruments to achieve "realism". Finally, through dynamic tracking, the speed and rhythm can be adjusted in real time to achieve "the same beat" as a real person.

In terms of practical applications, the Information Strategy Conference will demonstrate two interactive scenarios: "AI Music Partner" and "AI One-Man Group". "AI Music Partner" allows users to simply hum, and AI understands the music content and style, and then provides music generation and creative extension; "AI One-Man Group" presents AI smart concerto with guitar playing. When the player speeds up, slows down or changes the playing rhythm on the spot, the AI ​​virtual musician can also dynamically adjust the accompaniment to complete the concerto with the real person.

The Information Policy Council pointed out that the purpose of this technology is not to replace the profession of musicians, but to make AI a new creation and performance tool. For creators, a melody can extend different musical styles and arrangement imagination; for performers, AI can become a virtual group member that can interact in real time, allowing personal performance to add more parts and improvisational changes, and further apply it to personal creation, practice and live performances.

In addition to human-machine synergy, the Information Strategy Council will also focus on productizing the technology in the next step. Taiwan has advantages in manufacturing electronics, chips, audio equipment and musical instruments. The Information Policy Council hopes to use technologies such as AI music understanding, real-time concerto and timbre generation to further equip existing hardware with perception, understanding and interaction capabilities from sound playback and processing, and promote the introduction of technology into applications such as digital musical instruments, AI electric guitars, smart speakers and interactive performances.

At present, relevant results have been verified through actual music performances. In the future, we will continue to collect feedback from music creators, performers and industry players, and link musical instrument manufacturing, audio chips, electroacoustic equipment and digital content companies to accelerate the integration of technology and products and develop a new model of "hardware + AI audio engine + smart service". This set of AI music understanding and smart concerto technology will also be exhibited at the 2026 ITRI ICT TechDay.

Grasp the economic pulse with one hand. Click here to subscribe to Free Finance Youtube Channel