Artificial Intelligence Piano – My Hackweek Project

This article has been contributed by Lin Ma, Software Engineer and KVM Virtualization Specialist at SUSE. If you want to read more from Lin about virtualization, machine learning and artificial intelligence, have a look at the following articles:

• A Pen Plotter Powered by Artificial Intelligence
• How to Do Deep Machine Learning Tasks Inside KVM Guests with a Passed-through NVIDIA GPU
• Machine Learning: Oriented Text Detection from Natural Scene
• Face Recognition Based on *dlib* in a KVM Guest

With this article, I would like to introduce you to my SUSE Hackweek 19 project. If you worked on similar projects or topics, or if you would like to exchange experiences, please feel free to reach out to me.

Overview

As a Do-it-Yourself (DIY) enthusiast, I decided to have some fun during Hackweek with music and machine learning based on SUSE Linux Enterprise Server. Or, if you will, you could also think of my project as an Internet of Things (IoT) attempt based on SUSE products.

For those of you who don’t have much experience with music theory, it might be a painful process of learning how to read notes and playing them on a musical instruments. Thus, the goal of this project is to “read” a sheet music from a picture, then automatically translate it and play it on a physical musical instrument, in our case an electronic organ. And your problem is solved – thanks to our “Artificial Intelligence (AI) Piano” ;-).

Components of the Project

To create the AI piano, I used the following components:

• HUAWEI TaiShan 2280 ARM server running SUSE Linux Enterprise Server for Arm 15 SP1 and TensorFlow 1.4

Step 1 and step 2 run on the TaiShan 2280 server

• Raspberry Pi 3 running openSUSE Tumbleweed for AArch64 and Python 3

Step 3 runs on Raspberry Pi 3.

• A 61 keys electronic organ

My original idea was to buy a Yamaha P128 electric piano. But unfortunately it was too expensive for my project. Thus I used a 61 keys electronic organ instead.

• Solenoids (model 0730), x 21

The solenoids take on the role of the fingers.

• ULN 2803 driver, x 3

Each solenoid could be driven from a GPIO pin of the Raspberry Pi 3. However, as each model 0730 solenoid requires around 500 milliamps to actuation, this is beyond the ability of a Raspberry Pi 3. Thus I used ULN 2803 drivers to drive the solenoids.

Workflow

Regarding the workflow of how to create your AI piano, the following steps are involved:

Step 1: Use the neural network for the feature extraction of musical symbols from your sheet music image(s) serving as input.

Step 2: Use Optical Music Recognition (OMR) to recognize the musical notations (such as staves, measures, notes and so on) which were figured out by the neural network. Then save the result into a music XML file.

Step 3: Receive the music XML file via the TCP network and parse it. Then trigger the solenoids to play the piano.

Limitations of My Project

My piano (the electronic organ) has 61 keys and a range of 5 octaves. Based on cost-control considerations, I only bought 21 solenoids and mounted them on 21 white keys [C3 – B5] on the piano. In consequence, my AI piano can only handle 3 octaves. Moreover, it cannot play any half notes or semitones which are mapping to black keys on the piano.

This means that, while playing piano as indicated in step 3 above, any notation which raises the pitch of the note by one semitone or reduces the pitch of the note by one semitone will be ignored. Any standard pitch that doesn’t belong to these 3 octaves [C3 – B5], will forcefully be scaled up or scaled down to fit into these 3 octaves. The related part of music will seem to be out of tune in this situation.

Some Side Notes

• My experience with circuit diagrams is very limited. Thus the circuit design of this project might not be perfect.
• As the described project was a Hackweek project only, I didn’t take any action to reduce the noise generated by the solenoids.
• Running neural network inference and OMR on the HUAWEI TaiShan 2280 ARM server is much faster than on Raspberry Pi 3. The bigger the size of the sheet music picture is, the bigger is the performance gap. As an example, I have added here a sheet music picture of the song “Twinkle Twinkle Little Star” with an image resolution of 2480*3352 and a file size 115KiB:
  • It takes approximately 30 seconds to run the above step 1 and step 2 on HUAWEI TaiShan 2280.
  • It takes approximately 8 minutes or longer to run the above step 1 and step 2 on Raspberry Pi 3.

Results

Below you can find a couple of pictures that show the sheet music images I used and the YouTube links with the respective final AI piano musical outputs.

Twinkle Twinkle Little Star

My Heart Will Go On

Summer

Kiss the Rain:

Overworld – Super Mario Brothers

Have a lot of fun every day – with SUSE Linux Enterprise!