I recently tried to explain to a non-musician friend what exactly a conductor does. Try it – it’s harder than you think: most musical ensembles, at least after the first rehearsal, don’t need a person keeping time. Any choir master will tell you that trying to act as the arbiter of rehearsal discipline is usually an uphill battle! So, what does a conductor do? I have been fortunate in my life to work with a handful of great conductors; those musicians whose very presence makes everyone in the room play and sing better. If I break down what it is they do, they manage to communicate and negotiate the nuances of interpretation through gesture, through body language, even through facial expressions.
A recent project at Goldsmiths, University of London, has attempted to capture the magic of conducting via motion capture technology. Utilising a Leap Motion Controller, the participants attempted to distil the essence of the conductor’s art by mapping the movements of various participants. Ultimately, the results have to deal with the all-too-familiar problem of mapping empirical data on to the highly subjective landscape of music performance, but the experiment does raise a number of interesting potential applications.
The Leap Motion Controller, for anyone unfamiliar, is a relatively inexpensive motion sensor which connects to your computer via USB. The device tracks the movement of your hands and fingers in real time, using infrared cameras and LEDs to create a 3D model of your hands. Typically, this used to control software and games, but at 200 frames per second, the level of detail captured lends itself to a range of other applications.
Options on the market
Air guitar is an app from engadget which allows the users to do exactly what the name suggests. Transforming motions in the air into processed guitar sounds, either via a dedicated motion-sensitive plectrum, or converting the Leap Motion sensors into a virtual fretboard, the software does away with the need for a cumbersome ‘real-world’ instrument. While catering to everything from the casual gaming market to serious players, Air Guitar is capable of recognising a range of different strumming patterns and chord shapes, making virtual playing a serious possibility.
GECO takes the premise one step further, creating an interface which can control a wide range of software instruments. Tracking your hands, fingers, and even facial expressions, GECO from UWYN creates a holistic performance experience. Much of the effort has gone into tracking and recreating string players’ bowing motions, aiming to recreate the effect of a violinist carefully angling, changing pressure and weight and the myriad other nuances of right-hand technique. Used in isolation, GECO has proven a powerful tool for isolating that all-too tricky aspect of technique.
Motion capture is not the only technology to revolutionise practice. SmartMusic is an interactive music practice software from MakeMusic that provides sheet-music and accompaniments for various instruments. It listens to your performance and gives real-time feedback, helping you improve your timing, pitch, and technique. Incorporating a Natural Language Processing artificial intelligence (AI) technology, SmartMusic provides interactive and personalised music practice experiences for the young musicians. The AI aspect is crucial here: I will confess I went into this one expecting my performance to be mapped against a generic MIDI template, and for every nuance of pitch and rhythm to be flagged up as a ‘mistake’. And on the first few passes, they were. The more time I spent with the app, however, the more SmartMusic began learning the nuances of my playing style, and adapting its accompaniments to me.
SmartMusic also offers lessons in jazz and sight-reading/singing. The latter is incredibly useful – no matter how many times I tell students to keep going, keep the rhythm, not to self-correct, the temptation to deviate from their internal pulse in the name of perfection is just too great. Something about the almost videogame setup of SmartMusic embeds the need to keep playing in a way that I am just not able to.
SmartMusic from MakeMusic
Interactive options
Taking this process a set further, Yousician offers interactive lessons for guitar, piano, bass, ukulele, and singing. It listens to your playing and provides instant feedback, making practice more engaging and effective. Yousician invites you to engage in a play-along mode where you follow the on-screen instructions to play the notes, chords or melodies on your instrument, or voice. The app then uses audio analysis algorithms to detect and interpret the notes and rhythms you play. Yousician provides real-time feedback on various aspects of your performance. This can include: pitch accuracy, rhythm and timing, technique and dynamics, chord progressions and melodies. The app also incorporates practice tips specific to individual problem sections, providing a holistic practice support mechanism.
Yousician’s app where you can play along and earn rewards as you learn new skills
ABRSM has recently entered this field with their piano Sight-Reading Trainer, available for iOS and Android devices. The app, released in June, contains 155 specially composed pieces of sight-reading for piano Grades 1 to 5. Each piece is preceded by a series of games designed to improve your ability to spot the features, patterns and characteristics of the music before you play it.
The app provides an onscreen score, which scrolls in real-time. It also allows a customisable playback tempo, depending on how brave the student is feeling! In addition, the app includes a number of features to help students track their progress and improve their sight-reading skills over time. These include:
- Score tracking: The app keeps track of your scores for each piece, so you can see how you are improving over time.
- Practice streak: The app keeps track of your consecutive days of practice, so you can stay motivated to keep practising.
- Challenges: The app offers a variety of challenges to help you improve your sight-reading skills. These include Rhythm Match (Hear a rhythm and identify the corresponding notation); Pitch Patterns (Identify the pitch of a series of notes); Quick Fire Five (Answer five questions about a piece of music in a limited amount of time); and What’s the Difference (Identify the differences between two pieces of music).
Clearly, the app is aimed at covering more than just the sight-reading portion of ABRSM’s exams. The initial experience was solid, if affected by a slight latency at times. Given the broad skill base of this app, supporting sight-reading from a holistic perspective, I have no doubt that ABRSM will incorporate features available in other aural training apps in future.
EarMaster and ToneGym both offer the facility to sing pitches, and offer feedback on intonation and tuning. For the more adventurous, goodEar Melodies from iMobilTec provides training and feedback for whole melodic phrases, developing the skills for transcription and improvisation.
Conclusion
Practice is a highly subjective field. The idea of tracking motion, and offering realtime feedback on technique, posture and thus musical result is something of a musical Eldorado. Movement is ultimately highly individual; it largely depends on the musician’s physicality, range of motion, relationship to their instrument, and ultimately connects to their musicality and personality. Being overly didactic about aspects of technique can be destructive to anyone – if not all of these factors.
But neither is the integration of automated feedback something to be feared in musical education. Intelligent apps can help track progress and, more importantly, can help students to stay motivated. Automated feedback will never be a substitute for human interaction, but these tools are still in their early stages. They have the potential to save as much of my time as my students’, and, if that is the case, I hope that these tools become part of the future of music practice.
gb.abrsm.org/en/exam-support/apps-and-practice-tools/sight-reading-trainer
earmaster.comhttp://tonegym.com
apps.apple.com/gb/app/goodear-melodies-ear-training/id588125376
