It is increasingly likely that one or more of your instrumental pupils has access to a virtual reality headset at home, the most common being the Meta Quest. Though these headsets are most commonly used for gaming, several apps have recently appeared that claim to enable piano and keyboard learning, notably ‘PianoVision’ on the Meta Quest and ‘Piano: Flowing Tiles’ on the Apple Vision Pro headset.
What makes these different from YouTube cascade notation and apps like Simply Piano is that, instead of the notes being on a separate screen, notation is mapped over the notes of the user's actual piano in the form of coloured blocks. Not unlike Guitar Hero, but on a real instrument. This superimposition of virtual elements onto real-life objects is known as augmented reality, a technology that is already used to great effect in areas such as medical surgery and the military.
When trying this out with young people, I found that they could use this system to play the melodies of songs they already know instantly and in real time, leading to ‘wow’ moments and the engagement of pupils who were struggling to engage in other areas of music-making. But I had questions: What about technique? What about learning to read stave notation? And what about when the pupil took their headset off?
Designing the project
Previous research suggests that learning piano with a real-life teacher cannot (yet) be matched by any app or artificial intelligence-based technology. A teacher's rapport with their pupil and ability to be responsive and guide the learning journey is irreplaceable, and human-to-human teaching is an art and science that I very much believe in. So, the question was not ‘Can these apps replace teachers?’ but, instead, ‘How can teachers use a technology that is already in their pupils’ homes to increase engagement, progression and musicality?’.
This question is of increasing importance to music hubs, and we brought together a partnership of four hubs – Wiltshire Music Connect, Leicestershire, Lincolnshire and Norfolk – alongside Dr George Waddell from the Royal College of Music to carry out a piece of research that would allow us to develop evidence-based guidance for the sector.
After preliminary trials with various apps, we decided to use Meta Quest 2 headsets running the PianoVision app. This app offers both the ‘cascade’ style colour block notation as well as stave notation. It tracks the accuracy of pitch and timing and gives the user a score at the end of each piece. It was also possible to ‘cast’ the pupils' view onto a tablet or laptop, allowing the teacher to see exactly what the pupil is playing, and even whether they are looking at the stave or cascade notation.
The PianoVision app, as seen through the Meta Quest headset
Over ten weeks, six piano tutors from across the four hubs worked with 24 pupils. Of these pupils, 12 used VR headsets in their lessons and were given a headset to practise with at home. The remaining 12 had traditional piano lessons. All pupils were aged 13–16 and beginners on the piano, though some played other instruments.
We gathered data in the form of teacher focus-groups, teacher diaries, and pupil and parent questionnaires. Crucially, we focused on the teachers' perception of the technology and their pupils' responses, rather than attempting to quantify pupil engagement and progress: the teachers on the project all had at least five years' experience and would be able to compare the progress of pupils with all others that they had taught. Teachers are also the gatekeepers to their pupils' learning journey, and we wanted to find out why different teachers might react in different ways to the same technology, and how it might be further refined to work effectively in a teaching as well as learning context.
Using the technology
We wanted to create as ‘real world’ a scenario as possible, and gave teachers the freedom to use the headsets however they wanted in lessons. Some of them began by using it exclusively in their lessons, eschewing stave notation and focusing on progress within the app. Others taught a piece using stave notation, encouraging the headsets to be used as a practice tool. Some used it to ‘bookend’ the lessons, giving pupils homework on the headset and then asking them to show their progress at the next lesson.
Pupils taking part in the study were given a headset to practise with at home
Over time, all tutors agreed that it was much easier to teach basic technique – hand and finger position, posture, fingering and so on – with the headset off. By the end of the study, all tutors had also moved to an approach that had both VR and non-VR elements in each lesson.
Incidentally, one teacher also used the technology with a more advanced pupil outside the study, finding this produced much better results. ‘After ten minutes of using it, he was, by far, ahead of any of the pupils in the study’, the teacher said. ‘He knows what he's doing with his hands. He knows the keys. It's kind of just a new way of interacting with the music.’
Engagement and gamification
The strongest positive results were around engagement. At a time when attention spans in young people seem to be getting shorter, pupils responded well to the ‘instant success’ of playing both beginner pieces and more complex, well-known pieces with ease. Teachers felt that this would be particularly useful for pupils who may struggle with stave notation, including those with specific learning difficulties. One teacher felt that a pupil ‘without [the AR] wouldn't be coming to lessons’.
The gamification element of the technology, with pupils given a score at the end of each song, was another factor that increased engagement, which teachers felt ‘pushed pupils to improve’.
Cascade notation
Though stave notation was present in the app, it appeared that all pupils were focusing solely on the cascade notation. This sidestepping of notation was seen as a positive by some tutors, who contended that learning to sight-read is no longer necessary to enjoy and progress with playing in many styles.
Others saw this as a fundamental flaw of the technology, believing that it inhibited ‘longevity of learning’, and that notation allows you to ‘take that away and learn it across so many more things rather than being able to play one showy hard piece straight away to show off to your mates’.
Judging progress
Initially, the pupils with the headsets progressed quicker at repertoire acquisition. However, teachers found that set-up time and technical glitches were eating into teaching time, giving the non-headset cohort more focused teaching time. Also, as the pieces became more complex, teachers needed to take more time to teach off the headsets.
By the end of the project, there was no general increase in progression for pupils using the headsets. Two pupils were the exception; they were able to progress onto much more complex pieces using the headset. Their tutors encouraged them to stick with the cascade system, rather than ensuring that their knowledge of stave notation kept up with the repertoire they were playing in the headset.
Feedback and rapport
Teachers anticipated, before the study began, that the headsets would decrease the development of teacher-pupil rapport and inhibit the feedback process. This was true to some extent, as teachers stated that it was easier to conceptualise the structure of pieces, correct mistakes, and discuss musical elements such as dynamics and phrasing with the headset off. However, as the focus of the technology shifts more towards augmented rather than virtual reality, this is likely to become less of an issue.
What we learnt
Overall, the evidence suggested that using augmented reality as a teaching and practice tool enhanced some elements of teaching, but was unhelpful – and even hindering – in others. Teachers also felt that the technology was not refined enough to offer an easeful teaching experience. As one teacher put it: ‘I think it's got a load of untapped potential. I think it's just teething problems. I think it could be really good as a self-learning tool. If I'm meant to be teaching them with it, it needs to have more sort of detail to it.’
They suggested many improvements, including shared control of the app between teachers and pupils; a clearer relationship between the stave and cascade notation systems that would encourage pupils to develop their notation skills over time; and backing-tracks to support playing.
These are all great ideas, and my big takeaway from the research is the vital importance of teachers being part of the research and design of any new technology that aims to support the musical growth of our young people. Augmented reality, and our individual and collective roles as pedagogues, will be strengthened if we are part of the innovation process, and those relationships with industry need to be brokered by music hubs and wider sector organisations.
- Watch a video about the project on the RCM website
