The proof-of-concept is a very basic iPad app (video below). Nothing special, just playing around. Graphics are all made with Inkscape (but if you don't like them, believe me, it is not Inkscape to blame).
[the piano sound around 0:50 is quite loud... you're warned]
So here it is:
