Designing for hands, gesture and VR

Matt Corrall at UXDX Community: VR UX, Customer Success and Scaling Design Systems. Video: https://youtu.be/iw6Wq8-PyyM

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Designing products and interfaces controlled by gesture

[00:00:00] All right, let's dive into it. I could start introducing myself and what I do at Ultraleap, but frankly it's a lot easier to just show stuff and then it makes a lot more sense, so we'll get to it through the course of the next 25 minutes or so.

[00:00:15] Today what I'd like to do is give all of you an introduction to designing products and interfaces a bit like this. Everything you see here is controlled by gesture. Technology today allows devices to recognize your hands and your motions and your body and respond to you in a way that's quite natural and intuitive.

[00:00:39] As well as being controlled by gesture, the interfaces you're seeing are also 3D. They're what we call spatial, so they don't exist just on a flat screen, they exist all around you, perhaps even on your body. As an interaction designer, as a UX person, I find that fascinating, because in many ways these kinds of interfaces are starting to tear up the rulebook on what we know about good digital products and UX design today. It's not just about screens, it's about going into a third dimension. So for product teams that might be working on this kind of stuff in the future, there's this opportunity to extend beyond the screen and define this whole new world of interactions.

[00:01:23] I mentioned that gesture is quite natural and instinctive, and that's because you already know how to use your body. You naturally wave at friends and you point at things that interest you without thinking. Your hands are controllers you already know how to use, and you can do quite a lot with them. I believe that gestural interfaces are going to become an additional modality to the ones we already have as product teams, so voice and touch and that kind of thing.

[00:01:52] Alongside screens in the next few years I think you're going to see an increasing flexibility and variety in your choices of ways to interact. I'm talking about VR and AR headsets, voice, touchless displays, which I'll get to in a minute. The digital world that we are gradually building with these products is going to be opening up, and you're going to be able to experience it on different devices depending on the situation. Which is why I think knowing how to design for screens is not going to be enough in the years to come.

Why listen to me, and what Ultraleap does

[00:02:25] So why listen to me on that subject? Well, I've been doing design quite a while. I've been in this game about 20 years, and I've spent the last three focused purely on gestural and spatial interfaces, which is what I'm going to talk about. My background is product or industrial design, so the physical stuff, and over the last 20 years I've gradually moved more into digital products and service design. Today I spend my time working on emerging forms of interaction design, so that usually means virtual reality and gesture.

[00:02:59] Today I work for a company called Ultraleap. I'm the design director there, and I lead up a team of interaction designers and user researchers. Ultraleap is a tech firm that creates hand tracking technology and haptics technology. It's the creator of the Leap Motion camera, which might be a name that's familiar to some of you.

[00:03:21] The tech essentially tracks your hands in 3D space. I wave around a lot when I do these talks, you can see me gesturing a lot. We also have the ability, and I'll get to this a little later, to deliver haptic sensations to those hands as they gesture in the air. That tech is baked into a whole range of devices, from VR and AR headsets to touch screens, even some cars, and in all of these cases it translates the gestures you make with your hands into inputs for the interface.

[00:03:52] The result of that is digital devices that know what shape your hands are making, their positions in 3D space, the way they're moving, the speed of movement and that kind of thing. So you can wave and point and swipe and do all these things in the air and it can be recognized by your devices.

What you can do with hand tracking

[00:04:10] The question is, if you're a UX person or you're a product owner or you work in a team, what do I do with this stuff? What do I do with that toolkit? Well, if you're in VR and AR it means you don't necessarily need controllers all the time any more. For new or infrequent users, and in VR that still often means the majority of people, it takes away some of the barriers and that first awkwardness for those first user experiences. You don't have to fumble around to find the controllers, you don't have to remember what all the buttons do necessarily. If the experiences are well designed it means people can be having fun or getting work done a bit more quickly than before.

[00:04:52] It also enables you to better train for some real world scenarios. VR training is something we're finding a lot of our customers are doing at the minute. We're seeing it used for use cases like training airline cabin crews, and this example here is from Lufthansa, this is using our tech. We're seeing it used for hotel staff, even specialist things like training pilots and surgeons. The reasons are it's cheaper, it's more flexible, it's a bit more convenient than it might be to set up some of these training scenarios in real life. But it also means people can practice these procedures in a manner that's a bit more natural, a bit closer to the real thing.

[00:05:32] The same tech can be used to create different kinds of interfaces as well. Here you're seeing our same tech used to turn a touch screen into a touchless device. We can control it from a distance with gestures. The area in front of the screen is now technically part of the interface, it's part of a spatial interface now, where your hand's movements are controlling a cursor on the screen.

[00:05:55] For the designers on the call, and for your designers and developers, that means rollover states on touch screens are now possible. The UI can start to come to life as you navigate around. But also, because you're using a hand rather than just a simple pointer, it means you can do more complex things. You can pick up 3D objects and rotate them around, et cetera, from a distance. I think this can hugely enrich screen experiences, because it's starting to add layers of responsiveness and 3D interactions to a product that we already know and use every day.

Writing the rulebook: how do I push a button with a gesture?

[00:06:31] Now, I'm going to wager that most people on this call are quite used to designing their products for screens, and are used to working in this mature ecosystem that we've got. We know the UI components, we have accessibility standards and touchscreen gestures, and all stuff that's quite well understood. The tricky thing with these interfaces I'm showing is that it's all quite new. It hasn't really had the time and attention from product teams to get to that level of maturity, and that essentially is what my job is.

[00:06:59] My team and I at Ultraleap are doing the research to try and help write the rulebook for these interfaces. The ultimate goal is to deliver a toolkit or an SDK that would enable all of you really to build these interfaces in a way that's nice and easy. There's loads of stuff to do on that, but we do already know some of the components and the gestures and the design principles, and so today I'd like to give you a bit of a crash course in what we've learned over the last few years.

[00:07:29] I will start quite simply, and that's how do I push a button with a gesture? On the right you're seeing an example from VR, and on the left an example of one of our touchless displays. On the right we've got these virtual digital hands which map to our own, and so we can push these buttons in VR directly. There's loads going on behind the scenes, but actually that's quite straightforward from a customer point of view, it reacts a bit like real life.

[00:07:57] With the touchless displays it's a little trickier in first use, because the button is on screen but our user is standing a short distance back. My colleague here is a few centimetres away, but you could be as far as a metre away. So if you're not going to touch the button on the glass, what do you do? Are you going to move your hand forward like this, or are you going to tap down as if you're pressing on a mouse?

[00:08:19] In our research what we're trying to do is find the best interactions and UI and the way to pull off these things. What we find with the touchless display is that we settle on a kind of tap gesture, a short forward motion in the air. It's about looking for changes in direction and sudden changes in movement speed that enable us to see user intent.

Scrolling, swiping and the natural variance of gestures

[00:08:45] That same logic can then be extended to scrolling and swiping. So this time VR on the left, touchless display on the right. Again in VR the hands make contact with our UI panel there and that's quite straightforward. We just need to see that the hand's in the same space as the carousel, give it a swipe, look at direction, and we can initiate movement. With a touchless display, again we have to interpret it from a distance, and one of the tricky things with swiping in the air is that because I'm not touching the glass, swiping back and forth looks the same in both directions. The question is which direction is my user trying to go? Again it comes down to a lot of iterations and looking for tiny changes in direction and movement speed to enable us to interpret that correctly.

[00:09:33] Across the population, gestures have this natural variance. We all have different bodies, with different heights, different arm lengths, we have different mannerisms, different things we can do and find comfortable to do. So when we go out and do our user research, or we test these prototypes, we observe this variance across people. They're always trying to do different things even when they use the same interface and have the same tutorials.

[00:10:00] For example, the stuff on screen here are just some of the hand shapes we see when we ask people to try a touchless display and they try tapping buttons. It makes sense when you think about it, because our hands are our analogue controllers. They're all a bit different, they're not mass manufactured items like VR controllers. So a gestural interface has to be a little bit generous. It needs to work with that variance, because you can't expect all your users to be the same and do the same thing. The lesson here is that a gesture is not a precise fixed action that I have to learn, but it encompasses a small range of hand shapes and movements that all signify the same intent.

Grab select and virtual objects

[00:10:43] So far everything I've shown you in VR has been direct selection. It's been virtual hands making contact with UI panels and buttons, and that makes a lot of sense with hands. But there is another way to do it. This is our grab select gesture. It's a way to interact with objects from a distance in VR. This time it needs a cursor or a ray to be projected from your head, you need to see what you're pointing at, and it needs a new gesture. You need a way to select and deselect objects, so turn them on and off or whatever. For that we find a pinch or a grab gesture works very well. That's either putting the thumb and forefinger together, or all fingers in that clutching motion.

[00:11:23] If anyone's used hand tracking with VR you've likely done this. This is becoming something of a standard, it's very common these days. The interesting thing from a design point of view is that what we're doing here is very similar to what we just saw someone do on a touchless display, but here we're using a grab gesture. Back with the touchless display it was a tap gesture, it makes more sense in that scenario. So gestures are not universal. It's not like there's one gesture that means the same thing everywhere. It depends on the scenario and it depends on the device.

[00:11:55] Once you've got that grab gesture there's loads more you can do with it, because in spatial interfaces we can have these virtual objects, which are a whole new class of UI component. They're a lot like their real world counterparts. They react with gravity and physics. You can grab them, drop them, throw them, push them across the table, knock them over, and it's not about using precise hand poses to do it, you can just freely knock them about. In the airline training platform we saw earlier, using virtual objects is a very big part of the experience, it adds a lot to the realism.

[00:12:30] But because gestures have to be flexible, we need to cater for a lot of different ways to interact. So here you're looking at the physics system we've built, which enables you to grab these objects with any fingers and in any orientation you want, and it tells them to react in a manner that's realistic.

[00:12:47] But how people reach out to pick up objects is actually determined a lot by their design. If you look at the images on the right here, you can see that people pick up objects differently depending on the size. A small virtual object will often be picked up between thumb and forefinger, that's the green one. Slightly bigger objects get the whole hand, that's the yellow one. And then the pink object, the biggest, will usually be picked up with two hands. That's something we all do quite instinctively.

[00:13:13] These virtual objects also need a way to signify they're interactive. In these environments not every object is necessarily something you can interact with, and you need to know which ones you can grab and which ones you can't. So they need a signifier, something that tells me this is something I can interact with. In this example on the left we have a little text prompt that pops up as your hand gets in proximity, but it can also be things like the object lighting up, or snapping towards your hand as if pulled by a little magnet, things like that.

[00:13:42] The final bit of the puzzle is that these objects can communicate a lot about how you use them through their form and appearance, which is affordances. It's a common term, I'm sure a lot of you know it, which we use in 2D UI design. In 3D, if you think about an object like a bowling ball, it has these three holes where your fingers go, and those holes are how you pick it up. Or if you go to pull open a drawer in your kitchen, you know how to grab that handle. I'm an ex-industrial designer, and actually this makes me quite excited, because as our interfaces start to go 3D and have items like virtual objects, industrial or product design principles start to apply to them. So things I learned as an industrial designer can actually help to make a lot of these objects easier to read and easier to use.

Making the interface physical

[00:14:32] Now I'm talking a lot about 3D physical interfaces, and that really is the best way to do things when you're designing for hands. When you think about that button that I showed earlier, it makes sense that when you press with your finger you would expect the button to react as it would in real life. It should depress a bit like a mechanical button. When we started doing our research we had flat UI panels, as we would have on 2D screens, and over time they've become more three-dimensional. So now we raise our buttons off the panel surfaces, and have them react to your presses by clicking downwards.

[00:15:11] If you look below the UI panel on the left there, there's a little tiny ball shaped object, and the keyboard on the right has a few of them around. These are components we introduced to reposition and resize and angle our panels and our keyboards. They're tiny virtual objects, they're little grabbable handles that you connect with a little pinch like that. When you move from controllers to hands, you expect to be able to do things a lot more like real life. So going 3D, going a bit physical like this, actually makes the whole interface make a lot more sense for people. It gets more intuitive that way.

Wearable UI

[00:15:50] Another problem we have when we're designing VR experiences in particular is that people are moving around these virtual environments now, which presents a new problem for us, because people need access to menus and UI panels but they're effectively on the move. When you use VR controllers there's usually a dedicated button that brings up, say, the menu panel, but we have to figure out how do we offer an equivalent for hands? There's no buttons on here.

[00:16:17] Wearable UI is one of our solutions for that. We effectively anchor small UI components to the body, so we ensure you can carry them around with you, and they work especially well when they're anchored to the hands and the wrists and the forearms. But if they were always there and always present, that would be really annoying. They'd distract us, they'd get in the way when we're trying to do other things. So we also need a way to toggle them on and off and have them only appear when they're needed.

[00:16:45] I brought along a couple of examples of how we do this. In the example on the left I turn my wrist, I actually turn it towards my face, and it spawns that little UI panel off my wrist. That's really good for frequent use features. In the case of this, which is an art application, it contains my undo button and my brush size and things like that. The example on the right is a game where you're getting virtual plants and growing a little garden around a virtual space, and in this case we have a menu off the hand but it's activated by spreading the fingers. We have an item mapped to each finger, which you pluck off the finger with a pinch and drop into the environment.

Feedback, and mid-air haptics

[00:17:28] Unlike touchscreens, spatial interfaces can't give you tactile feedback when you interact with them. Think about picking up an object in the real world: you feel it against your fingers, you know it's there. When we design for gesture we get a lot more freedom, but we lose something in confidence and certainty. So to try and give users back that confidence, we've learned to exaggerate the responses of our UI a bit more than you might do if you were designing a web product.

[00:17:56] In the example on the left, as my hand approaches the keyboard the buttons light up. This is our signifier to say I'm interactive, you can push me. We then carry that feedback on throughout the whole interaction, so as I begin pressing a button down. In the example on the right, the touchless display, you can see the cursor has a little fill animation that fills up and down as the hand moves backwards and forwards. In both of these formats our buttons depress with these big clear animations, so we're giving you that feedback from the system to say it's working, the system sees you're pressing. And then of course when the button's fully down we play a nice audible click sound, we get a good visual state change. Particularly when people are learning, we're seeing that this kind of clear feedback is giving them that certainty. It's making up for a loss of tactility, and little details like this make a big difference.

[00:18:51] There are other ways to get that feedback as well, which is why at this point I'm going to introduce Ultraleap's other technology, which is mid-air haptics. I won't go into exactly how it works right now, but I'm happy to take questions if you want to know. Essentially this tracks your hands and can deliver haptic sensations to your fingers or to your palm. So there's another tool we have in our toolkit when we're creating these interfaces, but again the question is, what can I do with that?

[00:19:18] I brought along a prototype of an in-car interface that we're developing at Ultraleap right now. Here you can see our user, our driver, navigates around the interface with their hand, and we deliver a little haptic sensation to the palm as they move between button areas. It gives them more confidence. If they choose to push a button with a tap gesture in the air, we deliver a haptic sensation, a slightly more intense one, to the fingertip, to let them know the button's pressed.

[00:19:48] And you can even go a step further. Here you're seeing somebody forming a pinch and using these slider controls to change the volume of music or something like that. We use haptics there too, so as they move through the sections of the slider we raise and lower the intensity of haptic sensation as they adjust the level in the car.

Hand poses

[00:20:13] Now I said earlier that gestures aren't these precise shapes we've mastered, but wouldn't you know it, there are exceptions. Hand poses are when we do form a set shape with the hand, and the system recognizes the shape you hold up and triggers some action. We often use it for shortcuts or menus, or special features like this example on the left, where I can use a pinch and I can use it to draw in 3D space, which is a really nice fun interaction to test with.

[00:20:40] Examples of poses would be forming a pinch, or a thumbs up or down, or holding up different numbers of fingers. The tricky thing with these is that they're difficult to learn. You have to remember them, you've got to hold them in working memory while you're busy doing other things. It's a bit like trying to remember all the keyboard shortcuts you've got.

[00:21:00] Because they're difficult to retain, we usually recommend you use them quite sparingly, particularly for experiences where people are just dipping in and out, with first time users. But they come into their own with repeat uses. When someone's a regular user they can gradually build up that experience, they can retain them in working memory, and they can call on them when they need them.

Teaching people how these interfaces work

[00:21:22] One of the big challenges with designing this stuff right now is we need to help people understand what these interfaces are and how to use them. Remember, most people trying this stuff out, be it in VR or on a touchless display, it's going to be their first time. In VR we understand we're in a 3D space, because after all we just put a headset on, but we've also got these digital hands that react quite well, so we feel quite embodied. But there are still VR-specific concepts, like that wearable menu I showed you, that need to be taught through in-world tutorials and props. You can't have everything being instinctive, because it's not exactly like real life.

[00:21:59] The touchless displays have got an extra challenge, because people will walk up to these screens and not necessarily know it's anything different to a touch screen. It kind of looks the same. So when we're touchless we need to grab people's attention and we need to teach them very quickly. Whilst we use text props, like in the example in the very bottom left there, for hints and tips and things, they only go so far.

[00:22:23] When you're designing tutorials here, text and icons can't really be relied on very much. There's not really much prior knowledge for people to fall back on, so if you were to say "make a pinch", people wouldn't really know. You get all sorts of different responses. So we find the golden rule with gesture is to show, not tell. You can see some of our tutorial formats here that we've had more success with. A little robot character at the top shows you what to do. Videos and animations that you learn from. Or the example in the bottom right, where we teach you poses by getting you to hold your hands in guidelines in the interface. When we show people stuff we find it's much clearer, and we see more effective learning.

Ergonomics, accessibility and mixed reality

[00:23:08] You've seen by now that spatial interfaces can be quite different. You can have controls all around you potentially, you can have interactions that are a lot more physical than with a screen interface, and that means that human factors, or ergonomics, becomes a more important consideration when you're moving away from screens. Your team, if they were working with this, might be placing components in 3D space or choosing gestures, and they would have to start thinking about frequency of use and the effort and the comfort required for them.

[00:23:39] So that example with the wearable menu on my wrist: if I turn my wrist, that's great occasionally, but I really don't want to be doing that all day. If I use a touchless display and I reach up to tap a button high, that's fine momentarily, but you don't want me to be holding my arm up for long, because I'm going to get tired.

[00:23:56] There aren't really agreed accessibility or usability guidelines for this stuff yet, though we're doing our best to contribute to them and we're starting to see how important they're becoming. For example, in the next year part of our work will be trying to make sure everything I've shown you gets as inclusive as it can be, which will mean offering alternatives for things like two-handed interactions and hand poses for users that can't do that.

[00:24:20] The other thing we're going to be doing over this year is looking into mixed reality. We want to see how a lot of the VR stuff I've shown you today might need to adapt when we move into mixed reality. I'm going to show you another quick and dirty prototype. Here you're seeing a real world object, this cardboard box in our office, with a virtual button on top. You can see the green button there, so they're mixed together.

[00:24:43] As our user presses the button, our little friendly owl pops out, and our user will now use a hand pose with their left hand to call the owl over to sit on their finger. Then they're going to use their right hand to control this 3D spatial cursor, so they're guiding the owl around the room with that, and the depth of the cursor is controlled by these really subtle back and forth movements with the hands. This is really fun. This is the result of a design sprint the product team did a few weeks ago. It's just a first step, but it gives you a taste of what you might expect from Ultraleap in the year ahead.

[00:25:17] I've said there's a lot we still have to figure out as a team, but we already know a thing or two, and everything I've talked to you about today is available in our public design guidelines. This is all published on Ultraleap's site, it's available to everyone here right now. It's docs.ultraleap.com, and that covers everything I've talked about plus more detail than I could go into. I think we're about at the time, so I'm going to move on to the end. I'm going to wrap up for questions, and I'll leave this link and a few others for anyone that wants to know a little bit more. So yeah, thank you very much for listening, looking forward to a few questions.

Q&A

[00:25:58] Host: Thank you very much. It's incredible, some of the things. I'm just not in the AR and VR space, so there's a lot of things there that I learned of, some of the challenges and the things that you have to overcome.

[00:26:14] Host: Just a reminder to everybody, there is the chat box on UXDX or whichever platform you're watching this on, and if you put in your questions there I'll put them through to Matt. One thing, and again I'll say this as an outsider to the industry, but from my naive perspective I've always leaned more towards AR versus VR, because I've tried on the headsets and I've bumped into things and I've hit things. Is that a trend you see moving that way, as you brought up at the end there, or do you still see that there are very good use cases for both technologies?

[00:26:53] Matt: I think there's good use cases for both, to tell you the truth. It's quite a different experience really. AR is, let's see if I pull up some of the work examples, we've seen AR situations where somebody might be out in an industrial facility trying to do a repair job and they can overlay the information they might need on which part goes where. Or they could have an expert on the phone who could be telling them where to go to do a job, and actually giving them arrows in their heads-up display where to go, things like that.

[00:27:27] Matt: VR for us, like the training stuff I showed you earlier, that's about immersing you in an environment that you don't have available to you. Let's take, I mean surgery is a tough one, it's a high risk use case, but there's a lot of work going on in it on the AR side of that. There's augmenting what you get, so a surgeon's hands are busy but they might want access to your MRI scans or your medical notes, and they can have those overlaid above the table, so they can interact with that information and get that benefit without needing to put the tools down and press the buttons.

[00:28:06] Matt: But similarly, if you take surgical training in VR, that's about putting you in a hypothetical situation where you're dealing with whatever this person needs, and it's a lot easier than setting that up with, however they do it, dummies or actors or whatever it may be. It also means they can repeat that many times over and practice it and try different approaches quite easily. I'm just plucking one example, but I do think there's a call for both. We're seeing a lot of headsets now that can do both quite easily, and I think in both of those hand tracking makes a lot of sense whatever you're doing. So yeah, I don't think either one's going away.

[00:28:47] Host: Brilliant. And there's a brilliant question coming from Leanne. Have you found differences in interactions between different countries? Because I think a thumbs up is actually a negative thing to say in some cultures, in some countries.

[00:29:05] Matt: Yeah, and that one doesn't always go down very well either. It's absolutely right, it's a great question, and I don't have a comprehensive answer for you yet, because it's just something we're really starting to move into. One thing we've had to deal with is just natural gesturing in conversation, I do this a lot. Particularly when you're dealing with VR or a touchless display, you want to know that these movements of the hands mean something to the system. So am I trying to open my menu or press a button, or am I just talking to someone else in VR and gesturing a lot? Some cultures do that a lot more than others, and do different things.

[00:29:52] Matt: We know it's a thing. We've had some testing and research where we've had some of this and we've started to be able to design around it. I won't pretend that we've had the chance to optimize this for every culture, or that we really know all the differences, but our user researchers have started going out and testing and talking with customers in different regions. So we're starting, and it's something that I absolutely want us to get better informed at. So yeah, thanks for bringing it up, that's going to be one of the next steps as well.

[00:30:23] Host: Brilliant. Well, that brings us to time, so thank you very much. I really enjoyed that talk and I hope everybody out there did as well.

[00:30:29] Matt: You're welcome, thanks for having me.