Designing Social Robots

Minja Axelsson at UXDX Community: Helsinki. Video: https://www.youtube.com/watch?v=ooq54TSLQ3w

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Futurice and the InMoov robot

[00:00:09] I'm Minja, I'm a robotics designer from Futurice. Just very briefly about Futurice: probably everyone here has heard about Futurice, hopefully. No? Okay. We're a digital innovation consultancy. We have offices all around Europe, and we employ designers, business advisers and developers. We have a few other interesting design projects: we have the IA design kit, which is an intelligence augmentation kit, and also an IoT service kit. If you're interested in those things, I recommend you check those out.

[00:00:47] Then to designing social robots. This is about hypothesis-driven development, and I'm going to tell you about how a robot helps children with autism learn sign language. This design case was adapting an InMoov robot to teach children with autism. We did this at Futurice in collaboration with Satakunta [?] healthcare district, and Prizztech [?]; they were funding the project.

[00:01:23] Are you familiar with social robots, social robotics? A little bit. The working definition of social robotics that I like to use is that social robots are robots that have the primary task of interacting with people. For example, a robot in a factory that never interacts with a person is not a social robot. But a Pepper that you might see in a store greeting you, that's already a social robot. Maybe not always a great one, but it is. I also like to think of social robots as having a physical entity. Software robots have different design considerations, so I don't really want to go into that here. If at any point you have any questions, or if I use any terms that are confusing, please just stop me and ask.

[00:02:16] The InMoov robot was created by Gaël. He's a French sculptor, and he has made this open source. It's available online; you can print it at home if you want to. All the software is also available as open source online. It took us around six months to build this robot, so it's not an afternoon project, but you can still do it if you want to. We built an instance of InMoov at Futurice, and we called it Momo, which was unfortunate when this scandal with scary bots and children happened, but that turned out to be fake, luckily.

Who we're designing for

[00:03:03] Are you familiar with what autism spectrum disorder is, or what it entails? Some nods. I'll give you a brief definition, or some of the characteristics that are associated with autism spectrum disorder, so you have a better understanding of who we're designing for here. People with autism generally have problems in social interaction, communication and flexibility. Meeting new people could be very difficult; even meeting people they know could be difficult. Some of them follow a very tight routine, so any lapses in the routine might cause some upset.

[00:03:45] These are thought to stem from a lack of theory of mind, which means knowing that other people also have thoughts and feelings. The current theory is that people with autism have some lack in this understanding. They also have trouble integrating information into a whole, so they might get fixated on a certain detail in their environment and focus on that, and not really focus on what's happening around them, which leads to these social, communication and flexibility problems. However, it's a spectrum, so not every autistic person is the same, not every autistic child is the same. These are general things that you can see among the population.

[00:04:32] Having defined these characteristics of people with autism, we can define what they might need in a robot. Structure, safety and modularity were things that we thought would be very important. Since people with autism have problems with flexibility, having structure is really important. Since we're dealing with children, they might get scared by the robot. They might get scared by the fact that when they're coming to interact with the robot, it's a new situation for them, which is why we need to create an element of safety, and we did this by having the child's parent with them at all times. The robot would also need to be modular, since children with autism have different needs, and the robot may need to adapt to those needs.

[00:05:26] Okay, but why build a robot for autistic children in the first place, and why sign language? This brief came from the Satakunta [?] healthcare district. They had a psychologist and a speech therapist who worked with us, and they thought that this robot could potentially be easier to understand than humans. 50% of autistic people are actually mute in adulthood; they communicate with other means. Sign language is the most used form of assistive communication by people with autism. They've also been found to have a special interest in technology and robotics. Previous therapies with people with autism have found that when there are robotic components involved, they're more engaged. So both of these things logically lead to: maybe a robot could be easier to understand.

[00:06:22] How I approached this problem: this was my master's thesis project, so of course I had to do some literature review. I looked at a lot of previous robots that have been used for children with autism, and we did co-design with the psychologist and the speech therapist.

Ethical guidelines for social robots

[00:06:43] So we have a pretty good purpose for building this robot. Probably we need to take a look at the ethics. Are you familiar with robot ethics? Have you seen some of the news that's been going on? A little bit. I'll give you a few examples while I explain these ethical guidelines that I created for social robots.

[00:07:06] Firstly, of course, we need to think about physical safety. If machines are operating autonomously, they need to take human safety into account, so people shouldn't get pinched or pushed or hurt by machinery. We defined that the child shouldn't touch the robot at all, and we did this by having the speech therapist present with the child when they were interacting with the social robot.

[00:07:37] Safety of data. We just had a presentation on giving data to companies, so building on that, we want to keep the child's data encrypted, especially in fields where robots are used in health care, child care, elder care. These people are vulnerable. They don't always understand what their data is being used for, so it's important to keep it encrypted, and of course not collect data that is not needed.

[00:08:03] Correct behavior enforcement. This is quite specific to robotics. It's been shown that people can transfer learned behaviors from technology to humans. Have you read any news about Alexa? Children have been using Alexa at home, and they actually end up becoming more rude as a result. Because Alexa does not ask a child to say please and thank you, they forgot to say that to their parents or other children as well. Which is why a robot should also enforce these social rules that we have, which we owe to each other. In this situation, since this was a prototype, we did this by having the therapist enforce correct behavior. There was one child who really, really loved the robot. He was laughing throughout the whole experiment, but he also really, really wanted to hug the robot, so he was going at it quite aggressively, and the therapist was there to calm him down: this is not okay.

[00:09:08] We need to have equality across users. Probably you've heard about some of the racist or sexist algorithms that have been implemented, unfortunately. One really terrible example is an algorithm that was used in the US justice system to estimate recidivism rates, so to estimate how likely a person would be to commit a crime again after they've been convicted. The algorithm was racist against black people, because the data that was used to build the algorithm is racist. So when we build algorithms for robots, if they're dealing with this type of personal information, we need to take into account whether the data is biased.

[00:09:57] Another thing, actually specific to robots: male robots are often used in instances where the robot has to be powerful, and female robots are often used in service jobs. So that's something to think about. In this case, we wanted to make the robot gender neutral, because we were having both autistic boys and girls coming over, and with the therapist we wanted them all to feel equally welcome. So we decided no gender for this robot. Robots don't even need gender, so it's a weird question, I think.

[00:10:37] Transparency. This means that people should understand why the robot is doing what it's doing. The person should somehow be able to check if the robot is behaving correctly, so they know how to calibrate their level of trust in the robot. In our case, we told the parents and the children at the end that the robot was teleoperated. Teleoperated means it was remote-controlled by a human. We wanted to do this because we didn't want them to get any false expectations of what technology can do at this time. The therapy of autism is very difficult to begin with. We don't want to give any false hope. Even though technology is evolving, it's not there yet.

[00:11:24] Emotional consideration. This is also quite specific to robotics. It's been shown that people create emotional attachments to robots. Even though people know that they're not alive, we still treat them as if they were, especially if the robots have a name or a backstory. In this case, we wanted the child to know that the therapist was not being replaced. That's why the therapist was also in the room the whole time. And even if this robot were to be developed further from this prototype and taken into use in healthcare, we would still not replace the therapist. The robot would be there to support the therapist's job.

The experiment

[00:12:13] That was our whole hypothetical design that we created, and after this, of course, we wanted to perform some experiments. How were these set up? We had three different design conditions. The robot was speaking and signing; it was speaking, signing and showing a picture of the object that it's signing; and thirdly, it was speaking and signing and flashing some lights. We wanted to see if a combination of communication methods, if using the hands and an image, is better than these other methods.

[00:12:50] We had 10 children come over. They were asked to imitate nine signs. The experiments were from 6 to 28 minutes, which illustrates how much of a spectrum it is. Some children signed all the signs correctly right away; some of them did not sign correctly at all. So it really depends. Present in the experimentation room were the child, their companion, so their parent or other loved one, and a speech therapist. We measured the child's eye gaze and imitation success, so this was quite a scientific approach to measuring how well the robot is doing. We also did a survey with the child and their companion. And like I said earlier, the robot was teleoperated from the next room.

[00:13:38] Here we have a video of the robot in operation. Now it's signing "duck". These were nine signs that are taught to children anyway, so they're part of children's vocabulary. This robot was built quite quickly. We were still 3D printing some parts for it the previous evening in our hotel room. You can see that the screen is flashing. That's because it's pinging a server every few seconds to ask what image it should be showing. So it's not the most beautiful software package, but it's a prototype, and we wanted to illustrate that we can do this type of socially impactful prototyping quite quickly.

[00:14:35] So what was the outcome? Unfortunately I have an external link. I don't know if I can show that or not. Let's hope that there's no sensitive information in here. Sorry about that. Is it coming over? Oh yes, here we are.

[00:15:59] The speech therapist is showing the camera at the back of the room that it was incorrect, and then we're operating the robot accordingly. Now she's showing that it's correct. Now I've lost my cursor. Yes, great.

Results

[00:16:52] The results of the experiment: seven out of ten children imitated the robot. The girl you saw there, Maya, was one of the best ones. She signed everything correctly. Image was the best design condition according to the children's companions. By image I mean that the robot was showing the picture on the tablet at the same time. The children said that it was mainly a positive experience, although sometimes the robot was a little bit scary. It's a human-sized torso. It's bigger than me; it's adult-man-sized. So I think maybe that should be made a little bit cuter for children in the future.

[00:17:35] All of the companions said that they thought the children had a connection with the robot, and most of them said that the robot could be beneficial in the future. Of course this would need more development, and we need to verify whether the children understand the signs, whether they're capable of learning the signs from the robot, because now we only verified that the children copied the signs from the robot. Long-term learning is of course measured differently. Like I said earlier, we wanted to show that rapid prototyping of high tech is possible with the people who really need it the most.

A method for designing social robots

[00:18:09] I'll quickly share the method that I developed during my master's thesis for designing social robots. This is optimized for consultancy environments, or environments where a multidisciplinary team is working to create a prototype quickly. We used this in the autism case, but we have also used this with Oodi [?] to build some social robots for them. It's quite basic design stuff. Firstly, you define the problem space. Based on what the problem is, you make some design guidelines, which you then follow when you define the solution. And I think the solution of a robot has four components: the environment or the context it's operating in; its form, so how it looks; how it sounds and how it interacts; and then how it behaves.

[00:19:03] If you want to try it out, you can achieve these benefits: co-designing with users and experts, putting in the ethical considerations which we discussed earlier, and having a structured process with explicit design decisions. There are some canvases online which you can take a look at. There are eight in total. You don't have to do them all; you can take a few, or just take one, or do all of them if you're very excited about it.

[00:19:37] This is the method we followed with Oodi [?]. First you do the problem space canvas, then you do the ethical considerations, then you make design guidelines, and then you build an MVP. If you want to go more in depth into the design of the robot itself, you don't have to do this MVP. There are four different canvases which go into the four dimensions: environment, form, behavior, interaction. This is what it looks like to design a robot. This is from our workshop with Oodi [?]. We have canvases, we have Post-its: very, very basic design stuff. The end result is just a bit different.

[00:20:17] Thank you, and here's the link to the canvases if you want to see them. If you have some questions and you don't want to ask me now, you can ask me later on Twitter, or of course go to the Futurice website.

Q&A

[00:20:35] Host: Amazing topic, and a really interesting project for sure. Anyone want to ask something?

[00:20:41] Audience: Yes, thank you for the great presentation. I have a question concerning your hypothesis with social robots. Do I understand it right that you assume that the social robot will create an emotional connection to the users? Because I don't see why a screen or an animated figure wouldn't do the same. Was your hypothesis that this would be more efficient somehow than just learning with a screen or an animated character?

[00:21:27] Minja: Well, there's a lot of previous research on that, so I specifically did not examine that here. But it's been shown that if a character has its own physical form, as compared to a screen, it's more compelling. I think it's a combination of its physical presence, so it's actually there with you in the room and you can see it from all angles, and it's moving by itself, so that already ascribes some kind of intentionality to it. So people perceive it as if it were alive, even though it's not.

[00:22:04] Audience: One question: where was the control? And if you were to choose a control, what would it be? Would it be, like you said, an animation, or would the best control be a real human?

[00:22:17] Minja: In this case the control was the three different design conditions: speaking and signing; speaking and signing and showing the image; and speaking and signing and showing the lights. We were only comparing these things to each other; we were not comparing to a human. This was not a clinical study. This was verifying a prototype, and our hypothesis was that the children are going to sign, which was correct. Then we had some other hypotheses: we compared, in the survey, the opinions of the parents, like, do you think the connection to the robot happened? And they said that it happened. But of course, if this were to be developed into a medical clinical product, yes, definitely we would need a control study with a human. We would need long-term studies. This was only one instance, but unfortunately this study was only so big, and we could only do so much.

[00:23:29] I think the use case of this robot is to use it within therapy, in conjunction with the therapist, so we wouldn't really need to compare that with a human. It would just be either the therapist and the robot, or only the therapist, and see whether that has some effect. And one of the other effects that we know of is that it's not only potentially beneficial for the children, it's also potentially beneficial for the therapists. If the therapist has the robot there, their cognitive load is a little bit reduced, and then maybe they can have a longer therapy session, because generally speech therapy sessions are 45 minutes, because it's quite challenging.

[00:24:04] I was just viewing from the other room, but even I was a little bit exhausted after looking at those sessions, because there's a lot to do there. If the robot was there, maybe the therapist could have the robot do some of the signing, and then the therapist could spend more time planning the session and controlling the robot as needed, that type of thing. But yeah, further research is needed, as it's always said in science.

[00:24:32] Audience: I think you've sort of answered part of my question at least, but just on the ethical considerations, which you had some of: why replace a human with a robot, or why use a robot in conjunction with a human as opposed to just a human? What's the ethical basis for that?

[00:24:57] Minja: I wouldn't replace a human here at all. But I think the robot would be useful because previous research has shown that using a robot in therapy makes the autistic person more engaged in the therapy, as opposed to just a person, as opposed to other methods.