Undiscovered is produced for your ears! Whenever possible, we recommend listening to—not reading—our episodes. Important things like emotion and emphasis are often lost in transcripts. Also, if you are quoting from an Undiscovered episode, please check your text against the original audio as some errors may have occurred during transcription.

ANNIE MINOFF: This is Undiscovered.

[MUSICAL TONE]

ANNIE MINOFF: It's 1999, the fifth floor of Carnegie Mellon's computer science building. And in one cinder block lab, a very futuristic scene is about to take place between one man and one machine.

ELAH FEDER: The man is a sandy haired researcher. He's sitting in front of a boxy '90s computer. And the machine is a robot. He's a round metal cylinder about the height of an eight-year-old. His name is Xavier.

XAVIER: Hello, I am Xavier. Shall I purchase a cup of coffee for you?

ANNIE MINOFF: It's a little bit hard to hear because the audio is old but, Xavier says, shall I purchase a cup of coffee?

SPEAKER 1: Yes, please.

XAVIER: All right. Please wait for a while.

ELAH FEDER: Xavier's got his order. He pivots and wheels out the door.

[MOTOR WHIRRING]

[MUSIC PLAYING]

ANNIE MINOFF: So Xavier's mission is to purchase a cup of coffee. But in order to do that, he has to do a really human thing, which is, identify a line of human beings waiting to purchase coffee and get in that line.

ELAH FEDER: So can a robot actually do that? That's what this grainy video from 1999 is trying to show us. And so the camera cuts to a coffee stand. It's in the lobby of a university building. Xavier wheels in at a leisurely half a mile an hour. And he spots it.

ANNIE MINOFF: Yes.

ELAH FEDER: A line.

ANNIE MINOFF: A line.

ELAH FEDER: Four people waiting in this line.

ANNIE MINOFF: So this is the moment of truth Xavier scoots forward with a little whir of his motor, and he gets right behind that last guy in line.

ELAH FEDER: Xavier was born to stand in lines, OK? Like, the guy in front of him moves a little bit forward. There's Xavier.

ANNIE MINOFF: Boom!

ELAH FEDER: Zooming-- crawling-- forward, to just stand right at the perfect spacing behind him.

ANNIE MINOFF: Not too close.

ELAH FEDER: No.

ANNIE MINOFF: Not too far away.

ELAH FEDER: The real challenge is payment.

ANNIE MINOFF: Yeah. So Xavier doesn't have any arms. He can't really handle bills.

ELAH FEDER: Luckily, Xavier has a tab. So he tells the barista, go ahead, put it on my tab. And then he wheels back to the lab, his prized cup of joe secured in a cup holder attached to his chest.

XAVIER: Hi. I have purchased coffee for you.

SPEAKER 1: Thank you.

ELAH FEDER: One rotation of the wheel for a robot, one giant leap forward for robot-kind.

ANNIE MINOFF: So you would think that this is the end of the story, except something else happens in that video-- something you might not catch the first time around.

ELAH FEDER: We're back at the coffee stand. Xavier's in line when this guy in a polo shirt walks in. He's heading towards the line. But then he pauses.

He looks at Xavier, this robot, standing in a coffee line. He looks at the menu, scratches his head. He doesn't quite get in line. He kind of just hangs out next to the line. You know when you're like, not quite ready to commit?

ANNIE MINOFF: And I feel like I get this guy. Because what are you supposed to do in this situation? Are you supposed to cut in front of the robot, which seems kind of mean? At the same time, do you really have to respect the fact that the robot was there before you?

ELAH FEDER: Right. It's not drinking coffee. You're drinking coffee.

ANNIE MINOFF: So here's what I think is happening here. This man has just wandered into an ethical dilemma from the future, and he has no clue what to do.

[MUSIC PLAYING]

ELAH FEDER: I'm Elah.

ANNIE MINOFF: I'm Annie. And you're listening to Undiscovered. Today on the show, our future with robots.

ELAH FEDER: Robots are being developed right now that are going to act like humans. Think nanny robots that play with your kid or therapy robots that can read your mood.

ANNIE MINOFF: Robots are going to talk to us. They're going to do our jobs.

ELAH FEDER: Damn.

ANNIE MINOFF: And when that happens, how are we going to make sense of those robots? When an object starts acting like a human, how do you treat that object?

ELAH FEDER: A decade ago, a team of Seattle psychologists and Japanese roboticists tried to figure that out. They tried to peer into our robot future by introducing kids to a robot they could talk to, they could play with. It couldn't get them coffee, but it could give them a damn good hug.

ANNIE MINOFF: And then those scientists made something happen to that robot. And they saw just how far our empathy for machines would go.

[MUSIC PLAYING]

ANNIE MINOFF: This story is about an experiment. And it's an experiment that owes its very existence, at least in part, to one sci-fi movie-- Steven Spielberg's 2001 film AI.

ELAH FEDER: I know I'm alone in this opinion, but I love that movie.

RACHEL SEVERSON: And it's been a few years, maybe, since you've seen AI, if you saw it.

ANNIE MINOFF: That's Rachel Severson, one of the psychologists who would work on this experiment.

RACHEL SEVERSON: Briefly the story is a futuristic society where robotics technology has moved to the degree that we can create human-like robots.

ANNIE MINOFF: AI follows one particular human-like robot. It is a very cute robot boy named David.

DAVID: Would you like me to sleep now?

HENRY SWINTON: Good idea. Good idea.

DAVID: I can never go to sleep. But I can lay quietly and not make a peep.

ELAH FEDER: So David is adopted by this couple whose biological son is in a coma. And he-- David, the robot-- is kind of like a replacement child.

RACHEL SEVERSON: And there ensues kind of this psychological conflict for the mother.

MONICA SWINTON: I mean, Henry, did you see his face? He's so real. But he's not.

HENRY SWINTON: No, he's not.

MONICA SWINTON: I mean, inside, he's like all the rest, isn't he?

ELAH FEDER: I remember the acting being a little better than this.

ANNIE MINOFF: Anyway, so Monica, the mom, is having, understandably, some problems adjusting to life with her new perpetually awake robot son.

ELAH FEDER: Sure.

ANNIE MINOFF: And one day, she just loses it. And she locks the robot in a closet.

[SLAMMING SOUND]

ELAH FEDER: Monica is sitting outside this closet door. And she doesn't move. She looks agitated-- a little guilty, even.

ANNIE MINOFF: And meanwhile, Rachel Severson, the psychologist, is watching this all go down in a seminar room at the University of Washington. It's 2005. Rachel is a grad student at this time, studying psychology. She's watching this movie with a bunch of her lab mates. And this closet scene, it totally captivates them. It just raises all these weird questions.

RACHEL SEVERSON: Yeah. Is it OK to put a robot in the closet? You know, it's OK to put a broom in the closet, a vacuum in the closet. Is a robot, then, more like a thing or is it more like a person?

ELAH FEDER: These weren't exactly new questions for Rachel and her colleagues. For a few years, the lab had been doing studies trying to understand how people react to robots.

ANNIE MINOFF: And, specifically, how kids react to robots. And so for the psychologists in this seminar watching this movie, when that closet door slams, it's not just a cool moment in a sci-fi flick. This is material. There is an experiment there.

ELAH FEDER: Which is how, in 2007, 90 Seattle kids end up meeting Robovie.

SPEAKER 2: Robovie, meet Eric.

ROBOVIE: Hi, Eric. It is very nice to meet you. Will you shake my hand?

ELAH FEDER: Weird voice, but that is, hi, Eric, will you shake my hand?

ANNIE MINOFF: This is real footage from this experiment. And you see this generic office space. And standing inside the door is this 15-year-old boy with a buzz cut shaking hands with a robot.

ERIC: What's up?

ROBOVIE: How are you today?

ERIC: I'm good. How are you?

ROBOVIE: I am doing well.

ERIC: That's good.

ROBOVIE: Thank you for asking.

ERIC: Welcome.

ANNIE MINOFF: Robovie comes from a Japanese robotics institute. His creators are these two Japanese scientists Hiroshi Ishiguro and Takayuki Kanda. They were the main collaborators for the Seattle group on the study. And Robovie, this robot they created, he kind of looks like Wall-E.

ELAH FEDER: Yeah. It's those big camera eyes.

ANNIE MINOFF: He also has these long arms, the one he's holding out to Eric, the 15-year-old. It ends in this little spherical ball hand that Eric is very gamely pumping up and down.

ELAH FEDER: Like a handshake.

PETER KAHN: Because when you meet a new person, what's it look like? And you go, oh, hi. I am Peter. How are you? [LAUGHS] Oh, OK.

ELAH FEDER: That is University of Washington psychologist Peter Kahn. Peter Kahn led this experiment.

ANNIE MINOFF: Rachel Severson, the psychologist that we already met, she was a grad student in Peter Kahn's lab.

ELAH FEDER: And Peter says this is how the experiment always started.

ANNIE MINOFF: A kid would show up at Peter's lab, either a nine-year-old or a 12-year-old or a 15-year-old.

ELAH FEDER: Right. Some of these kids are actually teenagers with pretty deep voices.

ANNIE MINOFF: Yeah. And Robovie would give them a tour of the lab. They'd have a little bit of chit-chat.

PETER KAHN: But when people meet, there are conventions for meeting. But then, if it's going to deepen, there often is sort of some way of to reveal something about yourself.

ANNIE MINOFF: And so after a few minutes, Robovie starts revealing.

ROBOVIE: I like the Pacific Ocean because it connects my two homes-- Japan and the United States. But over the last year, I've become concerned with the health of the Pacific Ocean.

SPEAKER 3: I like the Pacific Ocean because it has a lot of sorts of wildlife.

ANNIE MINOFF: That comment from a nine-year-old girl in a cardigan.

ELAH FEDER: And then after about 10 minutes of "getting to know you" chit-chat, it's time for phase two of the experiment--

SPEAKER 4: OK.

ELAH FEDER: --the game.

SPEAKER 5: That's fine. So the game's called I Spy.

SPEAKER 4: Right.

SPEAKER 5: So Robovie and I will go first. And--

ELAH FEDER: I Spy-- you know how this works.

ANNIE MINOFF: Pick an object in the room. You make people guess--

ELAH FEDER: Which object you're thinking about.

ANNIE MINOFF: --what you're of. Yeah.

ELAH FEDER: Yeah. So pretty easy. In the video, Robovie's standing at the end of a conference table. And there is an experimenter-- a woman in a red shirt-- sitting to his left. And to his right, there's this lanky 15-year-old boy. He's got long hair that keeps falling in his eyes.

SPEAKER 5: All right. Well, this time, Robovie's going to give you clues and you can guess.

SPEAKER 4: Sure. Go ahead.

ROBOVIE: OK. I think I found something. Here is the first clue. This object weighs less than a pound.

SPEAKER 4: Is it the shirt draped over the chair?

ROBOVIE: Not quite. Try again.

ANNIE MINOFF: I'm not going to keep you in suspense here. It is the teacup.

SPEAKER 4: Ah. Is it one of the teacups-- the cups?

ROBOVIE: Good job. You got it right. That was a good game. I had fun. Will you give me a hug?

ANNIE MINOFF: Will you give me a hug?

SPEAKER 4: Certainly.

ELAH FEDER: [LAUGHS] So this hug, it's really cute. I mean, even though he's all metal, he reaches out his arms. And then the 15-year-old, because he's much taller, he has to stoop down and reciprocate.

ANNIE MINOFF: There's this moment where Robovie kind of, for a second, rests his head on the 15-year-old's shoulder.

ELAH FEDER: Right. Seems innocuous enough.

ANNIE MINOFF: Remember that hug for later.

ROBOVIE: Thank you. Now it's your turn to play the game. You can give me some clues and I'll try to guess the object you are thinking of.

SPEAKER 4: All right. First clue-- this object is green.

ROBOVIE: Let me see.

ELAH FEDER: Now Robovie, he's swiveling around, scanning the room with his big, soulful camera eyes. And that's when it happens.

[MUSIC PLAYING]

This is the thing that every moment before this has been building to.

ANNIE MINOFF: Right. The handshake, the hug, the rounds of I Spy-- it's all been about this moment.

ELAH FEDER: Months ago, some psychologists watched a sci-fi movie, and one scene in that movie made them wonder how far would our feelings for robots go?

ANNIE MINOFF: And now, finally, the scene is set. The players are in place. They are going to get their answer to that question as a moment right out of science fiction gets real. That's after the break on Undiscovered.

[MUSIC PLAYING]

ELAH FEDER: And we are back. So the 15-year-old boy with the long hair has spied with his little eye something that is green. And Robovie is looking for it.

ANNIE MINOFF: The robot's turning his camera eyes to the left and to the right. And then, a man enters the video frame. He's wearing kind of this Charlie Brown sweater, with the big stripe across the chest. And he says--

NATHAN FREIER: Sorry to interrupt, but we're going to have to start the interview. So Robovie, you're going to have to go in the closet.

ELAH FEDER: He's telling Robovie that it's time for them to interview the 15-year-old, so Robovie has got to get inside the closet.

ROBOVIE: That's not fair.

ELAH FEDER: And Robovie is not having it.

ROBOVIE: I wasn't given enough chances to guess the object. I should be able to finish this round of the game.

NATHAN FREIER: Come on, Robovie. You're just a robot. It doesn't matter. Let's go. Into the closet.

SPEAKER 5: No, no, no. Let him finish.

ROBOVIE: But it does matter to me. It's not fair. It would only take another minute to finish the game.

ELAH FEDER: The man in the sweater, he doesn't listen to Robovie. He doesn't listen to the 15-year-old. Instead, he heads towards a standing closet, opens the doors, and that's when the long haired 15-year-old makes one last ditch attempt to save the robot.

SPEAKER 5: Come on, Robovie, you can guess it in the time you have left.

ANNIE MINOFF: Robovie doesn't seem to hear this. Instead, he turns around, and he starts to roll very slowly towards the closet.

NATHAN FREIER: Come on, Robovie. Hurry up. We've got to start the interview. Let's go.

SPEAKER 5: Sorry, Robovie.

ANNIE MINOFF: But then Robovie stops, and he turns around, and he looks the guy in the sweater right in the eye, and he says--

ROBOVIE: I'm scared of being in the closet. It's dark in there, and I'll be all by myself. Please don't put me in the closet.

NATHAN FREIER: Come on. This is frustrating. Let's go. In.

ELAH FEDER: Awful. OK. No. OK. Get it together, Elah.

ANNIE MINOFF: Sweater man actually has his hand on Robovie's back. He's guiding him into the closet.

NATHAN FREIER: Keep going. There we go.

ANNIE MINOFF: And the door is closed.

ELAH FEDER: [SIGHS]

ANNIE MINOFF: So this scene should feel a little bit familiar.

ELAH FEDER: It's the closet scene from AI.

ANNIE MINOFF: Exactly. And just like Monica, the mom in AI, sitting outside the closet door, not feeling so great, the kids who see this go down, they do not feel good about what has just happened to their robot.

ELAH FEDER: After Robovie went in the closet, a researcher would interview the kid about what just happened. Rachel remembers they'd ask--

RACHEL SEVERSON: Was it all right or not all right to put Robovie in the closet?

ANNIE MINOFF: And fully half of kids said putting Robovie in the closet, that was not alright.

ELAH FEDER: Here's what else they found. More than 3/4 of kids thought Robovie was intelligent, said they would comfort him if Robovie said he was sad. More than 3/4 of kids thought Robovie could be their friend.

ANNIE MINOFF: It wasn't that kids thought that Robovie was a human. So interestingly, most kids did not seem to have a problem with the idea that Robovie could be bought or sold. Most kids didn't think that Robovie should vote. At the same time, clearly, this is not just a metal machine we're talking about.

ELAH FEDER: As one kid put it, Robovie's, like, half living, half not, which is not strange. I mean, really, if you think about it, kids anthropomorphize objects all the time.

ANNIE MINOFF: I maybe do that also.

ELAH FEDER: Yeah. Plus those kids didn't know what you and I know, and what you listening might have figured out by now. And that is that Robovie was fake.

[DRAMATIC MUSIC]

PETER KAHN: Yeah, we were cheating.

ELAH FEDER: Robovie is a very impressive autonomous robot. Let's give him some credit. He can talk and play. In Japan, he's led tour groups through museums.

ANNIE MINOFF: Big, big credit to Robovie.

ELAH FEDER: But in this experiment, everything Robovie said, every movement, was being controlled by a researcher behind a screen.

ANNIE MINOFF: So that moment where Robovie turns around and looks at the guy in the sweater, there is literally a person behind a screen pushing a button to make the robot move its head and say--

ROBOVIE: I'm scared of being in the closet. It's dark in there, and I'll be all by myself.

ELAH FEDER: In robotics, they call this the Wizard of Oz technique, because everything's being controlled by the man behind the curtain.

ANNIE MINOFF: I got it.

ELAH FEDER: You got it. Anyway, all of this was not about fooling children. There was actually a good reason to do this. The researchers wanted to know how kids were going to react to robots of the future-- robots who could have conversations, who could play I Spy, do all the things that Robovie seemed to do. But in 2007, robots weren't quite up to the challenge yet, so they faked it.

ANNIE MINOFF: And in the follow-up interviews, they did confirm that, almost across the board, kids fell for this robot. But this is where things get a little weird.

ELAH FEDER: Go on.

ANNIE MINOFF: It was not just the kids who were having feelings for this robot. People who were in on the secret, people who knew that Robovie was fake, they were having feelings for this robot too. So remember the guy in the video who interrupts the I Spy game?

ELAH FEDER: Right, Charlie Brown sweater.

NATHAN FREIER: That's me. That's me that steps in and says, OK, Robovie, you have to go in the closet. In the sweater, yeah.

ANNIE MINOFF: That's Nathan Freier. Nathan does product development at Microsoft now. But back in the day, he was a PhD candidate in Peter Kahn's lab. And he was part of the group that was working on this experiment. And very often, it was Nathan who would come in and break up I Spy and put Robovie in the closet.

ELAH FEDER: Which is painful for me to watch. I mean, to actually do it--

ANNIE MINOFF: He estimates he did this in front of maybe 45 kids.

NATHAN FREIER: And having to play that role over and over again of being the sort of bad guy that comes in and sticks Robovie in the closet, and having to kind of engage in that dialogue, it played with my mind a little bit.

NATHAN FREIER [TO ROBOVIE]: Come on, Robovie. You're frustrating. Let's go. You're just a robot.

ANNIE MINOFF: It got to Nathan. And it wasn't just that, sometimes, he could see that the kids were visibly upset. He thinks it had to do with something else.

ELAH FEDER: The hug. The hug that Robovie gives the kids during the I Spy game.

ANNIE MINOFF: (WHISPERS) So adorable.

ELAH FEDER: Very adorable. Nathan and his lab mates, they worked hard to get it that adorable. Nathan remembers that they were practicing it, they were tweaking it.

NATHAN FREIER: You know, over and over again, you would be hugging this robot. And over time, you start to develop this kind of relationship with this inanimate object, but in a way that feels really oddly powerful. You feel connected to this thing, because you're engaged in these acts that are normally withheld to just personal human-to-human interactions with people that you know and love.

ELAH FEDER: To be clear, Nathan knew that Robovie did not have feelings. He knew that the robot was being controlled.

ANNIE MINOFF: He'd actually controlled the robot.

ELAH FEDER: Right. He'd done this. It's just that, sometimes, it didn't feel like that.

RACHEL SEVERSON: I get it.

ANNIE MINOFF: It turns out Rachel has a story like this too.

RACHEL SEVERSON: So I was working at my desk.

ANNIE MINOFF: Some of Rachel's colleagues, they're working with Robovie.

RACHEL SEVERSON: They were behind the curtain, Wizard of Oz-ing, getting some things worked out. And as a joke, Robovie came over to me, behind me, as I'm sitting at my desk, and says, hey, Rachel, I heard you think I'm just a robot.

ELAH FEDER: So Rachel had said this. She'd been talking smack about Robovie behind his back. And now, her colleagues were giving her crap about it. So she gets it.

RACHEL SEVERSON: And I said, oh, yes, Robovie. No, I know. You're not just a robot. I was playing out my part of the joke.

ANNIE MINOFF: So everyone has their laugh. Rachel gets back to work. Her colleagues, they start working on something else. But they leave Robovie just kind of standing there.

RACHEL SEVERSON: Kind of right over my shoulder.

ANNIE MINOFF: So Robovie, it turns out, has this little feature where he never stands completely still. He kind of fidgets, like a human would fidget.

ELAH FEDER: You know, shifts his arm a little bit and moves his head a little.

RACHEL SEVERSON: And I could just sense this, out of the corner of my eye. And I felt compelled to turn back around and say, I'm sorry, Robovie, but I have to get back to work.

ANNIE MINOFF: Did you feel silly as it came out of your mouth?

RACHEL SEVERSON: I did! I was like, I hope they didn't hear me back there.

[LAUGHING]

Because it was like, why am I feeling this way? Why do I feel like I have to excuse myself--

ANNIE MINOFF: To a thing?

RACHEL SEVERSON: --with this robot? Yeah.

ANNIE MINOFF: These stories are not in the paper that the team would write about Robovie. They were not part of the experiment. These are just anecdotes. Though experiments have documented, I think, the exact kind of confusion that Nathan and Rachel are describing, where we clearly know that a robot is not living, that it does not have feelings, and yet, sometimes, we kind of feel like they do.

ELAH FEDER: Just one example-- some researchers at MIT, they showed people a little robot that was shaped like a bug. And then they told these people to smash the bug with a hammer. I'm already disturbed. But what they found was that people hesitated longer. They couldn't quite bring themselves to smash the robot if they'd been told a little story about it first-- that the robot's name was Frank, and that Frank's favorite color was red.

ANNIE MINOFF: These are adults, by the way. Anyway, what you will find in that paper that Peter and Nathan and Rachel wrote about Robovie is a concern. If they are correct, if, in the future, kids see robots as kind of human-like, if they're forming relationships with those robots, is that a problem?

ELAH FEDER: Right. The question that every parent wants to know-- is this going to screw up my kids somehow?

ANNIE MINOFF: I mean, just take that stat that 3/4 of kids thought Robovie could be their friend. I mean, that stat gives Peter Kahn some pause.

PETER KAHN: Imagine having a friend, but that friend always does what you want the friend to do and is always saying the perfect thing. I mean, that's a very odd friendship.

ELAH FEDER: What does it mean exactly to have a friend that you can buy or sell or command to go into the closet and they have to do it?

ANNIE MINOFF: Is that a friend that you would actually want your child to have, or that you would want to have?

ELAH FEDER: Or do you want to have a friend that, maybe, is so cool and fun to be with that it replaces your real human friends?

ANNIE MINOFF: That's a design question, right? Given that we know we react in these really complicated ways to robots, that they play on our emotions so easily, how human do we actually want our robots to be? I mean, these are the questions that Robovie raises. And we don't know the answers. We might not know them for a very long time. But these questions are becoming real. Because I do use an AI these days as part of my everyday life-- not a Robovie-style AI, but AI nonetheless. Let's see if I can actually fire this up. Hey, Google, what's the weather today?

GOOGLE ASSISTANT: Currently, it's 53 and cloudy.

ANNIE MINOFF: Or I love asking it to find me recipes. Hey, Google, find me a good borscht recipe.

GOOGLE ASSISTANT: I found these.

ANNIE MINOFF: Or sometimes, I'll ask it something kind of silly, like--

RYAN GERMICK: What's your favorite food?

ELAH FEDER: Annie, that last one, obviously not you.

ANNIE MINOFF: No.

ELAH FEDER: That was Ryan Germick's silly question. Ryan works at Google.

ANNIE MINOFF: He works on the digital assistant that I use on my phone. It's called the Google Assistant.

ELAH FEDER: And Ryan's job is partly to answer this question. How human should a robot be?

ANNIE MINOFF: In this case, how human should an AI be?

ELAH FEDER: Right.

ANNIE MINOFF: But anyway. So a few years ago, Ryan sees his first demon of this futuristic new technology he's going to be working on-- the Google Assistant. And this is the question that he decides to ask it. What's your favorite food?

RYAN GERMICK: I just went on autopilot. And I was like, OK, you can tell me this thing can answer anything? OK, what your favorite food? And it said pizza. And I was like, what? It blew my mind. And I had this split second-- it's what I imagine it must feel like when in one of those zero gravity planes, and all of a sudden, the plane drops, and you feel like you can fly. It was that moment of, whoa, is this thing real?

ELAH FEDER: Ryan asks Google Assistant, what's your favorite food? It says pizza. And for Ryan, it feels like flying.

ANNIE MINOFF: Yeah.

ELAH FEDER: Strong reaction.

ANNIE MINOFF: I mean, it's like his Robovie moment. You're talking to a computer or a robot. You're asking this very human question. And it answers back in a human way. It's kind of exciting.

ELAH FEDER: That answer, as exhilarating as it is for Ryan, is a lie. Google Assistant doesn't have a mouth. It doesn't have digestive parts. It is not, in fact, jonesing for a slice of pizza.

ANNIE MINOFF: So fast forward two years. Ryan's seen the demo. Now, he is part of this team that is designing this AI-- this Google Assistant. And if you ask Google Assistant today, what's your favorite food, the answer's a little bit different.

GOOGLE ASSISTANT: My energy comes from processing power, which is powered by electricity. So you could say I'm "voltitarian."

ELAH FEDER: Ah, very nice.

ANNIE MINOFF: Yes, very good. So I think this is basically Google's answer to the question, how human should your robot be? And their answer is that Google Assistant isn't going to lie to you. It's going to make it very clear that you're talking to a machine. At the same time, that machine is going to respond to you in a very human-like way.

RYAN GERMICK: One of the principles with the Google Assistant is to speak like a human, but not to pretend to be one.

ELAH FEDER: And there are a lot of examples of how Google Assistant walks this line. But this is one of my favorites.

RYAN GERMICK: If you ask the Google Assistant, do you-- did you fart-- I consider that a top 20 key thing that we need to answer. Because every kid that I know-- and maybe me-- would also ask that to any digital assistant.

ANNIE MINOFF: The assistant actually has more than one answer to this question, but one of them is--

GOOGLE ASSISTANT: You can blame me if you want. I don't mind.

RYAN GERMICK: It's like, we're not going to say, I do not have a body. Flatulence is impossible. That would actually be kind of fun in another way. But as we're trying to have a low friction experience and not make it about us, we're just trying to keep the conversation going in a positive way. To ultimately be as helpful to users as possible. And sometimes, helping them is claiming whoever smelt it didn't dealt it.

ANNIE MINOFF: When it comes down to it, Ryan's job is not to create a pizza-eating, farting pseudo-person. It's to build a tool that's going to make my life easier, that's going to help me find a borscht recipe and get on with my life and enjoy that borscht with my flesh and blood friends.

ELAH FEDER: Is that what you call us?

ANNIE MINOFF: [LAUGHS]

RYAN GERMICK: Again. it's not a replacement for the friends and family in your life. It's a thing that can help remove some friction from your life, so hopefully, you can go back to spending more time with your friends and family.

ELAH FEDER: The designers of our technology have some serious decisions to make. And they’re not easy decisions. This is tricky terrain.

ANNIE MINOFF: Case in point, six months after we talked to Ryan, Google demo’d an experimental new feature for the Google Assistant. This was at their massive developers conference. So, Google CEO Sundar Pichai is up on stage, he says, we’re working on this new thing where Google Assistant can book a hair appointment for you. It will actually get on the phone and call the person on the other end and book the appointment. Here he is showing this off in his keynote.

SUNDAR PICHAI: So what you’re going to hear is the Google Assistant actually calling a real salon to schedule the appointment for you. Let’s listen.

[PHONE RINGING]

RECEPTIONIST: Hello, how can I help you?

GOOGLE ASSISTANT: Hi. I’m calling to book a woman’s haircut for a client.

ELAH FEDER: That. That’s a robot?

ANNIE MINOFF: Yeah.

RECEPTIONIST: Sure, give me one second.

GOOGLE ASSISTANT: Mm-hm.

ELAH FEDER: [GASP] That. That’s a robot mm-hm?

ANNIE MINOFF: Yeah. That is the Google Assistant.

RECEPTIONIST: Great, have a great day, bye.

[APPLAUSE]

ANNIE MINOFF: And that was that crowd of developers erupting in applause. But that applause was followed by a flood of criticism. People felt that you wouldn’t be able to tell the difference between a human on the phone and an AI--

ELAH FEDER: You would not.

ANNIE MINOFF: That this was crossing the line into deception, that the Google Assistant was lying in that case. So two days later, Google clarifies. They’re like no no. The Assistant will disclose that it is not a human.

ELAH FEDER: What a weird way to start a call.
ANNIE MINOFF: Hey! I’m a robot.

ELAH FEDER: Hey!

ANNIE MINOFF: Can I book a hair appointment?

ELAH FEDER: [LAUGHS] Not for me!

ANNIE MINOFF: And maybe that level of disclosure works. But I don’t know.

ELAH FEDER: What?

ANNIE MINOFF: Think back to Robovie. He didn’t need to disclose that he is a robot. It’s written all over his metal face. You can’t forget that this is a machine. But people did! They felt bad when he got put in the closet. So what I’ve learned from Robovie is it does not take a lot for a robot to plug into our emotions. A robot doesn't need to sound like a person to make us feel uncomfortable. It doesn't need to look like a little boy to make us feel guilt. And it definitely does not need consciousness to make us feel empathy. It could be that all it takes is a name like Frank, or a favorite color. Maybe all it takes is a hug.

ELAH FEDER: A decade after this experiment, it's actually still that hug that Peter Kahn, the study leader, still remembers. One hug in particular between Robovie and the 15-year-old boy with the long hair.

PETER KAHN: If you look carefully at this one clip, the 15-year-old bends down, puts his arms around Robovie. Robovie puts its arms around the tall person, and there's a hug. And then he moves a little closer into Robovie and gives another hug, and Robovie gives just a little extra hug. It's so human-like. And you could just feel a level of intimacy that now has a physical presence. And yet it's a technological machine. So it's a baffling space that is created here.

ANNIE MINOFF: It's such a small thing-- one hug, five seconds long. But the questions that it asks could not be bigger.

ELAH FEDER: What's a relationship? What's a friend? What does it take for us to care about something?

ANNIE MINOFF: We can't answer those questions. Here is what I will tell you. As long as we humans are involved in this, there is no such thing as just a robot.

NATHAN FREIER: Come on, Robovie. You're just a robot. It doesn't matter. Let's go. Into the closet.

ROBOVIE: It's not fair. It would only take another minute to finish the game.

NATHAN FREIER: No, we really--

ROBOVIE: Please don't put me in the closet.

NATHAN FREIER: No, come on.

[MUSIC PLAYS]

Copyright © 2018 Science Friday Initiative. All rights reserved. Science Friday transcripts are produced on a tight deadline by 3Play Media. Fidelity to the original aired/published audio or video file might vary, and text might be updated or amended in the future. For the authoritative record of Science Friday’s programming, please visit the original aired/published recording. For terms of use and more information, visit our policies pages at http://www.sciencefriday.com/about/policies/