Skip to content Skip to navigation

Research & Ideas

Search this site

James Landay: What’s next in human-computer interaction?

With a user-centered mindset, engineers are exploring new ways to develop technology that more effectively serves people’s needs.

Illustration of a man sitting at a table, a woman running and a man with a backpack

The freedom we have in designing technology requires more intentional decisions about how we want to interact with it. | Illustration by Kevin Craft 

Computers are everywhere and humans are engaging with them in nearly everything they do.

Knowing this, the question becomes: How do we design a world around us so that technology makes life better, not worse? James Landay, an expert in human-computer interaction, says the key to thoughtfully integrating humans with digital technology is to put people first.

This perspective draws on a philosophy known as human-centered or user-centered design. Within this approach, the first priority is to understand the problem vexing a particular population by observing, interviewing, and working with that population. Only once the problem is clear does the development of a solution begin. Typically, engineers and technologists have done the opposite. They’ve worked to develop the coolest technology they can think of, and then once it’s ready look around for a way to use it.

With human needs at the forefront, Landay’s research focuses on finding ways to use artificial intelligence technology to augment human performance. His current projects range from leveraging technology to encourage positive behavior change, to enabling kids to stay engaged in their education, to helping professionals stay healthy while feeling more connected to their co-workers and workplace.

Tune in to this episode of The Future of Everything to hear more about how Landay draws on user-centered design to develop technology that supports human needs. You can listen to The Future of Everything on Sirius XM Insight Channel 121iTunesGoogle PlaySoundCloudSpotifyStitcher or via Stanford Engineering Magazine.

Full Transcript

Russ Altman: Today on The Future of Everything: The Future of Human and Computer Interactions.

I don’t think I need to tell anyone that computers are everywhere and humans are interacting with them increasingly and everywhere. Now let’s think back to the old days when it was through computer terminals. Where we sat down at a desk and we typed in stuff, this is the ancient era, and little TV screens showed you mostly text. It was mostly a black screen with little white green letters. Then a revolution happened when things got more graphical, so we could have white screens with black letters.

That blew our minds and was huge and I’m not really kidding. This was in the early 80s. The mouse emerged, Apple popularized the mouse as a way to interact, borrowing and popularizing technology from Silicon Valley and other innovators. Again, somebody added buttons to the mouse. That was a huge moment and of course I’m being a little facetious here, but the fact is that interaction with technologies in early days was driven by very limited interfaces and interface technologies and a model of a computer as like a very smart typewriter, calculator.

With the acceleration of computational capabilities, however both hardware and software provide many less limitations to what we can do, but the freedom we have now in like designing these systems means we actually need to make decisions about how we want to interact with these systems and there are not obvious answers.

Phones talk to us with a conversational speech. We can yell at our TVs, many of us have been yelling at our TVs for a long time, but now the TVs actually respond when we yell at them. We can embed computers in the environment around us for sensing and interacting. High-resolution monitors are approaching the same resolution as our eyeballs. They’re not there yet, but it is increasingly becoming difficult to tell the difference between a real image and a digital image. AI systems are now allowing software to be smarter, more responsive and sometimes do tasks normally associated with human intelligence and pretty well. They can identify objects, they can run robots that make decisions during surgeries, while cars are driving, etc.

Now we have all these choices and we have to make some decisions about how we want to design the world around us, so that this technology winds up improving our lives and not being a big bummer.

So what would improvement mean? Dr. James Landay is a professor of computer science at Stanford University and an expert on user interface design, cross-cultural interface design, mobile and ubiquitous computing and a use of technology importantly, I think, to support behavior change.

James, I made a little fun of the way we used to keep computers in a box with very rigid interactions, but now with the growth of AI there is an increasing sense that computers have gotten out of the box and escaped perhaps and there may be a need to rein them back. How do you approach this challenge of integrating humans with digital technology in a thoughtful way?

James Landay: So, the key to doing this in a thoughtful way is to put people first. So in human computer interaction as a field we like to talk about something called the human-centered design or user-centered design, which means we don’t even start with the technology in mind as much as trying to understand what is the problem we’re trying to solve for a particular user population. So you know you’ll hear the same thing from folks talking about design thinking in the Stanford d.school, it’s really about finding those user needs. What is the real problem they’re experiencing?

Often we as engineers think we know the problem, but it’s really not the actual problem. Until you go out and talk to people, observe people then you find a problem and then you see what technology you might have that can help you solve it. So it’s going to be the same thing with these ubiquitous AI technologies, which is yeah we might think hey we have some new technology, we could, you know, detect skin cancer better than a doctor or something like this, but in reality we want to understand okay, what’s a problem in that current situation. Is the doctor making a mistake? Is it taking too long? Is it hard? Is this something somebody needs to do at home? You know let’s figure that out and then we start to look at what technology might we bring to bear, to solve it.

Russ Altman: I’m struck by this because engineers are not necessarily traditionally trained to do the kind of humanistic needs finding, human-centered activities that you’re implying and have to happen. Engineers are trained to build stuff or at least that’s what they think they’re trained with. So it sounds like what you’re saying is actually a fairly revolutionary statement or is it not revolutionary in terms of how engineers have to work?

James Landay: I’d say it’s revolutionary to engineers. To those of us who have been practicing this for twenty or thirty years it’s kind of like basic knowledge of how you make things that work for people.

Now in reality we often do what I would say is technology push and technology pull. The traditional engineering is technology push, hey, I’ve got this new hammer where can we hit some nails. Technology pull is more of hey we forgot what the problem is and how to design for people and we look for a technology that works.

In reality, what often happens, especially in the research realm like the universities, is that we’re going both directions, because a lot of us have some new technology we’re developing and then we look for applications where it would make sense. So, it would be probably a little dishonest to say it always happens best when we start with users only, but we probably have to go both of those directions. You know looking at what the real problems are, seeing what interesting new technologies are coming down the line. Especially for us researchers we’re looking ten, twenty years out, right.

Russ Altman: You’re skating to where the puck will be.

James Landay: Right, skate to where the puck will be, exactly. Great Gretzky quote.

Russ Altman: Have you and your colleagues in the research domain operationalized and turned it into a science, figuring out the needs of users. Because my guess would be that they can’t always articulate where their problems are. Maybe sometimes they can. I mean what’s you observation about if you go into a situation where you have a high suspicion that you could be helpful with some engineering. How much do the users just give you their problems on a plate and how much kind of hard work is it to kind of deconstruct their world and figure out where the real problems are?

James Landay: I would say it’s not a science. It’s more of an art and this is what makes it kind of hard to teach and hard to be great at. It’s not like you can just get a formula and follow these steps and you will come up with the right thing. There are people who have this, what I would call design ethnography background, who are trained to observe, to interview and we call it that because it really comes out of the ethnography from the early 20th century of studying new tribes and other people that haven’t been studied before.

Russ Altman: Like these dermatologists are a tribe and we need to understand what’s up.

James Landay: Right you need to go into that community and observe them in the wild i.e. in their workplace where things are happening and with their patients and see how it’s doing. So you know we teach students to do this in our introductory human computer interaction class here at Stanford, but it’s something that they’re not good at until they’ve done it in you know three or four classes and in fact over in product design in the Stanford Mechanical Engineering Department they have a whole class just on need finding, how to do that. So it’s a skills, but I would say it’s not really a science. I try to teach the students that if you go interview five, ten people those aren’t scientific results. You’re not going to be able to say something statistically. It’s more of an inspiration to your design. You found something interesting that maybe no one else had seen that angle. So I see it more as art and inspiration than science.

Russ Altman: This is The Future of Everything. I’m Russ Altman. I’m speaking with James Landay about needs finding in human computer interaction and like, before you build your hammer you might want to understand your nail. Can you give examples from your work of where you feel like you’ve been successful at looking at a situation and saying this is what I think could help the practitioner or the person who has a problem be more effective?

James Landay: So, some of my research is simply been in tools that would help designers be able to apply new user centered design into a new domain. For example, a sub-domain of AI that we call activity inference. Where I want to figure out what a person is doing in the physical world, like are you running? Are you walking? Are you going up stairs?

We had developed some of that technology early on in a research lab I lead in Seattle for Intel, but there was no way a designer could then take that technology and easily develop a new application.

Let’s say that would let’s say track your exercise and give you feedback. We had made tools that would almost be like a design tool that a designer who didn’t have AI or machine learning background could use those concepts and try their design ideas really quickly and prototype new ideas. So that’s one area throughout my career where I’ve tried to enable designers to approach a new technology that’s coming out before it would really be accessible to them.

Russ Altman: So it strikes me that that must be an area that evolved incredibly quickly because I would guess you were doing this work before a Fitbit existed, before an iPhone, maybe before an iPhone, but certainly before the watch, the Apple watch. So how well were you able to kind of create these tools that, did they anticipate the availability of these new technologies or did they quickly become irrelevant because of the new technologies?

James Landay: I would say we did well at anticipating where that puck was going. In that there were no Fitbits, there were no smartwatches per say there may have been some prototype type thing, but nothing like an Apple watch or a Fitbit and we had horrible mobile phones at that point. The iPhone and the Android phone came out two or three years later than that kind of work, but we were right at anticipating that eventually those phones would have the processing power to do the sensing, machine learning on the phone.

When we first did this research you had your Windows mobile phone in one pocket and on your belt you had a little pager sized device that actually had another processor and 10 different sensors and it did all of the machine learning. We weren’t perceiving that people were going to wear one of those in the future. We knew that would eventually be built into the phone and so we were right in predicting that and we saw Fitbit come out and they actually picked up some of the metaphors that our early design showed the Apple watch. But I’m actually still working in this area because I actually think those companies haven’t picked up some of the lessons from that earlier research on how to change people's behavior using feedback.

Russ Altman: Great topic, so let’s talk about behavior, James cause I know that’s a big focus of your research which is we’re not just trying to help them do things, what they’re doing now, but sometimes people have goals to actually change their behavior for health reasons, for performance reasons. Tell me about the behavior aspect of all this.

James Landay: Yeah so I’ve been working in that area for ten or fifteen years, and one of the things that we originally noticed was people were starting to use their phones a lot. Now we know the research shows people pull their phones out of their pocket or purse a hundred or 200 times a day and glance at it. To just check the time or to see some SMSs or alerts or to unlock it and play a game, whatever. And so our idea was can we take advantage of those glances, those 200 glances to make people more aware of what their behavior was. We really thought the difference between somebody who is able to stick to a diet or stick to their exercise plan or be more thoughtful towards the environment was being more aware of what you were doing. Some people are more aware, some other people needed help. So what we wanted to do was every time you took a glance at your phone we were going to give you a way to know how well you were doing with respect to your exercise goals for example. So this system called You Be Fit on the background of your phone these flowers would grow and every time you exercised another flower would grow and the idea was not that you would look at your phone and go oh I did three runs and five walks and you know six aerobics.

Russ Altman: Like it’s not a graph.

James Landay: Right, it’s not a graph. It’s not quantitative. It’s more, hey the more flowers in my garden the more I know I’m doing towards the end of the week that I might be reaching my goal. So it’s glanceable, just to give you almost a imperceptible idea of how well you’re doing.

Russ Altman: This is The Future of Everything. I’m Russ Altman. I’m speaking with James Landay, now about growing flowers on your iPhone as a proxy for your exercise levels. So that’s interesting cause just to deconstruct it for a second. I pick out my phone because it buzzed and I’m getting a phone call or because I want to check the weather and in a real literally micro-seconds or milliseconds let’s say, I also get a visual cue. And do you expect that cue to be in the consciousness of the person or is it going to stay in the back, but still have some conscious effects on their behavior or do we even know that yet?

James Landay: I think it’s really an unconscious, in a subconscious. You notice it, but maybe not even consciously. Now every now and then you might notice it and go oh hey that pink flower grew. What did I do there? And then maybe you launch the app and look more at the analytical.

Russ Altman: Or if things start looking dark and dank and the flowers start to just dying.

James Landay: Or you might go, I really need to take my gym bag today and not you know not drive my car or something like this. So sometimes you might be conscious, but I think most of those glances is an unconscious and that’s what we were trying to get at and we’ve also looked at that using for example vibration on a smartwatch. Could we send you vibrations letting you know about your performance and would that cause people to do better than having to look at an image? So we’ve tried it visually, as well as using the tactile.

Russ Altman: So this is really interesting and I can imagine that for things like exercise and tracking of activity that this could work. Let me ask you what’s your, how do you think about somebody like me I might have multiple goals, right. So right now I want to make sure I get my exercise, but I’m also writing a grant proposal that I have a deadline for and let’s say there’s three or four other things going on. Do you imagine that we’re going to be able to have several of these cues in our life or are we single channel people and it will be hard to signal, multiple potentially competing behavioral goals through these technologies? How should I think about the evolution of this?

James Landay: So I think that’s an open research problem. I’ve written proposals about how we might interleave through your or different variables by showing you let’s say a different image or maybe one image and codes multiple variables. Like the sun moving across the sky represents your exercise and the flowers represented your green behavior.

Russ Altman: Progress on the grant.

James Landay: We have not shown which of those work and we haven’t really got to it. In this most recent version of this in an application we call WhoIsZuki. We actually tried to encode both physical activity and sustainable green behavior.

Russ Altman: So that’s two, that’s two.

James Landay: That was two, but we haven’t really shown, can people tell the difference? We’re just not there yet. We’re still at the point of trying to show that these interfaces lead you to longer term behavior change, to sticking with this. Because one of the big issues is, studies have shown things like Fitbits, a third of the people quit after three months, fifty percent of the people abandon after six months and it’s not because hey I’ve suddenly got fit. I met my goal. It’s because the novelty has worn off and in fact it starts to become a reminder of what you’re not doing that you’ve committed too. So some really great psychology research here at Stanford with professor Alia Crum in the Psychology Department really looks at health mindsets and how can we shape people to have healthy mindset towards physical activity. We’re trying to take some of the ideas from Professor Crum’s group and put that into some of these activity tracking applications. So that you have a positive attitude and will stick with it over a long period of time.

Russ Altman: So tell me a little bit more about the, I know the WhoIsZuki, tell me about that. I know that there’s a story and the idea is that there’s a narrative that proceeds over time and presumably this is one of the ways you’re going to grab the ongoing attention of the user.

James Landay: Yeah, so one of the issues that we’ve seen is again abandonment of these devices. In our prior work you just saw one image on the background screen of your phone and that would change over the week. Let’s say the flower is growing your garden and if you hit your goal a butterfly would appear, but then on Sunday it would all start again as an empty garden. You would start again. And so what we wanted to see is, is there a way to engage people longer? So what we’ve done is we’ve created a narrative arc that goes over thirteen different visual chapters and each chapter you move forward in the chapter and we actually have a plot and a background and a hero and a bad guy.

Russ Altman: Villagers. I’m sure there’s villagers.

James Landay: There’s ups and downs and all these things occur and the idea is that we’ll keep people engaged over a long period of time. And so in fact we’ve worked with a narratologist here at Stanford. Narratologist that’s a word I didn’t know about until a year ago. Which is an expert in narrative and in the English Department. A professor of English who’s helping us make those stories more engaging.

Russ Altman: You want stories that don’t suck.

James Landay: Right.

Russ Altman: Because that draws people in and you don’t have to be an expert on everything and if there are these narratologist who can tell you this is what makes a good story, then you’ll be able to take that and use it for these multiple uses. This is The Future of Everything. I’m Russ Altman. More with Dr. James Landay about human computer collaboration interaction and maintenance of attention next on Sirius XM Insight 121.

Welcome back to The Future of Everything. I’m Russ Altman. I’m speaking with Dr. James Landay about the use of technology to basically augment human performance and one of the areas, James that you’ve worked in very actively is smart intelligent tutors, education. What are the challenges and what are the opportunities there? And I know it connects to your work that we were just speaking about. About creating compelling narratives.

James Landay: When I think about this idea of human-centered artificial intelligence. The basic idea is how can we augment people and help them be better at what they do. Whether it’s being a better learner or a better teacher or better artist. And so when I think about education, it’s not about how do I replace the teacher with an intelligent tutor. It’s instead how do I help a student who maybe isn’t motivated by the current school system to be engaged more in learning. So when I see students at a place like Stanford, you know this is the cream of the crop, but there’s a lot of students who the traditional school system probably just bores them and we’re losing that potential. So one of the interests I had was, was there another way to engage younger children, kind of age six to twelve in learning, outside of the school day in a way that maybe would get them more engaged in the traditional school.

We started to think about story and what we were motivated by was by this 1995 science fiction novel by the author Neal Stephenson, the book he wrote at that time was called The Diamond Age. With the subtitle, this is the weirdest subtitle I’ve ever heard, it says “Or, A Young Lady’s Illustrated Primer.” Where primer is like a British school book.

In that story the key idea was that there was this tablet computer that this little girl got a hold of that her brother stole from somebody he mugged. And that story on the computer was a narrative. So she would read a story, but as she read that story she had to learn things in the real world. She had to learn to read first. She had to learn math. She learned physics. She learned stranger danger. She learned martial arts and survival skills all while reading a story that went on for years.

Russ Altman: And the motivation came from her fervent desire to understand what was going on in that narrative?

James Landay: The narrative is what drew her.

Russ Altman: Like I need to understand physics cause this doesn’t make sense.

James Landay: Right, the narrative is what pulled her in and the story and the characters pulled her in to learn these things and not just sitting behind a computer in the den. Some of these things you have to learn out in the real world. So when I read this, I was a graduate student and I thought that was impossible from an AI perspective. The AI just was not there to create this thing.

So roll forward to almost fifteen, twenty years, in 2010, the iPad came out. I was living in China on sabbatical and I saw that device and I thought that’s the Young Lady’s Illustrated Primer, that’s the hardware, then I started to think about it again. Going okay, can we build such a thing and software made a lot of progress in those fifteen years that I started think oh we could build it. Then when I got here to Stanford a few years ago, we seriously started the project and we’ve been creating narratives that embed fun, educational activities and try them with children and we iterate with kids and see oh is that story too complicated? Is that learning activity too complicated?

Russ Altman: And it’s very much based on the insights. So now did this author give you enough to work with in terms of how to motivate the young woman. Did he get it right? Or have you had to kind of flesh that out because she was a little bit too eager, maybe a little bit too willing to learn things and that doesn’t match reality?

James Landay: So there’s some things in the novel that are good to make a good novel. I did a thing here in maybe 2015 that I never thought I would do as a computer science professor which was I had a reading group with some undergraduates where we read a novel together and we actually tried to extract the spec of this device. What were the features that were described? And they’re still on my whiteboard in my office and some of those features we decided, oh those were just good for a story. So for example in this novel the little girl’s mom is always kind of shacking up with some bad guys and one of them takes the thing and throws it at her head and it just gently glides down. We’re not building that.

Russ Altman: So it was a little bit of metaphysics.

James Landay: Yeah, there’s some things there that are just good for the novel, but then there’s other things for example it can detect her emotion. Okay we haven’t quite put that in our prototype yet, but being able to detect whether a student is actually frustrated or not when you’re trying to teach them — that’s key. That’s a feature that we will try to build in. So that you can understand the state of the mind of the student and we’re also doing a lot with AI in terms of can you predict what problem you should show a student based on what they’ve done previously to both challenge them, but not challenge them too hard or not make it too easy. There is a lot of research that has been done there; we’re pushing those algorithms into this.

Russ Altman: This is The Future of Everything. I’m Russ Altman. I’m speaking with James Landay about this amazing idea of a narrative that motivates learning. So is there instantiation yet of a story with specific I guess learning goals, learning objectives for a narrative?

James Landay: Yeah, so we’re working with people in the graduate school of education as well as graduate students there. We’ve mapped out some learning goals and objectives and we’ve written a couple of short stories. There’s one where you go back in time into the time of ancient Greece, and you’re in a village, and people are arguing about whether the King’s crown is really made out of gold, and then you eventually become the apprentice to this scientist whose name is Archimedes and you do some experiments with actual water and containers and learn about volume and density and things like this. We’ve tested those with kids and you know they work well for some and not others, and we’re trying different math activities as well as science activities and writing activities in these types of stories.

Russ Altman: So how structured do you make, so this is great and so I’m imagining I’m reading and I come upon a set of challenges and now I realize oh I am the apprentice, I need to learn about volume. Do you have like quizzes where they have to demonstrate the, that seems a little rigid to the I’ve now acquired skill x. How do you allow them to move forward and do this trade off of: you’ve learned enough that we’re going to now expose the next part of this story versus we’re going to stall the story until you can measure the volume of a cup of water?

James Landay: So it’s less a quiz than you going through a learning activity. One of the pieces of advice I got early on in this project from Dan Schwartz, the Dean of the Graduate School of Education, was don’t make it such that people can’t move on if they can’t master something. Instead, make it about exposure.

I’ve thought that fit well with my idea of exposing kids to these learning concepts in a fun way rather than hey you didn’t master this, you’re stuck at this level. Now, one thing we did do is we’ve worked a lot on having a chat bot, an AI agent, that can help the student when they get stuck and so you can ask questions.

So again, some of the AI research is how do you train a chat bot on domain knowledge so it can help a student let’s say learning about volume and not have to hard code rules for every single possibility. So can we learn this material semi-automatically. And so that’s again some of the research the chat bot can help the student when they’re stuck at that step and allow them to move forward to the next part of the story.

Russ Altman: Much as a skilled tutor would know the right question to ask or have you thought about this or did you consider this, prompt the students and then they can make progress.

James Landay: Exactly.

Russ Altman: Well in the last couple of minutes I did want to ask you about your interesting work on hybrid physical spaces. This is actually not unrelated to what we’re talking about because you, right now you’re looking at a tablet, but some day you might be in a virtual world there might be a world around you where you’re learning. How have you approached that problem? Why is it important to you?

James Landay: Right, so this project, hybrid physical digital spaces, comes out of this observation that one of my colleagues in Civil Engineering, Sarah Billington and I had which was we spend 87 percent of our day in the built environment. The way that built environment is made effects your well-being, it effects whether you feel stressed, it effects whether you’re going to get any exercise, it effects your sense of belonging even in your organization, school. And we thought, well, how are buildings designed? Are they designed to take advantage of this? And we found there’s not a lot of good research doing this. So we wanted to first do the research to show how different factors effect those variables of things like stress, creativity, sense of belonging. So natural light versus no natural light. What’s the effect? Natural materials versus artificial materials. What’s the effect?

Russ Altman: So wood and plastic?

James Landay: Wood versus plastic or laminate. Or what is the effect of having imagery on the walls of people from diverse backgrounds versus let’s say all white men. What would be the effects? Research has been done here at Stanford and other places would lead us to believe people would feel they don’t belong. So we’re actually trying to build the scientific basis to measure these things carefully, but then in the future imagine buildings where we might notice that Russ is stressed out. How might we adapt his environment dynamically to change that and how do you do that in a non-creepy way. You know we don’t want to be the paperclip of buildings. Do we just make it such that Russ could notice this and easily put on his cool jazz and turn the lights down. So that’s part of the research is how to do it.

Russ Altman: It’s not unlike the very first thing you said about the flowers growing on my iPhone, but now you’ve abandoned the iPhone and it’s the entire environment that I find myself in cuing me perhaps in directions that would be better than other directions.

James Landay: Exactly so it’s gonna be this ambient awareness of not just the imagery on your phone, but it could be the large displays that are going to be on walls because they’re going to be ubiquitous due to the cost. It can be the light, it can be the airflow, it could be the sound, all of these things and so we’re trying to look at what would that future be, but can we do it in a privacy preserving way, so that people don’t feel like they’re just being tracked and surveilled by their boss and how do you balance those issues is key to our research.

Russ Altman: Thank you for listening to The Future of Everything. This is Russ Altman. If you missed any of this episode, listen anytime, on demand, with the Sirius XM app.