Skip to main content Skip to secondary navigation
Lvmin Zhang photographed outdoors in bright light

Lvmin Zhang

Spotlight
PhD candidate
Computer Science

I think of myself as an artist. I’ve always loved to sketch line drawings and do coloring, but my shading needs work. I’m currently researching AI-driven computational art in Stanford’s computer science program, and it makes me really happy that my work helps digital artists achieve more precise creative expression.

If I had to define my core research focus, I’d say human-centered computation. I’m working on aligning human needs with the design of computational mechanisms that humans can use. In 2023, my academic advisor, Professor Maneesh Agrawala, and I spent several months working on ControlNet, a mechanism for you to guide generative models with different control signals such as sketches, segmentation maps, object boundaries, and shapes. The control in the name refers to human control. People have plenty of ways to input their demands into AI now, but in 2023, generating images was like rolling dice. You’d type something into an image generator and have almost no way to control what it produced. Seeing the need for intervention, we developed a neural framework that allowed users to draw lines and have the machine interpret them. Our tool makes it easier for artists to get what they want from a text-to-image generator by giving them more control. When we published in 2023, the International Conference on Computer Vision recognized us with the Marr Prize, one of the highest honors in our field.

Last year we applied for and won a grant from the Brown Institute for Media Innovation, a research body at Stanford that innovates at the intersection of journalism and technology. Chuan Yan, a postdoc in our lab, was the lead on this project and I was his secondary. Together we created HiFiSketch, a tool that invented a way to make more precise line drawings by taking an artist’s freehand pencil or ink drawing and translating it into a digital drawing. It’s an important problem in artistic design and I’m the lab’s resident expert in line drawings, so it was great to be part of that team and have that university-supported opportunity.

Humans, computer software, and AI agents are currently in a triangle structure, and we’re looking to solve the communication and collaboration problems between them. Humans are the source and objective of everything. No matter what you want from a computer and no matter what AI or agents do, we humans always hold the highest level of demand. Agent AIs are very popular now because they can do lots of things on their own, and can keep working without supervision for long periods of time, but humans still own the highest level of demand because it all begins with a human issuing a command first. Our field has studied the interaction between humans and computers for decades. As agents get more intelligent, I think it is worth asking whether they should have their own place, so that the interaction becomes computer, human, and agent, and what alignment looks like then.

My work focuses on adding human control to AI systems to align with human intents and human priors, meaning what the models have already learned from human-made data. Now that people can generate images from ChatGPT, you can just say, “Generate a tree for me.” But we’re finding out that language is not enough. For example, if you say something is on the left of something, the AI does not know how left is on the left – maybe slightly left? Middle left? And even if you say slightly left, how slightly is slightly left? Our research recognizes this problem and offers a solution. If we weren’t studying this, the world would have a little less knowledge about how to align human intention with AI systems.

I’m aware of practical concerns about generative AI, like the many artists who are worried it will replace them. But I believe these concerns have arisen because AI is doing extra things that the artist has not asked for; in other words, a lack of control. I have lots of artist friends in animation and they use AI for tasks like filling in color that can be time-consuming and labor-intensive to draw themselves. While they actually prefer to use AI for these tedious jobs, they understandably don’t want AI to replace them in other tasks that allow them to explore their creativity. In other fields like programming and manufacturing too, people want AI to do very specific things, but AI begins to do extra things that nobody asked for. To me, this is the same problem. The system goes beyond the human intention, and the person has no good way to show the machine what they actually want.

Everything has a limit; humans have a limit, AI has a limit, and computers have a limit. The question is: What’s the limit and are we aware of it? Many things that we thought should be the limits of AI turned out not to be. Ten years ago, nobody would believe that you can chat with AI, but now we have ChatGPT. Today, AI is close to human ability, or actually beyond human in many areas. So the boundary I care about most is where a person’s precise intention ends. Inside that boundary, the system should do exactly what the person specified. Outside that boundary is the space for exploration, where the tool can propose many possibilities and the human chooses. Questions like where to draw this boundary, when a human should be convinced by AI, and how the power of AI changes human intention itself are the questions I want to keep thinking about.

This is a very exciting time to be doing AI research, and Stanford is the best place to do it. I’ve found the most important aspect of Stanford’s academic culture is empathy. My mentor Professor Agrawala and my peers are very empathetic. What that looks like for me is an understanding of how every individual works differently and has unique lifestyle habits and needs. Some prefer to take very small steps and after every small step report to their mentor and have a discussion. But others prefer to do everything and only then do they report to their mentor to have a checkpoint discussion. I’ve noticed mentors here don’t enforce a single work style; they align to students’ different styles. Allowing students that kind of flexibility is very empathetic. And I think that’s why so much innovation happens here.

Related spotlights

Portarit of Aldís Elfarsdóttir, standing alongside the marble spheres in the science and engineering quad.

Aldís Elfarsdóttir

PhD '26
Management Science and Engineering
My research is animated by the idea of “walking the talk.”
Read Aldís Elfarsdóttir's story
Portrait of Laura Kubiatko underneath a tree with bright green leaves.

Laura Kubiatko (she/her/hers)

PhD candidate
Civil & Environmental Engineering
As an engineer, my job is to bridge the gap between the theoretical and practical.
Read Laura Kubiatko (she/her/hers) 's story
Portrait of Margie Mene standing in a brightly lit hallway.

Margie Mene

Senior HR & Visa Specialist
I like to think of myself as an employee advocate.
Read Margie Mene's story