Stanford algorithm analyzes sentence sentiment, advances machine learning

Print view

NaSent is a powerful new 'recursive deep learning' algorithm that gives machines the ability to understand how words form meaning in context.

People express opinions every day on issues large and small. Whether the topic is politics, fashion or films, we often rate situations and experiences on a sliding scale of sentiment ranging from thumbs up to thumbs down.

As we increasingly share these opinions via social networks, one result is the creation of vast reservoirs of sentiment that could, if systematically analyzed, provide clues about our collective likes and dislikes with regard to products, personalities and issues.

Against this backdrop, Stanford computer scientists have created a software system that analyzes sentences from movie reviews and gauges the sentiments they express on a five-point scale from strong like to strong dislike.

The program, dubbed NaSent – short for Neural Analysis of Sentiment – is a new development in a field of computer science known as “Deep Learning” that aims to give computers the ability to acquire new understandings in a more human-like way.

“One of the main things about our approach is to get away from human experts having to create algorithms with so many rules and instructions,” said computer scientist Richard Socher, who developed NaSent in collaboration with Stanford computer science professors Christopher Manning and Andrew Ng, and Christopher Potts, a linguistics professor and Director of the Center for the Study of Language and Information.

Richard Socher, Andrew Ng and Christopher ManningThe Stanford team includes current and former students Alex Perelygin, Jean Y. Wu and Jason Chuang.  They will formally present NaSent at the mid-October Conference on Empirical Methods in Natural Language Processing (EMNLP) in Seattle.

Experts say NaSent advances the state of the art in creating machines that can extract information from language without constant reference to human-made dictionaries or rules.

“This is a beautiful example of (deep) learning,” said Yoshua Bengio, a professor at the University of Montreal and head of the LISA Machine Learning Laboratory.

“A lot of previous work has focused on (single) words,” said Bengio, a Deep Learning pioneer who is familiar with NaSent but was not part of its development team.

“Here the system is being trained to move beyond words to phrases and sentences, and to capture the sentiments of these word combinations,” Bengio said.

How did the Stanford team create NaSent?

During the summer the scientists started from a dataset of roughly 12,000 movie review sentences. They split these sentences into phrases, using automated techniques to “parse” groups of words into grammatical units of meaning. The result was 214,000 phrases and sentences. Each of these was read by three humans, who evaluated these expressions for intensity of like or dislike.

Computer scientists call this labeling the data.

Using the Stanford team’s NaSent algorithm, the machine “studied” this labeled data the way a student might study a grammar text.

Or, to be more accurate, the Deep Learning system assigned each labeled expression a set of mathematical attributes. Computer scientists call these numerical descriptions “feature representations.” They are roughly analogous to the concepts and definitions we understand as human beings.

The important thing about the Stanford work, from a computer science view, is how NaSent works.

The scientists provided this computational system with two inputs: one was the massive set of labeled data. The other was the algorithms, or mathematical tools, that the system used to analyze and organize the labeled data.

With no further human-made rules or interventions, NaSent analyzed the labeled data and computed its own framework for predicting the sentiments that these words, phrases and sentences conveyed.

Moreover, once NaSent computed that the words “great movie” belonged together, the system was able to use this mathematical understanding to further organize the data. Thus “great movie” might become an input that NaSent could use to recognize that “great movie for a date” and “great movie to snore through” were two different phrases, and that each conveyed a different sentiment. The Stanford team calls NaSent a recursive deep learning algorithm to signify that the system is learning from what it previously learned.  

Two sentences illustrate how NaSent works and how its multi-word approach improves upon previous generations of software that analyzed single words.

  • “Unlike the surreal Leon, this movie is weird but likeable.”
  • “Unlike the surreal but likeable Leon, this movie is weird.”

“I spent about five minutes cooking up these examples to show how, even when using the same words, the interpretation of words is dependent on how they are put together,” said team member Christopher Manning, a professor of computer science and linguistics at Stanford.

In evaluating these sentences, NaSent organizes words into what the scientists call grammatical tree structures that put the words into context.

Graphic analysis of sentence

Figure 1. NaSent's analysis of the sentence: "Unlike the surreal Leon, this movie is weird but likeable." Red nodes indicate that the model assigned a negative sentiment to the underlying word or phrase, such as in the case of "weird." The phrase "is weird but likeable" is correctly assigned a positive sentiment (indicated by blue).  

 

Graphic analysis of sentence

Figure 2. NaSent's analysis of “Unlike the surreal but likeable Leon, this movie is weird.” The model correctly focuses on the overall negative sentiment, even though this sentence uses exactly the same words as the one above.

This tree-structured approach allows NaSent to rate "surreal but likeable Leon" a positive phrase while predicting that "this movie is weird" would be a negative, giving the overall sentence a thumbs down. The reverse is true for the second example.

A machine-based system that merely tallied positive and negative words – the most commonly used approach in sentiment analysis today – would be unable to catch the subtle distinction and would assign both sentences the same sentiment, Manning says.

“We’re still a long way from having computers ‘understand’ language the way human beings understand it,” Manning said. “But language understanding is getting better all the time.”

Just how accurate is the system? Well, if NaSent were a student, it would get a B+.

“We have about an 85 percent accuracy rate,” said Socher, as compared with 80 percent for previous word-based evaluation systems.

In short, NaSent solves about one quarter of the errors of previous-generation systems.

Stanford’s efforts in Deep Learning are part of a larger trend in academia and industry. Google has shown off a Deep Learning system that taught itself how to recognize images of cats on the Internet. Facebook has embarked on a Deep Learning project that could help fine tune the flow of items in members’ news feeds.

More broadly all of these initiatives are moving in the direction of creating artificial intelligence – machine-based systems that can learn and make decisions without carefully specified directions. But the whole notion of AI became so overhyped in the past that serious researchers nowadays downplay the term even as they work toward that goal.

Meanwhile, the Stanford team has opened NaSent to public input, hoping to use crowdsourcing as a way to give the system new challenges.

“NaSent can use the inputs to continuously learn and refine itself,” Socher said.

You can experiment with NaSent, test its abilities and teach it when it makes mistakes in the online demo.

Learn more about Deep Learning in the tutorial at the Association for Computational Linguistics Conference.  Deep Learning for NLP (without magic).

Media Contact


Tom Abate, associate director of communications, Stanford Engineering, tabate@stanford.edu, 650-736-2245.

 

Last modified Thu, 10 Oct, 2013 at 9:15