While pollsters frantically changed their predictions as Donald Trump picked up Florida, Ohio, North Carolina (and eventually traditionally blue Wisconsin and Michigan), University of Illinois Urbana Champaign sophomores William Widjaja and Cody Pawlowski weren't surprised.
They predicted he would win in June.
The two are cofounders of Tweetsense, a sentiment analysis startup that mines social media and other internet comments to understand public opinion. Since launching their startup last year, they correctly predicted the 2015 Chicago mayoral election and the Brexit vote. And through analyzing the presidential race since June, they correctly predicted that Donald Trump would be the winner.
Now they think it's time for a radical overhaul of how we do polling.
"In order to fix the polling industry, we need to take a totally different approach," said Widjaja, an electrical engineering major at UIUC. "We need one that sort of throws everything out the window altogether."
Tweetsense combines natural language processing and machine learning to glean public opinion insights from social media, trained on hundreds of thousands of pieces of text. Widjaja explains the difference between Tweetsense and other social media analytics startups is that Tweetsense does sentiment analysis, rather than just keyword analysis, to understand the emotion behind words. ("It would be the equivalent of 100 human assistants looking through and understanding something," Widjaja told me earlier this year.)
Part of the problem is that undecided voters swung aggressively toward Trump, according to Sheldon Jacobson, the UIUC computer science professor behind Election Analytics (which predicted Clinton would win). That indicates they might have supported the candidate the whole time or at least were more motivated to vote when it came to Election Day than those leaning toward Clinton.
With sentiment analysis, Widjaja said undecided voters who might think twice about telling a pollster their opinion over the phone, wouldn't hesitate to dash off a Facebook comment or tweet that would indicate their political leaning.
"Among most circles of young voters, supporting Trump was extremely frowned upon," he said. "If a pollster asked them, do you support Trump, they have a reason to say no, especially if a pollster is the same age."
Since the beginning of the summer they mined up to 20,000 sentences per day from a combination of Twitter, public Facebook comments, Reddit and comments on news websites, putting it through their algorithms to analyze how people were leaning.
Even as controversies (such as Clinton's emails and Trump's lewd comments) and key moments (the National Conventions and endorsements) moved sentiment in one direction or another, they found it tended to settle back in Trump's favor.
Widjaja noted that their method isn't perfect--there's bias in who uses social media in terms of age and motivation, and they constantly had to program out political bots from their analysis--but they were able to have a more accurate sense of public sentiment in real time.
The Tweetsense cofounders recently hired a third employee, and they're looking to work with political campaigns as well as other clients interested in public sentiment, such as businesses (they can use their tech to micro-target customers, eliminate surveys and create personality profiles of a given population).
"The method of using sentiment is robust enough to work on any kind of future kind of election," he said. "And not just elections but also for anyone who wants public opinion. The polling industry is huge, but it’s going to die. And in fact it did kind of die today and yesterday, and it’s time for the world to have this."
