What did Twitter’s ‘open source’ algorithm actually reveal? Not a lot.

When Elon Musk first proposed taking over Twitter, one the first changes he claimed he'd make would be “open-sourcing” Twitter’s algorithm. Last week, Twitter finally followed through on that promise, publishing the underlying code for the site’s "For You" recommendations on GitHub.

Quickly, Twitter sleuths began sifting through the code to see what they could dig up. It didn’t take long for one eyebrow-raising finding: that Musk’s tweets have their own category (along with Democrats, Republicans and “power users”). Twitter engineers hastily explained that this was for “stat tracking purposes,” which has since been confirmed by other analyses. And though Twitter removed that section of code from GitHub within hours of its publishing, it’s still fueled speculation that Twitter’s engineers pay special attention to their boss’ engagement and have taken steps to artificially boost his tweets.

But there have been few other major revelations about the contents of the code or how Twitter’s algorithm works since. And anyone hoping this public code would produce new insights into the inner workings of Twitter will likely be disappointed. That’s because the code Twitter released omitted important details about how “the algorithm” actually works, according to engineers who have studied it.

The code Twitter shared was a “highly redacted” version of Twitter’s algorithm, according to Sol Messing, associate professor at NYU’s Center for Social Media and Politics and former Twitter employee. For one, it didn't include every system that plays a role in Twitter’s recommendations.

Twitter said it was withholding code dealing with ads, as well as trust and safety systems in an effort to prevent bad actors from gaming it. The company also opted to withhold the underlying models used to train its algorithm, explaining in a blog post last week that this was to “to ensure that user safety and privacy would be protected.” That decision is even more consequential, according to Messing. “The model that drives the most important part of the algorithm has not been open-sourced,” he tells me. “So the most important part of the algorithm is still inscrutable.”

Musk’s original motivation to make the algorithm open source seemed to stem from his belief that Twitter had used the algorithm to suppress free speech. “One of the things that I believe Twitter should do is open source the algorithm and make any changes to people's tweets — if they're emphasized or de-emphasized —that action should be made apparent,” Musk said last April in an appearance at TED shortly after he confirmed his takeover bid. “So anyone can see that action has been taken, so there's no sort of behind-the-scenes manipulation, either algorithmically or manually.”

But none of the code Twitter released tells us much about potential bias or the kind of “behind-the-scenes manipulation” Musk said he wanted to reveal. “It has the flavor of transparency,” Messing says. “But it doesn’t really give insight into what the algorithm is doing. It doesn't really give insight into why someone's tweets may be down-ranked and why others might be up-ranked.”

Messing also points out that Twitter’s recent API changes have essentially cut off the vast majority of researchers from accessing a meaningful amount of Twitter data. Without proper API access, researchers are unable to conduct their own audits, which would be able to provide new details about how the algorithm works. “So at the same time Twitter is releasing this code, it’s made it incredibly difficult for research to audit this code,” he wrote in his own analysis.

Alex Hanna, director of research at the Distributed AI Research Institute (DAIR) also raised the importance of audits when we talked last year, shortly after Musk first discussed plans to “open source” Twitter’s algorithm. Like Messing, she was skeptical that simply releasing code on GitHub would meaningfully increase transparency into how Twitter works.

"If you're actually interested in public oversight on something like a Twitter algorithm, then you would actually need multiple methods for oversight to happen” Hanna said.

There is one aspect of Twitter’s algorithm that the GitHub code does shed some new light on, though. Messing points to a file unearthed by data scientist Jeff Allen, which reveals a kind of “formula” for how different types of engagement are given priority by the algorithm. “If we take that at face value, a fav (twitter like) is worth half a retweet,” Messing writes. “A reply is worth 27 retweets, and a reply with a response from a tweet’s author is worth a whopping 75 retweets.”

While that’s somewhat revealing, it’s, once again, an incomplete picture of what’s actually happening. “It doesn't mean that much without the actual data,” Messing says. “And Musk just made data so insanely expensive for academics to get. If they want to actually study this now, you basically have to get a giant, massive grants — half a million dollars a year — to get a meaningful amount of data to study what's happening.”