For the past several weeks, my Twitter (X) feed, which normally seems to be contaminated with Elon Musk memes and other right-wing content, has been overrun with videos “proving” that the referees have been rigging NFL games in favor of the Kansas City Chiefs. Many videos show opposing players who appear to lightly bump the Chiefs’s quarterback Patrick Mahomes and are then immediately penalized. This question of the referees’ supposedly dubious motivations had gotten so intense (before the Chiefs’s obliteration in the Super Bowl put many of these conspiracies to bed) that it was genuinely difficult to find someone discussing football without mentioning it. Given this moment of heightened paranoia about cheating, I think it might be fitting to revisit one of if not the most dramatic, widespread, and systematic cheating scandals in the history of American professional sports: the Houston Astros’s sign-stealing. When the story broke, it garnered massive attention and some say that its effects are still being felt now, five years later, by players and teams across the league.
As a committed baseball fan who often gets as caught up in the off-season drama of the baseball world just as much as in the excitement and storylines of the games themselves, I thought I was at least clear on the basics of the cheating scandal. However, in the same way that a closer examination into some of the supposedly egregious penalties against the Chiefs showed a more complicated story, when I began to read about the effects of the Astros’s sign-stealing, I learned something pretty surprising: the Astros seem to not have benefited nearly at all from their entire scheme.
The story of the Astros’s sign-stealing was first made public in an article published in 2019 by Ken Rosenthal and Evan Drellich in The Athletic, which immediately took the baseball world by storm. Fans of the Astros’s rivals, including the Dodgers (whom they beat in the 2017 World Series) and the Yankees (who have been consistently unable to get past the Astros in the playoffs), were quick to point out that this scandal would irreparably tarnish the legacy of the Astros, who have recently been one of baseball’s most dominant teams. Rightfully so. Rosenthal’s article and subsequent reports and investigations detail the elaborate scheme, involving people at all levels of the Astros organization – from interns to players to the General Manager. During home games, the Astros would use a camera in center field to spy on the communication between catchers and pitchers regarding which pitch to throw next. Up until 2022, when the MLB introduced the PitchCom technology allowing catchers to signal to their pitchers electronically, catchers signaled which pitch to throw with their fingers. This practice was in plain view of the Astros’s center field camera. Using a glorified spreadsheet nicknamed “The Codebreaker,” the video would then be analyzed in real-time in the Astros’s clubhouse in an attempt to decode the catchers’ signs. Finally, someone in the Astros’s clubhouse would relay the signs to the batter by banging on a trashcan.
The fact that this scheme was so simple, so consistent, and was perceived to be so effective (given that the Astros went on to win the World Series) incited outrage and scrutiny across the MLB from fans to players, many of whom had already been suspicious of the Astros. Also, given that the trash can bangs needed to be loud enough to be heard by batters on the field, they were also often audible on the video broadcast. Thus, for fans carefully listening back over each game, this often made it quite easy to know on which pitches the Astros had attempted to let the batter know what was coming. In a viral video published a few hours after Rosenthal and Drellich’s article came out, beloved baseball YouTuber Jimmy O’Brien analyzed the video of an Astros’s at-bat in 2017 in which the trash can bangs are clearly audible (two bangs for changeup, none for fastball) and the batter definitely seems like he knows what is coming. I think one of the main reasons that this cheating scandal became such a phenomenon across baseball, beside the fact that it was egregiously against the rules and that the Astros were a great and well-hated team, is that it was just so easy for fans to clearly see (and hear) exactly how the scheme worked. It became common practice for everything that the Astros accomplished in 2017, including their World Series, to be chalked up to cheating. The Yankees’s General Manager has repeatedly said that his team had been robbed of the World Series, and a pitcher who got pummeled by the Astros before being sent down to the minor leagues even sued the team, claiming they ended his career. It would make sense to believe, as I did, that knowing consistently which pitch is coming would provide any team a massive advantage, and that they should be punished accordingly by at least having their 2017 World Series title revoked. However, quite shockingly, multiple different data analysts have come to the same conclusion: the Astros likely did not benefit very much from their sign-stealing.
There are two ways to examine how much of an edge the Astros got from cheating. First, we can simply compare each player’s performance at the Astros’s home stadium with their performance on the road. Since the sign stealing was only possible using a camera specially positioned and monitored by the Astros, they were not able to cheat while playing away from their own ballpark. Given this, one would think that most of the players should have played much better at home than away. This, however, is definitively not the case. The majority of players performed about the same home or away according to their Weighted On Base Average (wOBA) which is a good metric to analyze offensive production. A few players even became worse hitters at home where the sign-stealing was taking place, while only one, Jake Marisnick, played significantly better.
But what if the banging was done more selectively? For instance, what if the Astros only relayed pitches against specific pitches or during particularly important circumstances during the game? Not all hits are created equal; so even if, overall, the Astros’s hitters did not improve home vs away, they might have only needed to cheat in specific at-bats which would have an outsize impact on the game. To fully understand the effects of the Astros’s scheme, we would need to analyze every individual pitch before banging can be heard. Luckily, a diligent Astros fan went through all available footage of Astros home games during the 2017 season and analyzed the audio, noting every time a banging sound occurred (even if it could not be heard with the naked ear). He found a consistent pattern: the banging started sometime around the second inning (presumably during the 1st inning when the Astros were working on decoding the catcher’s signals) and their usage gradually declined from then on, possibly because catchers might change their signs later in the game. The consistency of this pattern seems to rule out the possibility that the Astros were only attempting to cheat on select pitches and were thus reaping an outsize reward that was not captured in general offensive metrics.
Robert Arthur, a statistician and writer for The Athletic, dug through this massive, publicly available database and determined that “Statistically, for all the work and effort that went into the cheating scheme, the grand result of it … turned out to be no runs at all.” One of the main reasons for this, it seems, is that the cheating was not always accurate: decoding and subsequent banging had to be done within the mere seconds before a pitch was thrown so that the batter could prepare. In fact, the banging correctly predicted a fastball only 65% of the time. When correct, Arthur found a modest improvement in performance, but when the banging came too late or got it wrong, hitters did significantly worse. This is likely because they had overcommitted to a certain pitch and were unable to adjust when the trash can banging proved to have been erroneous.
Others who have analyzed the data and published their results have come to pretty much the same conclusion: The Astros have forever tarnished their legacy, but it is very unlikely that they gained any real advantage from cheating. There was relatively widespread condemnation of the admittedly quite minor punishments the Astros received for the scandal. They were fined $5 million, stripped of several draft picks, and their manager and general manager were fired, and many called for their World Series title to be revoked. While I’m not sure what the adequate punishment should be, I do think it matters that the Astros did not benefit from the scheme. Whether or not the Astros’s World Series should forever be marked with an asterisk is an open question, but it’s a question that is complicated by knowing that their cheating, according to pretty much all the evidence, did not help them win.
