The following story was originally published at the UCLA Gaming Law Association website and is being republished with permission.
DraftKings and FanDuel were recently in court to fight an injunction sought by New York Attorney General Eric Schneiderman that would ban daily fantasy sports companies from operating in New York state. In advance of the hearing, the companies each released statistical evidence designed to prove their case that daily fantasy sports is not gambling.
My discussion will critique their statistical analysis and consider alternative statistical tests which may better support their argument. This post will focus on FanDuel’s data.
Why it matters
Schneiderman claims that daily fantasy sports fits the definition of “gambling” under New York state law, while the daily fantasy companies have consistently argued that their contests are “games of skill” and therefore not gambling. This case is crucial for FanDuel and DraftKings because it will set a precedent for other states that seek to challenge them in court. Here’s the relevant section of NY state law that is at issue:
New York gambling laws
- “Contest of chance” means any contest, game, gaming scheme or gaming device in which the outcome depends in a material degree upon an element of chance, notwithstanding that skill of the contestants may also be a factor therein.
- “Gambling.” A person engages in gambling when he stakes or risks something of value upon the outcome of a contest of chance or a future contingent event not under his control or influence, upon an agreement or understanding that he will receive something of value in the event of a certain outcome.
(http://ypdcrime.com/penal.law/article225.htm#p225.10)
Notes on the law
1) Under the NY law, daily fantasy sports can be a game of skill but still be gambling if winners are determined by a material degree of chance. New York courts have no recent rulings defining how much chance is material, but the court may choose to follow one of two opposing precedents.
In Missouri, courts have held that chance can be a material element without being the dominant element and that a game can be more than 50% skill-based but still involve a material degree of chance. See Thole v. Westfall, 682 S.W.2d 33, 37, n. 8 (Mo. 1984) (Holding that under Missouri’s material element test, “chance must be a material element in determining the outcome of a gambling game,” but “it need not be the dominant element”).
Alternatively, the court could follow a 1904 New York case that defined material degree of chance as the dominant element, the controlling factor in a game. People ex rel. Ellison v. Lavin, 71 N.E. 753, 755 (N.Y. 1904) (holding that to be considered gambling, a lottery need not be determined exclusively by chance). The legal briefs presented by DraftKings and FanDuel both appear to be using the Missouri standard in their reasoning, but on Wednesday their attorneys reportedly introduced the New York standard in court.
2) Unlike lotteries or casino-based games like blackjack or poker (where the probability of the appearance of a particular card is quantifiable), there are a number of external, non-quantifiable factors in daily fantasy that theoretically increase the degree of chance involved. These include injury, weather (in MLB/NFL contests), game situation, play-calling, and refereeing. Top daily fantasy players research these factors, write algorithms to control for them, and enter hundreds/thousands of different lineup combos to take advantage of market inefficiencies. But once the game starts, he/she is still at the mercy of these factors. There is little precedent for this type of contest except traditional sports betting.
3) When determining whether poker was a game of skill, courts analyzed the degree of chance involved in winning a single poker hand, as opposed to the influence of chance over a longer period of time. Joker Club, L.L.C. v. Hardin, 643 S.E.2d 626 (N.C. Ct. App. 2007). If the court applies the same reasoning to DraftKings and FanDuel, it will analyze the performance of a single lineup in a single contest, instead of analyzing contestants’ performance over thousands of contests.
This approach may hurt their case. In a single contest, the advantage of a skilled player over an unskilled appears to be minimal (FanDuel’s brief notes that that the top 10% of players score more points than the bottom 10% of players 59% of the time). A percentage that close to 50% indicates there is a significant degree of chance involved in a skilled daily fantasy player beating an unskilled player in a single contest.
Critiquing FanDuel’s statistical analysis
In its brief, FanDuel presented the results of several tests of its contest data designed to prove that chance is not a material element in determining the winner of a FanDuel contest. I have identified what I believe to be flaws with each of the models and suggest some alternatives FanDuel could present to better support their argument. Links to the raw data are here: FanDuel Prize Distribution Chart and FanDuel Brief.
1) FanDuel Prize Distribution (Exhibit 3)
What is it: FanDuel released data that showed about 50% of the prize money is won by 1% of the winners. In a game dominated by chance, you would expect a more equal distribution of prize money over time.
What’s wrong with it: The data also included the average number of entries per week and the top 1% of winners predominantly fell in the “500+ entries” category while the bottom 1% of winners all fell in the “25 or fewer entries” category. This suggests that one reason top players win more money is simply because they enter significantly more contests.
Alternative test: Calculating the total prize winnings/total number of entries for both skilled and unskilled players would generate the expected return on investment for both groups on a per-entry basis. If skilled players are winning significantly more money per entry, that would be better support for FanDuel’s argument.
2) Randomly-selected lineup vs. Average FanDuel user lineup (Pages 5-9)
What is it: FanDuel looked at the performance of lineups created completely at random (with no salary cap floor) and compared them to the performance of the average FanDuel user lineup across NFL, NBA, NHL, and MLB contests.
What’s wrong with it: Because the simulated lineups were selected randomly and without a salary floor, that means they would include low-priced reserve players who would not have been drafted by actual FanDuel users. Not surprisingly, the average FanDuel user lineups won most of the time. But all this proves is that even the worst FanDuel contestant is smart enough to avoid drafting reserves who don’t see playing time.
This conclusion is supported by the fact that FanDuel MLB and NHL users performed significantly worse than NBA users against the random lineups. In MLB & NHL contests, there are far fewer “scrub” options to choose from since the vast majority of draft-eligible players actually get playing time in the real games. In MLB contests, for example, starting pitchers who aren’t pitching that day are not draft-eligible. In NHL, even the worst player at the end of the bench is still going to play and be draft-eligible. Since NHL and MLB contests include a small percentage of draft-eligible players who are unlikely to ever be drafted by real users, the randomly selected lineups should look more like an actual FanDuel user lineup.
By contrast, NBA contests list all players on an NBA team’s 12-man roster as draft-eligible. But in a typical NBA game, only 6 or 7 of the 12 players on a roster will see significant playing time and 3 or 4 often won’t play at all. Given that, about half of draft-eligible NBA players are players you should never actually draft. Since the random lineups are drafting from this pool of players but FanDuel users are not, the random NBA lineups are going to be far worse than an actual FanDuel contestant’s lineups.
Alternative test: Compare the average point totals for the best FanDuel players against the worst players over time and across sports to show that there is a statistically significant difference that cannot be explained by luck or chance. It will be a more relevant analysis since FanDuel users don’t compete against computers or “the house,” but only against each other.
3) Randomly-selected lineup (with certain conditions) vs. Average FanDuel user lineup (Pages 7-9)
What is it: The same as #2 except now the randomly generated lineups had to use at least 85% of the salary cap and certain dollar values had to be used on certain positions.
What’s wrong with it: Imposing the 85% floor means the simulated lineups are are more analogous to actual lineups but their production is still artificially deflated by the 85% figure. While it’s hard to know what percentage of the salary cap the average FanDuel player uses, it’s safe to say that it’s close to 100%, putting the simulated lineups at a significant spending disadvantage. Even with these advantages, the FanDuel users only won 73% and 68% of the time in MLB and NHL contests, respectively. Those percentages would seem to hurt FanDuel’s case. Without the artificial advantage of the 85% floor, it’s logical that the winning percentage would be substantially closer to 50/50.
4) In multiple-entry tournaments, tracking the consistency of user performance using “best lineups” (Pages 13-15)
What is it: To show that user-performance is skill-based because it is relatively consistent over time, FanDuel tracked the consistency of lineup performance over time, as defined by its R value.
What’s wrong with it: FanDuel reasoned that it should only look at a user’s “best lineup” in any given multiple-entry tournament because that would be the best way to track success over time. The problem with that approach is top FanDuel players regularly enter hundreds or thousands of lineups into these multi-entry contests. The reason the users do this is to effectively minimize the element of chance through risk allocation. By only-looking at the player’s best lineups, the resultant R value is artificially high because it’s selecting from a non-representative and very limited data set.
Alternative test: Track the consistency of performance over time for users who only enter 1 lineup per contest into these multi-entry tournaments. If those point values are relatively consistent over time, then FanDuel has a stronger argument against chance because you’d expect a wider, more random distribution of the point totals in a game dominated by chance.
5) Comparing the consistency of those “best-lineups” to the consistency of performance from actual pro sports teams (Page 15)
What’s wrong with it: In general, professional athletes are competing against equally skilled players, so the element of chance should be expected to play a greater role in determining the outcome of a game. On FanDuel, the best players are often competing in tournaments with people who have never played before, so their skill plays a greater factor than in a contest between two equally skilled players.
Alternative test: See #4.
6) Calculating the probability that a top FanDuel user would beat an unskilled player over a 7-game series or a 162-game baseball season (Page 21)
What is it: In response to the data from the Attorney General of New York that showed the best FanDuel players beat the worst players 59% of the time, FanDuel re-calculated the probability of the skilled player beating the unskilled player over 7-contest and 162-contest periods resulting in percentages that supported its position.
What’s wrong with it: It’s completely irrelevant since there are no 7-game or 162-game contests on FanDuel or any other daily fantasy site. As mentioned above, the court will likely analyze the degree of chance involved in a player beating another player in a single contest. In a game determined completely by chance, the expected winning percentage of players would be 50/50. In a game determined predominantly by skill, chess for example, a grandmaster would be expected to beat a novice player close to 100% of the time. A 59% winning percentage suggests there is some skill involved in daily fantasy sports, but also a significant degree of chance.
Alternative test: Break up users into four quartiles based on winning percentage (i.e. top 25%, next 25%, 3rd 25%, and the bottom 25% of players). Then have the players in each of the four quartiles compete both against their own group and the 3 other groups over a statistically significant number of contests. Track each quartile’s performance against its own group of equally skilled players to set a baseline winning percentage. In a game dominated by skill, this number should still be close to 50% because equally skilled players are playing against each other.
Then track each quartile’s performance against all three other groups. In a game dominated by skill, a better quartile competing against a lower one should have a winning percentage significantly higher than 50%, and vice versa. The winning percentage should increase as the variance in skill increases so the 2nd quartile vs. 3rdquartile matchup should be closer to 50% than the 1st quartile vs. 4th quartile matchup.
If the results bear this out, then that would be a significant indication that chance is not a material element in determining the outcome of daily fantasy sports contests. Intuitively, this is how we know that real sports are not materially affected by chance.
