(Photo Credit: Keith Allison via Flickr Creative Commons)
Fundamental changes in the Major League Baseball amateur draft in 2012 have made it unique among entry drafts in professional sports. Due to escalating signing bonuses, MLB—via collective bargaining—changed the rules to assign each team a limited pool of money to sign players based on the number and position of each team’s picks in the first 10 rounds; higher and more picks earn a team more bonus money. While Major League Baseball recommends how much a player should be paid at a specific pick—or “slot”—it is left to a team’s discretion how to allocate their total bonus pool across the entirety of their draft class.
Complicating the situation is that prospects in the MLB Draft have leverage that their counterparts in the NFL and NBA do not have: they can head to, or return to, their respective collegiate teams if the signing bonus they are offered does not meet their demands. This gives prospects considerable leverage in negotiations and also allows them—and their agents advisors—to manipulate where they are drafted based on their pre-draft signing bonus demands and conversations with interested teams.
Given these dynamic—and economically fascinating—elements of the MLB Draft, teams have employed varying strategies in allocating their bonus pool since 2012. Six drafts later, one can now begin to look back at each of these strategies and evaluate their effectiveness. As a result, the research question in this study is as follows:
How does a MLB team’s bonus pool allocation strategy affect the quality of its draft class?
In an ideal world, the proper evaluation of a draft class would come 15-20 years afterwards when one simply adds up each player’s career performance (such as WAR) and weighs that against a team’s draft position and bonus pool amount. This approach, of course, is unavailable given that these rule changes were only put in place in 2012. However, even just a few years after a draft, one does have some perspective on the quality of a draft class. For this study, this perspective will be offered by prospect evaluations published by Baseball America three years after each draft. If a draftee develops into a highly-regarded minor-league prospect—or has already established himself as a major-league regular—at this point in their career, then this would retrospectively qualify as a successful selection three years after their selection in the MLB Draft.
(If pressed for time or attention, skip ahead to the “Conclusion & Caveats” section.)
In order to effectively evaluate the research question, this study must both (a) identify specific draft strategies and (b) develop ways to empirically measure the quality of a draft class. This article will categorize draft strategies on the basis of how a MLB team approaches the signing of its top selection in each draft. To those ends, this study advances three possible strategies based on the player’s signing bonus when compared to the assigned slot value at that respective pick number:
- Overslot Strategy: Signing bonus is 105% or more of the assigned slot value.
- Nearslot Strategy: Signing bonus is between 95% and 105% of the assigned slot value.
- Underslot Strategy: Signing bonus is 95% or less of the assigned slot value.
In the five MLB drafts between 2012 and 2016, teams have most often employed a nearslot strategy with their top pick (81 of 150 drafts), followed by underslot (46) and overslot (23).
While defining draft strategies is straight-forward, the development of empirical measures to evaluate the quality of a draft class is more complicated. This study uses two approaches, each drawn from the Baseball America Prospect Handbook. Published annually since 2001, these books represent the industry standard as the definitive source of information on baseball’s minor-league prospects. These volumes have stayed largely consistent in style and form since first released 16 years ago, allowing for their use in longitudinal empirical studies of baseball prospects.
This study will use two elements included in these handbooks. First, Baseball America provides a section that summarizes each MLB club’s drafting habits from the previous four years. This includes a subjective letter grade assigned to summarize the caliber of a team’s annual draft between two and four years prior to the date of the handbook; for example, the 2017 edition of the book provides grades for each team’s 2013 through 2015 drafts. By also adding the evaluation of the 2012 draft as published in the 2016 handbook, this study can compile four drafts worth (2012-15) of grades; these grades are then converted into standard academic grade values (A=4.0, B=3.0, etc.) and can be used in an empirical analysis of draft strategies in each of those four years.
While these letter grades may provide a simple one-number snapshot of a draft’s quality, these grades offer a limited—and entirely subjective—evaluation of a draft class. Drafts of similar grades may be completely different; it is possible that a “B” draft is attributable to either acquiring one great prospect or multiple good prospects. As a result, this study complements these grades by also including measures of the quality and quantity of good prospects that each team selects and signs from their respective draft class.
To evaluate individual prospects, this study uses the Prospect Handbook published three years after the draft is completed (i.e., the 2017 Handbook is used to evaluate 2014 draftees); while a longer time horizon might provide more accurate evaluations of a draft class, the choice of a three-year window was seen as best balancing competing demands for hindsight and sample size. In each team’s top 30 prospect list, Baseball America assigns each player a one-number score to capture their “realistic ceiling”. This number is on the standard baseball 20-to-80 scale, with 50 representing a “solid-average” big-league regular, 60 denoting an “occasional all-star” and so on (full description here). BA also provides an evaluation of the level of risk attached to a player reaching this ceiling. In the 2017 edition, this is represented with four possible levels of risk—low, medium, high and extreme—with the list respectively indicating decreasing probabilities of a player reaching this ceiling.
In podcasts and articles, Baseball America’s authors and editors have repeatedly articulated that, when evaluating a prospect’s value, they equate five points of ceiling (e.g., from 50 to 45) to a one-level change in risk (e.g., from high to medium). In reviewing BA’s handbooks over the years, they have consistently relied on this equation in ranking players in similar tiers; in other words, a particular tier of players in a team’s top 30 ranking is likely to feature an assortment of 45/Medium, 50/High and 55/Extreme prospects listed in consecutive order before BA drops down to a lower tier of prospects (e.g., those evaluated as 45/High or 50/Extreme).
This ceiling-to-risk equation is particularly useful in that it allows this study to categorize prospects into specific tiers of value. By counting the number of prospects in each value tier, this study has the ability to evaluate the quality and quantity of players that a team selects and signs from each draft class. This article proposes three tiers:
- Good Prospects: Players with a minimum score of 40/Low, 45/Medium, 50/High or 55/Extreme.
- Great Prospects: Players with a minimum score of 50/Low, 55/Medium, 60/High or 65/Extreme.
- Elite Prospects: Players with a minimum score of 60/Low, 65/Medium, 70/High or 75/Extreme.
The strength of MLB farm systems vary wildly, as some clubs may have 15-20 good prospects whereas others may only have a handful. But no team—with the possible exception of the 2017 Cardinals—had more than 30 good prospects over the time period to be studied.1 Given BA’s consistent approach to tiers and rankings over time, this ensures that this study has a complete count of the number of players who qualify for each of the three categories above. Note that players double-count across categories, as an elite prospect is counted as being included in all three classifications whereas a prospect that tops out as “good” is only included in the first category.
For completion’s sake, it should be noted that some draftees had already established themselves in the Major Leagues—and thus exhausted their prospect eligibility—three years after their draft selection. In those cases, prior years’ handbooks were consulted to provide the player’s last BA prospect grade. While this produced reasonable outcomes for most players—such as Kris Bryant being deemed an elite prospect and relievers Mike Morin and Kyle Crockett categorized merely as good prospects—it notably short-changed six players: Michael Wacha, Kyle Schwarber, Carlos Rodon, Aaron Nola, Michael Conforto and Marcus Stroman. In reviewing their career trajectories and MLB impact, each were reclassified as elite prospects for this study given where they would have likely been scored had they remained eligible for inclusion in the Prospect Handbook three years after the draft. Finally, note that players were assigned to the team that drafted them and not to team to which they were presented in Baseball America’s top 30 prospect lists.
With measures of a team’s draft strategy and the quality of player outcomes in place, this model proposes a basic regression model as follows:
Draft Outcome = f(Team Strategy, Pick Number of Team’s Highest Selection, Number of First-Round Picks)
The inclusion of the latter two variables is important and necessary, as a team’s place in the draft order and their number of first-round picks—including compensatory and competitive balance selections—control for each club’s level of resources available in the draft. A team’s position in the draft is particularly important in this analysis, as most clubs in the first 3-5 picks have employed an underslot strategy to considerable savings; as a result, it is necessary to control for draft pick position when evaluating the effectiveness of team strategy.
DATA & RESULTS
Table 1 (below) provides a summary perspective of the data analyzed in this study. The top panel of the table provides the grade point average for MLB teams based on the strategy they employed with their highest selection; teams that failed to sign their top pick are excluded from the analysis. In the four drafts between 2012 and 2015, the results indicate that Baseball America’s retrospective draft grades are nearly identical for each of the three strategies. While the overslot strategy may have a slightly higher GPA (2.54), the small sample of observations and minute disparities between the three groups fail to indicate any statistically significant differences between the three strategies (F=0.21).
The bottom panel of Table 1 presents the average number of prospects, by tier, which was selected and signed using each of the three draft strategies between 2012 and 2014. A couple of outcomes stand out. First, teams that employ an underslot strategy end up with a larger number (average=4.00) of good prospects when compared to teams using a nearslot (2.92) and overslot (2.38) approach. This difference is statistically significant with greater than 99% confidence (F=5.79). Second, teams that employ underslot and overslot strategies have higher rates of procuring an elite prospect from a draft class when compared to teams who use a nearslot approach, however the differences are small and not statistically significant (F=0.75).
While initially insightful, the findings in Table 1 ignore the potential influence of differences in teams’ initial draft capital derived from initial draft position and the number of first-round picks available to each club. Table 2 (below) therefore presents the results of the regression model as it used to predict the determinants of Baseball America’s grades for each team’s 2012-15 draft classes. The results demonstrate that the draft strategy that a team employs has no statistically significant effect on their draft grade years later. Instead, the only variable that has a statistically significant impact on a team’s draft grade is its number of first-round picks. The coefficient on this variable indicates that each additional first-round pick improves a team’s retrospective draft grade, on average, by 0.339; this roughly equates to a one-grade improvement (e.g., a B to a B+). The coefficient on the position of a team’s highest draft selection is predictably negative—indicating that higher pick numbers are associated with lower grades—but the effect falls just outside the bounds of statistical significance (p=0.128); it is possible that another year’s worth of data could nudge the effect into a significant range.
Turning to the number of prospects by tier that a team procures from a draft, the results from Table 3 (below) reveal a fascinating—but expected—story. Holding a team’s draft capital constant, the regression results indicate that MLB clubs that utilize an underslot strategy with their first pick are likely to acquire a higher number of good prospects from their draft class when compared to overslot and nearslot approaches; the effect is statistically significant with 95% confidence. In contrast, MLB clubs that employ an overslot strategy have an increased probability of procuring an elite prospect from their draft class when compared to an underslot approach; this effect is also statistically significant with 95% confidence.
Examining the regression models in Table 3 more closely, the results in the first column demonstrate that every variable in the model has predictive power in determining how many good prospects that a team is able to acquire in the draft; the coefficients also meet a priori expectations for direction and magnitude. Teams that pick earlier in the draft (i.e., lower numbers) procure more good prospects, all else equal. Further, each additional first-round pick is associated with an additional 0.637 good prospects; given the three-year developmental rate of first-round picks, that ratio seems appropriate. Finally, the variables on draft strategy indicate that teams employing an underslot approach procure 1.124 more good prospects compared to overslot teams, holding all else equal. Further, underslot strategies produce 0.730 more of these prospects when compared to nearslot clubs. Again, all effects are statistically significant with at least 95% confidence.
Turning to the middle columns of Table 3, the regression results indicate that MLB teams’ draft strategies have no significant effect on their ability to procure great prospects from a particular draft class. The positive coefficients on overslot and nearslot strategies may indicate some positive correlation between these approaches and the selection of such prospects (when compared to underslot), but the effects are not statistically significant at any reasonable threshold (p>0.5 for both variables). In other words, any relationship appears to be just noise at this point. That said, the other two variables in the model—draft position and the number of first-round picks—remain statistically significant and maintain coefficients that meet expectations for direction and magnitude.
The final columns of Table 3 reflect the determinants of an MLB team’s ability to procure an elite prospect from its draft class. Unsurprisingly, the most powerful predictor of this outcome is the position of a team’s highest draft selection; a higher pick (i.e., lower number) is associated with an increased probability of selecting an elite talent. This effect is statistically significant with greater than 99% confidence. Additional first-round picks have no estimated effect on producing an elite talent, another unsurprising outcome given that additional picks are typically toward the end of the first round. As mentioned above, the results also indicate that teams employing an overslot draft strategy produce 0.290 more elite prospects than clubs using an underslot approach.
While the magnitude of the overslot effect on the procurement of elite prospects was initially surprising, a deeper dive into the data reflects that this impact is largely driven by three elite prospects, all signed to above-slot bonuses between picks #11-20: Lucas Giolito (#16, 2012), Corey Seager (#18, 2012) and Trea Turner (#13, 2014).2 In the two subsequent MLB Drafts (2015-16) not included in this analysis, there have been five prospects signed to overslot deals in a similar range: Kolby Allard (#14, 2015), Brady Aiken (#17, 2015), Jay Groome (#12, 2016), Forrest Whitley (#17, 2016), and Blake Rutherford (#18, 2016). Given that the current study employs a ridiculously small sample—three drafts—it should be cautioned that the sustainability of the magnitude and statistical significance of the overslot effect will be largely dependent on the level of success attained by these five players moving forward.
The analysis to this point has evaluated the effect of whether a team has employed an overslot, nearslot or underslot approach in signing its top player. But does the amount of dollars gained or lost in a player’s signing bonus relative to his assigned slot value have any effect? To evaluate this question, Table 4 (below) re-estimates the previous three models but replaces the three draft strategy variables with a single variable that captures the difference between a player’s bonus and slot value; positive values indicate that a team went overslot with their top selection.
As presented in the first column of Table 4, the results of this revised regression equation indicate that every additional $100,000 paid to a team’s highest pick—holding the club’s draft position constant—lowers the number of good prospects a club procures in a draft by 0.100. This effect is statistically significant with 99% confidence. Multiplying the results by a factor of 10, the results can be interpreted to suggest that a team can add the equivalent of one additional good prospect—as evaluated three years later—with every additional $1 million spent in the draft under current rules. Given perspective on the developmental rate of prospects, this magnitude meets a priori expectations. Beyond the net gain/loss variable, the results indicate that every additional first-round pick is expected to increase a team’s haul of good prospects by 0.678; this outcome again meets expectations and the relationship is statistically significant with 99% confidence. The coefficient on the pick position of a team’s top selection continues to be negative, but falls just outside the bounds of statistical significance (p=0.103).
The middle columns of Table 4 continue to demonstrate that draft strategy has little effect on the ability of a club to procure a great prospect from the draft, as the coefficient on the bonus variable is not statistically significant at any reasonable threshold (p=0.754). The coefficients on the other two variables in the model are largely unchanged from what was presented in Table 3; while the coefficient on a team’s pick number is no longer significant with 90% confidence, it rests just outside this statistical boundary (p=0.103).
The final columns of Table 4 fail to demonstrate that the amount of a player’s signing bonus relative to the assigned slot value has any statistically significant effect on the probability of procuring an elite talent from the draft. That said, the positive value of the coefficient does indicate some correlation between paying a higher bonus relative to slot and the odds of acquiring an elite prospect, but the effect is not statistically significant (p=0.392). Just as was discovered in Table 3, the predominant factor predicting a team’s ability to acquire an elite prospect from the draft continues to be its position in the draft order.
CONCLUSIONS & CAVEATS
The goal of this study was to evaluate the effectiveness of MLB teams’ strategies in how they allocate their bonus pool money in baseball’s amateur draft. Analyzing drafts between 2012 and 2014 in the second half of the analysis, the results of this study are astoundingly clear even with the small sample size:
- MLB teams utilizing an underslot strategy with their highest draft selection procure more “good” prospects from a draft class.
Holding the level of a team’s draft capital constant, this study indicates that teams employing an underslot strategy with its highest selection procure 1.124 additional good prospects compared to an overslot approach and 0.730 more good prospects versus a nearslot strategy. Further analysis suggests that every $1 million saved from a signing bonus on a team’s top pick is expected to net that team one additional good prospect. All effects were demonstrated to be statistically significant with at least 95% confidence.
- MLB teams that use an overslot strategy with their highest draft selection have an increased probability of landing an “elite” player or prospect.
The primary determinant of a team procuring an elite talent from the draft is clearly its position in the draft order. But holding a team’s draft position constant, there is some evidence indicating that an overslot approach does increase the odds of finding a top-end prospect in the draft. The evidence is somewhat weaker and is largely dependent on a small number of overslot successes from teams making picks between #11 and #20 between 2012 and 2014.
On the surface, these two conclusions should serve to simply confirm common sense. After all, a team would likely employ an underslot strategy with its first pick in order to increase the odds of landing well-regarded prospects later in the draft. And teams that spend big on their first selection relative to the slot value at the pick are likely doing so because they believe that their choice has a disproportionate probability of developing into an elite prospect vis-à-vis the other players available at that pick.
While these are reasonable outcomes, what is particularly surprising is how quickly these patterns have emerged in statistically significant ways over the course of just three drafts. Given the highly variable nature of the MLB draft and minor-league player development, it was expected that any evidence of these effects might be marginal and would fail to be statistically significant with such a small sample size (n=87). Instead, the relationship between underslot drafts and the procurement of additional “good” prospects is astoundingly strong and provide estimated magnitudes that largely match a priori expectations of draft outcomes. The estimated effects presented in this study were consistent and robust across a variety of samples and model specifications.3
To recap, this study shows that an underslot strategy allows a team to procure more good prospects while an overslot strategy increases the probability that a team lands an elite talent. So which approach is better? Or should a team just play it down the middle and pay signing bonuses that match a pick’s value? To this point, there’s no clear answer to those questions. While this study attempted to address this question by looking at Baseball America’s draft grades, the results failed to find any statistically distinguishable difference in draft outcomes according to a team’s strategy with its highest selection. These answers may become clearer with more drafts and a longer time-horizon for drafted prospects.
Before concluding, a number of caveats must be addressed. First, while it may be a common mantra, one must be careful not to confuse correlation with causality. While the study finds that teams employing an underslot strategy are likely to acquire more good prospects in a draft, it could be that underslot teams are more advanced in their scouting and/or analytics. As a result, it may be that those teams’ relative success in later rounds in the draft has nothing to do with the strategy employed with their highest selection.
The second statistical caveat is that this analysis only encompasses a few drafts, thereby suffering from a small sample of observations. The inclusion of additional drafts will likely alter the relationships discovered in this analysis one way or another. As discussed earlier in the paper, the overslot advantage in procuring elite prospects was largely dependent on three players; the sustainability of this advantage over time similarly falls on the shoulders of a limited number of overslot signings in the last two drafts.
As a final caveat, the outcome variables used in this study—draft grades and prospect evaluations—are subjective. While Baseball America is the long-time industry standard for prospect information, BA staffers consult scouts, coaches and front office personnel in attaching ceiling and risk scores to individual players. It is possible that their evaluations—and that of the BA writers—could be influenced by a player’s draft position and signing bonus, thereby affecting the results of this study. I would not expect this bias to be large three years after a player was drafted, but it does reflect the benefit of more objective measures that may only be possible with a longer time horizon (i.e., a player’s career WAR).
1 – The ceiling and risk scores for the St. Louis Cardinals in the 2017 Prospect Handbook seemingly represent an anomaly when compared to other teams and past years. Of the 30 prospects in the system, nine players are graded as 60/Highs and an obscene 19 are evaluated as 50/Highs; every other set of team evaluations include a diverse set of ceiling and risk scores. This latter group of 50/Highs for the Cardinals includes clearly dissimilar players, with the scouting report for player #29 (2B Breyvic Valera) indicating a 25-year old utility infielder who hit .341 in AAA and player #30 (P Johan Oviedo) describing an 18-year old “impact-starter” who has yet to play a professional game in the United States. In every other year and every other system, these two players would likely have dissimilar ceiling and risk grades. Regardless of this inconsistency, the 2017 Cardinals’ rankings (i.e., from their 2014 draft) are nevertheless included in the analysis since it is possible that all of these 50/Highs might represent an equivalent score (i.e., 45/Medium or 55/Extreme).
2 – This result also reflects some weakness on how this paper defines a club’s draft strategy and its relevance for all of its picks. The only club between 2012 and 2014 to emerge with an elite prospect while using an underslot approach was the Toronto Blue Jays. In their 2012 draft, they selected D.J. Davis with their top overall selection (#17) and signed him to an underslot bonus. Five picks later, they drafted Marcus Stroman—identified as an elite prospect in this study—but signed him to a bonus that exactly equaled the slot value at that pick. Thus, while the Blue Jays were categorized as an “underslot” team, it is not clear that their strategy with Davis had any influence on their position with Stroman.
3 – Alternative specifications of the regression models were also evaluated. First, the models in Tables 3 and 4 were re-estimated on a limited sample that only included teams whose highest pick was between pick #6 and pick #50 in order to eliminate concerns about the standard use of underslot strategies by teams in the top five. But this revised sample produced largely similar outcomes as the full sample. Second, alternative models were estimated to indicate whether a team’s first pick was a hitter or pitcher and whether he came from the high school or college ranks. While there was weak evidence suggesting that high school hitters were better bets than high school pitchers, the inclusion of these variables had little to no effect on the other coefficients in the model (e.g., draft strategy). Given concerns about overfitting the model in a small sample, these variables were ultimately excluded from the final analysis.