2015 FRIAS Pitching Projections Review

Posted By Spencer Estey on 7:17am Oct 23, 2015 | 0 comments

(Title photo courtesy of Keith Allison, https://www.flickr.com/photos/keithallison/)

Back in March, I decided that it would be interesting to have a pitching projection system that only used one season of data to project the next season and so I created FRIAS, which projects the three basic DIPS components (walks, strikeouts, home runs) and folds them into a projected FIP. I thought that this was an interesting idea for a few reasons. The first is as a reaction to the sheer number of breakout pitchers that 2014 brought: Jake deGrom, Carlos Carrasco, Dallas Keuchel, Colin McHugh, Jake Arrieta and others came out of nowhere to be front-end starting pitchers. This year, all have kept it up. There is something about pitchers that makes mediocre ones more liable to break out than hitters. For these breakout pitchers, knowing their pre-2014 seasons and weighing them with any heft wouldn’t really have worked. They changed and that change was very evident during 2014. Perhaps looking at all pitchers from a one-year perspective is useful. The changes can be evident because many talent-relevant pitching statistics, such as fastball velocity, swinging strike rate or ground ball rate stabilize to an acceptable level fairly quickly, within a season even for reliever pitchers.

Another reason that I like the concept is that you can tell more clearly how much a skill a player displayed in the season in question. For popular projection systems like Steamer and ZiPS, it’s hard to tell exactly how much each year is affecting the projection or how a new season with certain inputs with affect future projections. With FRIAS we only have one season so the weight is clear. It’s phrased as a projection but also provides a look at the player’s skill in the sampled season and can be viewed as a peripheral indicator in the same way that xFIP is.

Anyway, while the postseason is still ongoing, the 2015 regular season has concluded. So I wanted to compare FRIAS with popular public systems, namely Steamer, ZiPS and PECOTA (and Fangraphs Depth Charts, which averages ZiPS and Steamer). I want to do this in two ways: first, by looking at individual pitchers and seeing if there are specific types of pitchers that FRIAS tended to over- or under-project. Second, on a overall accuracy level.

2015 FRIAS values (based on 2014 season) can be viewed in full from the original post. I grabbed Steamer, ZiPS and Depth Charts projections from Fangraphs a few days ago, before 2016 projections replaced them. PECOTA projections were taken from the Baseball Prospectus PECOTA spreadsheet, which was last updated on April 4th. Pitchers who pitched 20 or more innings in 2014 (and therefore have FRIAS ratings).

To compare the accuracy of the pitching projections, I adjusted each system very slightly so that they’d have the same weighted averages. This is because it’s really how players compare to other players that we care about. The systems don’t know ahead of time exactly what the league’s run environment will be but that’s okay because the differences between players that they forecast are what’s relevant.

Individual Pitcher Comparison

To review FRIAS’s effectiveness when compared to other systems when it comes to individual players, I needed to come up with a number that describes what other systems collectively think of a player. This number will be the average of Steamer, ZiPS and PECOTA.

Here’s what happened for players with 120 innings pitched. The x-axis is the difference between actual FIP and FRIAS projected FIP. The y-axis is the same with the consensus projections. A positive value indicates that the pitcher under-performed their projection; a negative value the opposite. The player names are scrambled in the middle of the plot but that’s fine because the stories behind those players are less noteable. The middle players were projected decently by both systems.

Many pitchers who over- or under-performed on one of the axes ended up doing the same in the other, which gives this graph its shape. The projection systems use different data and methodology but the outputs end up being similar. Jake Arrieta over-performed FRIAS by nearly a run and the consensus projection by nearly a run and a half. However, this graph doesn’t make that clear which players FRIAS “hit” on in comparison to the other systems. So I prepared another plot.

This time the x-axis is the FRIAS projection so we can tell which players were supposed to be good or bad. The y-axis is the difference between FRIAS error and consensus projection error. Error is the absolute distance from the actual 2015 value and positives on this axis denote players who FRIAS did worse than the consensus systems in projecting.

Mike Pelfrey is by himself again. All the numbers entering 2015 said he was bad and that the Twins were foolish to trust him, but he somehow turned in a somewhat competent system. FRIAS had him projected on an island, at nearly a 6.00 FIP and that made him the system’s biggest miss of the season. It’s interesting to see which pitchers FRIAS out-performed the other projections on. Some of these guys are the 2014 breakout stories (Carrasco, Arrieta) and others are pitchers who didn’t make that many starts in 2014 but posted surprisingly good peripherals (DeSclafani, Pineda).

Adam Warren was an effective Yankees reliever who was moved back to the rotation in 2015 and was very good, much better than expected. FRIAS may not have actually foreseen his 2015 success as much as it believed him to be a relief pitcher. On the other end of the scale are veterans who don’t really fit together in any way. FRIAS liked Jeff Samardzija much more than consensus and that blew up. It thought Max Scherzer wasn’t quite an elite pitcher but he seems to be and Steamer, ZiPS and PECOTA got that right. Chris Young was poorly projected all around, even though we’re talking about FIP and not ERA.

Overall Accuracy Comparison

Here are the results for overall accuracy. I calculated this in terms of two statistics: mean absolute error (MAE) and root mean square error (RMSE), and also in terms of four minimum innings thresholds.

From this, we can see that Steamer led the public projection systems although combining it with ZiPS (expressed as Depth Charts) provided gains for some of the innings samples. FRIAS was right there too and did better with the larger samples. It lagged Steamer and ZiPS when pitchers with fewer than 50 innings were considered but did better than ZiPS in the samples of seasons of 50+ and 120+ innings. PECOTA did worse than the other systems across the board, except when compared to ZiPS for seasons of 120+ innings.

I think that this demonstrates that FRIAS has a level of accuracy that can make it a useful tool. The 2015 results track pretty close with the 2014 test I did in the initial article. I was mindful of not including too many explanatory variables because of the dangers of over-fitting; that the system holds up in out-of-sample tests is encouraging.