TIPS, A New ERA Estimator

FIP, xFIP, and SIERA are all very good ERA estimators, and their predictability is well documented. It is well known that SIERA is the best ERA estimator over samples that occur from season to season, followed very close by xFIP, with FIP lagging behind. FIP is best at showing actual performance though, because is uses all real events (K, BB, HR). Skill is commonly best attributed to either xFIP or SIERA. ERA is also well known to be the worst metric at predicting future performance, unless the sample size is very large <500IP with the pitcher remaining in the same or a very similar pitching environment.

FIP, xFIP, and SIERA are supposed to be Defense Independent Metrics, and they are. Well, they are independent of field defense, but there is one small error in the claim of defense independent. K’s and BB’s are not completely independent of defense. Catcher pitch framing plays a role in K’s and BB’s. Catchers can be good or bad at changing balls into strikes and this affects K’s and BB’s. Umpire randomness and umpire bias also play a role in K’s and BB’s. It is unknown how much of getting umpires to call more strikes is a skill for a pitcher or not. Some pitchers are consistent at getting more strike calls (Buehrle, Janssen) or less strike calls (Dickey, Delabar), but for most pitchers it is very random (especially in small sample sizes). For example Jason Grilli was in the top 5% in 2013 but was in bottom 10% in 2012.

I wanted to come up with another ERA estimator that eliminates catcher framing, umpire randomness and bias, and eliminates defense. I took the sample of pitchers who have pitched at least 200IP since 2008 (N=410) and analyze how different statistics that meet this criteria affect ERA-. I used ERA- since it takes out park factors and adjusts for the changes in the league from year to year. I looked at the plate discipline pitchf/x numbers (O-Swing, Z-Swing, O-Contact, Z-Contact, Swing, Contact, Zone, SwStr), the six different results based off plate discipline (zone or o-zone, swing or looking, contact or miss for ZSC%, ZSM%, ZL%, OSC%, OSM%, OL%), and batted ball profiles (GB%, LD%, FB%, IFFB%). *Please note that all plate discipline data is PitchF/X data, not the the other plate discipline on FanGraphs, this is important as the values differ*

The stats with very little to absolutely no correlation (R^2<0.01) were: Z-Swing%, Zone%, OSC%, ZSC%, ZL% (was a bit surprised as this would/should be looking strike%), GB%, and FB%. These guys are obviously a no-no to include in my estimator.

The stats with little correlation (R^2<0.1) were: Swing%, LD%, and IFFB%. I shouldn’t use these either.

O-Contact% (0.17), Z-Contact%, (.302), Contact% (.319), OSM% (0.206), and ZSM% (.248) are all obviously directly related to SwStr%. SwStr% had the highest correlation (.345) out of any of these stats. There is obviously no need to include all of the sub stats when I can just use SwStr%. SwStr% will be used in my metric.

OL% (0.105) is an obvious component of O-Swing% (0.192). O-Swing had the second highest correlation of the metrics (other than the components of SwStr%). I will use it as well. The theory behind using O-Swing% is that when the batter doesn’t swing it should almost always be a ball (which is bad), but when the batter swings, there are a two outcomes, a swing and miss (which is a for sure strike) or contact. Intuitively, you could say that contact on pitches outside the zone is not as harmful to pitchers as pitches inside the zone, as the batter should get worse contact. This is partially supported in the lower R^2 for O-Contact% to Z-Contact%. It is more harmful for a pitcher to have a batter make contact on a pitch in the zone, than a pitch out of the zone. This is why O-Swing is important and I will use it.

Using just SwStr% and O-Swing%, I came up with a formula to estimate (with the help of Excel) ERA-. I ran this formula through different samples and different tests, but it just didn’t come up with the results I was looking for. The standard deviation was way too small compared to the other estimators, and the root mean square error was just not good enough for predicting future ERA-.

I did not expect/want this estimator to be more predictive than xFIP or SIERA. This is because xFIP and SIERA have more environmental impacts in them that remain fairly constant. K% is always a better predictor of future K% than any xK% that you can come up with. Same with BB% Why? Probably because the environment of catcher framing, and umpire bias remain somewhat constant. Also (just speculation) pitchers who have good control can throw a pitch well out of the zone when they are ahead in the count, just to try and get the batter to swing or to “set-up” a pitch. They would get minus points for this from O-Swing, depending on how far the pitch is off the plate, but it may not affect their K% or BB% if they come back and still strike out the batter.

So I didn’t expect my statistic to be more predictive, but the standard deviation coupled with not that great of RMSE (was still better than ERA and FIP with a min of 40IP), caused me to be unhappy with my stat.

I then started to think about if there were any stats that were only dependent on the reaction between batter an pitcher that are skill based that FanGraphs does not have readily available? I started thinking about foul balls and wondered if foul ball rates were skill based and if they were related to ERA-. I then calculated the number of foul balls that each pitcher had induced. To find this I subtracted BIP (balls in play or FB+GB+LD+BU) from contacts (Contact%*Swing%*Pitches). This gave me the number of fouls. I then calculated the rates of fouls/pitch and foul/contacts and compared these to ERA-. Foul/Contact or what I’m calling Foul%, had an R^2 of .239. That’s 2nd to only SwStr%. This got me excited, but I needed to know if Foul% is skill based and see what else it correlates with.

This article from 2008 gave me some insight into Foul%. Foul% correlates well to K% (obviously) and to BB% (negative relationship), since a foul is a strike. Foul% had some correlation to SwStr%, this is good as it means pitchers who are good at getting whiffs are also usually good at getting fouls. Foul% also had some correlation to FB% and GB%. The more fouls you give up, the more fly balls you give up (and less GB). This doesn’t matter however, as GB% and FB% had no correlation to ERA-. Foul% is also fairly repeatable year to year as evidenced in the article, so it is a skill. I will come up with a new estimator that includes Foul% as well.

I decided to use O-Looking% instead of O-Swing%, just to get a value that has a positive relationship to ERA (more O-looking means higher ERA), because SwStr% and O-Swing are negatively related. O-Looking is just the opposite of O-Swing and is calculated as (1 – O-Swing%).

The formula that Excel and I came up with is this: (I am calling the metric TIPS, for True Independent Pitching Skill)

TIPS = 6.5*O-Looking(PitchF/x)% – 9.75*SwStr% – 4.8*Foul% + C

C is a constant that changes from year to year to adjust to the ERA scale (to make an average TIPS = average ERA). For 2013 this constant was 2.57.

I converted this to TIPS- to better analyze the statistic. FIP, xFIP, and SIERA were also converted to FIP-, xFIP-, and SIERA-. I took all pitchers’ seasons from 2008-2013 to analyze. The sample varied in IP from 0.1 IP to 253 IP. I found the following season’s ERA- for each pitcher if they pitched more than 20 IP the next year and eliminated any huge outliers. Here were the results with no min IP. RMSE is root mean square error (smaller is better), AVG is the average difference (smaller is better), R^2 is correlation (larger is better), and SD is the standard deviation.

N=2316 ERA- FIP- xFIP- SIERA- TIPS-
RMSE 77.005 51.647 43.650 43.453 40.767
AVG 43.941 34.444 30.956 30.835 30.153
R^2 0.021 0.045 0.068 0.147 0.169
SD 69.581 38.654 24.689 24.669 15.751

Wow TIPS- beats everyone! But why? Most likely because I have included small samples and TIPS- is based off per pitch, as opposed to per batter (SIERA) or per inning (xFIP and FIP). There are far more pitches than AB or IP so TIPS will stabilize very fast. Let’s eliminate small sample sizes and look again.

Min 40 IP
N=1619 ERA- FIP- xFIP- SIERA- TIPS-
RMS 40.641 36.214 34.962 35.634 35.287
AVG 29.998 26.770 25.660 25.835 26.115
R^2 0.063 0.105 0.120 0.131 0.101
SD 26.980 19.811 15.075 17.316 13.843

 

Min 100 IP
N=654 ERA- FIP- xFIP- SIERA- TIPS-
RMSE 32.270 29.949 29.082 28.848 29.298
AVGE 24.294 22.283 21.482 21.351 22.038
R^2 0.080 0.118 0.143 0.145 0.095
SD 20.580 16.025 12.286 12.630 10.985

Now, TIPS is beaten out by xFIP and SIERA, but beats ERA and and is close to FIP (wins in RMSE, loses in R^2). This is what I expected, as I explained earlier K% and BB% are always better at predicting future K% and BB% and they are included in SIERA and xFIP. SIERA and xFIP take more concrete events (K, BB, GB) than TIPS. I didn’t want to beat these estimators, but instead wanted a estimator that is independent of everything except for pitcher-batter reaction.

TIPS won when there was no IP limit, so it obviously is the best to use in smaller sample sizes, but when is it better than xFIP and SIERA, and where does it start falling behind? I plotted the RMSE for my entire sample at each IP. Theoretically this should be an inverse relationship. After 150 IP it gets a bit iffy, as most of my sample is less than 100 IP. I’m more interested in IP under 100 anyhow.

Orange is TIPS, Blue is ERA, Red is FIP, Green is xFIP, and Purple is SIERA. If you can’t see xFIP, it’s because it is directly underneath SIERA (they are almost identical). This is roughly what the graph should look like to 100 IP:

Looking at the graph, at what IPs is TIPS better than predicting future ERA than xFIP and SIERA? It appears to be from 0 IP to around 70 IP.

Here is the graph for 1/RMSE (higher R^2). Higher number is better. This is the most accurate graph as the relationship should be inverse.

The 70-80 IP mark is clear here as well.

I’m not suggesting my estimator is better than xFIP or SIERA, it isn’t in samples over 75 IP, but I think it is, and can be, a very powerful tool. Most bullpen pitchers stay under 75 IP in a season. This means that TIPS would be very useful for bullpen arms in predicting future ERA. I also believe and feel that my estimator is a very good indicator of the raw skill of a pitcher. It would probably be even more predictive if we had robo-umps that eliminated umpire bias and randomness and pitch framing.

2013 TIPS Leaders with 100+IP. You can scroll horizontally, search for a specific player, and adjust number shown.

Rank
Name
Team
IP
ERA
FIP
xFIP
SIERA
TIPS
1Cole HamelsPhillies2203.63.263.443.483.01
2Matt HarveyMets178.12.2722.632.713.1
3Anibal SanchezTigers1822.572.392.913.13.23
4Yu DarvishRangers209.22.833.282.842.833.25
5Homer BaileyReds2093.493.313.343.393.28
6Clayton KershawDodgers2361.832.392.883.063.34
7Max ScherzerTigers214.12.92.743.162.983.36
8Francisco LirianoPirates1613.022.923.123.53.36
9Felix HernandezMariners204.13.042.612.662.843.4
10Hisashi IwakumaMariners219.22.663.443.283.43.4
11Justin VerlanderTigers218.13.463.283.673.683.46
12Jose FernandezMarlins172.22.192.733.083.223.46
13Patrick CorbinDiamondbacks208.13.413.433.483.643.48
14Tony CingraniReds104.22.923.783.493.393.5
15Madison BumgarnerGiants201.12.773.053.323.413.51
16Tyson RossPadres1253.173.23.433.413.54
17John LackeyRed Sox189.13.523.863.493.563.55
18Scott KazmirIndians1584.043.513.363.453.55
19Adam WainwrightCardinals241.22.942.552.83.013.57
20Jeff SamardzijaCubs213.24.343.773.453.63.57
21Tim LincecumGiants197.24.373.743.563.753.58
22Alex CobbRays143.12.763.363.023.263.58
23Mat LatosReds210.23.163.13.563.663.58
24Julio TeheranBraves185.23.23.693.763.623.59
25Cliff LeePhillies222.22.872.822.782.93.59
26Dan StrailyAthletics152.13.964.054.424.323.6
27Zack GreinkeDodgers177.22.633.233.453.673.61
28Chris SaleWhite Sox214.13.073.172.952.963.64
29Dan HarenNationals169.24.674.093.673.63.64
30A.J. BurnettPirates1913.32.82.923.13.65
31Derek HollandRangers2133.423.443.683.813.67
32Miguel GonzalezOrioles171.13.784.454.314.413.68
33Stephen StrasburgNationals18333.213.153.173.68
34Shelby MillerCardinals173.13.063.673.733.683.69
35Phil HughesYankees145.25.194.54.394.233.71
36Jake Peavy- - -144.24.173.964.033.953.73
37Gio GonzalezNationals195.23.363.413.513.763.74
38Kris MedlenBraves1973.113.483.553.763.75
39Lance LynnCardinals201.23.973.283.663.763.75
40Ricky Nolasco- - -199.13.73.343.583.723.75
41Jordan ZimmermannNationals213.13.253.363.483.673.75
42Matt Garza- - -155.13.823.883.733.793.75
43Jon LesterRed Sox213.13.753.593.93.973.76
44Tim HudsonBraves131.13.973.463.563.753.76
45Ian Kennedy- - -181.14.914.594.194.163.78
46Hiroki KurodaYankees201.13.313.563.63.793.78
47Marco EstradaBrewers1283.873.863.633.513.78
48Chris ArcherRays128.23.224.073.913.953.78
49Jeremy HefnerMets130.24.344.494.054.113.78
50Dillon GeeMets1993.6244.074.143.8
51Ryan DempsterRed Sox171.14.574.684.214.263.81
52Andy PettitteYankees185.13.743.73.884.133.83
53Matt CainGiants184.143.933.883.913.84
54James ShieldsRoyals228.23.153.473.723.853.85
55Clay BuchholzRed Sox108.11.742.783.413.663.86
56Gerrit ColePirates117.13.222.913.143.413.86
57Mike MinorBraves204.23.213.373.643.633.87
58Matt MooreRays150.13.293.954.324.313.88
59Ervin SantanaRoyals2113.243.933.693.853.89
60Rick PorcelloTigers1774.323.533.193.393.89
61CC SabathiaYankees2114.784.13.763.953.89
62Jeremy HellicksonRays1745.174.224.154.153.9
63Aaron Harang- - -143.15.44.794.384.223.93
64John DanksWhite Sox138.14.755.064.084.243.93
65Zack WheelerMets1003.424.174.214.43.93
66Justin MastersonIndians1933.453.353.333.43.94
67Tommy MiloneAthletics156.14.144.34.154.063.94
68Jorge de la RosaRockies167.23.493.764.084.453.95
69Edwin JacksonCubs175.14.983.793.864.043.95
70Roberto HernandezRays1514.894.633.63.663.95
71Bud Norris- - -176.24.183.864.224.263.96
72Corey KluberIndians147.13.853.33.13.323.96
73Chris TillmanOrioles206.13.714.423.883.953.97
74Randall DelgadoDiamondbacks116.14.264.993.944.093.98
75Wei-Yin ChenOrioles1374.074.044.144.233.98
76Joe BlantonAngels132.26.045.123.843.923.99
77Garrett RichardsAngels1454.163.663.583.74
78Eric StultsPadres203.23.933.534.134.284.01
79Carlos VillanuevaCubs128.24.063.863.973.964.01
80Jose QuintanaWhite Sox2003.513.823.863.924.02
81Bruce ChenRoyals1213.274.124.934.814.02
82Jerome WilliamsAngels169.14.574.64.244.424.02
83R.A. DickeyBlue Jays224.24.214.584.234.184.02
84Chris CapuanoDodgers105.24.263.553.673.874.04
85Brandon McCarthyDiamondbacks1354.533.753.774.084.04
86Ubaldo JimenezIndians182.23.33.433.623.744.04
87Travis WoodCubs2003.113.894.54.54.05
88Jon NieseMets1433.713.583.844.084.06
89Jarrod ParkerAthletics1973.974.44.414.484.07
90A.J. GriffinAthletics2003.834.554.184.014.07
91Alexi OgandoRangers104.13.114.364.644.654.07
92David PriceRays186.23.333.033.273.434.07
93Hyun-Jin RyuDodgers19233.243.463.674.08
94Erik BedardAstros1514.594.384.614.424.09
95Ivan NovaYankees139.13.13.473.683.664.12
96Jacob TurnerMarlins1183.744.434.714.924.12
97Wade DavisRoyals135.15.324.184.154.344.14
98Wily PeraltaBrewers183.14.374.34.134.394.14
99Jered WeaverAngels154.13.273.824.314.224.15
100Hector SantiagoWhite Sox1493.564.444.654.344.15
101Jason HammelOrioles139.14.974.934.564.564.15
102Dallas KeuchelAstros153.25.154.253.583.714.18
103Paul MaholmBraves1534.414.243.894.094.18
104Wade MileyDiamondbacks202.23.553.983.774.034.2
105Juan NicasioRockies157.25.144.254.324.484.2
106Charlie MortonPirates1163.263.63.693.524.21
107C.J. WilsonAngels212.13.393.513.934.084.21
108Jordan LylesAstros141.25.594.574.414.474.22
109Jason VargasAngels1504.024.094.294.354.22
110Edinson Volquez- - -170.15.714.244.074.344.23
111Martin PerezRangers124.13.624.234.044.254.24
112Kyle KendrickPhillies1824.74.014.154.344.29
113Andrew CashnerPadres1753.093.353.623.84.31
114Jeff LockePirates166.13.524.034.194.544.32
115Tom KoehlerMarlins1434.414.274.284.544.32
116Mike LeakeReds192.13.374.043.914.174.32
117Nathan EovaldiMarlins106.13.393.594.154.434.33
118Doug FisterTigers208.23.673.263.423.494.33
119Barry ZitoGiants133.15.744.924.814.934.34
120Kyle LohseBrewers198.23.354.084.034.234.35
121Felix DoubrontRed Sox162.14.323.784.144.264.36
122Bartolo ColonAthletics190.12.653.233.954.174.36
123Scott Feldman- - -181.23.864.033.964.14.37
124Joe KellyCardinals1242.694.014.194.314.38
125Esmil RogersBlue Jays137.24.774.734.064.184.39
126Kevin CorreiaTwins185.14.184.44.244.574.43
127Zach McAllisterIndians134.13.754.034.534.494.44
128Mark BuehrleBlue Jays203.24.154.14.094.214.45
129Henderson AlvarezMarlins102.23.593.183.974.184.47
130Dylan AxelrodWhite Sox128.15.685.444.914.894.48
131Samuel DedunoTwins1083.834.044.064.134.5
132Jhoulys ChacinRockies197.13.473.473.974.344.52
133Tyler ChatwoodRockies111.13.153.6644.24.53
134Mike PelfreyTwins152.25.193.994.544.634.53
135Yovani GallardoBrewers180.24.183.893.744.054.54
136Scott DiamondTwins1315.435.194.715.044.54
137Jonathan PettibonePhillies100.14.044.184.34.534.55
138Trevor CahillDiamondbacks146.23.994.264.114.374.58
139Joe SaundersMariners1835.264.724.234.494.63
140Jeremy GuthrieRoyals211.24.044.794.554.794.64
141Lucas HarrellAstros153.25.865.424.975.194.68
142Ryan VogelsongGiants103.25.734.914.54.714.7
143Bronson ArroyoReds2023.794.493.974.154.72
144Jake WestbrookCardinals116.24.634.624.955.14.78
145Jason MarquisPadres117.24.055.654.815.214.83

And Leaders from 30IP to 100IP

Rank
Name
Team
IP
ERA
FIP
xFIP
SIERA
TIPS
1Koji UeharaRed Sox74.11.091.612.081.361.78
2Aroldis ChapmanReds63.22.542.472.071.732.1
3Jason GrilliPirates502.71.972.211.792.36
4Greg HollandRoyals671.211.361.681.52.38
5Ernesto FrieriAngels68.23.83.723.492.72.41
6Paco RodriguezDodgers54.12.323.082.922.652.43
7Trevor RosenthalCardinals75.12.631.912.341.932.47
8Kenley JansenDodgers76.21.881.992.061.622.57
9Glen PerkinsTwins62.22.32.492.612.192.58
10Edward MujicaCardinals64.22.783.713.533.252.58
11Danny SalazarIndians523.123.162.752.792.59
12Craig KimbrelBraves671.211.931.951.742.6
13Jordan WaldenBraves473.452.813.512.862.62
14Tyler ClippardNationals712.413.824.023.282.63
15Steve DelabarBlue Jays58.23.222.723.352.822.71
16Chris WithrowDodgers34.22.63.573.032.592.73
17Jared BurtonTwins663.823.613.943.472.75
18Andrew MillerRed Sox30.22.643.052.442.52.81
19Sean DoolittleAthletics693.132.713.683.22.82
20Neal CottsRangers571.112.172.822.642.82
21Mark MelanconPirates711.391.642.051.82.85
22Kelvin HerreraRoyals58.13.863.72.862.572.88
23Luke HochevarRoyals70.11.922.962.92.332.92
24Al AlburquerqueTigers494.593.723.483.392.92
25Manny ParraReds463.333.072.792.562.95
26Michael KohnAngels533.744.564.94.252.98
27Gonzalez GermenMets34.13.932.94.163.822.98
28Mike DunnMarlins67.22.663.123.73.312.99
29Joaquin BenoitTigers672.012.873.162.863
30Boone LoganYankees393.233.822.712.433.01
31David CarpenterBraves65.21.782.833.112.683.02
32Tony WatsonPirates71.22.393.23.723.333.02
33Casey FienTwins623.923.162.712.293.02
34Jesse CrainWhite Sox36.20.741.522.942.53.06
35Joe NathanRangers64.21.392.263.272.823.06
36Luke GregersonPadres66.12.712.73.373.033.06
37Javier LopezGiants39.11.832.412.922.83.08
38Antonio BastardoPhillies42.22.3234.093.563.08
39Sergio RomoGiants60.12.542.853.22.943.09
40Jake McGeeRays62.24.023.413.112.753.1
41Craig StammenNationals81.22.762.823.082.893.11
42Yusmeiro PetitGiants483.562.863.483.423.11
43Bobby ParnellMets502.162.333.052.93.12
44Jake DiekmanPhillies38.12.582.52.873.063.12
45Wilton LopezRockies75.14.063.573.693.673.13
46Junichi TazawaRed Sox68.13.163.223.032.723.14
47Rex BrothersRockies67.11.743.363.493.443.17
48Justin De FratusPhillies46.23.864.014.354.113.18
49Danny FarquharMariners55.24.21.862.42.33.18
50Drew SmylyTigers762.372.312.992.653.19
51Johnny CuetoReds60.22.823.813.233.553.2
52Will SmithRoyals33.13.243.532.52.053.21
53Nick VincentPadres46.12.142.062.892.623.22
54Jim HendersonBrewers602.73.583.312.833.22
55David HernandezDiamondbacks62.14.484.363.993.363.22
56Tommy HunterOrioles86.12.813.683.633.283.22
57Pedro Strop- - -57.14.553.553.33.143.23
58Darin DownsTigers35.14.843.533.533.063.24
59Kevin GausmanOrioles47.25.663.993.043.153.24
60Mariano RiveraYankees642.113.053.092.923.27
61Jared HughesPirates324.784.114.324.143.3
62Alex TorresRays581.712.323.32.943.31
63Oliver PerezMariners533.743.263.362.823.31
64A.J. RamosMarlins803.153.244.073.73.31
65Blake ParkerCubs46.12.722.93.542.883.35
66Josh CollmenterDiamondbacks923.133.474.063.643.36
67Tanner ScheppersRangers76.21.883.743.883.553.36
68Robbie RossRangers62.13.033.183.393.213.36
69Carlos CarrascoIndians46.26.754.14.334.443.36
70Huston StreetPadres56.22.74.9243.633.37
71Matt BelisleRockies734.323.032.992.983.39
72Logan OndrusekReds554.093.943.443.183.39
73Addison ReedWhite Sox71.13.793.173.773.263.4
74Bryan MorrisPirates653.464.894.344.263.4
75Nate JonesWhite Sox784.152.642.772.643.4
76Fernando RodneyRays66.23.382.843.113.173.41
77Tony SippDiamondbacks37.24.784.884.994.13.41
78Ryan CookAthletics67.12.542.743.713.353.42
79David AardsmaMets39.24.315.274.634.083.43
80Charlie FurbushMariners653.743.063.373.023.44
81Michael WachaCardinals64.22.782.923.363.323.44
82Will HarrisDiamondbacks52.22.912.743.122.913.44
83Cody AllenIndians70.12.432.993.272.813.44
84Brian MatuszOrioles513.532.913.593.23.45
85Jonathan BroxtonReds30.24.114.684.53.913.45
86Burch SmithPadres36.16.445.474.093.873.46
87Nick HagadoneIndians31.15.464.84.674.53.47
88Vinnie PestanoIndians35.14.085.034.524.053.47
89J.J. HooverReds662.863.473.973.463.47
90Ross OhlendorfNationals60.13.284.033.963.913.48
91Brian DuensingTwins613.983.243.713.563.48
92Bryan ShawIndians753.243.073.583.393.48
93Darren O'DayOrioles622.183.583.593.063.49
94Kevin JepsenAngels364.53.383.933.543.49
95Scott RiceMets513.713.43.823.873.5
96George KontosGiants55.14.394.083.963.683.52
97Juan PerezBlue Jays31.23.693.713.393.213.52
98Joe Thatcher- - -39.13.23.383.553.273.52
99Jonathan PapelbonPhillies61.22.923.053.513.043.53
100Carlos Marmol- - -494.415.194.534.373.54
101Tom GorzelannyBrewers85.13.93.943.643.563.54
102Carter CappsMariners595.494.733.563.233.55
103Fernando AbadNationals37.23.353.263.973.583.55
104Josh OutmanRockies544.333.253.623.423.55
105Adam OttavinoRockies78.12.643.153.543.383.55
106Grant BalfourAthletics62.22.593.493.423.143.55
107Guillermo MoscosoGiants305.15.555.665.063.56
108Shawn KelleyYankees53.14.393.633.242.753.56
109Greg BurkeMets31.25.683.933.953.843.56
110Francisco Rodriguez- - -46.22.73.653.042.723.57
111Shaun MarcumMets78.15.293.644.224.153.57
112Caleb ThielbarTwins461.763.44.143.713.58
113Rafael SorianoNationals66.23.113.654.053.783.59
114Justin WilsonPirates73.22.083.413.823.653.6
115Brad ZieglerDiamondbacks732.223.43.322.773.62
116James RussellCubs52.23.594.454.774.313.65
117Ethan MartinPhillies406.085.654.534.293.65
118Alfredo SimonReds87.22.873.964.223.843.66
119Kevin SiegristCardinals39.20.452.2932.773.66
120Pat NeshekAthletics40.13.354.664.864.393.67
121Michael BowdenCubs37.24.34.35.044.743.67
122Dane de la RosaAngels72.12.862.993.493.413.67
123Marc Rzepczynski- - -30.23.233.373.553.073.67
124Preston ClaiborneYankees50.14.114.143.883.523.68
125Phil CokeTigers38.15.44.144.594.493.69
126Matt Thornton- - -43.13.744.044.133.943.7
127Tim StaufferPadres69.23.753.553.23.043.7
128Travis Blackley- - -50.14.835.934.214.183.7
129Josh RoenickeTwins624.354.645.34.873.71
130Brad BrachPadres313.194.144.434.23.71
131Chad QuallsMarlins622.613.323.252.983.72
132Joel PeraltaRays71.13.413.684.263.763.72
133Todd RedmondBlue Jays774.324.44.163.723.72
134Josh JohnsonBlue Jays81.16.24.623.583.733.72
135Dale ThayerPadres653.323.793.553.273.72
136Heath BellDiamondbacks65.24.114.13.082.783.73
137Nick TepeschRangers934.844.193.823.863.73
138Michael GonzalezBrewers504.684.873.883.433.75
139Carlos TorresMets86.13.444.33.53.363.75
140Jaime GarciaCardinals55.13.583.723.273.343.75
141Aaron CrowRoyals483.384.343.943.753.76
142Jesse ChavezAthletics57.13.923.013.833.433.76
143Cesar RamosRays67.14.143.74.023.823.77
144Yoervis MedinaMariners682.913.863.853.763.77
145Cory GearrinBraves313.774.344.424.283.78
146David Huff- - -37.25.54.613.613.463.8
147Adam WarrenYankees773.394.323.983.843.8
148Brett CecilBlue Jays60.22.822.882.992.793.8
149Hector RondonCubs54.24.774.44.424.163.8
150Brett AndersonAthletics44.26.043.853.263.313.8
151Taylor JordanNationals51.23.663.493.83.863.8
152Santiago CasillaGiants502.163.674.244.183.81
153Luis Ayala- - -333.273.683.773.683.82
154Brett OberholtzerAstros71.22.763.654.274.313.82
155Allen WebsterRed Sox30.18.66.515.185.113.83
156Josh FieldsAstros384.975.14.23.613.83
157Raul ValdesPhillies357.464.393.743.13.84
158Josh BeckettDodgers43.15.194.663.813.93.85
159Daisuke MatsuzakaMets38.24.424.324.814.493.86
160Scott Downs- - -43.12.493.093.33.153.86
161Jason FrasorRangers492.573.373.623.443.86
162Ronald BelisarioDodgers683.973.643.833.533.88
163Jeremy AffeldtGiants33.23.744.444.484.393.89
164T.J. McFarlandOrioles74.24.223.843.683.563.89
165Luis AvilanBraves651.523.284.023.843.89
166Tyler SkaggsDiamondbacks38.25.124.863.953.963.9
167Brandon BeachyBraves304.54.083.63.683.9
168LaTroy HawkinsMets70.22.933.063.123.033.91
169Jerry BlevinsAthletics603.153.884.253.633.92
170Jim JohnsonOrioles70.12.943.453.383.013.92
171Alex SanabiaMarlins55.14.885.855.35.313.92
172Steve CishekMarlins69.22.332.522.982.763.92
173Craig BreslowRed Sox59.21.813.64.374.313.93
174J.J. PutzDiamondbacks34.12.363.833.393.313.93
175Jose CisneroAstros43.24.124.244.314.023.93
176Anthony BassPadres425.364.244.274.33.94
177Tom WilhelmsenMariners594.123.694.574.583.94
178Alfredo FigaroBrewers744.144.873.643.63.94
179Erasmo RamirezMariners72.14.984.834.264.33.94
180Aaron LoupBlue Jays69.12.473.323.312.813.94
181Juan Gutierrez- - -55.14.233.734.093.843.95
182Brandon KintzlerBrewers772.692.542.932.823.95
183Roy OswaltRockies32.18.633.083.393.453.96
184Blake BeavanMariners39.26.134.994.334.033.96
185Zach Duke- - -31.16.034.24.354.213.96
186Rich HillIndians38.26.283.824.123.863.96
187Matt Guerrier- - -42.24.013.824.444.263.97
188Jamey WrightRays703.093.133.483.23.97
189Wesley Wright- - -53.23.693.923.323.093.98
190Luis GarciaPhillies31.13.735.124.925.063.98
191Chad JenkinsBlue Jays33.12.73.954.254.413.99
192Tim CollinsRoyals53.13.543.44.293.973.99
193Jake Arrieta- - -75.14.784.844.614.834
194Neil WagnerBlue Jays383.794.133.533.54
195Matt LindstromWhite Sox60.23.123.153.873.74
196John Axford- - -654.024.343.563.424.01
197Sam LeCureReds612.662.973.323.134.01
198Tyler LyonsCardinals534.753.733.843.864.02
199Drew StorenNationals61.24.523.623.713.394.02
200Jeff FrancisRockies70.16.274.543.823.874.02
201Jose MijaresGiants494.223.053.93.454.02
202Brandon LyonMets34.14.983.984.484.384.02
203J.A. HappBlue Jays92.24.564.314.824.694.03
204Dan OteroAthletics391.382.123.1834.04
205Dan JenningsMarlins40.23.762.683.63.464.04
206Kevin SloweyMarlins924.113.814.013.924.05
207David RobertsonYankees66.12.042.612.62.374.06
208Edgmer EscalonaRockies465.674.874.484.014.06
209Jeanmar GomezPirates80.23.353.853.953.984.06
210Vin MazzaroPirates73.22.813.3143.834.06
211Troy PattonOrioles563.74.424.13.764.07
212Josh LindblomRangers31.15.464.424.54.644.07
213Brandon WorkmanRed Sox41.24.973.433.183.184.07
214Luis MendozaRoyals945.364.754.734.934.09
215Chad BettisRockies44.25.644.934.684.754.1
216Ross DetwilerNationals71.14.043.664.254.444.11
217Randy ChoateCardinals35.12.292.573.32.884.12
218Seth ManessCardinals622.323.433.132.654.13
219J.P. HowellDodgers622.182.893.483.34.14
220Anthony SwarzakTwins962.913.283.813.614.14
221Joba ChamberlainYankees424.935.644.64.444.15
222Chad GaudinGiants973.063.3443.994.15
223Ryan PresslyTwins76.23.873.674.294.254.16
224Donovan HandBrewers68.13.695.014.574.544.17
225Brad PeacockAstros83.15.184.984.444.234.18
226Wade LeBlanc- - -555.44.765.024.854.19
227Alex WoodBraves77.23.132.653.183.384.19
228Joe OrtizRangers44.24.233.973.833.984.2
229Brandon LeagueDodgers54.15.34.934.073.74.23
230Casey JanssenBlue Jays52.22.562.743.092.884.24
231Jose AlvarezTigers38.25.825.194.44.364.24
232Jose Veras- - -62.23.023.623.953.374.26
233Brandon MaurerMariners906.34.94.254.14.26
234Joe SmithIndians632.293.63.73.534.26
235Freddy Garcia- - -80.14.375.4944.344.28
236Jeff ManshipRockies30.27.045.594.434.74.28
237Robbie ErlinPadres54.24.123.833.984.24.28
238Tanner RoarkNationals53.21.512.413.143.344.3
239Liam HendriksTwins47.16.855.444.774.464.3
240Kyle GibsonTwins516.535.174.774.784.3
241Philip HumberAstros54.27.95.024.614.584.31
242Clayton MortensenRed Sox30.15.344.835.034.674.31
243Roy HalladayPhillies626.826.145.14.934.31
244Stephen FifeDodgers58.13.74.353.893.984.32
245Anthony VarvaroBraves73.12.823.474.194.224.33
246Justin Grimm- - -985.974.594.224.294.34
247Rob ScahillRockies33.15.134.974.374.074.35
248Matt AlbersIndians633.143.493.823.614.35
249Wandy RodriguezPirates62.23.594.4243.954.36
250Chris RusinCubs66.13.934.754.464.784.37
251Sonny GrayAthletics642.672.72.923.114.38
252Ryan WebbMarlins80.12.913.63.793.74.39
253Brandon MorrowBlue Jays54.15.635.424.494.394.39
254Sandy RosarioGiants41.23.023.654.584.774.4
255Hector AmbrizAstros36.15.75.664.424.124.41
256Brad LincolnBlue Jays31.23.985.485.915.154.42
257Tommy HansonAngels735.424.654.834.674.42
258Brandon CumptonPirates30.22.052.623.43.474.43
259Chris PerezIndians544.335.083.833.474.43
260Kyle Farnsworth- - -38.14.74.143.733.644.46
261Jeremy Bonderman- - -555.45.125.165.214.46
262Ryan MattheusNationals35.16.373.444.043.94.46
263Tyler ThornburgBrewers66.22.033.114.434.454.48
264Paul ClemensAstros73.15.45.695.074.624.48
265Vance WorleyTwins48.27.215.534.764.824.48
266Lucas LuetgeMariners374.863.754.164.074.49
267Tyler CloydPhillies60.16.564.494.774.834.49
268Andrew AlbersTwins604.053.964.424.714.5
269Ross WolfRangers47.24.154.544.434.614.51
270Manny CorpasRockies41.24.544.394.144.084.51
271Kevin GreggCubs623.484.14.384.134.54
272Scott AtchisonMets45.14.373.754.023.954.57
273Jon GarlandRockies685.824.934.544.94.57
274John LannanPhillies74.15.334.374.714.844.62
275David PhelpsYankees86.24.983.814.033.984.66
276Clayton RichardPadres52.27.016.544.665.024.69
277Darren OliverBlue Jays493.864.053.743.544.72
278Alfredo AcevesRed Sox374.866.355.015.284.74
279Jarred CosartAstros601.954.354.685.344.76
280Ramon TroncosoWhite Sox304.55.284.914.814.76
281Pedro HernandezTwins56.26.835.545.175.294.77
282Andre RienzoWhite Sox564.825.854.764.944.84
283P.J. WaltersTwins39.15.955.035.225.324.93
284Zach BrittonOrioles404.954.84.594.775.03
285Burke BadenhopBrewers62.13.473.533.43.395.12
286Johnny HellwegBrewers30.26.757.066.816.545.47

 

Fielder for Kinsler, and Big Contract Hot Potato
The Other Guys

Author: Chris Carruthers

Chris has been a baseball (and Jays) fan for 15 years. He has also played since the age of 6, working his way through Little League and Babe Ruth Baseball as a catcher and first baseman. He got interested in sabermetrics after viewing the movie Moneyball. His continuous self-learning in sabermetrics and advanced stats is driven by his engineering background and love for numbers. Chris's go-to website is FanGraphs, where he has had a few previous community submissions. Chris also enjoys music and plays guitar in his sparse spare time from his studies. He also follows hockey and his favourite team, the Calgary Flames. Follow Chris on Twitter @CCBreakingBlue.

Share This Post On

23 Comments

  1. Well done, though I’m unsure where, as you say, catcher framing is removed from this. Couldn’t O-Looking% be affected by this?

    Post a Reply
    • No O-Looking% is different than ball%. These are pitches that are outside the zone as determined by PitchF/X. Whether the pitch is called a strike or a ball is irrelevant. This is therefore independent of framing and umpire calls.

      Post a Reply
    • PitchF/X uses various camera angles to determine where the ball actually crossed the plate. PitchF/X is not fooled by the Molina brothers’ ridiculous framing skills as umpires are. One example is David Price, he is generally caught by Jose Molina (known as a top 3 framer, if not the best) and Jose Lobaton (also an above average framer). TIPS believes Price showed merely average (or even below) skill this year, where as ERA, FIP, xFIP, and SIERA show Price as ace quality.

      Post a Reply
  2. First of all, this is awesome.

    Second, could you post the 0-30 IP results? TIPS is still roughly as accurate (RMSE) as FIP over that range. It could be of some use/interest.

    Post a Reply
    • Thanks, and yes I can add the 0-30 IP when I have time. I am also going to post leaderboards for previous years soon as well.

      Post a Reply
      • Awesome.

        Also did you look at any differences between pitchers used as starters and relievers?

        Obviously, you already did a lot of work, but that might be interesting as well, as well as lefty/righty.

        Post a Reply
  3. Nice job!

    Would love to see you test TIPS against the various other run estimators using only pitchers who switch teams (to completely take park and defense out of the equation).

    Post a Reply
  4. Chris, interesting stuff.

    I couldn’t follow from the article whether you tested out-of-sample or in-sample… Did you use the same data for the regression that produced the formula and for evaluating its performance against FIP, xFIP and SIERA (all pitchers, 2008-2013)? Or did you use a different data set for doing the regression?

    Post a Reply
    • Data set was completely independent of test data sample.

      MGL, I ran that last night and it provided similar results. Will post it sometime soon.

      filihok, I decided to not use starting IP or relieving IP in the formula. If relievers see boosts due to being able to throw with all of their velocity at once, it should show up in the data.

      Post a Reply
      • Cool on the in-sample/out-of-sample. What was the in-sample data? Did you reserve certain pitchers? Since you tested 2008-2013 I assume you didn’t reserve any specific years.

        Post a Reply
        • No, all pitchers from every year, regardless of IP, were tested against the next year IP provided they pitched at least 20 inning the next year.

          For creating the formula, I used pitchers “career” stats from 2008-2013 who pitched at least 300 innings in that span. I used 300 IP to ensure I had bullpen arms in the sample.

          Post a Reply
          • That’s not really an independent sample then. You estimate 2009′s ERA based on 2008′s pitch stats, but the formula was derived using data from both seasons.

            In other words, the formula has been fit to the same pitchers over the same period that you’re testing. Whereas FIP hasn’t been adjusted retroactively to provide the best fit for the specific years 2008-2013, other than the adjustment of the constant. I assume the same is true of SIERA and xFIP.

        • I see what you are saying, but it’s not like I had much of an option. PitchF/x data was only available 6 years ago. It is also over a span of 6 years, and the testing is year to year. Correct me if I’m wrong, but wouldn’t that be more of an issue in larger samples (not my niche of 0-70 IP)?

          I also didn’t use 2007 data, can I run the tests and get back to you for the predictability of 2007-2008? This should be more of the independence you are looking for. TIPS can also be tested after this year as well.

          Post a Reply
          • 2007 data might be kind of screwy.

            One thing you could do is redo the regression based on 2008-2012, and use the new coefficients to evaluate 2013. It’s entirely possible the formula will come out about the same and the predictive value for 2013 will remain – I’m not trying to poopoo the concept. :)

          • I will do this and get back to you!

  5. “One thing you could do is redo the regression based on 2008-2012, and use the new coefficients to evaluate 2013.”

    This resulted in 7o – 9.25s -4.8f +2.0. O-looking co went up slightly and SwStr went down slightly. Difference in correlation to sample of this formula and previous TIPS was .0002699. Standard deviation of the formula for 08-12 was noticeably smaller (too small for my liking), but it’s predictability for 2012 to 2013 was slightly better (not significant at all) than TIPS. 41.020 RMSE compared to 41.087 RMSE for TIPS-.

    Post a Reply
    • A discussion on TangoTiger’s blog is about the same issue, although it looks to be rectified with what I just did. 2008-2012 sample used on 2012 to predict 2013 worked just as well, even slightly better, but with a smaller standard deviation.
      More years of data would obviously ideal, but with pitchf/x being very recent, this is not possible.

      I will be actively testing 2013 to 2014 once the season starts and 2014 in season as well.

      Post a Reply
  6. Pretty cool. Casey Janssen’s TIPS caught my eye. More evidence that he should probably be traded.

    Also, go Flames!

    Post a Reply
  7. So now I ask the obvious question … why on earth would something be more predictive under 70 IP, but not 70 IP and over? Wouldn’t this indicate simply mashing numbers to try to get a superficially predictive match, and ignoring logic in the process? What, that’s included in the other derived statistics, suddenly gains more importance/relevance to a pitcher’s success/failure as they cross the 70 IP threshold that wasn’t as important/relevant below it? Those are the sorts of questions that need to be posed and answered via peer review before these things get out there and gain footing amongst the masses. “We got a match!” shouldn’t cut it.

    Post a Reply
    • I’m thinking that what might be likely is that the inputs of TIPS, O-Looking%, SwStr%, and Foul%, simply stabilize quicker than the FIP inputs do. I recall from a piece on BP that pitcher K and BB rates take something like 1/5 and 1/2 of a full season to stabilize (so say 40 IP and 100 IP). I don’t remember how long HR rate takes to stabilize, but it’s probably much longer. So in light of that, the crossover points between TIPS and xFIP at ~70IP and TIPS and FIP at ~170 would make a lot of sense.

      Past 70 IP, K and BB rates are stable and K and BB rates are stronger indicators of skill than their inputs.

      And really, I think Chris’ thought process is well outlined so any concerns about “number mashing and ignoring logic in the process” are not warranted here.

      Post a Reply
    • Yes, it’s due to the denominator of the statistics. SwStr, Foul% and O-Swing% have the a smaller denominator (pitch or contact or o-pitch) than other estimators (IP or PA). This means TIPS stabilizes much faster, and therefore is more predictive that the others. The other estimators use more real events (K, BB, HR) that can be affected by an environment (K, BB). If this environment stays constant (catcher) it is more likely to continue this way. This is why the other estimators are likely more predictive in larger samples.

      Post a Reply

Trackbacks/Pingbacks

  1. Knockout Radio - […] you only read one link in this tank, make it this one. Chris Carruthers brings us TIPS, a new …

Submit a Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>