Using Football Outsiders' game charting data to evaluate 3rd & long playcalling

For years I have watched fans boo and roll their eyes when the home team has run a screen pass on third and long, apparently on the assumption that the play call constitutes giving up on the drive and settling for a punt. I have never agreed. This is partly because I find the screen to be among the most satisfying plays in football when executed properly. Subjectively, screen plays seem to gain significant yards quite frequently, and I have long hypothesized that in a third and long situation a screen may give you as good a chance as a longer pass of gaining a third down, partly because the defense is expecting a long pass, and as such the pass rushers will likely be going all out for the quarterback, which should of course be an ideal set up to exploit them with a nice little screen. But let’s test this with a little data.


I just bought myself Football Outsider’s very useful game charting data for 2010 (you will be duly invoiced). Unlike the standard NFL play by play, the game charters mark all plays that are screens and put the data into a nice little sortable excel file you can play with. So here we have the capability to compare screens versus other play options and see if it is in fact a reasonable 3rd & long play call. We will also look at every color commentator’s favorite target for invective – the dread 3rd down pass short of the sticks, as well as draw plays (which seem to never work, right?).

I guess the first thing we need to do is define “3rd and long.” Seven yards seems like a good place to start it because I think it lets us conveniently divide our play calling options. At seven yards, it is an obvious enough pass situation that screens will benefit from the aggressive pass rush as mentioned above. And the traditional shorter routes such as slants and smashes will pretty much always end up short of the sticks, and so would only be called if the offensive coordinator is planning to rely on yards after the catch for a first down.

We also have to set an upper limit, because in extreme yardage situations the OC may be conceding the drive and simply trying to pick up a few yards of free field position before punting. I chose 12 yards as the upper limit here, simply because it feels like an offense could reasonably think that a successful screen or catch and run of a short route could attain this yardage. Longer than this and any such short play calls are probably not intended to pick up a 1st down anyway.

So to get a baseline, let’s first look at all 2010 regular season plays on 3rd & 7-12:

3rd & 7-12 plays: 2,924

1st downs (including touchdowns): 854

Success rate: 29%

So we can surmise that if any of our playcalling options fall well below this 29% first down rate then they should not be considered reasonable third and long plays.

Screens

There were surprisingly few screens thrown in this circumstance – only 150. I thought they would have been much more common, but perhaps I just remember them more because whether successes or failures they are particularly noticeable. Of those 150 screens 37 resulted in 1st downs, for a success rate of 25%. We have a pretty small sample size, but I think it probably means something that that the success rate is at least in the ballpark of 3rd and long plays as a whole. Given that there are no good options on thirds and long, I think this supports the idea that a screen is as good a means as any of trying for a 1st down. Another argument for running screens here is to set yourself up for future 3rd & longs. If the defense considers there to be a reasonable threat of a screen it will take some of the gas out of the pass rush, and possibly improve your chances of completing longer passes.

Draw plays

As a TV football viewer, draw plays seem almost utterly futile. According to the game charting data, of 33 third & long draw plays, only 5 were first downs. This is an even smaller sample than for screens, so we have an indication here that fans may be right when they consider a draw to be a give-up play it’s hardly conclusive.

Passes short of the sticks

Nothing drives color commentators up the wall quite like a third down pass route that is run short of the first down marker, but it’s not always fair. When a receiver runs his out route a yard short of the sticks then he certainly earns criticism. But in many cases, the plays are clearly designed to get the ball into the guy’s hand with room to turn it upfield and run for the first down. Given how low completion percentages get as you go deep downfield, the short pass with complementary routes designed to clear space for YAC has always struck me as a reasonable idea.

The NFL play by play info contained within the charting data handily splits passing yards into yards in the air versus yards after catch, so we can isolate the pass plays where the ball was thrown short of the marker. We have to trim this set down a bit further though. Screen passes have already been looked at separately, so we have to filter those out. We also need to kick out the cases where a quarterback is looking downfield but due to pressure or a lack of open receivers if forced to dump it off to a running back. Fortunately dumpoffs are tracked by the charters. Hopefully most of what is left covers the slants, shallow crosses, and other routes designed to hit a receiver in stride. There are a total of 644 of these short passes, with 157 first downs for a 24% success rate.

That puts us in the neighborhood of the success rates we know by now to expect, so short routes look like a perfectly reasonable call for 3rd and long. And it probably understates their effectiveness. Remember the receiver two paragraphs up who lost track of himself on the field and ran his out route a yard short? He is included in this sample. So it seems likely that plays which are designed to create running room after a short pass would have a significantly better success rate than the 24% we’re looking at.

Passes past the sticks

OK, now let’s look at passes that are thrown to or past the first down marker. There were 906 of these passes, and 432 of them resulted in first downs. This is an impressive 47% success rate. However, it is inflated for a number of reasons. Obviously, what we’re really measuring here is completion percentage since we’re already to the sticks. So these numbers only track passes that get to the intended receiver. On these longer passes you’re doing a lot of 5 and 7 step drops, against a pass rush that is pinning its ears back for 3rd & long. So the actual success rate when intending to throw past the sticks would be much lower than 47% because so many of these plays would turn into sacks, pressures, or checkdowns and so are not included in those numbers.

The obvious problem here is that it’s impossible to know (from the play by play and charting data at least) what a quarterback was intending to do before he got sacked. But it’s a good bet that the majority of those aborted passes were due to the longer drops and slower releases necessary for deeper passes. So it would be very conservative to assume that half of all dumpoffs, scrambles, and sacks occurred when the quarterback was intending to throw past the first down marker. Using that 50% number, which I think understates things, and assume that half of the numbers below come from attempted passes past the sticks. This is very, very rough of course, but it will give us some idea of how these aborted passes would impact overall success rate.

Dumpoffs: 148 1st downs: 47

Scrambles: 287 1st downs: 87

Sacks: 217 1st downs: 0, obviously.

Adding everything up and using the conservative 50% number, we end up with 1,232 plays and 499 first downs, pulling the success rate down to 41%. That’s a noticeable drop, but still much better than any of the other options we’ve looked at. And while I’m pretty sure that much more than 50% of those aborted pass scenarios happen when attempting to throw long, it’s still a long drop down to the 25% range we’ve been seeing (for example if you increase the proportion to 60% the success rate only comes down to 39%).

So I guess fans and commentators may have a point when they argue that offenses should quit messing around and just throw the ball to the sticks. There would be some exceptions of course. If your team lacks a quality offensive line that can protect a deep drop (like a certain local football team you and I are familiar with), your success rate might be much lower than the average level we’re looking here and screens or reliable short passes with the hope of YAC may be a more appealing option. And as mentioned above you have to keep defense honest by making them at least be conscious of underneath routes or a screen. Nonetheless, the data here suggest that those options should only be for the occassional change of pace.

What we’ve looked at here isn’t conclusive. Ideally we would combine the data from several years together to get a more reliable sample size. And of course the data itself is subject to simple errors by the volunteer charters, though things like screens and dumpoffs are pretty easily identified by experience football watchers (as the game charting volunteers are). I am also stymied as to how to solve the dilemmas of the sacks and scrambles and how to apply them to the various passes. But I think it does give us enough to have a basis for discussion about third and long play calling options.

Let me know if you have any ideas of how to do improve this small attempt at research. Also, since I have the game charting data on hand feel free to toss me suggestions of other things we should try to learn from it.