Update: I’m happy this post is getting the occasional link and I hope people enjoy my exhaustive look at voting procedures (who wouldn’t!). However, further reporting after the announcement in the change in the Best Picture nominating rules – particularly by Steve Pond at The Wrap – revealed that some of my initial assumptions were incorrect. I have inserted some updates to clarify where necessary. The original post:

This morning the Academy announced changes to the Best Picture nomination process. After two years of ten nominees, the number of nominated films may now vary between 5 and 10. Only films that receive at least 5% of first place votes may qualify for a nomination, though five is the minimum.

The intention of this rule change is great. Ten felt unwieldy at times, with a few also-also-rans filling out the slate. Allowing the quality of that year’s contenders determine the number of slots makes a lot of sense.

It’s too bad they bungled the math so bad.

Breaking 5%

In the Academy press release, Executive Committee members claimed they pored over the data from recent years to see what would have happened under the new 5% scenario. “In studying the data, what stood out was that Academy members had regularly shown a strong admiration for more than five movies,” said retiring executive director Bruce Davis. In the eight years before the expansion to ten nominees, the new system would have resulted in slates of 5, 6, 7, 8, and 9 nominees. Note that there’s one number missing: 10.

I know this isn’t a large sample size, but in all the years they tested, a 5% threshold never reached the maximum number of allowed nominees. This means that the 5% rule is not tweaking the list of eligible candidates, it is the sole determinant of the nominees. (Update: Not entirely true. See below.)

An Elegant System Tossed Away

Why does that matter? Because the current system is surprisingly well-devised. A good voting system should accurately reflect voters’ preferences and diminish the temptation to vote strategically (“game the system”). The Oscar’s current voting process does this well.

Alternate Voting: Rather than voting for one film to nominate for Best Picture, voters submit a ranked list 1-10 of nominees. To count these ballots, the accountants make a pile for each film that receives a #1 vote. Any film that receives 1/11th of all #1 votes gets a nomination and those ballots are set aside. Next, the movie that receives the fewest #1 votes gets eliminated. Those ballots get transferred to the #2 film on each list. It does get a little more complicated, but essentially round after round of eliminations eventually results in ten films crossing the nomination threshold. Even though voters submit up to ten films, each voter only has one vote.

This process frees the voter to vote however she wishes. She doesn’t have to consider the “electability” of her number one choice. She can put any film at #1, no matter how remote its chances. If that film doesn’t have enough support, her vote transfers to her #2 choice. Her vote is not wasted. This isn’t like a political race where a vote for a third party is essentially meaningless. (Update: It turns out that any film that receives fewer than 1% of first place votes does get redistributed.)

Surplus Votes: Realistically, voters have preferences beyond seeing just one film nominated. Maybe they want a certain one to win the Oscar, but would also really like to see another one nominated. In such a scenario, the voter may be tempted to vote for the one she likes less just to help it get nominated if she thinks the one she prefers will get lots of other support. This is especially true in races where it’s dead certainty that a contender will receive a nomination (Avatar, The King’s Speech, etc…)

The Oscar vote tabulators take this into account. If one nominee receives 20% more votes than it needs, all that film’s votes get reassigned to the next film on the ballot on a pro-rated basis. So if a film needs 500 votes to get nominated and receives 1000 votes, all those votes will be reassigned to the next film on those ballots and be worth 50% of a vote. So now the voter doesn’t even have to worry about wasting her vote on an obvious front-runner! A portion of her vote will go to another favorite. (Update: The surplus rule isn’t entirely thrown out either, it turns out. A film that receives 20% more of the 1st place votes it would have needed to cross the 1/11th threshold still has its votes redistributed on a pro-rated basis. The result is a distribution for films that receive more than roughly 11% of first place votes.)

The combination of Alternate Voting and reassigning surplus votes makes for a system that removes most incentives for gaming the system. Voting strategically isn’t going to get a voter much further than simply voting with her heart.

(Update: With Steve Pond’s reporting, we now know that the surplus rule is applied first. Then any film with less than 1% of first place votes gets redistributed. After that one round of redistribution, all films with 5% or more of first place votes receives a nomination for Best Picture. The result is many fewer wasted ballots than I feared, but still many more than under the previous voting system.)

An Arbitrary Threshold

No need to fill all of this out

I understand the desire to adjust the number of nominees based on the qualities of the contenders. “A Best Picture nomination should be an indication of extraordinary merit,” says Davis. “If there are only eight pictures that truly earn that honor in a given year, we shouldn’t feel an obligation to round out the number.” But requiring at least 5% of #1 votes is not the way to do it.

First, the number of films receiving 5% of #1 votes is not necessarily indicative of the strength of that year’s slate. More than anything it indicates the strength of the front-runners. If there is a strong front-runner or two, those two films could easily account for more than a third of #1 votes. Even if other potential nominees are broadly well-respected, it will be tough for many to hit that 5% level.

Awards Daily, a popular Oscar site, did a simulated ballot for its readers last year using the same voting system as the Academy. The top three vote-getters netted 65.6% of all #1 votes. As you can see, this left very little room for other films to also hit 5%, even well-respected ones that garnered lots of support once votes were reallocated. I know the audience for an Oscar blog is going to be different from the Academy membership, but similar patterns could certainly emerge.

A year that has many nominees wouldn’t mean that the crop of films that year was better. It most likely means the field is more even with no one or two films leading the pack and snapping up extra #1 votes.

So the 5% rule doesn’t have much to do with film quality, despite the justification from the Academy. What it DOES do is eliminate the also-rans. In a year with just a couple true contenders to win Best Picture, let’s just abandon the pretense and not nominate a bunch of films that have no chance to win, even if the consensus considers them great. In years that have a lot more films in the running for the win, let them in even if not all of them are that good. I don’t think this is what the Academy is trying to do but it least it would make sense.

The usual voting system spelled out above comes into play to determine nominees among eligible films. But the 5% target is so high that never once in the eight test cases the Academy studied was the usual counting system necessary. Eleven films would have to receive 5% of #1 votes before the weighted ballots came into play. That is very, very unlikely.

(The usual voting process could also come into play if less than five films hit 5%. In fact, only four films did so on the Awards Daily 2009 simulated ballot.)

Barring the unlikely event that more than ten or less than five films reach the 5% threshold, the slate of Best Picture nominees will be entirely composed of those films that hit the threshold. No alternate voting. No reallocation of excess votes. And therefore lots of incentive to vote strategically as voters try not to waste their ballots on long-shots.

An Example in Screwy Voting

We’ll use my 2009 Best Picture mock ballot as an example. My top five votes would have been:

1. In the Loop
2. Zombieland
3. The Informant!
4. An Education
5. Up

I had little expectation that In the Loop would garner a Best Picture nomination. But it wasn’t impossible. It was in the conversation for a screenplay nod and with ten nominees something could sneak in out of left field. Its chances were low but not nil. There was no risk to voting for it, however, because if it got eliminated my vote would move down to my #2 film, then #3, etc… Realistically this ballot would have resulted in a vote for An Education or Up as numbers 1-3 got eliminated.

Now, I really loved An Education. I wanted it to get nominated and I was concerned it was on the bubble. With the 5% rule, it doesn’t just need my vote to get nominated. It needs my #1 vote. So now I have a dilemma: do I vote for In the Loop, the film that I loved the most even though its chances were very slim under the old rules and are much slimmer with a 5% threshold, knowing that if it doesn’t hit 5% my vote will count for absolutely nothing? Or do I vote for An Education in case it needs my vote to cross 5%? The rational vote is the latter and I don’t think that’s a good thing.

Bad incentives make economists angry!

Think about it like this: I have two preferences. The primary preference is to get a nomination for In the Loop. The secondary one is to help An Education elbow out its competition for the final few slots. The old system lets me have both my preferences and assigns my vote based on how others vote. If my vote for In the Loop helps it, then my vote goes there. If my vote for In the Loop doesn’t help, then at least my vote can help my secondary objective. Under the 5% rule, I must make the choice and risk wasting my vote if I choose wrong.

Ultimately, there’s no reason to submit a list of ten films on the nomination ballot any more. In fact, there’s no reason to vote for more than one. Either your #1 choice gets 5% of votes and it is nominated or it doesn’t reach 5% and it’s not nominated. That’s it.

(Update: With the subsequent further clarification, the problem of strategic voting is somewhat diminished but still prominent. A voter can vote for a real long-shot with no risk. Once that film is eliminated the vote will redistribute to the next film on the ballot. However, a ballot is still wasted if the first place vote goes to a film that receives between 1% and 4.999% of first place votes. This is enough to give an informed voter pause and strategically alter her vote.)

Rational Voters?

All this analysis depends on voters being rational. Strategic voting is an issue only when voters understand the voting system. I think it’s fair to say Academy voters never really understood it to begin with and there’s a good chance the 5% rule will make it seem closer to their misunderstanding of what the process is anyway (i.e., they think it works like a political race). In 2009 there was a nonsensical campaign to list The Hurt Locker at #1 and Avatar at #10 thinking that it would somehow hurt Avatar.

I also don’t think Academy voters have been voting with their heart much anyway. I suspect many put at #1 the Oscar contender they liked the most, not their favorite film regardless of its place in the Oscar race. Therefore, the history of a combination of laziness and being too stupid to realize they don’t have to vote strategically could mute the effect of the new rule.

The new rule was devised with the help of PriceWaterhouseCoopers, the firm that does the ballot counting for the Academy. These are smart accountants. Did they not advise the Academy on the obvious problems of an arbitrary 5% threshold? It’s possible they realized that the 5% rule doesn’t affect the vote too much due to the unoriginality of Academy voters who all vote for the same group of potential nominees.

I sing the praises of the current voting process with its alternate voting system, but the Awards Daily experiments show that #1 votes are still king. Categories with five slots rarely have a nominee that finished outside the top six (or seven) in #1 votes. Very often after the lengthy vote reallocation process, the nominees are just the ones who got the most #1 votes. So maybe the results end up about the same.

I do think it makes it harder for smaller films. Does something like Winter’s Bone even bother with a Best Picture campaign knowing that 5% is going to be tough to reach? Because now it’s not only facing the normal challenges of a minnow candidate, but it also must face people thinking it has no chance to reach 5% so they won’t be wasting their vote on it. The type of momentum a smaller film needs to generate simply never materializes.

(Update: Now that we know there is a round of surplus rule reallocation and vote redistribution for films that received less than 1% of first place votes, this problem is somewhat diminished. Realistically, a film can receive a little less than 5% – maybe even as low as 4%? – and pick up some support via reallocation and redistribution.)

John’s Perfect Solution

All that said, I love the idea of altering the number of nominees based on the quality of films in the running. The Academy just needs better criteria. I’m sure they could unleash an egg head (like me! I’ll do it!) to create some sort of complex formula that measures broad-based consensus support for films.

But I think a workable solution is easier. The current ballot counting system is iterative. Votes are counted, films are eliminated, votes are reallocated, and the votes are counted again. Repeat until there are ten nominees. Simply cap the number of times you eliminate and reallocate votes. I don’t know what the optimal number of rounds is, but analyzing the data of past years should come up with a good number. 15? 20? If round after round passes without the slate of ten filling out, that’s a sign that there is no broad consensus on what films are quality enough to be nominated. That is the whole point, right?

In the 2010 Awards Daily example, the first six nominations were secured in three rounds. The next didn’t come until round 17. The last three came in round 20 when literally every other film had been eliminated. I think in this case it’d be fair to take the first six or seven qualified nominees and call it a day.

Realistically the round limit would probably have to be determined by how far apart the remaining potential nominees are. If there’s not a lot separating them then there is no consensus and you can be okay nominating none. But looking at it from a round-perspective should be a much better indicator of quality than an arbitrary 5% target.

Academy, please feel free to use my system. Just toss me a few tickets to the ceremony!

(Update: The rules clarification doesn’t prevent my solution from being the perfect one!)