Every year when the Hugo Award nominees or winners are announced, someone finds something to complain about. This year one of the obvious targets is the fact that only three works made it onto the ballot for Short Story because of something called the “5% Rule”. I have seen this described as “shameful”, as if some dreadful moral failing can be ascribed to, well, someone. Perhaps the mysterious “They”, who are often cited as the secret cabal that decides who gets what in the Hugos. As is usually the case, this is more an issue of statistics and rules than any deliberate malfeasance.
We should start with a few facts. This rule is not new. It has been in force longer than I have been involved with the Hugos, and has been invoked before. Kevin says it dates back to 1980. It does not apply only to the Short Story category. It is simply that the typical distribution of nominations in that category makes it more likely to fall foul of the rule than other categories. There is no conspiracy to defraud short story writers of their rightful nominations. Indeed, given that the Hugos have categories for three different lengths of short fiction, you could argue that they have more chances at a rocket than anyone else.
So what is this rule? What does it say, and why is it there? Here is the actual text from the WSFS Constitution:
3.8.5: No nominee shall appear on the final Award ballot if it received fewer nominations than five percent (5%) of the number of ballots listing one or more nominations in that category, except that the first three eligible nominees, including any ties, shall always be listed.
The first thing to note here is that whoever wrote the rule was well aware that it could result in a fairly thin category; hence the stipulation that there must always be at least three nominees. It is possible, though not likely, that none of the three stories on the ballot this year achieved 5% of the vote.
The purpose of the rule, fairly obviously, is that works must have a reasonable groundswell of support in order to get on the ballot. However, the reason why that might be a concern is not so obvious. So I’d like to take you all the way back in time to 2007.
That’s not long ago, but we’ve seen a huge increase in interest in the Hugos in recent years. In 2007, with Worldcon being held in Japan, only 409 people participated in the nominating stage of the awards (compared to 1343 this year). In Short Story only 214 people submitted nominations, which was actually quite high. For Fan Artist the number was 141 (statistics here). So the cut-off for getting on the ballot in Short Story was just 11 votes. For Fan Artist it was only 8.
There are two points to consider here. The first is, would you really want someone to be able to get on the ballot with less than 8 votes? Thankfully, for Fan Artist that year it wasn’t a problem. All of the people who did get on the ballot had well over the limit, because it was a case of the usual suspects getting nominations. For Short Story, however, there are different stories each year, and the spread of votes can be very broad and flat. All five nominees got over the 11 vote limit, but in the runners up there was one story on 14, one on 13, two on 12, two on 11, one on 10 and four on 9 (data here). It is all very tight. Ties for 5th place in Short Story are by no means unknown. And if the cut-off point is very low the chances of a 3- or 4-way tie for 5th place are quite high. We don’t want to be in a position where one vote could make a massive difference to how many people get on the ballot.
These days, of course, we have a lot more participation. The number of votes required to meet the magic 5% in Short Story this year was 34. We have no idea what the 4th and 5th place stories got, but I venture to suggest that the chances of a multi-way tie for 5th are a lot less than they would be if the cut-off was 11.
If we do want to make a change to the rule, what I would suggest is that we replace the limit of 5% with something like “5% or 30 votes, whichever is lower”. Obviously 30 is a number I have plucked out of the air, and I’m sure that there will be people who think it is shameful that any work should get on the ballot with less than a much larger number of votes. It is, however, a simple and workable solution, and in my opinion far better than abandoning the 5% rule altogether, which some people were calling for last night.
It would, of course, help us to make a decision if we knew the actual nominating numbers for Short Story this year. LoneStarCon 3 can’t release the numbers of votes for the three nominees, as that could influence the final ballot. However, they might feel that they can release the numbers (but not titles) of the stories that finished in 4th down to 10th. If it turns out that none of them got more than, say, 10 votes I’d venture to suggest that we are better off without them. If, on the other hand, 4th and 5th got 33 and 31 votes respectively then I think they deserved to be included.
Now it is up to fandom to decide what it wants. While not everyone can attend the Business Meeting, I’m sure that online debate will influence the opinions of people who can go. Have at it.
Wouldn’t we need to see a trend, first, before adjusting the 5% rule, though? I would be loath to change it before seeing if it has further impact on the category, really. It might be just a temporary blip, after all:
1965: 3 stories
1968: 3 stories
2013: 3 stories
It’s pretty rare when it happens. I’d wait another year, maybe two, and see what happens?
If we were being sensible, yes. However, according to Cheryl’s Second Law of Fandom reduced nominee lists are only a Dangerous Trend That Must Be Resisted when they are one-offs. If they repeat then they become a Scared Tradition That Must Be Preserved.
I do find it odd that people are insisting that there should be a minimum number of slots, though, something more than three (as set by the 5% rule), but where do you draw the line? Four? Five? More?
Also:
2011: 4 stories
And while Short Story is where this has happened most often, it also applied to the short-lived Best Original Artwork category in 1994 and 1995, when there were only three nominees.
“Obviously 30 is a number I have plucked out of the air, and I’m sure that there will be people who think it is shameful that any work should get on the ballot with less than a much larger number of votes.”
That’s a pretty interesting observation. If the minimum number of nominations had been 50 last year, there wouldn’t have been much left to vote for. http://www.thehugoawards.org/content/pdf/2012HugoVotingReport.pdf (From page 20.)
This is what would be left in the categories with less than 50 nominations required to be on the final ballot last year (, ignoring the three nominees minimum):
-Four in best novella
-Two in best novelette
-None in best related work
-One in best graphic story
-Three in best dramatic presentation, short form
-Three in best editor, long form
-One in best professional artist
-Two (+Clarkesworld) in best semiprozine
-One in best fanzine
-Three in best fan writer
-None in best fan artist
-Three in best fancast
-Two in the John W. Campbell award
My point being, the number of nominations required to be on the final ballot are so low (,the highest being 105 in dramatic presentation, long form, ) that even the 5% rule seems silly.
In my opinion the Hugo Awards should stick with five nominees in each category, or remove alltogether the categories that don’t get enough nominations. As it stands now the 5% rule is clearly a disadvantage for the short fiction categories where the number of potential nominees is high.
-I don’t see much chance of any rule change though, since setting a minimum numberr of nominations would more or less kill the fan categories, unless there was block voting of course…
You are right, of course, but people who complain often know very little about the awards. There are people around who are outraged, OUTRAGED, that you can get on the ballot with less than 1000 votes.
It’s 5% of the category, not 5% overall.
I suspect that this shows that there is too much short fiction out there, and not enough recommended lists.
I think if something got on the ballot with only 1%–quite possible if there are, say, 4 blockbuster movies and a scattering of no-name indie movies (the BDP-long usually gets at least 600 ballots), I’d be unhappy, even if it did round out the field.
Doing the f/m numbers is much more straightforward if there are always five nominated works per category and each work is created just by one person, although two is OK. Now, I’m not some sort of Procrustean bean counter who runs around lopping off legs or crushing people just a little bit to make them fit into narrow categories but I don’t think it is so outrageous of me to hope the voters will lop off the odd foot for my numerical convenience.
Didn’t people complain about the six nominees for novella last year? Six is a much less pretty number than five.
You can always rely on fandom to find something to complain about.
Having thought a bit more about it, I have come up with an idea for replacing the 5% rule. One I think qould be fairer to categories with a large number of nominated works. It looks like this:
(Ballots cast/nominated works) x ? =cut off point for inclusion on final ballot
Where ? would be how many times more than the average votes of the nominated works/individuals in any category that would be needed to make the final ballot.
For example, let’s say there’s 700 ballots cast for short story, and 70 different works, and you need three times the average to get on the final ballot, it would look like this.
(700/70) x 3 =30
I don’t have the access to how many works/individuals are actually nominated in each category, so I don’t know how that would work out for all the categories. But does seem a fair way of doing things for the short fiction categories.
Was there some event that inspired the 5% rule?
I don’t know. The rule was adopted as part of a huge number of changes to the WSFS Constitution in 1980-81 (something of a wholesale rewrite, I gather), and my own history with WSFS only goes back to 1984. Ben Yalow was one of the authors of the proposal, but I haven’t asked him about it.
Note that the Business Meeting periodically seems to want to monkey around with the rules to make it more likely that No Award wins; that was the genesis of the complex-but-unlikely-to-ever-affect-a-category No Award Showdown rule. There seems to me to be a strain of rules lawyer who is troubled about what they perceive as “unworthy” works winning, and who try to devise ways to prevent those “unworthy” works from even making it onto the ballot.
Nonetheless, I’m in favor of something that prevents an eighteen-way tie for last place on the ballot, which is likely to happen in a category with a very long, flat tail of nomination interest. I think Cheryl has a good compromise, although of course one can argue about what the alternative absolute count should be, and I have no strong feeling about it myself.
I’m glad that every year the Hugo list inspires debate. The time to start worrying is when it doesn’t.
I’m also glad that we don’t summarily dismiss complaints by lumping them together, but by carefully examining them.
Here’s my concern: in the year 2013, with perhaps more magazines and anthologies than ever before, with almost 700 ballots cast, the best the ballot can provide is 3 choices in the short fiction category. There are 10 movies or television shows to choose from, but only 3 short stories. 3 short stories, in a field rich with amazingly talented writers sharing a lot of great stories. 5 short fiction editors, but only 3 short stories. And woe to voters who took the time to vote with choices that didn’t make the 5% rule, because we’re only going to have 3, and 2 of them from the same (very excellent) magazine. That’s not what I call plentiful options expressing the wide diversity in short fiction today.
Actually, I would suggest that the result points to too much diversity, given that there is so little consensus about what it good in the field of short fiction that you couldn’t get five works that could muster at least 34 nominations out of 700 cast.
Thanks Sandra. There’s a lot to unpack there, so I’m going to elevate this to a post in its own right, hopefully later today, so I can discuss it more.
I guess I don’t understand; if a rule sometimes results in something deeply stupid happening–only three nominees when there are literally dozens or even hundreds of worthy stories–change the rule. What’s the difficulty?
Because what if something considered even more deeply stupid — twelve or fifteen candidates on the ballot due to the long, flat distribution — happens if you change the rule? Or what if works end up making the ballot having polled only 1% of the total votes in the category and when you see the top-15 you realize that there is essentially no difference between fifth and sixth, seventh, eighth places? It’s not as clear-cut as you make it out to be. Are there really “dozens or even hundreds of worthy stories” if only three of them (or possibly fewer) can make the ballots of less than 5% of those people making nominations in the category? The members of WSFS don’t make rules without reason, and there’s a reason for the 5% rule. You may not agree with the reason, but it wasn’t done without thought, and changing it needs more than simply dismissing the existing rule as “deeply stupid.” (Indeed, there’s a good chance that telling the people who passed the rule that they are “deeply stupid” is likely to make them stop thinking and simply defend the existing rule for its own sake rather than pay attention to a reasoned argument. Politics is like that.)
I didn’t say the people who passed the rule were stupid; I didn’t say that the rule itself was stupid; I said the rule “sometimes results in something deeply stupid happening”. Very specific. Perhaps too harsh; perhaps I should have qualified “something that seems to me deeply stupid”. But I did not “dismiss the rule as deeply stupid”, nor did I accuse “the people who passed the rule” of being deeply stupid. No.
But seriously: Is it really the case that out of the many (hundreds?) stories submitted, only three are really worthy of being considered for the Hugo? Truly? That seems statistically unlikely in the extreme.
The reasons don’t matter, honestly; the results matter. In an internet environment, it is extremely easy to imagine so many stories being nominated that only 3 achieve the 5% threshold, not unlike when two actors from the same film are nominated for an Academy award, splitting the vote and potentially throwing the award to a third actor.
With regard to your example, that’s actually a simple one to fix; if you end up with 12 nominees, have a second “runoff” round of voting to trim it down to 5 or what have you. That’s an easy thing to solve.
But to reiterate: I did not say the rule was stupid; I did not say the people who created the rule were stupid; I said it sometimes produced stupid results.
Trust me, I know the people who regularly attend the Business Meeting. No matter how much you protest otherwise, unless you tone down your rhetoric, you’ll be perceived as telling the regular attendees they’re stupid. I didn’t say this was fair; I’m telling you what is actually going to happen on the ground. If you really want to make something happen, you have to win the hearts and minds of the people most likely to actually turn up and vote at the Business Meeting.
I know it sounds simple to you, but trying to add an additional round of voting to the Hugo Awards for your runoff is probably a non-starter. It already takes months to conduct the balloting, and adding another round would shorten the amount of time the final ballot could be out. Now possibly you’re thinking you could change the nominating phase to some form of instant-runoff voting, where people would have to rank the relative merits of all of their nominations rather than all of them being equally weighted. That is slightly less complicated — after all, we already use IRV for the final ballot — but still something unusual and unusual things are generally viewed with great skepticism by the Business Meeting, which is a fundamentally conservative (in the sense of “reluctant to change,” not related to any form of American political party) institution.
(Feel free to complain that it’s a gathering of science fiction fans that are resistant to change. That doesn’t change reality, and as a WSFS politician of long standing, I’m mostly interested in what is likely and possible, not what is ideologically pure. That’s how I make change happen.)
I think you need to convince a majority of the regular Business Meeting attendees that the current system is actually broken. Many of them are unlikely to think it is. It’s like the Iditarod “Dead Dog” Rule that disqualifies a racer if any of his/her dogs die on the course; the first time a dog died and a popular musher was accordingly disqualified, there were protests about how “unfair” it was. But the rule was there for a reason, and the fact that it affected someone popular shouldn’t have anything to do with it.
Anyone wanting to make a change needs to make the case that works receiving less than 5% of the votes cast in the category are more than just statistical noise. The burden of proof is on those who want to make the change, not upon those who made the rule in the first place.
Note that I personally think that there is some change that might be beneficial here (I favor Cheryl’s change because, among other things, it’s the least disruptive, and therefore the most likely to actually pass) but I do in fact understand the reason for the 5% rule and do not dismiss it out of hand.
[Sigh] Nor did I “dismiss it out of hand”.
You actually answered my question tho, Kevin; 1) the people charged with rules oversight are a conservative bunch with lots of inertia to overcome, and 2) a runoff is too complicated.
With regard to my, I dunno, “tone” I guess: It’s not a matter of me “protesting otherwise”; I made several clear, declarative statements. If other people impugn meaning to them that isn’t there, I can’t do much about that. (I am reminded of Heinlein in “Expanded Universe” lamenting about what he considered unfair criticisms of his work, “usually based on a failure to understand simple indicative English sentences, couched in simple words.”)
I have no plans whatsoever to try to change hearts and minds. I do not care nearly enough about the issue to go diving in and playing politics–something I am inordinately bad at–with a bunch of people I do not know. I think it’s unfair to the writers left out, but there my interest ends. And if me posting a couple of sentences–literally two sentences–in a comments section on this blog is enough to inflame their passions, then I really don’t want to engage in the politics.
So there you have it.
The reason I do all of these long posts is to try to explain that what might seem “deeply stupid” is more complex than it seems, and that many of the proposed solutions are likely to cause even more outrage than the issue that they are trying to fix. The trouble with people who are angry on the Internet is that they often want a benevolent dictator who will fix things the way that they want, immediately. They don’t consider that other people may have valid reasons for wanting things to stay they way they are, or that the process of change might be long and complicated. And, as Kevin notes, if you do make the change that people are demanding, you may well find yourself being yelled at the next year for having done it, when it has exactly the effect you predicted it would. Sometimes there are no easy solutions, no matter how much we might want them.
Thank you for the thoughtful response, Cheryl! Very much appreciated.