On the morning of the eliminations the bracket goes up at the entrance to the field, and next to the first name there is a 1. Everyone reads that number the same way, as a verdict already delivered: that archer is the strongest in the field, shot better than anyone else the day before, and from here on every result will be weighed against the label. If that archer goes out in the quarter-finals, it counts as a disappointment, and if the 32nd seed reaches the semi-finals, it counts as a surprise.
That number says less than it is asked to say, and the gap can be calculated. In a top-level field simulated with archers of known ability, the top qualifier really is the best archer present a little over once in seven; the top eight really are the best eight four times in ten thousand; and every archer enters the bracket carrying a label that is off, on average, by eight and a half places. Miscounting and inattentive judges have nothing to do with it: the cause lies in the relationship between how fine the question is and how precise the instrument answering it can be.
The model's figures come from work I published in the Lab, where the calculation is set out in full and every assumption is declared one by one. Towards the end I set them beside what the results of major international competitions show, which tell the other half of the story: the ranking often misses the fine order, but the band in which you finish qualification weighs heavily on who reaches the medals.
How a competition is built
In Olympic recurve it opens with a qualification round in which everyone shoots 72 arrows at 70 metres, on a face 122 cm across divided into ten scoring zones of equal width, 6.1 cm each. Inside the ten sits the X, an inner ring 6.1 cm across that scores ten like the rest of the zone and is counted separately to break ties. The theoretical maximum is 720 points, and the totals from that day produce the ranking.
The ranking awards no medals: its job is to set the pairings. In a bracket of 64, as at the Olympic Games, first meets sixty-fourth, second meets sixty-third, and so on. From there the competition becomes single elimination, with six rounds to survive and 63 matches to produce a winner, 32 of them in the first round. Individual matches are shot in sets of three arrows, five at most, with two set points to whoever wins a set, one each for a tied set, and six set points to close the match: that is the arithmetic I have written about elsewhere.
This article concerns only the first link in that chain: what the ranking measures, and how precisely. The set format and the shoot-off arrow come later and are not needed to answer the question.
What "the best" means here
The criterion has to be stated before the numbers, because without it the percentages mean nothing. In the work these figures come from, the best archer is defined in one way and a fairly narrow one: the archer who, stepping onto the line, expects the highest average score. No other quality enters the definition, and the exclusion is deliberate.
So nerve in a close finish, the ability to read a turning wind, coolness on a shoot-off arrow and the knack of recovering after a lost set all sit outside the calculation, and those are qualities archers recognise perfectly well and nobody considers irrelevant. Anyone who rejects that definition of ability has no reason to correct the numbers below, because what they are rejecting is the question those numbers answer: the disagreement concerns the definition and leaves the calculations intact.
The three numbers
To measure what a ranking is worth, you have archers of known ability shoot, meaning archers whose expected score is fixed in advance, and count how often the order produced by their 72 arrows matches the actual order. Repeat that over sixty thousand qualification rounds for each scenario and the answer stops depending on the luck of any single day. For a compact field, with 64 archers spread between 690 and 665 points, the results are these.
| What the ranking is asked to do | How often it manages |
|---|---|
| Put the best archer in the field first | 14.2 times in 100, a little over once in seven |
| Put the best eight in the first eight places | 4 times in 10,000 |
| Put each archer in the place they deserve | mean error of 8.6 places |
There is another way of putting the first row that conveys the order of magnitude better than the percentage does: on a circuit of seven competitions a year, the number one seed is on average the strongest archer present just once, and in the other six it is somebody else, without anyone noticing, because the bracket does not say so.
The second row is the one that surprises most. The eight archers sitting in the first eight slots turn out to be the eight best in the field four times in ten thousand, which is fewer than one competition in two thousand. So whenever protecting the seeds comes up for discussion, the group being protected is almost never the right eight.
The third row says the same thing in a more usable form. With a mean error of eight and a half places on a grid of 64, the number next to a name marks a stretch of the ranking rather than a single slot: an archer who deserves tenth place ends up, on average, anywhere between second and nineteenth.
Why a good measurement makes a bad ranking
The explanation does not lie in any defect of the qualification round, because as a measuring instrument 72 arrows work well. Asked to compare two archers 10 points apart, they put them in the right order 95 times in 100, which means one error in 20.
The trouble comes from the question put to the instrument. In the compact field all 64 competitors sit inside 25 points, so two neighbours in the ranking are separated on average by four tenths of a point rather than by ten. Over a gap that size the same 72 arrows point to the better archer 52.9 times in 100, barely better than tossing a coin; and since a ranking of 64 names rests on 63 comparisons between neighbours, that uncertainty accumulates down the whole list instead of staying confined to one pair.
The image the essay uses is a kitchen scale. The scale works, and it will tell one bag of flour from another that weighs 100 g more without hesitating. What it is being asked to do here is put 64 bags in order when they differ from one another by a gram, and no kitchen scale can do that however good it is.
Four tenths of a point, seen on the target
To see how out of proportion that request is, it helps to turn the points into something physical. The model describes each archer with a single number, the spread of the group, meaning the typical distance at which the arrows come to rest from the centre (the statistician's sigma). For a 700-point archer it is 3.78 cm; at 690 it is 4.46, at 680 5.14, at 670 5.81 and at 660 6.49, and the lower the score, the larger it gets, with an almost perfectly regular step.
The spread is not the diameter of the group you see on the face, which is always much wider: an archer with a spread of four and a half centimetres puts two arrows in three inside a circle a little over 13 cm across. Put in golds, a 700-point archer puts 72 arrows in 100 in the ten, and a 630-point archer 22.
With that conversion to hand, 10 points out of 720 stop looking like a detail. Going from 680 to 690 means reducing the spread by almost 7 mm, about 13%, and between 690 and 700 the same ten points are worth 15%. Two archers 10 points apart differ by something that 72 arrows recognise in 95% of cases.
Four tenths of a point, by contrast, are worth a little over half a percentage point of spread, which on four and a half centimetres comes to a little over a quarter of a millimetre. Asking 72 arrows to separate two archers who differ by a quarter of a millimetre is asking for a resolution that number of shots does not have, and the ranking answers with the only thing it can produce, which is an order decided largely by chance among neighbours.
It depends who is in the competition
All the figures so far are for a compact field, and the same qualification round performs a great deal better when the field is put together differently. Without that caveat the whole argument would read as an accusation against the measurement rather than a description of its limit.
| Shape of the field | Top qualifier is the best | Top eight are right | Mean error |
|---|---|---|---|
| Compact, 690 down to 665 points | 14.2% | 0.04% | 8.6 places |
| Bimodal, 16 archers from 692 to 680 plus 48 from 655 to 630 | 24.8% | 1.9% | 6.8 places |
| Spread, 695 down to 625 points | 32.4% | 5.1% | 4.1 places |
In a spread-out field, where 70 points separate first from last instead of 25, the mean error more than halves and the top qualifier is the best archer almost once in three. Anyone reading the results of a regional competition rather than a World Cup stage is therefore looking at a ranking that carries more information. The essay also notes that in a competition like that the best archer is visible to the naked eye, so the measurement matters least exactly where it works best.
The row that carries weight is still the first, because a top-level field tends to be compact: you get there through selection, and selection removes the distant archers and leaves the ones who resemble each other. As the level rises, then, the qualification ranking becomes less informative, not more, which is the opposite of how it is read.
Doubling the arrows does not fix it
The obvious proposal is to shoot more, and it moves in the right direction without changing the substance. Take the qualification round to 144 arrows and, in a compact field, the top qualifier is the best archer in 18.9% of cases instead of 14.2%, the top eight come out right 0.3% of the time, and the mean positional error falls from 8.6 places to 6.5.
That is a real gain, and it doubles the time spent on qualification at every event on the calendar. What remains, though, is still a ranking that puts the best archer first less than once in five, and the reason it returns so little is arithmetic: the precision of a repeated measurement grows with the square root of the number of trials, so halving the uncertainty would take four times the arrows.
Ties and distances from the centre
Inside the qualification round there are two smaller levers, and they have been measured too because they come up often on shooting fields. The first is the set of criteria used to break ties on score, meaning the count of tens and of Xs. Against a ranking in which ties are drawn by lot, counting Xs takes the probability that the top qualifier really is the best from 14.0% to 14.2%, and adding the count of tens does not move it.
The second lever is more radical and consists of ordering the qualification round by the distance of the arrows from the centre rather than by score, recovering the information the scoring zones throw away every time they round an impact to the value of its zone. That one works: the top qualifier becomes the best archer 15.9 times in 100 instead of 14.2, and the mean error falls from 8.6 places to 7.7. On the title, though, the effect stays at half a percentage point, for a reason that becomes clear once you look at how the bracket uses the ranking.
The toll of seeding
An error of eight and a half places does not stay confined to the first round, because the bracket goes on trusting the ranking for the whole competition. The essay measures that cost separately and calls it the toll of seeding: you run the same structure twice, once with seedings that reflect actual ability and once with the ones the qualification round produced, and look at what is lost between the two.
Every structure pays in proportion to how often it consults the ranking. The standard bracket, which turns to it once at the start, loses seven tenths of a percentage point on the probability that the title ends up with the best archer. Quarter-final repechage, which consults it again to place the archers who have been knocked out, loses 1.6 percentage points. Reseeding, which rebuilds the pairings every round and therefore consults the seeding six times, loses 4.8.
The structures that look on paper like the best correctors of bad pairings are therefore also the ones that lean hardest on an imprecise measurement, and they pay most for doing so. The toll does not make them worse in absolute terms, since the advantage they bring can outweigh it, but it does explain why the net balance always comes out thinner than the logic of the format would lead you to hope.
Why a fairer ranking is not enough
From all this it would be natural to conclude that the fix is a better qualification round. The essay tests that route as well, with a result that runs against intuition: raise the agreement between the ranking and the actual order from 0.81 to 0.99, which is to say buy an almost perfect measurement, and the standard bracket goes from 9.7% to 10.4% of titles awarded to the best archer, a difference that stays within the noise of that simulation and so amounts to no gain at all.
The reason is that the bracket consults the ranking to decide who meets whom, not to find out who is best: what matters is how it distributes opponents across the matches, and on that distribution a perfect ranking changes little, because it still sends half the field to knock each other out in the first round. Applied to a structure with reseeding, by contrast, the same data take the figure from 11.1% to 14.0%: the same improved measurement returns four times as much the moment the structure is able to use it.
The order of operations, then, is the reverse of the one most often proposed: change the structure first, then improve the measurement. Doing it the other way round, which is also the easier decision because it leaves the rules of the elimination rounds untouched, produces no measurable result in the model.
Who actually wins the title
Moving from the ranking to the whole tournament changes the question: how often does the gold end up with the best archer in the competition? In the model, with the structure currently in force, that happens 9.7 times in 100, which is one competition in ten.
The two figures measure different things and have to be kept apart. The 14.2% is about the ranking, meaning how often the top qualifier is the best archer in the field, and it is a property of the measurement; the 9.7% is about the whole tournament, meaning how often the title ends up in the hands of the best archer after six rounds of single elimination, and it is a property of the competition. Neither says how often the number one seed wins the competition: both are about who is the best archer, not about who wins.
What actually moves the second figure is where the seeding comes from, more than how precise the qualification round is. Use the season rankings in place of the score from the day, and rebuild the pairings at every round, and the title goes to the best archer 13.7 times in 100 instead of 9.7: four more titles in every hundred editions, without an extra arrow or an extra minute of competition, but with a change in the way the bracket is built. Even then, in six competitions out of seven the title still goes to somebody who is not the best archer present, and that is a property of single elimination rather than a fault anyone can correct.
What the calculation does not look at
These numbers come out of a modelling study and not from a collection of real results, and the essay says so at the outset: no competition data was used. The archers are constructed profiles defined by their expected score, and the three fields of competitors are put together on paper, with their score ranges declared one by one. Even the 25 points separating first from last in a compact field are a stated assumption rather than a measurement taken on World Cup fields.
On the simulated values the essay recommends reading two significant figures rather than three, and treating differences below half a percentage point as absent. When I write 9.7 or 13.7, those figures carry a margin of a few tenths, and comparisons have to be made with that margin in mind. The same quantity, measured by different runs of the same model, appears elsewhere with slightly different digits.
Finally, the direction of the limitations is declared: four of the model's five simplifications push towards a smaller ability to tell archers apart, not a larger one. Wind, for instance, widens everyone's groups, and wider groups make neighbours even harder to separate. That is why the essay reads its own numbers as upper bounds rather than central estimates.
What international competitions show
The model shows how badly the ranking handles the fine order. The results of major international competitions show the other side, which is how much the band you finish in still matters. At one World Cup stage and the World Championships in 2013, almost nobody reached the medal matches without having finished qualification inside the top twenty; across all 182 major international competitions from 2000 to 2014, gold went to an archer in the top 16 of qualification in 90% of cases, in the top eight in 76% and in the top four in 54%. In the registry the claim that medals almost always go to archers who finish the qualification round in the first 20 is rated reported: both analyses are by the same author, on partly overlapping data and on competition formats that have changed since, so the threshold of twenty is best read as an order of magnitude.
The two readings do not contradict each other, because they answer different questions. The model asks whether the ranking puts the best archer exactly in first place, and the answer is hardly ever; the competitions ask which band of the ranking the winner comes from, and the answer is nearly always the top. A mean error of eight and a half places moves an archer by a few slots, not from the top to the tail: the archer who deserves tenth place, as we saw, stays on average inside the top twenty. The bracket, in turn, sets the archers at the top against the lowest-ranked in the early rounds.
In qualification, moreover, points turn into places quickly. In the middle stretch of international outdoor rankings over 72 arrows, one place is worth between 0.39 and 0.84 points depending on the division: one point, in other words, is worth from just over one place to about two and a half, and indoors over 60 arrows it reaches almost four. There are usually a few outstanding scores at the top and a collapse in the tail, so the figure holds only for the middle. The claim that in the qualification round one point is worth from under one place to almost four is also rated reported, because all the figures come from the same author.
The four tenths of a point between neighbours in the compact field sit at the dense end of that range, close to the values for the men's divisions. The comparison is only a guide: the scores in a real ranking also contain the luck of the day, which widens the gaps compared with those between the archers' actual levels.
What to do with a ranking
For you as an archer there are two consequences, and they belong together. The first is that your finishing position, as a measure of ability, is a noisy label, and so is your opponent's. Four places in the middle of an outdoor ranking are worth between a point and a half and a little over three, and a difference like that does not establish which of you is more precise: with a three-point gap over 72 arrows, confidence in the order of the two archers is no higher than 68%, as I worked out in how many points it takes to say one archer is better than another.
The second is that, as a fact of the competition, your position counts all the same. In the middle of an outdoor ranking every point moves your name more than one place, and the places at the top decide whom you shoot against in the early rounds and, judging by international competitions, who has a realistic chance of reaching the medals. The point you dropped in the first end says little about what you are worth, but in the ranking you pay for it anyway.
The reverse holds too, and it is the thing to keep in mind the evening before a match: meeting the number one seed is not the same as meeting the best archer in the field, because in the model that number points to the best archer a little over once in seven.
For coaches, the implication is that a finishing position from a single competition is not material to build an assessment on, while the series of finishing positions across a season is. The reason is the same one that makes the average of several competitions tighten the uncertainty around a score, while the score from a single competition leaves it wide.
Where this goes next
The volume the model's calculations come from goes well beyond the qualification round. It measures alternative bracket structures, repechage, double elimination, match formats and season rankings, and it arrives at a price list of what each choice of rule costs and what it returns. The result that holds it all together is the mismatch between what gets discussed and what actually counts. Changing the format in which the match arrows are shot changes the outcome of about a third of a match out of 63, while the instrument that decides who meets whom leaves the order of neighbouring pairs largely undetermined, and it seldom gets questioned, because it produces a list of numbers that looks like a verdict.