It happens more or less the same way every time. An archer changes something in the shot, a slightly different bow grip, a corrected anchor, a few weeks of work on the back, and at the next competition scores four points more than at the one before. By Monday the conclusion is already drawn: it worked. Had the four points gone the other way, the opposite conclusion would have been drawn with equal confidence, and in both cases the same instrument would have decided it, namely the total from one competition.
That total, though, holds two things together that are better kept apart: the archer's ability and the luck of that particular afternoon. The share belonging to luck is much larger than most people imagine before seeing it calculated, and knowing its size changes several practical things: how a ranking should be read, when it makes sense to say one athlete is better than another, and above all how long it takes before a score can confirm a technical change.
The calculation can be done, and the answer is a number: between two top-level archers, saying with 95% confidence which of the two is the more precise takes a gap of a good ten points or more over 72 arrows, and below five points the ranking says almost nothing. The calculations come from work I published in the Lab, where they are set out at length and the code to redo them can be downloaded; the studies in the registry add the rest, namely what competitive archers actually score, what a shorter test round is worth and how much a level label tells you.
Seventy-two arrows are a finite sample
A qualification round is made of 72 arrows, and 72 arrows are a finite sample in exactly the way 72 measurements of anything else would be. If the same person, with precisely the same ability, shot that round ten times over, they would post ten different totals, and the difference between the highest and the lowest would not be a negligible detail.
This happens because every arrow comes out of a distribution of impacts around the aiming point, and where any individual arrow lands inside that distribution is down to chance rather than ability. Ability decides how tight the distribution is, meaning how close the arrows sit to one another over the long run, while the total from one specific competition also depends on how those particular 72 draws happened to fall. A very strong archer can have an afternoon where the arrows settle unluckily on the ring boundaries, and a weaker archer can have one where they all land on the good side of the line.
To gauge the size of that effect, you need a way of turning points into something physical, because as long as the argument stays in points the number remains abstract and everyone projects onto it whatever importance they prefer.
One point is seven tenths of a millimetre
Such a conversion exists, and it is surprisingly stable. The essay works it out for recurve at 70 metres on the 122 cm face, and it measures the group by its typical spread, meaning how far the arrows usually stray from the centre in each direction: this is not the diameter of the group, which is several times larger, but the statistical measure on which everything else depends. Between 630 and 700 points out of 720 the relationship between expected score and that spread is almost a straight line, and a single qualification point corresponds to about seven tenths of a millimetre, less than a millimetre.
The consequences show up immediately: two archers separated by three points on a ranking, who look worlds apart on paper, differ by barely two millimetres, and eight points, which look like a chasm, come to five and a half millimetres. For a point of reference, an archer who averages 690 has a spread of about four and a half centimetres and puts roughly six arrows in ten in the 10; one who averages 650 sits a little over seven centimetres and reaches the 10 three times in ten. Between those two levels, which are a very long way apart on the scoreboard, lies a little over two and a half centimetres.
Bear in mind, too, that 630 out of 720 is already close to nine points per arrow, the range of archers who shoot for a national squad. In the studies that have measured competitive archers, six samples in five countries, the mean per arrow sits around seven, not nine: the most complete sample, sixty Indian archers with national competition experience, starts from about 7.4 over 72 arrows. At that level the group is wider and the total fluctuates more, to the point that in a group of Indonesian club archers averaging around seven points per arrow the standard deviation between arrows from the same archer is about two and a half points. The figures that follow are calculated for the top level, and for a club archer they describe the most favourable case.
Asking 72 arrows to separate a difference of two millimetres reliably is asking more of an instrument than it can give. Seventy-two arrows are not a small number in absolute terms, but the natural variability of the movement, arrow after arrow, produces swings in the total of the same order of magnitude as the difference you are trying to measure. The signal, in short, is smaller than the noise that comes with it, and the next step is to measure how large that noise is.
Two identical archers finish five points apart on average
The cleanest way to measure it is an experiment you cannot run in the field but can run by calculation. Take two top-level archers who really are exactly identical, meaning the same group, the same ability, no difference of any kind, and have them each shoot 72 arrows as in a qualification round. Then look at the gap that appears between them on the ranking, knowing that gap cannot have been produced by anything other than chance, because by construction there is nothing to tell them apart.
The mean gap between two identical archers comes to five points, and one time in three it exceeds six and a half. For two people who are indistinguishable on the target, you would instinctively expect a point or two of difference, but the calculation gives five, with tails that regularly reach further.
Anyone who has looked at a qualification ranking recognises the order of magnitude. In the middle of the ranking, away from the top and the tail, one point is worth from under one place to almost four depending on format and division, and in the men's recurve qualification at the London 2012 Olympics it was worth about two. Five points therefore separate archers several places apart, and it is the gap usually read as a difference in ability. The calculation says it can be produced entirely by chance, between two athletes whose ability is identical.
How much confidence a gap gives you
From the mean gap you reach the question that actually matters, which is how much confidence to place in an observed gap. When one top-level archer sits five points above another on a ranking, the probability that they really are the more precise of the two, rather than simply the one whose afternoon went better, can be calculated for any gap.
| Gap in qualification | Confidence in the actual order | How to read it |
|---|---|---|
| 1 point | 56% | Barely better than a coin: the order is undetermined |
| 3 points | 68% | About one time in three the actual order is reversed |
| 5 points | 79% | An indication, not yet evidence |
| 8 points | 90% | The signal starts to lift clear of the noise |
| 12 points | 97% | A difference in precision hard to put down to chance |
| 16 points | 99% | Practically certain |
The conventional threshold for saying that something has been established, in any discipline, is 95%, and reaching it takes a gap of a good ten points or more. Below five points you are largely reading noise, and every time two consecutive archers on a ranking are separated by one, two or three points, which is the ordinary situation in any closely matched field, their order relative to each other is essentially undetermined. The table holds between two top-level archers: further down the total fluctuates more, and the same gap says even less.
Why these numbers are optimistic
Two things need saying about that table, and neither makes it more forgiving. The first concerns what the calculation treats as unknown: the confidence is derived as though nothing were known about the two archers beforehand, meaning as though, before looking at the scores, every possible difference in ability between them were equally plausible. In a real competition, though, it is well known that archers close together on a ranking are almost always close in ability too, and allowing for that would make an observed gap even less conclusive than the table claims.
The second is that the calculation measures only arrow-by-arrow chance, holding the archer's form perfectly constant from the first shot to the last. It is the noise that would remain if the only source of variation were the dispersion of the group, whereas in reality everything the model does not look at is added on top: the previous night's sleep, the archer's condition that afternoon, what is going on in the head; nearly all of these widen the range rather than narrow it. The only measurement of day-to-day variation comes from kyudo: fifteen university archers, twenty arrows at 28 metres on two or three mornings of the same week, and between the best and the worst day the average hit rate goes from 83.7% to 61.7%. Part of that gap is produced by chance, and the study does not separate it; that is why in the registry the claim that an archer's accuracy varies a great deal from one day to the next is reported, with a single study behind it.
There is one exception, and it runs the other way: a disturbance that affects both archers alike, such as a wind that turns for everybody or light failing across the field, partly cancels out in the difference between the two scores, and in that case the gap becomes more informative. The net sign of everything the model leaves out therefore remains an empirical question, but the sources that affect each archer separately are the more numerous, which is why the five points of mean gap, and the ten that follow from it as a threshold, should be read as an optimistic estimate of the actual ambiguity of a ranking.
What to do with a ranking
For anyone coaching, that optimistic estimate turns into a rule to apply every time a results sheet comes up: if an athlete finished fourth rather than second, and three points separate them from second place, that gap on its own establishes no stable difference in precision between the two, and building a technical analysis on top of it means explaining something that may not have happened. If instead they came twelfth rather than fourth, with fifteen points between them, that is a reliable signal, and it makes sense to find out where it came from.
A qualification ranking sorts distant groups well and neighbouring ones badly: it says something solid about an archer at 650 sitting below one at 690, and almost nothing about the order between two neighbours on the list. Anyone measuring themselves against the clubmate a couple of points above or below is chasing a difference that the ruler used to establish it cannot see.
The same logic applies to comparing two competitions by the same archer, which is the commonest case of all. Last Sunday's total and next Sunday's will almost certainly differ by a few points, and that difference, taken on its own, says nothing about what happened in between. That is why an archer who wants to know whether they are improving needs a strategy other than comparing two consecutive numbers.
How many competitions it takes to demonstrate an improvement
There is a way out, and it calls for more measurement rather than better shooting: the luck of a single competition cannot be removed, but it can be diluted by averaging across competitions, and the dilution is quick, because the variability of a mean falls with the square root of the number of rounds. Four competitions halve the uncertainty compared with a single one, and nine reduce it to a third.
From there you reach the practical question, which is how many competitions are needed before and after a technical change for an improvement of a given size to become demonstrable. For an archer shooting around 650 points, the calculation gives these figures.
| Actual improvement | Competitions needed per period | What that means in practice |
|---|---|---|
| 3 points | 29 | About two seasons of competitions, before and after |
| 5 points | 11 | A full season per period |
| 8 points | 5 | Within reach of a normal calendar |
| 10 points | 3 | Visible within a handful of competitions |
| 15 points | 2 | A jump the score recognises straight away |
Three points gained over a season is excellent technical work, and demonstrating it through score would take about thirty competitions, which in practice is two seasons. Ten points, by contrast, show up in three competitions per period. Most real technical changes fall between those two extremes, and that is why score on its own rarely settles the question within the season in which the change was made.
Performance level then shifts things again, and in the least convenient direction. The same three-point improvement, for an archer scoring 690, takes thirteen competitions rather than the twenty-nine needed at 650, because the total of an archer who groups tighter fluctuates less. Cross-sectional studies point the same way, and in the registry the claim that stronger archers are also the most consistent ones is reported, with a caveat that matters here: the link is partly geometric, because an archer shooting close to the 10 has little room above. The archer still in the middle of changing things, who has most need of quick feedback, is therefore also the one who has to wait longest to get it.
A test round and a level label
If 72 arrows struggle this much, a shorter round struggles more. Clubs, and some training studies too, use a 36-arrow round shot on a single day as an accuracy test. With half the arrows the expected gap between two archers is halved, while the noise falls much less, because it falls with the square root of the number of arrows: at the same level, a short round separates two close archers less well than a full qualification. An independent model gives the scale: for a top compound archer at 50 metres on the 80 cm face, with the sight set correctly and never touched, the total varies from one round to the next with a standard deviation of 2.7 points over 36 arrows and 3.8 over 72, enough, according to the author, to mask small improvements or declines. For an advanced barebow archer at 50 metres, a laboratory study estimates a standard deviation of about six points for a 30-arrow round, and concludes that getting an average for each weight set-up stable enough to compare with the others would take between 900 and 1,050 arrows.
How repeatable a 36-arrow round is from one day to the next, when archers shoot it, nobody has measured. Two Indonesian training studies credit it with a reliability of 0.825, but neither calculates the coefficient on its own sample, which is why in the registry the idea that a round of thirty-six arrows measures an archer's level reliably enough is marked as not studied. A test round may be enough to separate archers of very different levels; nothing measured so far supports reading differences of a few points into it.
A level label says even less about the individual archer. When level is assigned from outside, through national-team selection or membership of a squad, and the score is measured afterwards, the average points the expected way: ten Turkish national-team archers averaged 1,296 and 1,322 FITA points in the two subgroups, against 1,157 and 1,147 for club beginners, and sixteen from the Chinese national squad scored 8.7 points per arrow at 70 metres against 6.8 for the Beijing squad. The gap holds for the group, though, not for the individual: in the Beijing squad someone reached 8.2, above the weakest member of the national squad at 7.7, and between high-school and university teams in Taiwan the step is 74 points in recurve and just 7 in compound, where scores are already close to the maximum. That is why in the registry the claim that archers classed at a higher level score more points is reported: it holds on average, on comparisons between different groups, and says little about the individual. Two small studies add that in the weaker groups scores are also more spread out from one athlete to the next, and in the registry that claim is reported too: the standard deviation between athletes was 0.5 points per arrow in the Chinese national squad and 1.4 in the Beijing squad.
Score is a complete measure and a slow one
All of this risks sounding like a demolition of the score, and that would be the mistaken conclusion. The total from a competition remains the most complete measure available, because everything contributes to a final total at once: technique as much as staying power over the distance, the wind as much as what happens in your head when the points matter. No partial indicator, however precise, holds as many things in a single number.
What it lacks is speed. A measure that holds everything also collects all the variability of everything, and for that reason it needs time to settle. Score is therefore an excellent instrument over a season and a poor one over a Sunday, even though it is the same instrument: behind one Sunday's number sit 72 arrows, behind a season's average sit a few thousand.
It follows that your own results should be tracked differently: instead of comparing the last competition with the one before, which is the least informative comparison available, keep a moving average of the last four competitions and watch how it moves across the months. That average carries half the uncertainty of a single competition, and above all it moves very little, which makes it dull to check every week and reliable when it does move.
There is one last consequence, and it bears on training decisions more than on reading results. If score takes months to register three points of improvement, then a season is not enough to try more than one or two different approaches, and the archer who changes setup every six weeks because "there are no results" throws away, each time, the run of competitions the measurement would have rested on, and starts again from zero. For score to say anything about a technical choice, that choice has to stay in place for the number of competitions in the table, which for three points of improvement is about thirty.
What to judge a change on, then
The working rule is this: a technical change is not judged on score in the short run, because in the short run score does not have the resolution to see it. The limit lies in the instrument before it lies in the patience of whoever is waiting, since the measurement is being asked to detect a difference smaller than its own margin of error.
That does not leave you without feedback for two seasons, as long as you change the question. Video shows directly whether the movement you meant to change has changed, which is a different question and a much easier one than "am I shooting better", and the answer comes at once. Repeatability of the movement and the shape of the group carry information the total hides, because two archers with the same score can group very differently, and a group stretching vertically or horizontally suggests where to look, as I describe in the article on how the group opens up with distance. The group, though, is worked out from the same arrows, and over a single round it remains uncertain: in the essay, a 690 shot in one competition is compatible, with 95% confidence, with a spread anywhere between 3.85 and 5.07 centimetres. So it pays to record where the arrows land, not just what they score, and to look at it over many ends.
Score comes in afterwards, as confirmation, once the competitions have built up. That is the exact reverse of the usual order, score first and feel later, and in practice it changes one thing above all: it removes the temptation to start rebuilding your technique on the Monday after a bad competition, when the only data available is a single total, and a single total cannot tell a real decline from an afternoon in which the arrows fell on the unlucky side of the line.
The same measurement, inside a match
A further consequence concerns what happens after qualification. If 72 arrows struggle to separate two archers who are close, an elimination match, which has at most fifteen arrows plus a possible shoot-off arrow, separates them even less well, and the calculation expects upsets to cluster between archers of similar level. On real brackets the picture is less tidy, and in the registry the claim that in head-to-head matches, upsets happen when the two archers are of similar level is contested: at the first World Cup stage of 2013 the upsets show up mainly where the bracket brings together archers with close qualification ranks, but at the World Championships the same year, with gusts above 20 metres per second, many low-ranked compound archers reached the top places.
This is where the argument joins a larger discussion, the one about how points are counted inside a match. In the essay on the set system I calculated what it costs, in precision, to decide a match by sets instead of by adding up the arrows, and the interesting result is the proportion rather than the cost itself: people argue fiercely over arithmetic that moves outcomes by about two percentage points, while the instrument that decides who is the favourite, namely the qualification round, leaves the order of almost every neighbouring pair undetermined. The set system is a slightly rough judge working from an entry ranking that is far rougher than it is.
The essay carries the full calculation, the tests of how far the result changes with the shape of the group, and the code package with which anyone can redo the sums and check that the published numbers are the ones the calculation produces.