Statistics of a sample

In 1918 Harlow Shapley measured the distances of the globular clusters, ball-shaped swarms of hundreds of thousands of old stars, and found that they are not spread evenly round the sky. Most of them lie in one half of it, towards Sagittarius, and he argued that they swarm round the centre of the Milky Way, far from the Sun. IOAA 2024 gave a catalogue of 25 globular clusters and asked for his argument to be repeated with modern distances. Turning 25 scattered positions into one distance and its uncertainty takes the statistics of a sample, which describe where its values lie and how widely they spread, how well its mean is known and what to do with a value that does not seem to belong. In this section we learn these, and then how to combine measurements of different quality.

D7.1The centre of a sample

The mean of $N$ values, $\overline{x}=\sum x/N$, uses every value, and when the values scatter at random about the truth it is the best single estimate of it. The median is the middle value once they are put in order, or the mean of the two middle values when $N$ is even. It ignores how far away the outer values are, so a few wild values hardly move it. When the two differ a lot, the sample is lopsided or contains values that do not belong, and the median is the safer description.

The catalogue of IOAA 2024 gives each cluster's galactic longitude $l$ and latitude $b$, the coordinates of Section 1.6, and its distance modulus $\mu$. From these we find its distance $d={10}^{(\mu +5)/5}$ pc and its distance in the direction of the Galactic centre, $x=d\cos b\cos l$. For NGC 6522, with $\mu =14.3$, $l=1.025°$ and $b=-3.926°$, this gives $d=7240$ pc and $x=7230$ pc. The other two coordinates, $y=d\cos b\sin l$ along the Sun's motion round the centre and $z=d\sin b$ out of the plane of the Milky Way, follow in the same way. For the 25 clusters the mean of $x$ is 6.16 kpc and its median is 6.81 kpc.

D7.2The spread of a sample

The standard deviation $s$ measures the typical distance of the values from their mean. We square each difference $x-\overline{x}$, so that values below the mean count as much as values above it, add the squares, divide by $N-1$ and take the square root. Dividing by $N-1$ instead of $N$ allows for the mean having been worked out from the same values, which brings it closer to them than the true value is, just as the uncertainty of a least-squares line divides by $n-2$ in Section D6.

Random errors usually scatter values in the bell-shaped pattern called the normal distribution. In it 68% of the values lie within one standard deviation of the mean, 95% within two and 99.7% within three. For the 25 clusters $s=4.06$ kpc, and 21 of them, 84%, lie within one standard deviation of the mean. That is more than 68%, because the two clusters far from the rest make $s$ larger.

D7.3How well the mean is known

The mean of many values is more certain than any one of them, because their random errors partly cancel. Its uncertainty, the standard error of the mean, is the standard deviation divided by $\sqrt{N}$.

For the 25 clusters the standard error of the mean of $x$ is $4.06/\sqrt{25}=4.06/5=0.81$ kpc. The means of $y$ and $z$ are −0.32 and 0.43 kpc, both close to zero, so the clusters are centred on a point almost straight towards the Galactic centre, $\sqrt{{6.16}^2+{0.32}^2+{0.43}^2}=6.19$ kpc away. The distance from the Sun to the centre of the Milky Way comes out as about 6.2 ± 0.8 kpc. Today's value, found from the orbits of stars round the black hole at the centre, is about 8.2 kpc. Our estimate is low by more than twice its uncertainty, which suggests that the sample is not a fair one, and indeed clusters beyond the centre are the hardest to see through the dust of the Milky Way's disc. Shapley's own estimate was too large, because he did not know that dust dims distant stars.

D7.4Quartiles, box plots and histograms

The median splits the sorted values into a lower and an upper half, and the medians of those halves are the first and third quartiles, $Q_1$ and $Q_3$. A quarter of the values lie below $Q_1$ and a quarter above $Q_3$. Conventions differ slightly in how they treat the middle value, and we follow the official solution of the 2024 problem, which leaves the median out of both halves when $N$ is odd. The distance between the quartiles, the interquartile range $Q_3-Q_1$, measures the spread of the middle half, and like the median it is hardly moved by wild values.

A box plot draws the sample as a box from $Q_1$ to $Q_3$ with a line at the median, and whiskers that reach out to the farthest values lying within 1.5 interquartile ranges of the box. Values beyond that are drawn on their own as outliers. A histogram shows the shape of a sample as bars, each as high as the number of values that fall in its bin. Bins that are too narrow break the histogram into spikes, and bins that are too wide hide its shape. About $\sqrt{N}$ bins, five for our 25 clusters, is a fair first choice.

For $x$ the quartiles are 3.29 and 8.22 kpc, so the interquartile range is 4.93 kpc. The whiskers may reach 1.5 times this beyond the box, to −4.11 kpc below and 15.62 kpc above. The nearest cluster in the direction away from the centre, NGC 2298 at −3.97 kpc, just stays inside, while NGC 6426, at 20.9 kpc from us and 17.70 kpc towards the centre, lies beyond the upper limit.

D7.5Outliers and sigma clipping

An outlier may be a mistake, or it may be real and interesting. NGC 6426 is a real cluster, one of the old clusters of the Milky Way's halo, 5.8 kpc above the plane of the disc, and the box plot simply tells us that it is unlike the rest. When a sample is known to contain a few values that do not belong, a common way to remove them is sigma clipping. We find the mean and standard deviation, remove every value more than a chosen number of standard deviations from the mean, often two or three, and repeat with what is left until no more values are removed. A problem may also set a one-sided limit, as IOAA 2021 did when it removed planets more massive than one standard deviation above the mean to find a sample of low-mass planets.

Clipping the values of $x$ at two standard deviations shows how it works. The first round has limits at $6.16\pm 2\times 4.06$ kpc, from −1.95 to 14.28 kpc, and removes the two clusters beyond them. The second round, with a mean of 6.10 kpc and a standard deviation of 2.69 kpc, has its lower limit at 0.72 kpc and removes NGC 288 at −0.09 kpc. The third round removes nothing, and the 22 clusters that remain have a mean of 6.38 kpc and a standard deviation of 2.38 kpc. The standard deviation has fallen a lot while the mean has hardly moved. Clipping is a decision about the data, so an answer should always say which values were removed and by what rule.

D7.6The weighted mean

When the values come with different uncertainties, the plain mean gives a poor measurement as much say as a good one. The weighted mean gives each value a weight $w=1/{\sigma }^2$, so a measurement twice as precise counts four times as much. It counts as much as the mean of four measurements of the poorer kind would, since that mean has half their uncertainty. Before combining values in this way we should check that they agree within their uncertainties, since values that disagree by much more point to a systematic error in one of them.

For the two measurements of the guess the weights are 1 and 1/4, so the weighted mean is $(10\times 1+13\times 1/4)/(1+1/4)=10.6$, uncertain by $1/\sqrt{1.25}=0.89$.