Three Guesses Beat One

There are three quantities you want to estimate, written ${\theta }_1,{\theta }_2,{\theta }_3$. The model gives one measurement of each, $x_1,x_2,x_3$, by adding independent, zero-centred normal errors with the same known variance ${\sigma }^2$. An individual error is not known; the variance describes its distribution over repeated measurements. The theorem uses these assumptions, so it does not apply automatically to unrelated real-world measurements with different units or error scales.

The obvious estimate of ${\theta }_1$ is $x_1$, and the obvious estimate of the three together is $(x_1,x_2,x_3)$. Score a set of guesses by the total squared error, which is the squared distance from the guesses to the truth.

Now do something that ought to be pointless. Compute one number from all three measurements at once,

where $d$ is how many quantities there are, ${\sigma }^2$ is the size of the noise on each, and $\Vert x{\Vert }^2$ is the sum of the squares of the three measurements. Then multiply all three guesses by that factor. For example, with three measurements 4, 3 and 0 and variance 1, the factor is 1 − 1/25 = 24/25, giving adjusted estimates 3.84, 2.88 and 0. If the squared length is very small, the ordinary formula can instead produce a negative factor and move past zero. The positive-part version clips that factor at zero; the laboratory shows both versions.

The total squared error of the pulled-in guesses is smaller, on average, than the total squared error of the measurements. Not for a lucky choice of the true values: for every choice of them. And the pulling towards zero is not because zero is a good guess. It works pulling towards any fixed point you like, chosen before you look at the data.

At three quantities it works. At one or two it does not, and the same formula says why: the $d-2$ in it is negative at $d=1$, zero at $d=2$, and only becomes positive at $d=3$. Those are the failure presets on this page, and they are not accidents of the simulation.

The first thing to give up is the idea that the estimator knows anything about the three quantities being related. It does not, and they are not.

Here is the trade being made. Multiplying a guess by a factor slightly below 1 introduces a bias: the guess is now systematically a little short of the truth. In exchange it reduces the variance, because a shrunken random quantity varies less. Squared error is bias squared plus variance, so shrinking is worth doing exactly when the variance saved exceeds the bias bought.

For one quantity, that trade is never worth it in the way you would need. You do not know which direction the truth lies in, so shrinking towards zero helps when the truth is near zero and hurts when it is far away, and there is no way to choose the factor from the data that wins everywhere.

The second thing to notice is what happens to a length in several dimensions. If each measurement error has variance ${\sigma }^2$, then the squared distance from the measurement to the truth has average $d{\sigma }^2$. It grows with $d$. But something else grows too: $\Vert x{\Vert }^2$, the squared length of the measurement, comes out on average around $\Vert \theta {\Vert }^2+d{\sigma }^2$, which is systematically larger than $\Vert \theta {\Vert }^2$. The measurement is reliably too long. Not in any particular direction, which would be a bias you could correct, but in length, which is a bias in a quantity that all $d$ coordinates contribute to.

That is the whole mechanism. The measurement vector is too long by an amount you can estimate, because $d$ separate coordinates each contribute their variance to its length, and correcting the length is one correction learned from $d$ pieces of evidence. One coordinate cannot tell you its own error. Three coordinates can tell you, collectively, roughly how much error there is in total, and that collective knowledge is what gets spent.

Why three and not two? Because the correction has to be paid for. Shrinking the length costs you accuracy in the direction the vector points, and that cost does not shrink with dimension, while the saving does. The formula sets the exchange rate at $(d-2){\sigma }^2$, and below three dimensions there is nothing left over.

Two sentences carry this page, and they point in opposite directions.

The result is a theorem. In three or more dimensions, the shrunken estimator has strictly smaller expected squared error than the measurement, at every possible true mean. Stein proved this in 1956 and James and Stein gave the explicit estimator in 1961.

This page has not shown you that. Everything on it is a finite number of draws around one true mean at one seed. A simulation samples finitely many true means, finitely often, and the statement above is quantified over all of them.

Those two live together uncomfortably and that is the point of putting them next to each other. The race view separates, convincingly, and would separate just as convincingly if the theorem happened to be false at some true mean the simulation never visits. The picture is consistent with the theorem. It is not evidence for it, in the ordinary sense that evidence is something which would have come out differently had the claim been wrong.

There is a good reason this page still runs the simulation. What a simulation is genuinely good at is showing you the mechanism: one trial at a time, you can watch the factor being computed from the measurement's own length, watch every coordinate move by the same proportion, and watch the shrunken estimate lose about a third of the individual trials while winning the average. None of that is available from reading the statement, and none of it is an argument.

And there is a third thing the laboratory does that the theorem does not. It shows what the theorem looks like when it is true and invisible. Set the dimension to ten and move the truth out to fifty units from the target: the improvement is still there, still at every true mean, and too small for any figure at that scale to render. A reader who judged by the picture would call it a tie. That is worth more than another convincing separation, because it is the case where trusting the figure over the statement would give you the wrong answer.

Let $x\sim N(\theta ,{\sigma }^2{I}_d)$ with ${\sigma }^2$ known, and score an estimator $\hat{\theta }$ by its risk, the expected squared error

Write ${\hat{\theta }}_{\text{obs}}(x)=x$ for the measurement itself, and for a fixed $c\in {\mathbb{R}}^d$ chosen in advance,

1. $R(\theta ,{\hat{\theta }}_{\text{obs}})=d{\sigma }^2$ for every $\theta$. Proved, in layer 5.
2. For $d\ge 3$ and every $\theta \in {\mathbb{R}}^d$, $R(\theta ,{\hat{\theta }}_{\text{JS}})<R(\theta ,{\hat{\theta }}_{\text{obs}})$. Proved by Stein and by James and Stein, cited here. The deduction from their risk identity is one line and is given in layer 5; the identity itself is not derived on this page.
3. For $d\le 2$, ${\hat{\theta }}_{\text{obs}}$ is admissible: no estimator has risk at most $d{\sigma }^2$ at every $\theta$ with strict inequality somewhere. Proved by Blyth for $d=1$, and recorded by Stein as the reason three is the threshold. Cited, not reproduced.
4. The factor $1-(d-2){\sigma }^2/\Vert x-c{\Vert }^2$ exceeds 1 when $d=1$, equals 1 exactly when $d=2$, and is below 1 when $d\ge 3$ and $\Vert x-c{\Vert }^2>(d-2){\sigma }^2$. Proved, in layer 5, in one line.
5. ${\hat{\theta }}_{{\text{JS}}^{+}}$, the same estimator with the factor replaced by $\max (0,\text{factor})$, has $R(\theta ,{\hat{\theta }}_{{\text{JS}}^{+}})<R(\theta ,{\hat{\theta }}_{\text{JS}})$ for every $\theta$ when $d\ge 3$. Proved by Baranchik, cited.
6. The mean squared errors printed by the laboratory are what they are at the seed, dimension, true mean and trial count shown. Observed. They are not evidence for statement 2, and no number of trials would make them so.

Statement 2 is the subject. Statement 6 is what this page produces. The distance between those two rows of the ledger is the reason the page exists.

The risk of the measurement. The coordinates of $x-\theta$ are independent with mean zero and variance ${\sigma }^2$, so

for every $\theta$. It does not depend on $\theta$ at all, which is why the plain estimator's mark in the dimensions view lies on a straight line through the origin. ∎

The sign of the factor. The factor is $1-(d-2){\sigma }^2/\Vert x-c{\Vert }^2$, and ${\sigma }^2>0$ and $\Vert x-c{\Vert }^2>0$. So the sign of the subtracted term is the sign of $d-2$. At $d=1$ that is $-1$, so the subtracted term is negative and the factor is $1+{\sigma }^2/\Vert x-c{\Vert }^2>1$: the estimate is pushed away from $c$. At $d=2$ it is $0$, so the factor is exactly $1$ and ${\hat{\theta }}_{\text{JS}}(x)=x$ identically, not approximately. At $d\ge 3$ it is positive, so the factor is below 1, and it is additionally non-negative precisely when $\Vert x-c{\Vert }^2\ge (d-2){\sigma }^2$. ∎

Why $d\ge 3$, from the risk identity. Take $c=0$ without loss of generality. James and Stein compute the risk of ${\hat{\theta }}_{\text{JS}}$ exactly:

That identity is cited and not derived here; it follows from Stein's integration-by-parts lemma for the normal distribution. Granted it, the conclusion is one line. For $d\ge 3$ the quantity ${\mathbb{E}}_{\theta }[1/\Vert x{\Vert }^2]$ is finite and strictly positive, and $(d-2{)}^2>0$, so the subtracted term is strictly positive and $R(\theta ,{\hat{\theta }}_{\text{JS}})<d{\sigma }^2$ for every $\theta$. ∎

And why the same line does not work below three. It is tempting to read the identity at $d=1$, notice that $(d-2{)}^2=1$ is still positive, and conclude that shrinking wins there too. It does not, and the reason is the expectation. For $d\le 2$, ${\mathbb{E}}_{\theta }[1/\Vert x{\Vert }^2]$ is infinite: the normal density does not vanish at the origin, and $1/\Vert x{\Vert }^2$ is not integrable against it in one or two dimensions. The identity has no content there, and the laboratory shows what actually happens instead, which is that the $d=1$ estimator is much worse.

What is not proved here. Statements 2, 3 and 5 of layer 4, in the sense that matters: the risk identity above, the admissibility of the measurement in one and two dimensions, and Baranchik's improvement are all taken from the literature. This page can substitute numbers into a formula and average the results, and that is a different activity from establishing any of them.

What this laboratory computes. Up to ten quantities at once and up to ten thousand trials, with the noise on each coordinate having variance exactly 1. The measurements come from a seeded generator: uniform values from a 32-bit generator, turned into normal ones by Box and Muller's transform. Nothing here uses the browser's own randomness, so the address bar reproduces a run exactly, and the seed is written into it even when it is the default one.

What the numbers on the page are. Observations, in the strict sense the ledger uses. Every mean squared error printed here is an average over finitely many draws around one true mean, and it is reported as a property of that run. The seed makes it checkable. It does not make it general.

Where the figures stop being able to show the truth. At ten dimensions with the true mean fifty units from the target, the improvement is real, is guaranteed by the theorem, and is smaller than the width of a line. The dimensions view has the same problem at $d=1$ in the other direction: the shrunken estimator's mean loss is many times the plain one's, so it is drawn as an arrow at the top of the plot and the number is left to the table. Both are cases where the figure cannot carry what is true, and saying so is cheaper than redesigning the figure to hide it.

What is left open. Not the theorem, which is seventy years settled. What remains genuinely difficult is everything about using it: which point to shrink towards when you have no reason to prefer one, how to report an uncertainty for an estimate that borrowed strength from unrelated quantities, and what to say to somebody who only cares about the beetle and is being told that the best guess for its length depends on the price of tin. The improvement is in the total squared error across all the quantities, and a reader who wants a guarantee about one of them separately does not get it from any of this.

Read this next to Every Even Number, Two Primes. There, a search never fails and the conjecture is open, so the computation is evidence for something unproved. Here, the theorem is settled and the computation is evidence for nothing at all. Two pages of convincing pictures, and the pictures are doing entirely different work.