What “margin of error” really means

The short version

• A poll's margin of error is about one candidate's share. The uncertainty on the lead between two candidates is roughly twice as large.
• It only counts the luck of who got sampled. Real polls miss by more: the result fell outside a poll's own 95% range for the lead 26% of the time, not 5%.
• A lead inside the margin of error is not a tie. When the poll average was that close, the leader still won 67% of the time (363 of 546 elections) and lost 34%.
• A lead outside it is not safe. 76 candidates lost despite a poll lead bigger than the margin of error.

What the number measures

A pollster can't ask every voter, so it asks a sample, usually 500 to 1,500 people. Even a perfectly drawn sample won't match the whole electorate exactly, just by the luck of who was picked. The margin of error says how far that luck could plausibly move a number: in 95 samples out of 100, a candidate's share lands within that many points of the truth.

It shrinks slowly as the sample grows, with the square root of its size. A poll of 400 people has a margin of about ±4.9 points, 600 people ±4.0, 1,000 people ±3.1, and 2,000 people ±2.2. Quadrupling the sample only halves the margin. The typical poll in our data asked 643 people, for a margin of ±3.9.

The lead's margin is about twice as big

Here is the catch most coverage misses. Say a poll of 600 people has Smith at 48% and Jones at 44%, with a margin of error of ±4. That ±4 applies to each number. If the sample happened to run 2 points too high for Smith, it probably ran about 2 points too low for Jones, because those people were counted in one column instead of the other. The errors move in opposite directions, so the error on the gapis roughly double: Smith's 4-point lead is really 4 ± 8. That lead is “outside the margin of error” the way it's usually reported, and it still includes Jones winning.

What it leaves out

The margin of error assumes the only problem is random sampling. Real polls have others: guessing who will actually vote, people who won't answer the phone (and differ from those who do), how the question is worded, and voters changing their minds after the poll. None of that is in the ± figure, and it doesn't shrink with a bigger sample.

That shows up in the record. If the stated margin captured all the uncertainty, the result would land outside a poll's 95% range for the lead about 5% of the time. Across 16,897 polls taken in the final three weeks, it landed outside 26% of the time. The typical poll missed the final margin by 4.3 points. In practice, a poll's real uncertainty is about double what it says.

Within the margin of error is not a tie

If “within the margin of error” meant “anyone's guess”, the leader would lose about half the time. They don't. Of the 4,998 final polls with a lead inside their margin of error, the leader went on to lose 39%. Being ahead still tells you something; it just tells you less than it seems. And even a lead of one to two margins of error, comfortably “outside” it, was wrong about the winner 23% of the time.

Who won and lost on the margin of error

Single polls are noisy, so we also averaged each election's final-three-week polls, the way most coverage summarizes a race, and compared the average lead with the typical margin of error of those polls.

In 546 elections the average lead was inside the margin of error. The leader won 363 of them and lost 183, one in three. In the other 1,601 elections the lead was bigger than the margin of error, and the leader lost 76 (5%). Altogether the poll leader lost 259 of 2,147 elections.

The pattern is steady: under 3 points, the trailing candidate wins about a third of the time or more; from 3 to 7 points, about one time in six; past 10 points, almost never.

Do the polls predict the result?

Yes, strongly, which is easy to forget in a piece about misses. Across all 2,147 elections, the final poll average and the result have a correlation of r = 0.93 (1 would be perfect). The fitted line is nearly the diagonal, shifted about 1.0 points toward the Republicans: a poll average of D+10 became D+9.1 on average, and R+10 became R+11.1. Over these 26 years, the polls leaned slightly Democratic. The chart zooms in on the close elections, where the question matters.

Much of that strength comes from landslides, where nobody needed a poll. Among the 1,082 competitive elections, with the poll average inside 10 points, the correlation falls to r = 0.66: still a real signal, with a lot of scatter around it. Most red dots sit near the middle, where the polls said the race was close, but some are well out from it: Maine in 2020 had the Democrat up 5.6 and the Republican won by 8.6.

One more correlation says a lot about the margin of error itself. If it measured how wrong a poll is likely to be, polls with bigger margins would miss by more. Across all 16,897 polls, the correlation between a poll's stated margin of error and how far it actually missed is r = 0.14, close to none. A bigger sample makes a poll a little steadier, but most of what decides a poll's miss isn't in the number.

Ahead inside the margin, and lost

These candidates led in the final polls by less than the margin of error and lost, mostly by less than a point. The polls weren't badly wrong in any of them. The leads were never as solid as they looked.

Ahead inside the margin, and held on

The same kind of lead often holds. Every one of these was “within the margin of error”, and the leader won, some by even less than the polls showed.

Ahead outside the margin, and still lost

These are the famous misses. Each leader was ahead by more than the margin of error in an average of many polls, which should have washed out sampling luck. What it can't wash out is an error that every poll shares: in 2016 and 2020, polls across the Midwest and in Maine and North Carolina understated Republican support together.

How to read a poll

• Double the margin of error before judging a lead. A 4-point lead with ±4 is not safely outside it.
• Then allow for more. Real-world misses run about twice the stated sampling error.
• Prefer averages to single polls, but remember that polls can all miss in the same direction.
• Think in odds, not verdicts. A 2-point lead is roughly a 2-in-3 chance, not a win.

How VotePredictor handles it

That last point is what a forecast is for. VotePredictor doesn't use the stated margin of error. Its win probabilities come from how far polls and models have actually missed in past elections, including the misses every poll shares. That's why a race can show a clear polling lead and still be close to a toss-up. See the methodology, and how the forecast has done against everyone else who calls elections.

About the data. Polls taken in the final three weeks of general elections for president (by state), Senate, governor and House, 1998–2024: FiveThirtyEight's archive through 2022, and its 2024 poll feeds for president, Senate and governor (no House) scored against the official results. In all, 16,897 polls with a sample size, in 2,147 elections where the Democrat and the Republican together won at least 80% of the vote (so races an independent won are left out). Margin of error is the standard 95% figure for one candidate's share, 98 divided by the square root of the sample size; an election's figure is the median of its polls'. The analysis is aielect/margin_of_error.pyin the project's repository.