Introduction

Sampling is the essence of statistics. Normally a researcher cannot get access to an entire population and must make do with a sample, even if a sample is not exactly the same as the population. There is going to be some difference between the results of a sample and the truth, and if another sample is taken, the results would be different again. Ideally, the results of the sample would be relatively close to the truth. Remember that the real objective in statistics can best be summarized in the definition of inferential statistics – using a sample to make inferences about a population.

Confidence Intervals

When a statistician makes inferences, they are estimating. The value of a population mean is rarely known, but the sample mean usually is known. Considering the sample mean by itself is not accurate and may be too large or too small, an interval is used to estimate. The confidence interval is basically the sample mean + / - a margin of error. The margin of error is computed from the z-score (based on the level of confidence) times the standard deviation divided by the square root of the sample size. The confidence interval is essentially saying that we are X% confident that the true population mean is within the confidence interval.

The z-score corresponds to the standard normal distribution. A 95% confidence interval captures the middle 95% of the normal curve, so the z-score is 1.96 since the area between -1.96 and +1.96 equals 95%. A 90% confidence interval captures a smaller area and has a z-score of 1.645. A 99% confidence interval captures a larger area and has a z-score of 2.576. Thus, the larger the level of confidence, the larger the margin of error.

The rest of the confidence interval formula shows that the larger the sample size, the smaller the margin of error. Additionally, the larger the standard deviation, the larger the margin of error. The standard deviation cannot be controlled (it is what it is), but the level of confidence and the sample size can be affected by input.

To help illustrate confidence intervals, imagine throwing a dart at the bull’s-eye on a dartboard from 12 feet away. If only one throw is available to hit the target, the confidence in that happening would be low. If the target were expanded to include the ring around the bull’s-eye, confidence in hitting the target would grow. If the target were expanded further to include the entire dartboard, confidence would grow more. Essentially, the larger the margin of error, the greater the confidence at hitting the target.

Now aim for the ring around the bull’s-eye again and assume several practice throws. If the target is hit 8 out of 10 times the next dart could be expected to hit a relatively small area. The expected target is probably much smaller than it was prior to the first throw. Essentially, the larger the sample size, the smaller the margin of error.

There are two formulas for computing the confidence interval of the mean. The only difference between the two has to do with one's knowledge of the population standard deviation. If it is known, use a formula that uses the z-table, but if it is unknown, the formula uses a t-table. When using Excel, the two tables are transparent on the computer, but there are two different menu items for the two situations.

Confidence intervals can also be computed for proportions. They are essentially the same in that there is still a confidence level (translated into a z-score) and sample size, but there are no standard deviations. An election poll that reports a margin of error is using a confidence interval. For example, if one candidate is shown to have 52% of the vote with a 3-point margin of error, it means that candidate truly has between 49% and 55% of the vote. The 52% is based on a sample, but the confidence interval implies that if the entire population were polled, the results would be in that range. The certainty that the truth is in that range is the level of confidence. Notice how often the newspapers report the margin of error but not the level of confidence. A range of 49 – 55% means that the candidate might win or might lose, and the race should be labeled as undecided. Making a conclusion based on being close is a mistake. If the confidence interval were 51 – 55%, then it could be stated that the candidate is winning, at X% confidence.

Sample Sizes

Probably the most common question often asked of a statistician is "how large of a sample do I need to take…" To understand the answer to this, it is important to look back a bit. When data is normally distributed, probabilities can be computed if only the mean and standard deviation are known, and life is good. However, what if the population mean is unknown? In that case, the best estimate of a population mean is a sample mean. So take a sample and compute the mean to produce an estimate. How good is that estimate? That is what margins of error are for, based on the level of confidence. But what if the confidence interval is computed and it is just too large (after all, what good is it to estimate that the presidential candidate has 52% of the vote with a margin of error of 20%)? If the confidence interval is too large, a larger sample can be taken to tighten the confidence interval, but how much is enough? It would be nice to know exactly how large a sample is needed to get the confidence interval desired. Thanks to the sample size formula, that is possible.

To compute the minimum sample size, two questions need to be answered: how much error will be tolerated AND how much confidence is wanted in the truth being within that margin of error. The more confident is wanted, the larger the sample must be. The greater the margin of error, the smaller the needed sample. Normally the answer given by those not in the know is to say they want 0% error and 100% confidence, but that would require sampling the entire population. Anything less than the population involves incurring a degree of error, but the question is "how much is acceptable?" Once a margin of error and confidence level are decided on and the sample size is computed But what if this sample is 2,000 people? In a phone survey, up to 20,000 people may have to be called just to get 2,000 responses, but if the sample size is cut, the confidence is cut and the margin of error increases. It is truly a balance game.

Here is another interesting sampling misconception. Would a sample of 400 people from a neighborhood be a good sample? It might, but what about 400 people from a city? What about 400 people from a state or even 400 people from the United States.? Unbelievably, they are equally good. The size of the population has nothing to do with the quality of a sample. A sample of 400 is just as good if it comes from a population of 100,000 or 100,000,000. The only issue is that the sample must be representative (i.e., do not sample 400 from a city and use it to draw conclusions about the U.S.A.). The sample must be randomly chosen too.

To help illustrate, imagine if making chili for a family and the mother-in-law likes it mild while everyone else likes it spicy, so one quart of mild chili and 2 gallons of spicy chili are made. When it is time for a taste test, a spoonful of the mild chili is tasted and declared perfect. How many spoonfuls of the larger pot do need to be tasted? Since it is 8 times larger, does that mean 8 spoonfuls? Of course not; one spoonful of the larger pot reveals just as much as one spoonful of the smaller pot, providing the pots are stirred first. The idea is that the spoonful is representative of the pot and has a representative sample of molecules of chili from the population known as the chili pot. Therefore, the ideal sample size is not a percentage of the population; population size only comes into play when the population is small, in which case the sample size can be smaller.

Conclusion

      Surveys are often misinterpreted. Knowledge of confidence intervals and sample sizes (what a margin of error truly means, why the error is preferred to be small, and how larger sample sizes translate to higher levels of confidence with smaller margins of error) represent an insiders knowledge of survey interpretation and will help to minimize those misinterpretations.