15.7 Sample Size
Several things determine the sampling error in a random-sample survey. One major factor is the size of the sample. A general rule is that the larger the sample, the smaller the sampling error. After a certain level, however, any increase in the sample size results in such a small decrease in the sampling error that it is not worth the effort to increase the size of the sample any further. This is why most major polling organizations use a random sample of between 1,200 and 1,500 respondents. That size sample results in a sufficiently small sampling error (between 2.4 to 3.4 percent) so that a larger sample size is not justified.
It is important for you to look for a stated sampling error in the results of any survey you may be contemplating using. Also, when a stratified random sample or a disproportionate random sample is used, the results may include an analysis of sub-samples, or responses from smaller groups within the larger sample. The margin of error for sub-sample results will always be larger than for the total sample, thereby requiring even more careful scrutiny to determine if differences in responses are real. For example, if a survey of 250 people shows results by region (Midwest, South, East and West), you should be cautious of these results. When the sample is divided in this way, recognize that only about 62 people represent each region in the sample – almost never a large enough sample to be truly representative of a particular region.
Another factor determining sampling error is the level of confidence used with the analysis of the results. The level of confidence refers to the odds that the results of a specific survey are within the estimated sampling error range. In other words, if the survey were repeated, the researchers are 90 percent (or 95 or 99 percent) confident that they would get the same results within the same range of error. Again, confidence level increases as the size of the sample in a probability sample increases.
When you are evaluating the results from a poll that used a random-sample method, you should look for the statement of the sampling error and the level of confidence. If you see these figures, you can be sure the results are based on a probability sampling method. A common goal is to look for a poll with a sampling error of no more than plus or minus 5 percent at the 95 percent confidence level. If you are evaluating a poll with those kinds of figures, the data are likely to be more credible than those from a poll with higher sampling error and lower confidence levels.
