Polls, polls, polls.

As any election draws nearer, the number of news stories reporting the results of public opinion polls grows larger.

And so it should. A public opinion poll -- properly conducted and properly reported -- promotes the democratic process. It gives people a voice. It allows minority viewpoints to be heard. In this era of big-money politics, the public opinion poll lets people of more limited means participate in the public discussion.

The abundance of polls, and their often contradictory contents, makes it difficult for citizens to determine which surveys, and which sets of results, to believe. There is no simple or fail-safe way to solve this problem. Not all polls are created equal, even though they are sometimes treated that way by journalists.
The following are guidelines for citizens and journalist to ponder when considering stories reporting the results of public opinion surveys.


Wording matters

The most important thing to consider in evaluating the findings of a survey is the wording of the questions asked. The answers a survey respondent gives are obviously affected by the questions he or she is asked. Consider the following two questions:

• Do you approve or disapprove of the performance of Bill Clinton, a pot-smoking, draft-dodging, liar?

• Do you approve or disapprove of the performance of Bill Clinton, a president who has lead the nation during a period of peace, prosperity, and moderation?

Which question yeild the most positive responses for the president? Obviously, the second one. But differences in question wording don’t have to be that blatant to yeild different results.

Pollsters rarely agree on a single "correct" way to ask questions about topics such as presidential approval or candidate support. For example, both Clinton items listed above are bad questions if the goal of a survey is to measure the President's current level of public support. However, the first item is a good question for identifying the President's hard-core support (whoever says "approve" to this item will probably support Clinton under any circumstance). Similarly, the second item is useful for measuring Clinton's maximum support.

Thus, evaluating the results of surveys requires us to look closely at the wording of questions and, based on this, consider what the item is really measuring. With this information, readers can reach reasonable conclusions about what a survey has "found," and what these results "mean."?


"Who" is often more important than "how many"

The method used to select the sample is another important factor to consider in evaluating the results of a survey. Sampling involves selecting some units out of a population or universe. Most public opinion surveys are based on samples because it is quicker and cheaper to interview some people rather than the entire population. Still, the purpose of most surveys is to gather information about the preferences or behaviors of an entire universe (i.e. citizens, registered voters, likely voters). The difficult part of sampling, then, is to select respondents in such a way that the results of a sample can be legitimately generalized to the entire population from which it was selected.

In general, larger samples are more accurate in making these generalizations than are smaller samples. However, it is often the case that the most critical thing to consider in evaluating a sample is not its size, but how it was selected.

At the present time, most publicly reported surveys are based on samples in which each respondent has a roughly equal chance of being selected. Thus a telephone survey of Alabama adults begins with a list of every possible telephone number (i.e. residential and non-residential, listed and unlisted) in the state. From this list, a random sample of numbers is selected. "Random" means that each number, and thus each household, has a roughly equal chance of selection. The selected numbers are then called. When a household is reached, a method (different organizations use different procedures) is used to randomly select a specific individual to interview. That person is then interviewed. The result is a random sample of people in Alabama.

Surveys based on different sampling procedures require careful evaluation. For example, care is needed in interpreting surveys which do not give each unit in a population an equal chance of being selected to be part of a sample. Thus surveys based on "man-in-the-street" interviews of people found at the local mall, for example, are highly suspect. Each person does not have an equal chance of being at the mall at a particular time on a particular day.

A poll based on a list, such as of registered voters, is questionable because such lists are rarely complete or up-to-date. Similar problems are found in surveys based on numbers listed in telephone directories. Not only are telephone directories out-of-date by the time they are printed, but they also exclude unlisted numbers. Thus they are not based on a universe of "all potential" numbers. Surveys based on interviews with whichever adult answers the telephone are also subject to question. Men and women, younger and older people are not equally as likely to answer a telephone when it rings.

One also needs to consider is the population to which the survey refers. For example, other things being equal, a survey based on a sample of registered or "likely" voters provides more useful information about an upcoming election than does a study using a sample of all adults. (It should also be noted that most survey researchers consider the identification of likely voters to be as much art as science.) Similarly, if one is interested in studying public opinion in Alabama, then a statewide sample is needed not one based, for example, on Tuscaloosa County residents.

Finally, one has to consider how many of the selected respondents actually participated in a survey. The higher the refusal rate, the greater the chance that the sample used in a survey will largely consist of abnormal people.


Be careful

Unfortunately, question wording and sampling are not the only factors that may influence the outcome of a survey. Research has also found, for example, that who the interviewers are (in terms of gender, age or race), the order in which questions are asked, and when a study is conducted, may, but do not always, affect the results of a survey.

Despite these difficulties, properly-conducted surveys produce accurate and reliable information about the preferences and behavior of citizens. Also, most (but unfortunately not all) polling firms and organizations who sponsor surveys work hard to gather and report accurate information, because in the long run it is in their best interest to do so.

Still, citizens need to be careful in interpreting the results of surveys.

And so do journalists. At minimum, a reporter should include in a story about poll results the exact wording of the questions and how the survey's sample was selected.

Readers then should carefully consider this information. Properly informed, they have as much ability to interpret survey results as anyone else.


Patrick R. Cotter and James G. Stovall are co-directors of Southern Opinion Research, a survey company based in Tuscaloosa.

* * *
Permission is granted to anyone who would like to reprint this article. We ask only that proper credit be given (including the website address of Southern Opinion Research) and that we be notified of its use.