14.17 Evaluating Data

An important strategy in evaluating data is to find out as much as possible about where the data were obtained. Avoid estimates that appear too precise to be true, use common sense when presented with numbers that appear counterintuitive, and maintain skepticism with claims that could just as easily be based on coincidence.

• A good starting question to ask about any statistical data claim is, “How do you know that?” Is the person or organization providing the data just telling you something they “know” or have “found to be true?” Or can they provide studies, experiments, documentation of the claim? And even if someone says, “I’ve done a study,” you should ask, “What kind? What were the possible flaws in the study?” An honest researcher will always report the limitations of his or her data and recognize that a single study rarely proves anything.

• Another good question is, ”How sure are you about your data?” There are many obstacles to gathering certain types of information. How certain can we be about the number of people in the U.S. who shoplift, carry the AIDS virus, are homeless, commit white-collar crimes or drink more than three beers a day? People are unwilling to provide truthful information for certain purposes, many things go undocumented and there are physical barriers to accurate observation of many types of activities.

As this lesson has demonstrated, communicators have many opportunities to generate and use data from both inside their own media organizations and from other organizations. The evaluation skills we discussed in Lesson 10 should be applied to data sets as rigorously as they are to the content of interviews or the claims of clients. Asking where the data originated, how they were collected, what the original purpose or use for the data was, and all of the other information evaluation questions will help you avoid making a mistake or misinterpreting what you find.

In some cases, you will find yourself in over your head with a particular data set. The best strategy then is to seek out the help of a data expert, either inside your own organization or from a reputable outside organization (academic center, statistical consulting firm, polling organization, etc.). Your request for help from those experts, however, will be much better informed if you have at least a basic familiarity with how data are gathered and used.