14.10 Data from Scholarly Sources

Scholars collect and report on data as part of their work as new knowledge creators. Physical scientists collect data from the innermost reaches of quantum space to the outermost reaches of the universe. Social scientists gather huge datasets about individuals, communities, governments and businesses as they try to understand how society works. Humanists gather data about languages or codes of writing, architectural developments, music, and other topics related to human beings’ creativity. Scholars also generate data when they report on their research work.

The scientific method is designed to generate publicly-verifiable results. Scholars are supposed to do their work under conditions that are reproducible by others. Scholars also try to minimize factors that affect the accuracy and interpretation of their results. Hence, they try to conduct their work under controlled circumstances as much as possible.

This is sometimes easier for physical scientists than for social or policy scientists, however. Physical scientists can do their work in the laboratory and/or with multiple observations of the phenomena they’re studying. Social scientists do much of their work in the social environment and are studying complex human behavior, thereby making control harder to achieve. Nonetheless, all reputable scholars strive to be precise in their methods and careful in their interpretation of research findings.

As much as scholars want to achieve this perfection of the scientific method, they are, in the end, human beings. The data they generate must be scrutinized and used with caution, just as with any other source of information. Critical thinking experts M. Neil Browne and Stuart Keeley suggest that you keep a number of factors in mind when you are considering using scholarly research data as a source of information. These include:

• Research varies in quality, and because the research process is so complex and subject to so many external influences, even well-trained scholars sometimes make mistakes. Just because something was published in a peer-reviewed journal does not mean it is not flawed in other important ways. This is why it is important to try to interview several scholars about the findings of a particular study in order to understand the reputation of that work.

• Research reports often contradict one another. Thus, single research studies presented out of the context of the body of work in that area often present misleading conclusions. You should look for studies that have been repeated by more than one researcher or group of researchers.

• Research data do not “prove” conclusions. All someone can do with a set of data is say it does or does not “support” a particular conclusion. Scholars must always interpret the meaning of their findings, and all findings can be interpreted in more than one way. Thus, when you see the statement, “research findings show…,” you should translate that statement into “researchers interpret their findings to show…”

• Despite best intentions, scholarly work is not neutral and value-free. Scholars, just like the rest of us, have emotional investments in particular ideas and ways of understanding the world. Regardless of how objective a piece of scholarship may appear, subjective elements are always involved. Therefore, you must always try to put an individual scholarly finding or piece of evidence into the larger context of what we think we know about that topic, and try to understand the full set of influences on that scholar’s work.

• A scholar’s work may be mischaracterized by another scholar who is citing or using that original piece of evidence in a new study. Do not rely on someone else’s characterization of another person’s data. Look at the original piece of scholarship yourself and make sure you understand what that particular scholar said he or she found.

• Research findings change over time. What we once thought we knew is no longer true, or has been superseded by more recent scholarship. As with so many other sources of information you will use for your messages, be sure you have the most recent findings in front of you.

• Some studies are done using artificial circumstances because trying to do the study in the “real” world would be impractical, unethical or otherwise not feasible. For instance, many studies about the effects of media messages on children involve exposing kids to messages with particular characteristics (violence, pro-social behavior, etc.) and then studying how the children interact with one another or with dolls or toys in a laboratory setting. How do we know that this artificial setting isn’t affecting the outcome?

• Other studies are conducted with research subjects chosen because of their easy availability rather than their appropriateness for the study. For example, an advertising scholar may use a classroom full of freshmen advertising students to study the effects of several different commercials for house paint (some using humor, some using a more rational appeal, etc.). How many of the freshmen in the room are actually in the market for house paint (in other words, how many of them are in the target audience for that product)? You need to ask whether the scholarly study you are thinking of using is flawed by the artificiality of the research setting and/or the selection of research subjects.

• As we’ve already said, many scholars do work that is funded by outside sources such as research grants or contracts from large companies or government agencies with an interest in the outcome of the work. This does not, of course, automatically mean that their findings are biased. However, it is not uncommon for drug companies, for instance, to require a scholar funded with one of their grants to seek permission from the company before submitting any findings to academic journals for peer review. Understandably, the drug companies are unlikely to give permission for studies that are not favorable to their products or treatment regimens. As a communicator considering using any type of scholarly research findings, you need to know how the work was funded, and by whom. You also need to ask the author(s) what restrictions they might have been working under as a result of that funding source.

As Browne and Keeley advise, you should approach any scholarly study with the skills of a critical thinker. Just because you are not an expert in the subject area, you are not relieved from the responsibility to make sure you get your facts right and your interpretations verified.