What is Reliability in Science?
(testing reliability)

I'm impressed with the effort most of you are putting into learning how to determine reliability of references. It also is clear that different criteria and approaches are used, and that there are differences in understanding of what is meant by "reliability" of the sources. In a sense all are right, but not all are addressing the issues I want you to focus on. I'll explain some of why, and what, seems to contribute to confusion.

Many fields of science have applications in engineering, medicine, social services, politics, and business. Our society might even be addicted to science and it's applications -- we're technology addicts!! The perspective I want to encourage is characterizing our society as using technology intelligently and that we develop technology with intellectual and scientific sophistication. Academia and associated research is one of our ways to implement this development.

Over time, there are social forces that can undermine the quality of the academic disciplines, and genetics has had lots of attention since Mendel's work was rediscovered in 1900. In the early 1900's eugenics threatened the new science of genetics, and led to Nazi Germany's efforts to eliminate Jewish populations -- which was detrimental socially and scientifically. In the 1960's the debates over inheritance of IQ and racial differences caused social backsliding and erosion of basic social values in this country -- another example of "bad science and inappropriate social application." All of these have illustrated how the "science" can be exploited by certain groups of people for reasons that create social disasters as well as undermine the quality of the scientific research. Today, we have similar forces in genetics and medicine that are reflecting economic interests from patented genes and techniques, and biology and medicine are being impacted in ways that undermine the quality of genetics research in similar ways and degrade both society and our ecosystem.

Your defense for your society and for the quality of genetics research depends on your being able to detect the non-science presented as science and the subjective and self-serving claims of "scientific fact" that are made without the authority of science and the power of the scientific process. First, you need to be sophisticated in the scientific process, and second you need to be skilled in the detection of misuse for socially and environmentally inappropriate uses.

Topics in class also reflect our social paradigms about science and society. After seeing the paradigm video, please spend some time thinking about your beliefs, values, and where you acquired them. You may wish to focus on some of them and evaluate them as consistent with your ideas of science, social processes that apply science through technology, and the effects on society and the ecosystem. Our deeply held beliefs are paradigms, and act as filters for what we observe, and how we integrate our ideas for guiding our actions. Most of them were "conveyed" to us by our culture without our agreement, and without our evaluation of their benefits or detriments. Note the people in the last century who we judge from today's perspective as misguided, but thought they were doing the "right thing" with the science of genetics and tools of society.

Daniel Quinn, who has spoken in previous classes calls this "Mother Culture whispers in our ear." He is the author if "Ishmael," which I highly recommend that your read if you haven't already.

To help focus on the confusion about reliability, consider the following words:

validity
accuracy
propaganda
advertising
rhetoric
trustworthy
integrity
authentic

Let's examine how "reliability" is used for evaluation of references. It depends on the context of your use. We are using reliability in the context of science and how it is used in reporting scientific information. Generally, reliability means high integrity, accuracy and validity. But, there are reliable sources that openly do opposites of these and they are reliably untrustworthy, inaccurate and/or invalid in the information presented. Of course, we would avoid this "social aberration" unless we were illustrating actions and perspectives we likely consider socially disruptive. (We usually find such "sociopaths" in prison. -- Or do we?)

A current example is the use of words that mean the opposite to their use, such as "sound science" when discussing political or economic objectives disconnected from the scientific process. This may sound strange, but it arises by exploiting people's confusion about advertising, rhetoric, and validity. The "sound science" slogan exploits a misunderstanding of the scientific process. Such misunderstanding is encoded in "scientific proof" which you should recognize as a logical impossibility. Indeed, "scientific proof" is an oxymoron -- a pair of linked words (adjective and noun) that are inconsistent or illogical. Logic and mathematics have "proofs" but science does not have theoretical "proofs" by the nature of it's discovery processes and inability to evaluate accurately all possibilities.

Reliability -- "Information that can be depended upon with confidence and certainty." (Information need not be "true" but intent of communication must be clear.) We hope the intent is to be trustworthy and logical in communication. For me, this is communication with high integrity. The source is reliable. The information may be found to be false, incomplete or misleading, but is then corrected as best possible. Reliability is maintained as a result.

Science is always changing our understanding, and the models are changed as new information is obtained. The models and assumptions made in building the models alert us to potential weaknesses in our models, and suggest ways we should be cautious, and skeptical of overextending their applicability in technology. Scientific models that seem to agree well with experience are applicable, but must be continually checked to determine if, in a new context, existing models are appropriately describing observations and relationships. In this discussion, the "experience" in testing is observational on the "natural world" as contrasted to the "intellectually and socially created world" of social science. The "paradigm filters" are socially and experientially created, and also filter our observations and interpretations of the natural world. This is why we must be alert to our potential "paradigm paralysis" effects when we are observing and modeling the natural world. Our "two worlds" overlap, sometimes more than we like to admit! Science continually produces new – hopefully improved – usually more reliable, models for new circumstances.

Old models may be useful in special cases. Examples in genetics are Mendel's models of inheritance, or the "Central Dogma of DNA and proteins," even though they are greatly oversimplified descriptions, limited by our ignorance at the time they were created. We now know from our observations over the past 4 decades that the Central Dogma is a gross over simplification, as is Mendel's model of "particulate inheritance." Both belong in the historical archives of biology, along with the "flat earth" model in geography and the Earth-centric model of our solar system in astronomy. (Note: We still use these absurdly oversimplified models when they are convenient – highway maps, "sunrise and sunset" terminology are examples – but we have no problem understanding their irrelevance in science.)

Unreliability may be caused by subtle conflicts of interest. You are less reliable describing a friend's mistake than a person you only observe. Subjectivity may be unconscious, and filter your observations and your interpretations. If you depend on pleasing someone, you will "edit" negative information and possibly "enhance" positive information given to others about them, or you will "flatter" them directly. This is why scientific journals are becoming more demanding of authors, requiring them to list sources of income that may create a real or perceived "conflict of interest." This information becomes public, and readers are now able to judge the results and interpretations with appropriate caution or skepticism.

Validity -- relates to logical arguments, and most of the time is not an issue in science since models are formed logically and new observations that are unexpected lead to new models. "Testing validity" and "proof" is a domain for concern in rhetoric, mathematical logic (theorems) and philosophy. The models that are illogical usually are not long-lived because the observations and inconsistencies of meaning reveal their failure to predict or even explain. In more subjective areas of human activity, this is a problem. Invalidity causes irrelevance of perspectives or statements. In science, claims of "correctness" or "proof" always are invalid by the nature of the scientific process, which is tentative and evolving as information accumulates. Models are either adequate or inadequate.

Accuracy is always a concern, and refers to the ability to measure and describe in detail. "Precision" is related to accuracy, but refers to the repeatability of measurements. Accuracy depends on how close you get to the "true" value of a measurement. High precision may have a consistent bias, and be inaccurate. Our observations may be unreliable. This is a technical problem. It also depends on the skills of observers.

Propaganda (including most modern advertising and political rhetoric) is intended to affect your decisions and limit your understanding to fit a predetermined conclusion. Typically this conclusion is unscientific, inaccurate, self-serving and generally untrue -- that is, it is unreliable. The purpose of this rhetoric is to manipulate your behavior in a way that others prefer, and may be designed to be accomplished without your recognizing that you are being manipulated while you perceive it to be your "best choice." Your choice becomes "a part of you" and acts as a mental filter, which means that it creates a paradigm to influence your perspective and actions. The paradigm becomes more difficult to recognize (unconscious or rationalized) and distorts the information to become consistent with the paradigm. You find it difficult to "believe your eyes" when you see unexpected things. When our assumptions become "automatic and unconscious" we are now under the "control" of our paradigms. This effect is illustrated in the video, The Business of Paradigms, and is called "paradigm paralysis." All of us have paradigms we obtain from our culture, belief systems, habitual activity, etc. When this happens, our observations as well as our interpretations become unreliable. This is a human belief problem that requires effort to recognize, control and replace with more "objective" thought and understanding.

Rhetoric is related to this above in that it is language used to convince you that a certain understanding is "correct" or "beneficial" to you. Rhetoric contrasts with dialog, which is an attempt to understand multiple perspectives, rather than to convince another that one perspective is "best." Debate is a formalized form or rhetoric.

Trustworthy means that interpretations and observations have been viewed with caution and scrutiny for interference of inappropriate paradigms. It also implies when information is communicated that there is consistency of delivery of information that is not manipulative for ulterior motives, leaving understanding and conclusions to individuals receiving the information. When the reliability of a source is lost the source becomes untrustworthy. There is a conflict between objectives of your source and your independent understanding – conveying information vs. manipulating your conclusions. It means that, as a recipient, your individual thought and observation processes are respected. A trustworthy person will respect your self interests and attempt to give full support to your having accurate information and understanding of alternative relationships.

Integrity of another person implies that you readily observe respect for your interests and perceptions, you are given complete information as far as possible, and that you are supported in your development of understanding and judgment. This also is a way of identifying a "reliable" person or information from a reliable person.