Evaluating Internet Research Sources

By Robert Harris
Version Date: June 15, 2007

Introduction: The Diversity of Information

Information is a Think about the magazine section in your local grocery store. If you reach out
Commodity with your eyes closed and grab the first magazine you touch, you are about as
Available in likely to get a supermarket tabloid as you are a respected journal (actually more
Many Flavors likely, since many respected journals don't fare well in grocery stores). Now
imagine that your grocer is so accommodating that he lets anyone in town print
up a magazine and put it in the magazine section. Now if you reach out blindly,
you might get the Elvis Lives with Aliens Gazette just as easily as Atlantic
Monthly or Time.

Welcome to the Internet. As I hope my analogy makes clear, there is an

extremely wide variety of material on the Internet ranging in its accuracy,
reliability, and value. Unlike most traditional information media (books,
magazines, organizational documents), no one has to approve the content
before it is made public. It's your job as a searcher, then, to evaluate what you
locate in order to determine whether it suits your needs.

Information Information is everywhere on the Internet, existing in large quantities and

Exists on a continuously being created and revised. This information exists in a large
Continuum of variety of kinds (facts, opinions, stories, interpretations, statistics) and is
Reliability and created for many purposes (to inform, to persuade, to sell, to present a
Quality viewpoint, and to create or change an attitude or belief). For each of these
various kinds and purposes, information exists on many levels of quality or
reliability. It ranges from very good to very bad and includes every shade in
between.

Getting Started: Screening Information

Pre-evaluation The first stage of evaluating your sources takes place before you do any
searching. Take a minute to ask yourself what exactly you are looking for. Do
you want facts, opinions (authoritative or just anyone's), reasoned arguments,
statistics, narratives, eyewitness reports, descriptions? Is the purpose of your
research to get new ideas, to find either factual or reasoned support for a
position, to survey opinion, or something else? Once you decide on this, you

Evaluating Internet Research Sources 1

 will be able to screen sources much more quickly by testing them against your
research goal. If, for example, you are writing a research paper, and if you are
looking for both facts and well-argued opinions to support or challenge a
position, you will know which sources can be quickly passed by and which
deserve a second look simply by asking whether each source appears to offer
facts and well-argued opinions, or just unsupported claims.

Select Sources Becoming proficient at selecting sources will require experience, of course, but
Likely to be even a beginning researcher can take a few minutes to ask, "What source or
Reliable what kind of source would be the most credible for providing information in
this particular case?" Which sources are likely to be fair, objective, lacking
hidden motives, showing quality control? It is important to keep these
considerations in mind so that you will not simply take the opinion of the first
source or two you can locate. By thinking about these issues while searching,
you will be able to identify suspicious or questionable sources more readily.
With so many sources to choose from in a typical search, there is no reason to
settle for unreliable material.

Source Selection Tip:

Try to select sources that offer as much of the following information as possible:
Author's Name
Author's Title or Position
Author's Organizational Affiliation
Date of Page Creation or Version
Author's Contact Information
Some of the Indicators of Information Quality (listed below)

Evaluating Information: The Tests of Information Quality

Reliable You may have heard that "knowledge is power," or that information, the raw
Information is material of knowledge, is power. But the truth is that only some information is
Power power: reliable information. Information serves as the basis for beliefs,
decisions, choices, and understanding our world. If we make a decision based
on wrong or unreliable information, we do not have power--we have defeat. If
we eat something harmful that we believe to be safe, we can become ill; if we
avoid something good that we believe to be harmful, we have needlessly
restricted the enjoyment of our lives. The same thing applies to every decision
to travel, purchase, or act, and every attempt to understand.

Evaluating Internet Research Sources 2

 Source Source evaluation--the determination of information quality--is something of
Evaluation is an an art. That is, there is no single perfect indicator of reliability, truthfulness, or
Art value. Instead, you must make an inference from a collection of clues or
indicators, based on the use you plan to make of your source. If, for example,
what you need is a reasoned argument, then a source with a clear, well-argued
position can stand on its own, without the need for a prestigious author to
support it. On the other hand, if you need a judgment to support (or rebut)
some position, then that judgment will be strengthened if it comes from a
respected source. If you want reliable facts, then using facts from a source that
meets certain criteria of quality will help assure the probability that those facts
are indeed reliable.

The CARS The CARS Checklist (Credibility, Accuracy, Reasonableness, Support) is

Checklist designed for ease of learning and use. Few sources will meet every criterion in
the list, and even those that do may not possess the highest level of quality
possible. But if you learn to use the criteria in this list, you will be much more
likely to separate the high quality information from the poor quality
information.

The CARS Checklist for Information Quality

Credibility Because people have always made important decisions based on information,
evidence of authenticity and reliability--or credibility, believability--has always
been important. If you read an article saying that the area where you live will
experience a major earthquake in the next six months, it is important that you
should know whether or not to believe the information. Some questions you
might ask would include, What about this source makes it believable (or not)?
How does this source know this information? Why should I believe this source
over another? As you can see, the key to credibility is the question of trust.

There are several tests you can apply to a source to help you judge how
credible and useful it will be:

Author's The author or source of the information should show some evidence of being
Credentials knowledgeable, reliable, and truthful. Here are some clues:

Author's education, training, and/or experience in a field relevant to the

information. Look for biographical information, the author's title or
position of employment
Author provides contact information (email or snail mail address, phone

Evaluating Internet Research Sources 3

 number)
Organizational authorship from a known and respected organization
(corporate, governmental, or non-profit)
Author's reputation or standing among peers.
Author's position (job function, title)

Evidence of Most scholarly journal articles pass through a peer review process, whereby
Quality Control several readers must examine and approve content before it is published.
Statements issued in the name of an organization have almost always been seen
and approved by several people. (But note the difference between, "Allan
Thornton, employee of the National Oceanographic and Atmospheric Agency,
says that a new ice age is near," and "The National Oceanographic and
Atmospheric Agency said today that a new ice age is near." The employee is
speaking for himself, whereas a statement in the name of NOAA represents the
official position of NOAA.)

Evidence of quality control of Internet material includes these items:

Information presented on organizational web sites

On-line journals that use refereeing (peer review) by editors or others
Postings of information taken from books or journals that have a quality
control process

Meta- Meta-information is information about information. Information workers

information (sometimes called knowledge workers) all over the world are constantly poring
over, processing, and evaluating information--and making notes. As the
challenges produced by the increasing quantity of information continue, access
to high quality meta-information will become increasingly important. Meta-
information can take many forms, but there are two basic types: summary and
evaluative.

Summary meta-information includes all the shortened forms of information,

such as abstracts, content summaries, or even tables of contents. This type of
meta-information gives us a quick glance at what a work is about and allows us
to consider many different sources without having to go through them
completely.

Evaluative meta-information includes all the types that provide some judgment
or analysis of content. This type includes recommendations, ratings, reviews,
and commentaries. Even the search results order of pages from a search engine
like Google represents a type of evaluative meta-information, since pages are
ranked in part by the number of other pages linked to them (and hence "voting"

Evaluating Internet Research Sources 4

 for them in some sense).

And, of course, these two types can be combined, resulting in the best form of
meta-information, providing us with a quick overview and some evaluation of
the value. An example would be a World Wide Web yellow pages or directory
which describes each selected site and provides evaluations of its content.

Indicators of You can sometimes tell by the tone, style, or competence of the writing
Lack of whether or not the information is suspect. Here are a few clues:
Credibility
Anonymity
Lack of Quality Control
Negative Meta-information. If all the reviews are critical, be careful.

Bad grammar or misspelled words. Most educated people use grammar fairly
well and check their work for spelling errors. An occasional split infinitive or
comma in the wrong place is not unusual, but more than two or three spelling
or grammar errors are cause for caution, at least. Whether the errors come
from carelessness or ignorance, neither puts the information or the writer in a
favorable light.

Accuracy The goal of the accuracy test is to assure that the information is actually
correct: up to date, factual, detailed, exact, and comprehensive. For example,
even though a very credible writer said something that was correct twenty years
ago, it may not be correct today. Similarly, a reputable source might be giving
up-to-date information, but the information may be only partial, and not give
the full story. Here are some concepts related to accuracy:

Timeliness Some work is timeless, like the classic novels and stories, or like the thought-
provoking philosophical work of Aristotle and Plato. Other work has a limited
useful life because of advances in the discipline (psychological theory, for
example), and some work is outdated very quickly (such as technology news).
You must therefore be careful to note when the information you find was
created, and then decide whether it is still of value (and how much value). You
may need information within the past ten years, five years, or even two weeks.
But old is not necessarily bad: nineteenth-century American history books or
literary anthologies can be highly educational because they can function as
comparisons with what is being written or anthologized now. In many cases,
though, you want accurate, up-to-date information.

An important idea connected with timeliness is the dynamic, fluid nature of

information and the fact that constant change means constant changes in

Evaluating Internet Research Sources 5

 timeliness. The facts we learn today may be timely now, but tomorrow will not
be, especially in technology, science, medicine, business, and other fields
always in flux. We must remember to check and re-check our data from time
to time and realize that we will always need to update our facts.

Note: Many Web pages display today's date automatically, regardless of when
the content on the page was created. If you see today's date on a page other
than from a news site, be extra careful.

Comprehensiveness Any source that presents conclusions or that claims (explicitly or implicitly)
to give a full and rounded story, should reflect the intentions of
completeness and accuracy. In other words, the information should be
comprehensive. Some writers argue that researchers should be sure that
they have "complete" information before making a decision or that
information must be complete. But with the advent of the information age,
such a goal is impossible, if by "complete" we mean all possible
information. No one can read 20,000 articles on the same subject before
coming to a conclusion or making a decision. And no single piece of
information will offer the truly complete story--that's why we rely on more
than one source. On the other hand, an information source that deliberately
leaves out important facts, qualifications, consequences, or alternatives may
be misleading or even intentionally deceptive.

Audience and For whom is this source intended and for what purpose? If, for example,
Purpose you find an article, "How Plants Grow," and children are the intended
audience, then the material may be too simplified for your college botany
paper. More important to the evaluation of information is the purpose for
which the information was created. For example, an article titled, "Should
You Buy or Lease a Car?" might have been written with the purpose of
being an objective analysis, but it may instead have been written with the
intention of persuading you that leasing a car is better than buying. In the
latter case, the information will most likely be biased or distorted. Such
information is not useless, but the bias must be taken into consideration
when interpreting and using the information. (In some cases, you may need
to find the truth by using only biased sources, some biased in one direction
and some biased in the other.) Be sure, then, that the intended audience and
purpose of the article are appropriate to your requirements or at least clearly
in evidence so that you may take them into account. Information pretending
to objectivity but possessing a hidden agenda of persuasion or a hidden bias
is among the most common kind of information in our culture.

Evaluating Internet Research Sources 6

 Indicators of a In addition to an obvious tone or style that reveals a carelessness with detail
Lack of Accuracy or accuracy, there are several indicators that may mean the source is
inaccurate, either in whole or in part:

No date on the document

Vague or sweeping generalizations
Old date on information known to change rapidly
Very one sided view that does not acknowledge opposing views or
respond to them

Reasonableness The test of reasonableness involves examining the information for fairness,
objectivity, moderateness, and consistency.

Fairness Fairness includes offering a balanced, reasoned argument, not selected or

slanted. Even ideas or claims made by the source's opponents should be
presented in an accurate manner. Pretending that the opponent has wild,
irrational ideas or arguments no one could accept is to commit the straw man
fallacy. A good information source will also possess a calm, reasoned tone,
arguing or presenting material thoughtfully and without attempting to get you
emotionally worked up. Pay attention to the tone and be cautious of highly
emotional writing. Angry, hateful, critical, spiteful tones often betray an
irrational and unfair attack underway rather than a reasoned argument. And
writing that attempts to inflame your feelings to prevent you from thinking
clearly is also unfair and manipulative.

Objectivity There is no such thing as pure objectivity, but a good writer should be able to
control his or her biases. Be aware that some organizations are naturally not
neutral. For example, a professional anti-business group will find, say, that
some company or industry is overcharging for widgets. The industry trade
association, on the other hand, can be expected to find that no such
overcharging is taking place. Be on the lookout for slanted, biased, politically
distorted work.

One of the biggest hindrances to objectivity is conflict of interest. Sometimes

an information source will benefit in some way (usually financially, but
sometimes politically or even emotionally or psychologically) if that source can
get you to accept certain information rather than the pure and objective truth.
For example, many sites that sell "natural" products (cosmetics, vitamins,
clothes) often criticize their competitors for selling bad, unhealthy or dangerous
products. The criticism may be just, but because the messenger will gain

Evaluating Internet Research Sources 7

 financially if you believe the message, you should be very careful--and check
somewhere else before spending money or believing the tale.

Moderateness Moderateness is a test of the information against how the world really is. Use
your knowledge and experience to ask if the information is really likely,
possible, or probable. Most truths are ordinary. If a claim being made is
surprising or hard to believe, use caution and demand more evidence than you
might require for a lesser claim. Claims that seem to run against established
natural laws also require more evidence. In other words, do a reality check. Is
the information believable? Does it make sense? Or do the claims lack face
validity? That is, do they seem to conflict with what you already know in your
experience, or do they seem too exaggerated to be true? "Half of all Americans
have had their cars stolen." Does that pass the face validity test? Have half of
your friends had their cars stolen? Is the subject on the news regularly (as we
might assume it would be if such a level of theft were the case)?

It is important, of course, to remember that some truths are spectacular and

immoderate. Over the past few decades, Michel Lotito, a French performer
with the stage name of Monsieur Mangetout (French for "eats everything") has
actually eaten 18 bicycles, several TV sets, a few shopping carts, and a small
airplane by first having them ground into a fine powder and sprinkling a few
teaspoonfuls on his breakfast cereal each morning. So do not automatically
reject a claim or source simply because it is astonishing. Just be extra careful
about checking it out.

Consistency The consistency test simply requires that the argument or information does not
contradict itself. Sometimes when people spin falsehoods or distort the truth,
inconsistencies or even contradictions show up. These are evidence of
unreasonableness.

World View A writer's view of the world (political, economic, religious--including anti-
religious--and philosophical) often influences his or her writing profoundly,
from the subjects chosen to the slant, the issues raised, issues ignored, fairness
to opponents, kinds of examples, and so forth. World view can be an
evaluative test because some world views in some people cause quite a
distortion in their view of reality or their world view permits them to fabricate
evidence or falsify the positions of others. For some writers, political agendas
take precedence over truth. If you are looking for truth, such sources are not
the best.

Indicators of a Writers who put themselves in the way of the argument, either emotionally or

Evaluating Internet Research Sources 8

 Lack of because of self interest, often reveal their lack of reasonableness. If, for
Reasonableness example, you find a writer reviewing a book he opposes by asserting that "the
entire book is completely worthless claptrap," you might suspect there is more
than a reasoned disagreement at work. Here are some clues to a lack of
reasonableness:

Intemperate tone or language ("stupid jerks," "shrill cries of my extremist

opponents")
Overclaims ("Thousands of children are murdered every day in the United
States.")
Sweeping statements of excessive significance ("This is the most important
idea ever conceived!")
Conflict of Interest ("Welcome to the Old Stogie Tobacco Company Home
Page. To read our report, 'Cigarettes Make You Live Longer,' click here."
or "The products our competitors make are dangerous and bad for your
health.")

Support The area of support is concerned with the source and corroboration of the
information. Much information, especially statistics and claims of fact, comes
from other sources. Citing sources strengthens the credibility of the
information. (Remember this when you write a research paper.)

Source Where did this information come from?

Documentation What sources did the information creator use?
or Bibliography Are the sources listed?
Is there a bibliography or other documentation?
Does the author provide contact information in case you wish to discuss an
issue or request further clarification?
What kind of support for the information is given?
How does the writer know this?
It is especially important for statistics to be documented. Otherwise, someone
may be just making up numbers. Note that some information from corporate
sites consists of descriptions of products, techniques, technologies, or processes
with which the corporation is involved. If you are careful to distinguish
between facts ("We mix X and Y together to get Z") and advertising ("This
protocol is the best in the industry"), then such descriptions should be reliable.

Corroboration See if other sources support this source. Corroboration or confirmability is an

important test of truth. And even in areas of judgment or opinion, if an
argument is sound there will probably be a number of people who adhere to it

Evaluating Internet Research Sources 9

 or who are in some general agreement with parts of it. Whether you're looking
for a fact (like the lyrics to a song or the date of an event), an opinion (like
whether paper or plastic is the more environmentally-friendly choice), or some
advice (like how to grow bromeliads), it is a good idea to triangulate your
findings; that is, find at least three sources that agree. If the sources do not
agree, do further research to find out the range of opinion or disagreement
before you draw your conclusions.

What you are doing with corroboration, then, is using information to test
information. Use one source, fact, point of view, or interpretation to test
another. Find other information to support and reconfirm (or to challenge or
rebut) information you have found.

Corroboration is especially important when you find dramatic or surprising

information (information failing the moderateness test, above). For example,
the claim that a commonly used food additive is harmful should be viewed with
skepticism until it can be confirmed (or rebutted) by further research. The
claim may be true, but it seems unlikely that both government and consumer
organizations would let the additive go unchallenged if indeed it were harmful.

External While the test of corroboration involves finding out whether other sources
Consistency contain the same new information as the source being evaluated, the test of
external consistency compares what is familiar in the new source with what is
familiar in other sources. That is, information is usually a mixture of old and
new, some things you already know and some things you do not. The test of
external consistency asks, Where this source discusses facts or ideas I already
know something about, does the source agree or harmonize or does it conflict,
exaggerate, or distort? The reasoning is that if a source is faulty where it
discusses something you already know, it is likely to be faulty in areas where
you do not yet know, and you should therefore be cautious and skeptical about
trusting it.

Indicators of a As you can readily guess, the lack of supporting evidence provides the best
Lack of Support indication that there is indeed no available support. Be careful, then, when a
source shows problems like these:

Numbers or statistics presented without an identified source for them

Absence of source documentation when the discussion clearly needs such
documentation
You cannot find any other sources that present the same information or
acknowledge that the same information exists (lack of corroboration)

Evaluating Internet Research Sources 10

Summary of The CARS Checklist for Research Source Evaluation

Credibility Trustworthy source, author’s credentials, evidence of quality control,

known or respected authority, organizational support. Goal: an
authoritative source, a source that supplies some good evidence that allows
you to trust it.

Accuracy Up to date, factual, detailed, exact, comprehensive, audience and purpose

reflect intentions of completeness and accuracy. Goal: a source that is
correct today (not yesterday), a source that gives the whole truth.

Reasonableness Fair, balanced, objective, reasoned, no conflict of interest, absence of

fallacies or slanted tone. Goal: a source that engages the subject
thoughtfully and reasonably, concerned with the truth.

Support Listed sources, contact information, available corroboration, claims

supported, documentation supplied. Goal: a source that provides
convincing evidence for the claims made, a source you can triangulate (find
at least two other sources that support it).

Living with Information: The CAFÉ Advice

Here is one last piece of advice to help you live well in the world of
information: Take your information to the Café (Challenge, Adapt, File,
Evaluate).

Challenge Challenge information and demand accountability. Stand right up to the

information and ask questions. Who says so? Why do they say so? Why
was this information created? Why should I believe it? Why should I trust
this source? How is it known to be true? Is it the whole truth? Is the
argument reasonable? Who supports it?

Adapt Adapt your skepticism and requirements for quality to fit the importance of
the information and what is being claimed. Require more credibility and
evidence for stronger claims. You are right to be a little skeptical of
dramatic information or information that conflicts with commonly accepted

Evaluating Internet Research Sources 11

 ideas. The new information may be true, but you should require a robust
amount of evidence from highly credible sources.

File File new information in your mind rather than immediately believing or
disbelieving it. Avoid premature closure. Do not jump to a conclusion or
come to a decision too quickly. It is fine simply to remember that someone
claims XYZ to be the case. You need not worry about believing or
disbelieving the claim right away. Wait until more information comes in,
you have time to think about the issue, and you gain more general
knowledge.

Evaluate Evaluate and re-evaluate regularly. New information or changing

circumstances will affect the accuracy and hence your evaluation of previous
information. Recognize the dynamic, fluid nature of information. The
saying, "Change is the only constant," applies to much information,
especially in technology, science, medicine, and business.

Other Internet Related Items

World Wide Web Research Tools

Internet Search Tips and Strategies
Education-Related Sites
Resources for English Language and Literature
Gullibility Virus Warning
Miscellaneous Links
Health of My Family

Retrieved on August 5, 2010, from http://www.virtualsalt.com/evalu8it.htm.

Evaluating Internet Research Sources 12