Back to the Physics 3333/ CFB 3333 Home
Page
Espa??ol, Korean, Français
One could easily find a large number of books on the subject of baloney detection, pseudoscience, and pathological science and deception. A complete bibliography would fill several pages. Michael Shermer, Robert Park, Carl Sagan, and others have written books and articles about these subjects. All have produced lists of things to look for and questions to ask in order to detect bogus claims or even outright fraud.
What follows is a synthesis of a number of these. All writers will say that no scheme provides completely foolproof detection of bogus claims; however, if you poke and prod the claim with the questions that follow, the odds are in your favor. The questions are in no particular order, so you'll have to mine out the applicable ones.
1. How is the claim/discovery announced?
A scientific discovery is checked and reviewed before being
accepted for publication in a peer-reviewed journal. If a claim or
discovery is announced by holding a press conference, beware. Such
a claim may be unverified - not checked by an independent
investigator. The cold fusion debacle is often cited. Robert Park's
description of this affair in "Voodoo Science" is recommended
reading. Also beware of "discovery" announcements that ask you to
subscribe (pay) to some newsletter.
2. Does this source often make such claims?
If this particular source frequently makes claims that bear no
resemblance to the best current knowledge, beware. There was
someone in Dallas who, as the year 2000 approached, made a series
of predictions based on the book of Revelations. He described how
the sky would open up, trumpets would sound, and the world would
end. Obviously, nothing happened.
3. Is anecdotal evidence cited?
Anecdotes, or individual stories, are not scientifically useful.
Such a story suggests only that the individual did so-and-so
(assuming no fakery). Nothing about larger populations can be
inferred from it. Any description of an invention or product which
uses large numbers of anecdotes or testimonials is suspect. You
also must remember that testimonials can be bought; you can pay
someone to do it for you. Also - the Federal Trade Commission will
prosecute false claims, but testimonials are not regulated.
4. Does the source claim that "the establishment is trying to
suppress this discovery?"
Such a claim does not provide any evidence that the "discovery" is
really valid. You need to poke these with other questions to find
out more.
5. Does the claim fit in with what we know about the
world?
Does it fit into the larger context of knowledge? Shermer describes
the claim that the Earth is only 6,000 years old. Accepting that
claim requires that you also concede that everything learned about
the Earth and solar system through astronomy, physics, geology,
biology and other sciences is completely wrong.
6. Was the "discovery" made in isolation?
The reality of this world is that someone not trained in the
science and its methods and working alone in a garage is not likely
to make any kind of significant scientific discovery. Such a person
making a great "discovery" is likely to be misinterpreting a known
effect. They will likely persist with their belief in their
discovery even after being shown that it is a well-known
effect.
7. Has anyone tried to disprove the claim?
It is very important to find out if other investigators have tried
to replicate the work. Toward this end, real scientists will
publish complete information about the work, enough for another
competent investigator to attempt replication. Others will repeat
the work and publish the results.
8. Is this source offering a new explanation for observed
phenomena, or simply attacking the existing explanation?
As with claims of suppression, this claim does not offer any
evidence that the discovery is valid. Anyone attacking an existing
explanation had better offer evidence. Absent evidence, all you
know is that they do not like the existing explanation, for
whatever reason.
9. Does the source claim that "this knowledge has been around
for so long it must be good?"
A lot of interesting "discoveries" were made a long time ago. These
include such gems as "the Earth is flat", "fire is a result of the
phlogiston escaping from the burning material", and "the primary
components of everything are Earth, air, fire and water." That they
have been around a long time doesn't make them right. Again, what
is the evidence?
10. Is the observed effect very small and accompanied by lack
of success in increasing it?
Along this line, is the experiment a multi-trial statistical one or
a direct measurement? Park's description of mind-over-matter
experiments is recommended. Although balances capable of measuring
micro-gram (very tiny) forces exist, mind-over-matter experiments
use statistical trials, possibly because no claimant has ever been
able to move a micro-balance with thoughts. The small effect
observed in statistical trials is likely due to some small systemic
bias in the experiment itself. The polywater affair is an example
of a small effect that could not be increased.
11. Does the evidence for the "discovery" not get any better
with time?
See number 10 above. If the observed affect is really a tiny bias
in the experiment, nothing the investigator can do will increase
the effect. Another thing to check is the results of better
experiments. If, as the experimental methods improve, the observed
effect gets smaller, it is likely that a perfect experiment will
show no effect at all.
12. What does the bulk of evidence point to - the new claim
or something else?
Is the source focusing on one small thing and ignoring a huge base
of accumulated evidence that points to something else? Be sure to
look at the claim in the context of current knowledge.
13. What kind of reasoning has been used?
Beware of "Post hoc, ergo propter hoc" thinking; this implies
confusion about causation. This latin adage translates as "after
this, therefore because of this." If A happened then B happened,
then obviously A caused B. This is a very bad assumption. A claim
that A caused B must be accompanied by credible evidence showing
how A caused B. To put it simply, correlation does not mean
causation.
Also beware of such things as questionable techniques (like hypnotic regression), anecdotal evidence (see 3 above), conspiracy theories (like government cover-ups), poor quality evidence (low-resolution photos) and simple misperceptions.
14. Does the new claim offer a new explanation for something,
does it account for as many phenomena as the old one did?
Einstein's relativity explained more phenomena than did Newtonian
mechanics. At velocities much less than the velocity of light,
relativity reduces to Newtonian mechanics.
15. Is there any indication that the source's personal
beliefs and biases are driving the conclusions?
Has contradictory evidence been ignored or pushed aside? It is
necessary to consider all the evidence, not just the favorable
bits. The process of science tends to sort this out.
16. Is it possible to test the claim?
Obviously, a statement that "God did it" cannot be tested in any
way. Such a claim is an untestable construct. Explanations that do
not make any testable predictions are of no use. They do not add to
knowledge.
17. Is a chain of evidence (links) offered?
If chain of evidentiary links for a claim is presented, every link
must be solid. A proposed chain of evidence fails completely if
even one link fails. If someone claims that A causes B, B causes C,
C causes D, and D causes E, they had better be prepared to
demonstrate EVERY link. If, for example, all links are proven
except that C causes D, which cannot be in any way demonstrated,
then there is no proof that A results in E.
18. In extreme cases, such as claims about UFOs, can hoax be
confidently ruled out?
Evidence for "flying saucers" is usually photographic. Modern image
processing technology is so good that it is no longer possible to
reliably detect a hoax photograph by examining it. A claim that a
photograph has "passed all photographic analyses" may mean nothing
more than that the hoax is very well done.
The 1967 Patterson film of a Bigfoot has long been considered the best evidence for the existence of that creature. The film was widely thought to be genuine until an investigator named Greg Long did several years work interviewing everyone in the area who knew anything about Roger Patterson. What Long turned up was evidence that was more than sufficient to show that the famous film is a hoax. Long even talked to the costume manufacturer who sold Patterson the gorilla suit!
If hoax cannot be ruled out, there remains a small probability that the evidence is fake.
19. Would a claimed invention violate the laws of
thermodynamics?
Be aware of the laws of thermodynamics. The universe operates
according to a set of physical laws that we understand reasonably
well. Those laws govern what can and cannot be done. No amount of
cleverness or piety will allow anyone to violate those laws.
The First Law of Thermodynamics says, simply, that "you can't win." Energy is conserved; it is not magically created. Always remember the old phrase " Non gratuitum prandium", which means "no free lunch." No thermal energy can be extracted from a single source - a heat flow from hot to cold is required.
The Second Law of Thermodynamics says "you can't even break even." Any energy conversion process has losses - the output energy will be less than the input. The difference is wasted and contributes to increasing the entropy of the universe. No one has ever found a way to get around this.
20. Is the claim/discovery really spectacular?
Scientists will tell you that "extraordinary claims require
extraordinary evidence." They mean exactly that. If the claim is
that life has been found on Mars or that someone has found a way to
eliminate aging, really extraordinary evidence will be required to
demonstrate it. A vague observation or a few anecdotes will not do
the job.
21. Beware of special pleadings.
Be especially wary of any claim or excuse that the claimed effect
cannot be measured for some reason. It may be that "the presence of
a skeptic contaminates the effect", or "attempting to measure the
effect destroys it", or "bad vibrations interfered", or something
like that. An "effect" that cannot be measured likely does not
exist.
22. If the effect is measured from a sample, how was the
sample obtained?
Obtaining a proper random sample for a statistical measurement is
more difficult than most people realize. There are many problems
that can occur with sampling, problems that can seriously bias the
results.
For example, suppose you want to take a poll about something and you want a random sample of the population of your city. How to do it? You might think of randomly choosing names from the phone directory - but that will exclude all people who either have no phone or have unlisted numbers. Try going downtown and randomly picking people who pass by. That will exclude all people who do not go downtown. Getting a real random sample is not easy.
23. Beware of "It can't be, so it isn't" thinking.
In the Cottingley Fairies incident, in 1917, two young girls took
photographs which supposedly showed fairies cavorting with them in
the woods. The two girls who photographed the fairies were
considered incapable of perpetrating such a hoax, and thus followed
the conclusion that they had not. The photos were therefore fawned
over as genuine pictures of fairies. Truth was that the girls had
indeed cut the "fairies" out of paper and photographed them in a
prank which got out of hand when others saw the photos. This was
not figured out until 1978.
24. Look at spelling and grammar.
When reading any web site, flyer, or anything else, be aware of how
well the stuff is written. A web site or printed item that is
filled with misspellings and bad grammar is highly suspect.
A number of excellent books on the subject of pseudo-science exist. Here are some good ones. Please remember that there are many more. The book on hoaxes is included because it is a good collection of historical (and hysterical) hoaxes.