One of the most frustrating events that I encounter when debating topics goes roughly as follows:
Me: Data indicates that 90% of cats are shy.
Other person: I had a cat when I was a kid that was friendly with everybody!
The reason it particularly annoys me is I’m a mathematician. I’ve studied statistics, taught statistics, and have to use it on a semi-regular basis in my job. What I know, then, is that a sample size of 1 cat, or 3 people, or 5 dogs means basically… nothing.
The Logical Fallacy
This is actually an example of a fallacy called The Hasty Generalization.
A hasty generalisation draws a general rule from a single, perhaps atypical, case. It is the reverse of a sweeping generalisation.
(1) My Christian / atheist neighbour is a real grouch.
(2) Christians / atheists are grouches.
This argument takes an individual case of a Christian or atheist, and draws a general rule from it, assuming that all Christians or atheists are like the neighbour.
The conclusion that it reaches hasn’t been demonstrated, because it may well be that the neighbour is not a typical Christian or atheist, and that the conclusion drawn is false.
Also note The Sweeping Generalization, which is the opposite extreme:
A sweeping generalisation applies a general statement too broadly. If one takes a general rule, and applies it to a case to which, due to the specific features of the case, the rule does not apply, then one commits the sweeping generalisation fallacy. This fallacy is the reverse of a hasty generalisation, which infers a general rule from a specific case.
(1) Children should be seen and not heard.
(2) Little Wolfgang Amadeus is a child.
(3) Little Wolfgang Amadeus shouldn’t be heard.
No matter what you think of the general principle that children should be seen and not heard, a child prodigy pianist about to perform is worth listening to; the general principle doesn’t apply.
What both of these fallacies involve is ignoring the difference between aggregate data and specific instances. In the first case, the mistake is to think a specific (grouchy) person can tell us anything about any other people with a similar and arbitrary characteristic (religious affiliation). In the other, the mistake is to think that the typical value of a child’s audio emissions necessarily apply to a specific child’s music.
To a statistician, it is the difference between a probability distribution, and a statistical sample.
Distributions vs Samples
A probability distribution is a mathematical function that describes the behavior of a random variable (usually a physical or mental characteristic of interest). For some things, like flipping a coin, it can be very simple:
P(heads) = 0.50
P(tails) = 0.50
There are two possible results, and each has a 50% chance (0.50) of happening. For something like the height of an adult, the formula can be quite complicated, as there are numerous possible heights. These are generally approximated as one of several common distributions such as the Normal Distribution (colloquially known as the “Bell Curve” because of its shape). Even that name is somewhat deceptive, as the Normal Distribution is actually a family of distributions with similar properties.
The key to a probability distribution is that it describes all representatives for the variable in question. For example, the Normal Distribution describing the IQ of people represents all people. It has a mean of 100 and standard deviation of 15, which means 68% of people have an IQ between 85 and 115, and 95% of people have an IQ between 70 and 130. Note that it describes the portion of people, overall, who lie in a range of values.
A Statistical Sample, by contrast, pulls a limited number of examples out of that distribution and examines them. For example, if you pick five men and measure their height, that would be a Statistical Sample. However, several things can affect what heights you are likely to get. For example, in Japan the average height is 5’7″, but in the United States the average height is 5’9″ (see List of Heights worldwide). In NBA locker rooms, it’s 6’6″. So where you get your sample from makes a huge difference in what values you’re likely to get.
Statistics vs Personal Experience
When discussing a topic, a good plan is to collect and discuss statistics related to the topic. Christian attitudes towards various social issues, Muslim attitudes towards various social issues, or even what topics are important to registered voters are important for understanding many social topics. This data, when collected by reputable groups with valid techniques, can be very useful for giving us insight into the Probability Distributions that describe the topic at hand.
Personal experience, by contrast, is always a sample, and generally a Biased Sample. For example, somebody might say, “Teenagers generally have poor planning and reasoning skills.” I could respond, “Most of the teenagers I interact with have quite high planning and reasoning skills.” Both statements can be completely true. To understand why, consider the fact that the teenagers I normally interact with are on Minecraft servers. These are people who enjoy playing a game where they have to plan what they want to do, figure out how mechanisms will interact, etc. I didn’t take my sample from a representative pool of teenagers.
This type of analysis applies to a wide variety of things. For example, if I identify a population group that has high levels of anti-social behavior, you could probably cite several friends from that population group who don’t exhibit that behavior. But would you be friends with them if they did? We generally befriend people with similar values and interests. We generally do not make friends with people who are significantly different from us.
As another example, one person could claim that churches are very open to homosexuals, while another could claim the opposite, both based on personal experience. If you live in Topeka, Kansas, home of the Westboro Baptist Church, you are likely to view churches as hostile to homosexuals. If you live in New York City, you are likely to view churches as far more favorable to homosexuals.
Personal experience is very important in shaping who we are, and how we think about various people and groups. Unfortunately, when discussing issues that involve people or groups well outside our personal experience, personal experience is just that: personal. If someone has statistics supporting their view, the way to attack it is to attack the methodology of how they were generated, or some other method related to the argument. Your experience means nothing.