3. Recognize the purposes of and differences among sample surveys, experiments, and observational studies; explain how randomization relates to each.

Students should be aware of and understand the similarities and differences between the three most common types of data gathering methods: sample surveys, experiments, and observational studies. Students should know that randomization is the process selecting a random sample so that the sample represents the population as accurately as possible.

Sample Surveys

Sample surveys consist of randomly selecting a subset of people from our target population and measuring whatever parameter we're interested in. If these groups of people aren't selected randomly, the integrity of the study may be compromised. How can this happen?

Lazy researchers only sample the people who are easiest to sample. (They'll often make excuses like, "I think I'm stuck to the chair.") Sometimes these easily sampled individuals are predominantly English speakers, adults, or just people nearby. Sampling only from this limited group will yield results that aren't indicative of the larger population.

Biased researchers select certain people based on their preference either for the people or for how they want the study to turn out. (They'll often say things like, "I like you! You're in!") This, of course, isn't random at all. So the results of this random sample won't be useful in analyzing the population as a whole.

If the above non-random samples are used, then the only inferences that can be drawn are about populations similar to the ones tested and not the entire population. In order to be able to draw inferences about the entire population, we need to randomize how we obtain our sample survey.

Experiment

An experiment is a process of trial and error that is used to test a hypothesis. We force a "cause" to observe an "effect." (This doesn't mean that our cause-and-effect relationship is valid; we're just measuring one.) After enough data is collected, we either reject the hypothesis or fail to reject it.

Most experiments we hear about are lab or controlled experiments. It's called "lab" for a reason. In these types of experiments, a control population is compared to a test group to see if there is a significant difference between the two. 

The random allocation of test subjects to control and test groups helps ensure randomization and is aptly called a randomized trial. That way, both the control and test groups receive a representative mix of test subjects and the results can apply to the entire population in question.

Double blinding is another method of randomization that ensures that neither the researcher nor the test subject introduces any bias. For example, in a study measuring the effects of the drug, both the doctor administering the experimental medication and the patient receiving it do not know whether the "medication" is the one being tested or the control medication.

Some experiments cannot occur in a lab (oftentimes because there are too many aspects that cannot be controlled in a laboratory environment). These are called field experiments or natural experiments.

An example can be a study of elephant behavior based on weather patterns. It'd be difficult to randomize a study like this based on many factors. Not only that, but it's nigh on impossible to control the weather, let alone isolate behavioral patterns solely due to the weather. Instead of studying the elephants, let's ride 'em!

Observational Study

An observational study is something like a randomized controlled trial (where subjects are randomly assigned to test and control groups), but not quite. In some scenarios, whether due to time, money, or legal or ethical concerns, it's not possible to have a specific control group and a treatment group. That means investigators must rely on existing control and trial groups in order to observe their study outcomes.

For instance, a scientist who wants to study radiation effects on humans 48 hours after a nuclear disaster probably wouldn't get many volunteers. That's not even mentioning the ethical, legal, and financial issues involved.

What can be done instead? An observation study, of course! That's the subject we're on, isn't it? This same scientist may have to wait for a terrible catastrophe to occur, like a nuclear meltdown, before he can begin his observations and compare it to a control group.

As such, observational studies are rarely, if ever, considered random (unlike controlled experiments and probability based sample surveys).

Based on a description of the study, students should be able to identify the kind of study, the methods that promote or hinder randomization, and the effects on the results of the study were certain elements of the study altered.