Inappropriate Use of Research Results: What We Do Wrong and Why It Matters

Generalization Issues
Ed Tech Pioneers envisioned the potential of personal computers to deliver personalized lessons to students with a wide variety of learning styles, strengths, weaknesses and needs. This vision is only partly realized today. Students are accessing a wider variety of materials through the web both within and outside the formal classroom, anytime and anywhere there is a wifi connection. But conscious efforts to embed the educational content in teaching materials specifically designed to make learning easier for “special” students, those whose learning needs put them outside the statistical norm, has not caught the attention of many instructional software developers. We are focused on “going viral”, on creating a product, albeit educational, that appeals to the many while we continue to let the few fall by the wayside.
Terry Burnham writes about the cautions behind assuming one result works for all, even if it is an academic trial.
Liza Loop expands into the greater implication with the comment she posted there that we reflect here.

A deeper problem is highlighted in this story: We, as a society, are not skilled at generating statistically significant findings or interpreting experimental results appropriately. Results from a study population of 40 individuals, such as the CRT mentioned here, only tell us that an outcome **can** happen. They do not imply that the same result is **likely** to happen. This mistake is epidemic in learning and education research as well as the popular press. We experiment with a small number of students, find something that works for them and suddenly mandate that treatment for every school child in the English-speaking world. In education, we should be using small population studies to inform ourselves about how to personalize teaching, not how to generalize our practices. These small-scale results challenge us to investigate just who — what kind of student — will respond as our study subjects did, not to assume that our small sample was representative of the whole population. Even more troubling is to contemplate how this tendency toward flawed generalization from small studies impacts democratic decision-making. It is “too bad” if scholars fail to update their citations to reflect emerging revisions in psychological theory. It will be catastrophic if voters and legislators base the laws that control the many on effects found only in a few.

Advertisements

About lizaloop

Social philosopher, educator, teacher, parent, grandmother, musician, horseback rider, skier, dog owner, writer, trouble-maker, idea generator.

2 thoughts on “Inappropriate Use of Research Results: What We Do Wrong and Why It Matters

  1. Every learning environment is different; so rather than apply one solution to all, should we instead test the environment to find the right tool before we try to teach?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s