A deeper problem is highlighted in this story: We, as a society, are not skilled at generating statistically significant findings or interpreting experimental results appropriately. Results from a study population of 40 individuals, such as the CRT mentioned here, only tell us that an outcome **can** happen. They do not imply that the same result is **likely** to happen. This mistake is epidemic in learning and education research as well as the popular press. We experiment with a small number of students, find something that works for them and suddenly mandate that treatment for every school child in the English-speaking world. In education, we should be using small population studies to inform ourselves about how to personalize teaching, not how to generalize our practices. These small-scale results challenge us to investigate just who — what kind of student — will respond as our study subjects did, not to assume that our small sample was representative of the whole population. Even more troubling is to contemplate how this tendency toward flawed generalization from small studies impacts democratic decision-making. It is “too bad” if scholars fail to update their citations to reflect emerging revisions in psychological theory. It will be catastrophic if voters and legislators base the laws that control the many on effects found only in a few.
Ed Tech Pioneers envisioned the potential of personal computers to deliver personalized lessons to students with a wide variety of learning styles, strengths, weaknesses and needs. This vision is only partly realized today. Students are accessing a wider variety of materials through the web both within and outside the formal classroom, anytime and anywhere there is a wifi connection. But conscious efforts to embed the educational content in teaching materials specifically designed to make learning easier for “special” students, those whose learning needs put them outside the statistical norm, has not caught the attention of many instructional software developers. We are focused on “going viral”, on creating a product, albeit educational, that appeals to the many while we continue to let the few fall by the wayside.
Terry Burnham writes about the cautions behind assuming one result works for all, even if it is an academic trial.
Liza Loop expands into the greater implication with the comment she posted there that we reflect here.