I’ve long respected Dan Willingham’s work, but I’m concerned about a study he tweeted favorably about on twitter this week.
I and others tweeted some criticisms, and Dan posted a rebuttal on his blog. In this post, which I’ll also share on twitter, I’m going to respond, point by point, to that rebuttal.
First, a few notes about my respect for Dan and his work. When his first book came out, I gave it a 5-star review on Amazon; I linked to an article of his on phonics here just 3 months ago; and I’ve had cordial, even friendly, exchanges with him over the years.
However, the study Dan cited has raised red flags for me. Its object of investigation—Rapid Prompting Method—is similar to a discredited method called “Facilitated Communication” (FC). Both have been used with minimally-verbal children with autism, and both are things I’ve blogged about, most recently here.
RPM has never undergone the rigorous testing that FC has, and a number of us, long familiar with the litany of non-rigorous tests, anecdotes, and unsubstantiated claims that purportedly support RPM, took a look at who was involved in this study and immediately had concerns.
One concern: the conflicts of interest of the first author, Vikram Jaswal. First, there’s his long-time collaboration with the person who runs the clinic, whose teaching method (a form of RPM) the study investigated and drew favorable conclusions about. Second, there’s the fact that this method is used with Jaswal’s autistic daughter. From an article in the Washington Post:
Another concern: Jaswal’s collaborator, whom various indicators suggest was also the assistant overseeing the experiment. She–I’ll avoid using her name–has a history of claiming credentials she did not have, including in her dealings with paying clients: dealings relevant to the questions addressed by the experiment.
Unimpressed with our remarks, Dan ultimately tweeted:
Soon a number of threads spun out containing discussions of the study’s methodology and data, including this one from @24shaz. @angelrabanas analyzed one video in detail and posted her analysis on YouTube.
So I pinged Dan, and a day or two later he notified us that that he’d put up a post on his blog in response entitled “Responding to a Study You Just KNOW Is Wrong.”
Dan begins his post with a discussion of the learning styles controversy, and how educators who think they are “in the know” will pounce “with glee” on anyone who mentions learning styles. He goes on to say:
This made me wonder whether any of us had said anything bullying or snobbish about Jaswal et al or their studies. So I went through the various threads, and the only things I found that I thought might possibly be interpreted as snobbish or bullying were these:
Anyway, Dan goes on to explain how if you uncritically dismiss disproven theories you might miss new developments. As an example of a new development that he has not missed, Dan cites new research on learning styles theory and notes that his views have changed as a result.
I’m familiar with Dan’s skepticism about learning styles, and so I was intrigued to hear he’d changed his mind. I followed the link he gave to his new views, read through to the conclusion, and this is what I found:
But anyway, back to Dan’s blog post. Dan next turns to Jaswal et al’s paper:
Missing from this discussion is the most likely way that cuing could occur: not through a distinct cue for each of the 26 letters of the alphabet, but through the movement by the facilitator of the letter board. Such movement is a regular occurrence in videos of RPM, including those that are included in this study.
In particular, the facilitator may (unwittingly) shift the letter board such that a particular letter approaches the subject’s finger and/or enters the path of his/her eye gaze. To pick up on such cues, the subject does not look up at the assistant.
A moving letter board also makes it hard to draw reliable conclusions about intentional eye fixations or about typing rhythm.
Anyway, moving on, Dan turns to some of our concerns. Noting (correctly) that I mischaracterized Jaswal’s relationship to something called “The Tribes” (thus mischaracterizing the precise nature of Jaswal’s collaboration with clinic’s director), he turns to a tweet by Jason Travers, observing that:
Dan leaves it at that. But Jason is making an important point. Yes, he’s asking for a different experiment, but not one that differs in the ultimate questions being explored (i.e., the authorship of the messages). Rather, Jason is asking for an experiment that addresses these same questions using a more rigorous, well-established methodology: one where the possibility of facilitator influence is eliminated, as it is in a message passing test.
Jaswal et al attempt to explain why they think message-passing tests are problematic, making some claims that anyone with any background in language acquisition and linguistic pragmatics would instantly recognize as absurd:
Years of prompting and feedback are needed before children are able to convey new information? Tell that to the parent of a young toddler! Then there’s the fact that all of the participants, although they’re called “nonspeaking,” can talk—just not fluently and interactively.
What would cause these kids to be more fluent and interactive when typing on a letter board with an index finger? That is a question that Jaswal et al leave unexplained.
Even if Jaswal et al had good reasons for rejecting message passing, they’d still need to explain why they didn’t take other influence-reducing steps–like blindfolding the assistant and/or placing the alphabet board on a stationary stand. Such options are never even mentioned.
Dan does not discuss these huge design flaws. Nor does he address our concerns about how a moving board (1) provides cues that don’t involve subjects looking up at the assistant and (2) seriously warps the eye gaze results. Nor does he address various concerns we had about how the data was coded.
Instead, having dispensed with Jason, Dan then reacts to two tweets that, he grants, do address the data and its methods.
I’ve reviewed these and other tweets of @24shaz and @angelrabanas re the videos, and as far as I can tell, Dan’s reaction here is a non sequitur that does not rebut their critiques. Doing that convincingly would require quite a bit more discussion.
Dan then turns to a tweet I posted:
So I went back to the methods section to see if I had missed something:
There’s nothing here, so far as I can tell, that’s relevant to the issue of oral cues being delivered ahead of letter selection. Nor does this issue come up anywhere else in the methods section.
Having dispensed with the above tweets, Dan turns to one regarding the study’s small sample size and explains how these are routine in neuropsychology. He does not address concerns about how the subjects were non-randomly selected–they were hand-picked by the clinic–or about the grounds for eliminating one of the subjects.
Dan then dispenses with the rest of our tweets, among them many more that focused on methodology and data (and a mysteriously missing video) with this:
Alluding to his opening remarks, Dan then contrasts his take on our tweets with his take on his openness to revising his views on learning styles.
He goes on to say that people would be best advised to just ignore studies that they aren’t going to bother to read and that are outside their areas of expertise:
Regarding the second point, on expertise, what’s most relevant here, as far as our critiques go, is the expertise involved in evaluating RPM and FC videos for signs of manipulation. Some of us have spent many hours doing this—perhaps more hours than this paper’s authors, reviewers and advisors have.
Regarding the first point, about bothering to read things, the same applies to Dan himself. If he has the feeling that a conclusion (say one regarding Jaswal’s paper) is probably wrong, but doesn’t want to take the time to properly engage with the arguments, he might be better off ignoring it.
Incidentally, if Dan has a feeling that Jaswal’s critics are probably wrong, he has left out one possible reason why: he and Jaswal are colleagues in the same department at UVA, and Dan is mentioned in the paper’s acknowledgements:
Dan concludes with this:
If Dan were to inform himself about the harsh and abusive practices in both FC and RPM (evident in some of the videos of kids who are subjected to it), and about the linguistic and educational opportunity costs that they potentially impose, and about the evidence of greed and malice within the FC/RPM industrial complex, he might hesitate to interpret our passionate concern as nothing more than righteous indignation.
Catherine & Katharine 