An ecologically valid experiment captures relevant real-life conditions so as to make it viable to generalize the conclusions of the experiment beyond the laboratory walls and to the relevant population (which, for my purposes, is the population of experts). Though it is difficult to say what exactly it is for an experiment to be ecologically valid, there are a number of features of the varied-focus experiments that—though I think do not tell the whole story—intimate trouble.
One arguably ecologically invalid feature of these experiments that might seem significant to their outcome is that what subjects are asked to perform in the skill-related supplemental task is not something they would normally do. Christensen et al. (forthcoming) explain this with an analogy. Asking players to continually monitor their feet, they tells us, is like asking a driver to continually monitor the rearview mirror while driving. On Christensen’s view, just like a driver would perform suboptimally while doing this, even though it is important to occasionally monitor the rearview mirror, a soccer player would perform suboptimally while continually monitoring their feet even if such monitoring is part of typical play.
This seems reasonable, however, it may be worthwhile to ask whether it is actually known that even continually monitoring one’s feet while playing high level soccer would interfere with performance. I would like to see a study that looks into this, a study that simply asks players to continually monitor their feet, yet does not also ask them to report on what they have identified via such monitoring. Of course, in such a situation it would be more difficult to know whether subjects are complying with the request. But on the other hand, such a setup would seem to be more ecologically valid. Participants in Ford et al.’s study, however, are not merely asked to monitor various aspects of their skills; they are asked to monitor and report them, and sometimes to report on actions that have already gone by: individuals are instructed to attend to the side of the foot that had just been in contact with the ball. This does make it less like real-life conditions since even if expert soccer players were to consciously focus on their feet in real-life situations, it is highly likely that they do not reflect back on past foot action. Thus Christensen’s analogy should be to a driver who is asked to continually monitor the rearview mirror and then when she hears a randomly generated tone, report what she has just noticed. If continually monitoring the rearview mirror was already going to interfere with performance, this would make driving even harder.
Another way in which the experiments are not ecologically valid is that they fail to capture the high-stakes situations athletes find themselves in when they are playing a real game. Because of this, it is very likely that an athlete will find herself in at least slightly different psychological state during an experiment than during an actual game; in particular, the controlled environment might not elicit the type of intense focus that is characteristic of expert level performance in situations that matter.
Varied-focus experiments are often seen as supporting the view that, as Beilock et al. put it, expert skill is “governed by proceduralized knowledge that does not require explicit monitoring and control,” and that an extraneous task (such as listening to a recording of words and aurally identifying a target word) “should not degrade performance in comparison with skill execution under single-task conditions, as attention should be available to allocate to secondary task demands if necessary without detracting from control of the primary skill” (2002: p. 9). But even if attention to a secondary task does not degrade performance in an experimental setting, it might degrade performance in the wild. In a psychology experiment, subjects may be motivated, but it is unlikely that they are as highly motivated as there are in an actual game. Thus they might not engage their full conscious powers while performing the tasks. And because of that an extraneous task might not degrade their performance. However, in a high stakes situation, this might not be the case. Moreover, it is interesting to note that some athletes claim that the reason they take the drug Adderall—a drug often given to those with ADD—is because it increases focus and eliminates distractions.56 Thus, there seems to be ample room to doubt the view that experts tend to perform better or even just as well when distracted by skill-unrelated supplemental tasks.
However, that the varied-focus experiments fail to capture real-life settings cannot be the entire explanation for why they appear to show that certain forms of expert action are harmed by self-monitoring, since both the less and more experienced participants perform ecologically invalid tasks, yet ability is differentially affected. Why is this, if not for the reason, as Beilock et al. (2002) conclude, that “well-learned performance may actually be compromised by attending to skill execution” (p. 9)?