Monday, October 1, 2007

Chase and Simon (1973) Perception in chess, Cognitive Psychology 4:55-81. A scientific blunder.

Blogging on Peer-Reviewed Research

Here´s an email I´ve sent some months ago to a number of very bright people.

The 1000 dollars offer holds until the end of this year.

Imagine if two famous biologists published a study, over 30 years ago, with two parts: in the first part, they unequivocally showed that sharks and dolphins had a strikingly different nature. In the second part, however, they tried to explain that difference by looking at the habitats of a dolphin and the habitat of a shark (i.e., the same data). Imagine that that paper would be cited by hundreds of people, for decades.

Now imagine that Chase and Simon, writing a study entitled "Perception in chess", in Cognitive Psychology 4, p.55—81 (1973), divided it into two parts. The first part (p.55—61) of the paper showed that when chess masters looked at a board for 5 seconds, they could reproduce it with enormous accuracy, while beginners could not reproduce it for more than a few pieces. This difference could not be explained by masters' greater memory, for, in randomized positions, the effect disappeared, with masters and beginners able to reproduce only a few pieces of the board. Sharks and Dolphins, it was clear, were different.

Now, what was the nature of the chunks like? The second part of the paper devised two tasks, a 'perception task', and a 'memory task'. These tasks looked at masters and beginners 'interpiece interval times' (within glances at the board, and in between glances) in reconstructing the boards. The results were unequivocal: the data was exactly the same for masters and beginners (figs 3 and 4). They pointed this out clearly:

[Perception task, p.65] "The first thing to notice is that the data are quite similar for all subjects. The latencies show the same systematic trends, and, for the probabilities, the product moment correlation between subjects are quite high: Master vs Class A=.93; Master vs Class B=.95, and Class A vs Class B =.92. The same is true for the between glance data… Thus, the same kinds and degrees of relatedness between successive pieces holds for subjects of very different skills."

[Memory task] "Again the pattern of latencies and probabilities look the same for all subjects, and the correlations are about the same as in the perception of data: Master vs Class A=.91, Master vs. Class B=.95, and Class A vs. Class B=.95".

The obvious conclusion is, of course, that whatever difference exists between Masters and Class B players, it cannot be obtained from this dataset. Nothing about the "nature of the chess chunk" can ever be obtained here.

Yet, with that dataset at hand, the authors proceeded to study the nature of the chess chunk: "These probabilities are informative about the underlying structures that the subjects are perceiving" (p. 68). How can they be, if a Master subject perceives the global meaning of the position, and a Class B perceives nothing?

"Our data gives us an operational method of characterizing chunks, which we will apply to the middle-game memory experiments of subject M[=Master]" (p.78). One wonders: why bother? Send the master home. They could gather all they needed a from a Class B subject, or from a yanomami, after that non sequitur.

Chase and Simon 1973 explained the difference between sharks and dolphins by looking at their habitats, and the whole world bought it. At the risk of running into utter humiliation, I will paypal one thousand dollars to the first person on this list that proves me wrong. The deadline for your thousand shiny dollars is 24h before the deadline for submission to cogsci in Nashville, when I will go on and commit scientific suicide.

Any takers?


bstp said...

Simple and brilliant !

PS : Have you ever thought of proposing the bet to Ferdinand Gobet ?

Best regards,

Benoit St-Pierre

Alexandre Linhares said...

Well, Dr Gobet has suggested rejection of all my papers in chess, so I think my ideas--while having quite a large overlap with his work--won't be too appreciated by him.