Wednesday, October 13, 2010

Inference From Confirming Evidence

Every once in a while I read papers in Archaeology claiming about there allegedly scientific methods failing to choose the right type of hypothesis testing for their models. While reading Taleb's "The Black Swan" I come across a very nice  psychological experiment conducted 50 years ago by Wason (1960).
The basic idea is that you have a data-set with a specific pattern, and in order to explain the underlying rule you conduct a series of experiments to test and propose a model. Of course in archaeology you cannot strictly do experiments all the time, but you can look for other data which will support or not support your model.
Now the experiment is based on a simple sequence of number, and one should simply "discover" the underlying rule. The player can propose another sequence of numbers and the experimenter will tell you whether such sequence can been generated from the same algorithm of the original data-set or not.

The sequence is:

2-4-6

Now, most people will most likely propose something like following sequence:

8-10-12

which will basically test the model increase by two. Now if the experimenter will tell you "Yes", you'll be probably quite happy about that, and probably you will write a paper for the Journal of Integer Sequences, which believe it or not, actually exists) with something lines:
"Our experiment confirmed our hypothesis of the increase by two rule". Very few will claim that this model is wrong and will be part of the scientific knowledge of your field (I'm sure that none of the editors of Journal of Integer Sequences will accept your paper, nor will fall on this trap...)
Now the big problem here is that the algorithm with which 2-4-6 has been generated was numbers in ascending order. Thus actually 5-6-7 would also have been accepted by the experimenter. However, it's vert likely that most people uses these experiments to confirm their model rather trying to falsify it. Testing 5-6-7 would have allowed a re-evaluation of the originally proposed rule and might have lead to the right answer.
The question now is: what sequence of number are you proposing in your archaeological research?


References:

  • Taleb, N.N.,2007, The Black Swan: The Impact of the Highly Improbable, Random House, 
  • Wason, P.C., 1960, On the failure to eliminate hypotheses in a conceptual task, The Quarterly Journal of Experimental Psychology, 12: 3, 129-140.