Quite a few experiments, like counting the number of coins in a jar & guessing the weight of the cake have been conducted to test out the ‘wisdom of crowds’ concept. Recently I got a chance to conduct an experiment at a gathering of approximately 150 people — all software/computer professionals
We put up a jar filled with éclairs and asked the audience to guess the number of éclairs in the jar and note it down on a piece of paper and hand it over to us. Obviously, we were excited to know the results – if the experiment worked, did someone guess the correct number?
Here are the results:
We had responses ranging from 25 éclairs to 786 éclairs. The actual number of éclairs in the jar was 204. Out of the 134 responses, only 1 person got the number right. The average, the ‘collective wisdom’ of the ‘crowd’ was 161.4 – an accuracy % of 79.14.
The results have left me wondering as to why did the results not concur with the ‘wisdom of crowds’ theory. After all, the following elements of ‘wisdom of crowds’ were present in the experiment:
- There was enough diversity among the participants. even though all the participants were software engineers, they came from different project teams, had varying levels of experience, roles etc.
- Each person was making his/her own independent choice and was not influenced by any other member
- There was means for us to collect and aggregate the responses.
The fourth element ‘Decentralization — people being able to specialize and draw on local knowledge’ can be debated to be missing from the experiment. How can we assume that there were people specializing in guessing the number of éclairs in a jar? How could they draw on local knowledge?
In Galton’s experiments, there were people who were butchers and farmers who one can assume to be specializing in ox/cattle. In ‘Guess the weight of the Cake’ experiment, people could feel the cake in their hands, there were people who were traders and hence one can assume that ‘knowing the weight of an object’ was something that they were familiar with.
The ‘guess the number of coins’ experiment was an online experiment and had no option for people to see, to feel the jar, hold it in their hands. Similar aspects were present in the experiment we conducted – we just showed people the jar filled with éclairs. We did not have people holding the jar, touching it, feeling it. Could such seemingly insignificant things allow people to make more informed choice? I don’t know.
BTW, ‘Guess the weight of the cake’ experiment was 99% successful even though it had just 120 participants whereas ‘guess the number of coins’ experiment was just 88% successful even though it had 1760 participants. This indicates that while the sample size could be a factor, but it does not seem so in the above examples.
I am wondering if aspects of ‘cognition, cooperation, coordination’ are present in such experiments?
What about the sample composition itself – all software/computer professionals? Do they bring diversity or are software professionals just not the right sample for any kind of experiment?