collective intelligence

Wisdom of crowds did not work. Is it something to do with software engineers?

Posted on Updated on

Quite a few experiments, like counting the number of coins in a jar & guessing the weight of the cake have been conducted to test out the ‘wisdom of crowds’ concept. Recently I got a chance to conduct an experiment at a gathering of approximately 150 people — all software/computer professionals

We put up a jar filled with éclairs and asked the audience to guess the number of éclairs in the jar and note it down on a piece of paper and hand it over to us. Obviously, we were excited to know the results – if the experiment worked, did someone guess the correct number?

Here are the results:

We had responses ranging from 25 éclairs to 786 éclairs. The actual number of éclairs in the jar was 204. Out of the 134 responses, only 1 person got the number right. The average, the ‘collective wisdom’ of the ‘crowd’ was 161.4 – an accuracy % of 79.14.

The results have left me wondering as to why did the results not concur with the ‘wisdom of crowds’ theory. After all, the following elements of ‘wisdom of crowds’ were present in the experiment:

  • There was enough diversity among the participants. even though all the participants were software engineers, they came from different project teams, had varying levels of experience, roles etc.
  • Each person was making his/her own independent choice and was not influenced by any other member
  • There was means for us to collect and aggregate the responses.

The fourth element ‘Decentralization — people being able to specialize and draw on local knowledge’ can be debated to be missing from the experiment. How can we assume that there were people specializing in guessing the number of éclairs in a jar? How could they draw on local knowledge?

In Galton’s experiments, there were people who were butchers and farmers who one can assume to be specializing in ox/cattle. In ‘Guess the weight of the Cake’ experiment, people could feel the cake in their hands, there were people who were traders and hence one can assume that ‘knowing the weight of an object’ was something that they were familiar with.

The ‘guess the number of coins’ experiment was an online experiment and had no option for people to see, to feel the jar, hold it in their hands. Similar aspects were present in the experiment we conducted – we just showed people the jar filled with éclairs. We did not have people holding the jar, touching it, feeling it. Could such seemingly insignificant things allow people to make more informed choice? I don’t know.

BTW, ‘Guess the weight of the cake’ experiment was 99% successful even though it had just 120 participants whereas ‘guess the number of coins’ experiment was just 88% successful even though it had 1760 participants. This indicates that while the sample size could be a factor, but it does not seem so in the above examples.

I am wondering if aspects of ‘cognition, cooperation, coordination’ are present in such experiments?

What about the sample composition itself – all software/computer professionals? Do they bring diversity or are software professionals just not the right sample for any kind of experiment?

Book Review: “The Wisdom of Crowds” by “James Surowiecki”.

Posted on Updated on

Traditional belief, particularly in IT industry, suggests that if you place put together a group of outstanding persons/experts along with a mix of average persons and some below average persons and ask them to work on a problem then what you will most probably get is sub- optimal results.

This book turns the above notion upside down and suggests that certain types of problems and decision making are best solved by groups (the larger the better) of reasonably informed and engaged people. The group’s answer, he shows, is almost invariably better than any expert’s answer, even better than the best answer of the experts in the group.

The author also places great emphasis of what constitutes an ideal group:

Intellectual diversity: Different opinions and perspectives (unlike most management teams and boards, who tend to select others who think the same way they do.). The author suggests having a physicist in a group which is trying to solve a chemistry problem.

Independence: Freedom from the tendency to want to agree automatically with what one or more other group members says, and

Decentralization with Aggregation: Individual access to different, specialized knowledge, and a mechanism for effectively sharing that knowledge with the rest of the group.

The book begins with a taxonomy of three types of problems that individuals and groups try to solve:

Cognition problems: Problems with one definitive answer that we try to accurately assess after considering available and missing information (e.g. what’s a stock worth, who will win an election, or what caused a disaster).

Coordination problems: Where an optimal combined solution is needed for a problem that affects a whole group, and where this optimal solution is usually sought by having each individual act in personal self-interest (e.g. finding buyers and sellers for products, or determining the best route to work in traffic), and

Cooperation problems: Where an optimal combined solution is needed for a problem that affects a whole group, and where this optimal solution usually depends on individuals trusting each other and acting fairly and in what they perceive to be collective self-interest rather than just their own (e.g. how to deal with pollution, devise a tax system, or remunerate employees).

The author provides numerous examples and observations from our daily lives, corporate settings, social examples to make this book interesting. He also points out that answers/solutions coming out from group are not exactly “average” or “consensus” answers. The author terms the groups answers to be a “collective” answer with the superiority of the collective answer depending importantly on the group’s members having ‘Intellectual diversity’, ‘Independence’ and ‘Decentralization with Aggregation’

However what the author does not do is provide how his findings can be applied. I think that is left to the reader to figure out.

All in all it is an excellent intellectually stimulating book that conveys it’s message through simple observations and findings.