Surprisingly popular

The surprisingly popular answer is a wisdom of the crowd technique that taps into the expert minority opinion within a crowd. For a given question, a group is asked two questions:

• What is the probability that this answer is correct? (Which answers are most likely to be right?)
• What is the average probability others will give to this answer? (Which answers will be most popular?)

The answer that maximizes the average difference between the "right" and "popular" answers is the "surprisingly popular" answer. The term "surprisingly popular" was coined in a 2017 paper published in Nature entitled "A solution to the single-question crowd wisdom problem", which outlined the technique.

Algorithm

Suppose we'd like to determine the answer to the question "Is Philadelphia the capital of Pennsylvania?" The two questions asked of the group, and the average responses, are:

Is Philadelphia the capital of Pennsylvania? ("Right" question)

• Yes: 65% (average probability)
• No: 35% (average probability)

What is the average probability people will assign to "Philadelphia is the capital of Pennsylvania"? ("Popular" question)

• Yes: 75%
• No: 25%

The difference between the answers to the right question and the popular question:

• Yes: 65% − 75% = −10%
• No: 35% − 25% = 10%

Thus, the No answer is surprisingly popular (10% > −10%). (The capital is not Philadelphia, but Harrisburg.)

Explanation

The technique avoids the "double-counting" of prior probabilities across participants, a major issue for belief aggregation rules under the naive assumption that participants' answers are independent. Say a crowd has two groups:

• Experts, who have some valuable piece of evidence which is not common knowledge. They combine this evidence with their prior probability (coming from common knowledge) to get an improved posterior probability.
• Non-experts only have common knowledge to go off of, and therefore provide only the prior probability.

When asked to answer a question, non-experts will tend to give equal answers to both questions. This is because they have no reason to expect they are wrong in either direction—their answer is just as likely to be an overestimate as it is an underestimate. (If the participants expected to change their probability estimates after learning more information, they already would have.)

However, the experts have access to both the prior probability and the posterior probability, which allows them to make a better estimate of the group's opinion. Because they know the group contains both experts and non-experts, they will expect the average probability to be in between the prior and the posterior. This means that, unlike the non-experts, their answers will not tend to cancel out when the prior probability (as proxied by the "popular answer") is subtracted out.

Looking again at the capital example, say there are two groups, experts and non-experts:

• Experts – "Philadelphia is/is not the capital, but most others won't know that."
  • This group thinks they have unknown information about whether Philadelphia is likely to be the capital. (They likely know Harrisburg is the capital.)
  • This group thinks the probability that Philadelphia is the capital is low, but that not everybody will realize this.
  • Therefore, the group will tend to assume others assign a "bad" (high) probability to Philadelphia being the capital.
• Non-experts – "Philadelphia is/is not the capital, and others will agree."
  • This group is answering based on common knowledge.
  • This group has no reason to think the average probability that Philadelphia is the capital will be different from their own estimate.
    • Thus, their estimate for the popularity of Philadelphia is roughly equal to their estimate for the probability that Philadelphia is the capital.
    • This means that when subtracting the two probabilities, the group's contributions to the overall probability cancel out.

The strength of the method is that it causes the two non-expert groups to cancel out, thus identifying the opinions of the expert group. (It is assumed that most people who think they have "inside" knowledge are correct and knowledgeable, rather than misled.)

For rankings

For m>2 candidates, the Surprisingly Popular Algorithm requires votes from an infinite number of voters on all possible ranked permutations (m!) of the alternatives to recover the ground-truth ranking with complete certainty, as discussed in the Nature article. However, the algorithm can be extended to recover rankings using various elicitation formats.

See also

• Keynesian beauty contest
• Guess 2/3 of the average
• Family Feud

Further reading

• Prelec, Dražen; Seung, H. Sebastian; McCoy, John (25 January 2017). "A solution to the single-question crowd wisdom problem". Nature. 541 (7638): 532–535. Bibcode:2017Natur.541..532P. doi:10.1038/nature21054. PMID 28128245. S2CID 4452604.