imagine you’re at the greengrocers and there you place a huge bunch of grapes
there you place a huge bunch of grapes
there you place a huge bunch of grapes on the scale and get a nasty surprise
on the scale and get a nasty surprise
on the scale and get a nasty surprise because the numbers shown on the scale
because the numbers shown on the scale
because the numbers shown on the scale is not what you expect but you you need
is not what you expect but you you need
is not what you expect but you you need the grapes but you would hate to pay too
the grapes but you would hate to pay too
the grapes but you would hate to pay too much and so you think to yourself never
much and so you think to yourself never
much and so you think to yourself never mind but then you notice the greengrocer
mind but then you notice the greengrocer
mind but then you notice the greengrocer look in your direction and he’s looking
look in your direction and he’s looking
look in your direction and he’s looking too intensely at you you think and to
too intensely at you you think and to
too intensely at you you think and to make make matters worse he’s holding a
make make matters worse he’s holding a
make make matters worse he’s holding a large wad of cash in his hand which by
large wad of cash in his hand which by
large wad of cash in his hand which by now you’re thinking he’s probably
now you’re thinking he’s probably
now you’re thinking he’s probably appropriated from the poor fools who
appropriated from the poor fools who
appropriated from the poor fools who stood there before you but little does a
stood there before you but little does a
stood there before you but little does a green grocer know that you are no fool
green grocer know that you are no fool
green grocer know that you are no fool you know something about collective
you know something about collective
you know something about collective intelligence and now you plan to conjure
intelligence and now you plan to conjure
intelligence and now you plan to conjure up the wisdom of the crowds right there
up the wisdom of the crowds right there
up the wisdom of the crowds right there in the green growth grocery store
in the green growth grocery store
in the green growth grocery store because you’ve watched all against one
because you’ve watched all against one
because you’ve watched all against one the block posted Danish gameshow on
the block posted Danish gameshow on
the block posted Danish gameshow on based on wisdom of the crowds and you’ve
based on wisdom of the crowds and you’ve
based on wisdom of the crowds and you’ve seen just how easily the average opinion
seen just how easily the average opinion
seen just how easily the average opinion of ordinary viewers competed with the
of ordinary viewers competed with the
of ordinary viewers competed with the particularly bright participants in the
particularly bright participants in the
particularly bright participants in the studio so now you long to approach the
studio so now you long to approach the
studio so now you long to approach the greedy green green greedy green grocer
greedy green green greedy green grocer
greedy green green greedy green grocer with accurate information and tell me
with accurate information and tell me
with accurate information and tell me about the error of his ways and a scale
about the error of his ways and a scale
about the error of his ways and a scale so you begin to ask your fellow shoppers
so you begin to ask your fellow shoppers
so you begin to ask your fellow shoppers what they think about this bunch of
what they think about this bunch of
what they think about this bunch of graves so that you can calculate the
graves so that you can calculate the
graves so that you can calculate the average opinion and after many awkward
average opinion and after many awkward
average opinion and after many awkward moments you have your answer
moments you have your answer
moments you have your answer 875 grams 25 grams short of the 900
875 grams 25 grams short of the 900
875 grams 25 grams short of the 900 shown on the scale now admittedly this
shown on the scale now admittedly this
shown on the scale now admittedly this 25 gram deviation is a lot less than you
25 gram deviation is a lot less than you
25 gram deviation is a lot less than you thought but cheating is cheating you
thought but cheating is cheating you
thought but cheating is cheating you think and triumphantly you make your way
think and triumphantly you make your way
think and triumphantly you make your way to the counter the green grocer has been
to the counter the green grocer has been
to the counter the green grocer has been expecting you which obviously bugs you
expecting you which obviously bugs you
expecting you which obviously bugs you bought what really troubles you is what
bought what really troubles you is what
bought what really troubles you is what you instantly see behind his cancer it’s
you instantly see behind his cancer it’s
you instantly see behind his cancer it’s a large pile of literature on collective
a large pile of literature on collective
a large pile of literature on collective intelligence and it even includes
intelligence and it even includes
intelligence and it even includes similar works on the wisdom of crowds
similar works on the wisdom of crowds
similar works on the wisdom of crowds and then the Green Grove spree
and then the Green Grove spree
and then the Green Grove spree I see what you’ve been doing he says but
I see what you’ve been doing he says but
I see what you’ve been doing he says but there’s something that you should know
there’s something that you should know
there’s something that you should know while your strategy is remarkably
while your strategy is remarkably
while your strategy is remarkably powerful under normal circumstances it
powerful under normal circumstances it
powerful under normal circumstances it and not my scale is wrong here because
and not my scale is wrong here because
and not my scale is wrong here because you’re dealing with an item far from our
you’re dealing with an item far from our
you’re dealing with an item far from our typical experience and then it begins to
typical experience and then it begins to
typical experience and then it begins to lecture now we have long known that the
lecture now we have long known that the
lecture now we have long known that the average opinion of many people provides
average opinion of many people provides
average opinion of many people provides excellent information about the truth
excellent information about the truth
excellent information about the truth but there’s no reason to believe that
but there’s no reason to believe that
but there’s no reason to believe that the average opinion errors for reasons
the average opinion errors for reasons
the average opinion errors for reasons of psychology when the magnitude being
of psychology when the magnitude being
of psychology when the magnitude being presented is extreme in some way now
presented is extreme in some way now
presented is extreme in some way now what you have to do he almost whispers
what you have to do he almost whispers
what you have to do he almost whispers is look at the entire distribution of
is look at the entire distribution of
is look at the entire distribution of judgments and not just the central
judgments and not just the central
judgments and not just the central tendencies in particular you have to
tendencies in particular you have to
tendencies in particular you have to look at the way judgments skew or slant
look at the way judgments skew or slant
look at the way judgments skew or slant because that’s information that you can
because that’s information that you can
because that’s information that you can use to infer when the average opinion is
use to infer when the average opinion is
use to infer when the average opinion is erroring you could say concludes the
erroring you could say concludes the
erroring you could say concludes the knowledgeable greengrocer that skew is a
knowledgeable greengrocer that skew is a
knowledgeable greengrocer that skew is a cue for our collective error and with
cue for our collective error and with
cue for our collective error and with that you can do even better perhaps even
that you can do even better perhaps even
that you can do even better perhaps even as well as my mechanical scale now the
as well as my mechanical scale now the
as well as my mechanical scale now the greengrocer is correct and that’s
greengrocer is correct and that’s
greengrocer is correct and that’s important for all the reasons why M good
important for all the reasons why M good
important for all the reasons why M good decisions and good judgments matter in
decisions and good judgments matter in
decisions and good judgments matter in our lives but let me explain why it’s
our lives but let me explain why it’s
our lives but let me explain why it’s correct and turn to the question of
correct and turn to the question of
correct and turn to the question of significance as I conclude my talk and
significance as I conclude my talk and
significance as I conclude my talk and along the way I’ll invoke three things
along the way I’ll invoke three things
along the way I’ll invoke three things an old probability machine and two
an old probability machine and two
an old probability machine and two psychological theories now psychologists
psychological theories now psychologists
psychological theories now psychologists have long established that when we face
have long established that when we face
have long established that when we face the same magnitude many times our
the same magnitude many times our
the same magnitude many times our judgments is subject to variation it’s
judgments is subject to variation it’s
judgments is subject to variation it’s kind of like when you go home from work
kind of like when you go home from work
kind of like when you go home from work and it did 30 minutes it actually takes
and it did 30 minutes it actually takes
and it did 30 minutes it actually takes sometimes feels longer and sometimes
sometimes feels longer and sometimes
sometimes feels longer and sometimes feels shorter and the probability
feels shorter and the probability
feels shorter and the probability machine I’m invoking does an excellent
machine I’m invoking does an excellent
machine I’m invoking does an excellent job in capturing this phenomenon it was
job in capturing this phenomenon it was
job in capturing this phenomenon it was invented and built by the half cousin of
invented and built by the half cousin of
invented and built by the half cousin of Charles Darwin Sir Francis Galton
Charles Darwin Sir Francis Galton
Charles Darwin Sir Francis Galton in 1873 their golden coal this machine
in 1873 their golden coal this machine
in 1873 their golden coal this machine the Queen comes but we’ll just call it
the Queen comes but we’ll just call it
the Queen comes but we’ll just call it the golden board it consists of displays
the golden board it consists of displays
the golden board it consists of displays rows of pins at the top and equally
rows of pins at the top and equally
rows of pins at the top and equally spaced their compartments at the bottom
spaced their compartments at the bottom
spaced their compartments at the bottom that serve the purpose of capturing
that serve the purpose of capturing
that serve the purpose of capturing balls have dropped into and dropped
balls have dropped into and dropped
balls have dropped into and dropped through the machine now you may think
through the machine now you may think
through the machine now you may think about the individual path of
about the individual path of
about the individual path of such a ball as a line of thought in the
such a ball as a line of thought in the
such a ball as a line of thought in the cognitive process and you may think
cognitive process and you may think
cognitive process and you may think about the particular compartment that a
about the particular compartment that a
about the particular compartment that a ball ultimately settles into as an
ball ultimately settles into as an
ball ultimately settles into as an estimate of magnitude in our case an
estimate of magnitude in our case an
estimate of magnitude in our case an estimate of weight now along the way
estimate of weight now along the way
estimate of weight now along the way this thought process is disrupted by
this thought process is disrupted by
this thought process is disrupted by random events as captured by bolts at
random events as captured by bolts at
random events as captured by bolts at head pins in the otherwise straight fall
head pins in the otherwise straight fall
head pins in the otherwise straight fall and where exactly this ball will
and where exactly this ball will
and where exactly this ball will ultimately end up and we cannot actually
ultimately end up and we cannot actually
ultimately end up and we cannot actually predict for certain but we can say
predict for certain but we can say
predict for certain but we can say something about the pattern that emerges
something about the pattern that emerges
something about the pattern that emerges when we drop many balls into the system
when we drop many balls into the system
when we drop many balls into the system one after the other and that pattern you
one after the other and that pattern you
one after the other and that pattern you might know it’s a famous bell curve but
might know it’s a famous bell curve but
might know it’s a famous bell curve but the bell curve is symmetric and skewed
the bell curve is symmetric and skewed
the bell curve is symmetric and skewed judgment distributions occur much more
judgment distributions occur much more
judgment distributions occur much more frequently than predicted by the golden
frequently than predicted by the golden
frequently than predicted by the golden board and so we must change it in some
board and so we must change it in some
board and so we must change it in some way in order to learn why that is and
way in order to learn why that is and
way in order to learn why that is and the first change that we make to the
the first change that we make to the
the first change that we make to the golden board is to combine that with
golden board is to combine that with
golden board is to combine that with what’s called adaptation level theory
what’s called adaptation level theory
what’s called adaptation level theory according to which we become used to the
according to which we become used to the
according to which we become used to the typical magnitudes of our experience and
typical magnitudes of our experience and
typical magnitudes of our experience and presume these magnitudes as we then
presume these magnitudes as we then
presume these magnitudes as we then confront new items in our case we become
confront new items in our case we become
confront new items in our case we become used to the typical weight of a bunch of
used to the typical weight of a bunch of
used to the typical weight of a bunch of grapes from our shopping experience in
grapes from our shopping experience in
grapes from our shopping experience in the setting of the the Queen conquers
the setting of the the Queen conquers
the setting of the the Queen conquers are the golden board this implies that
are the golden board this implies that
are the golden board this implies that the ball begins its journey and starting
the ball begins its journey and starting
the ball begins its journey and starting at a magnitude that is typical in our
at a magnitude that is typical in our
at a magnitude that is typical in our case let’s assume that’s 500 grams but
case let’s assume that’s 500 grams but
case let’s assume that’s 500 grams but of course it’s often important
of course it’s often important
of course it’s often important differences between the particular and
differences between the particular and
differences between the particular and the typical and the second change that
the typical and the second change that
the typical and the second change that we make to the golden board is to assume
we make to the golden board is to assume
we make to the golden board is to assume that people will try to account for
that people will try to account for
that people will try to account for these differences and do so according to
these differences and do so according to
these differences and do so according to what’s called sequential sampling theory
what’s called sequential sampling theory
what’s called sequential sampling theory now the basic idea here is that what we
now the basic idea here is that what we
now the basic idea here is that what we judge has features and these features
judge has features and these features
judge has features and these features contain information and this information
contain information and this information
contain information and this information is consistent with one of two hypotheses
is consistent with one of two hypotheses
is consistent with one of two hypotheses that relate to the typical value in our
that relate to the typical value in our
that relate to the typical value in our case heavier than typical or lighter
case heavier than typical or lighter
case heavier than typical or lighter than typical for example a bunch of
than typical for example a bunch of
than typical for example a bunch of grapes can have particularly plump
grapes can have particularly plump
grapes can have particularly plump fruits which would tends to suggest that
fruits which would tends to suggest that
fruits which would tends to suggest that we’re dealing with an item that’s
we’re dealing with an item that’s
we’re dealing with an item that’s heavier than typical but that same bunch
heavier than typical but that same bunch
heavier than typical but that same bunch of grapes can have particularly sparse
of grapes can have particularly sparse
of grapes can have particularly sparse fruits in places which would suggest
fruits in places which would suggest
fruits in places which would suggest suggests
suggests
suggests opposite now the these features are now
opposite now the these features are now
opposite now the these features are now captured by the rows of pins in the
captured by the rows of pins in the
captured by the rows of pins in the golden board with the probability of a
golden board with the probability of a
golden board with the probability of a ball moving in the same direction as the
ball moving in the same direction as the
ball moving in the same direction as the evidence provided by the particular
evidence provided by the particular
evidence provided by the particular feature capturing the expertise or the
feature capturing the expertise or the
feature capturing the expertise or the adeptness of the judge at the task for
adeptness of the judge at the task for
adeptness of the judge at the task for example a wine producer will be much
example a wine producer will be much
example a wine producer will be much more adept at extracting useful
more adept at extracting useful
more adept at extracting useful information from a bunch of grapes then
information from a bunch of grapes then
information from a bunch of grapes then will your typical shoppers so it’s by
will your typical shoppers so it’s by
will your typical shoppers so it’s by combining the gulten board with
combining the gulten board with
combining the gulten board with adaptation level theory and sequential
adaptation level theory and sequential
adaptation level theory and sequential sampling theory that we finally reach M
sampling theory that we finally reach M
sampling theory that we finally reach M the Augmented Queen Kong’s or the aq for
the Augmented Queen Kong’s or the aq for
the Augmented Queen Kong’s or the aq for short and the aq can tell us why skew is
short and the aq can tell us why skew is
short and the aq can tell us why skew is a cue for our collective error the aq
a cue for our collective error the aq
a cue for our collective error the aq predicts then when the percent in
predicts then when the percent in
predicts then when the percent in magnitude is typical than for the people
magnitude is typical than for the people
magnitude is typical than for the people in the crowd then these people form
in the crowd then these people form
in the crowd then these people form their average opinion of these people
their average opinion of these people
their average opinion of these people will be perfectly aligned with the truth
will be perfectly aligned with the truth
will be perfectly aligned with the truth irrespective of how they use information
irrespective of how they use information
irrespective of how they use information moreover the judgment distribution say
moreover the judgment distribution say
moreover the judgment distribution say together form will be symmetric under
together form will be symmetric under
together form will be symmetric under these circumstances the wisdom of the
these circumstances the wisdom of the
these circumstances the wisdom of the crowds is almost magic but when the
crowds is almost magic but when the
crowds is almost magic but when the percent of magnitude is in some way
percent of magnitude is in some way
percent of magnitude is in some way extreme then how we use information
extreme then how we use information
extreme then how we use information matters for both collective error and
matters for both collective error and
matters for both collective error and for skewness in actual fact it’s when we
for skewness in actual fact it’s when we
for skewness in actual fact it’s when we have some level of expertise that skewed
have some level of expertise that skewed
have some level of expertise that skewed judgment distributions are predicted to
judgment distributions are predicted to
judgment distributions are predicted to occur in the first place it’s almost
occur in the first place it’s almost
occur in the first place it’s almost like a swarm of birds where the majority
like a swarm of birds where the majority
like a swarm of birds where the majority head towards one clear direction but a
head towards one clear direction but a
head towards one clear direction but a few laggards trailed behind leaving the
few laggards trailed behind leaving the
few laggards trailed behind leaving the impression of a long tail in the
impression of a long tail in the
impression of a long tail in the distribution so for example when the
distribution so for example when the
distribution so for example when the presented magnitude is smaller than
presented magnitude is smaller than
presented magnitude is smaller than typical then these swarm not of birds
typical then these swarm not of birds
typical then these swarm not of birds but of judgments head towards small
but of judgments head towards small
but of judgments head towards small values whereas it heads towards large
values whereas it heads towards large
values whereas it heads towards large values when the percent of magnitude is
values when the percent of magnitude is
values when the percent of magnitude is is larger as in the case of the
is larger as in the case of the
is larger as in the case of the greengrocers store and you might think
greengrocers store and you might think
greengrocers store and you might think that well that’s all well and good but
that well that’s all well and good but
that well that’s all well and good but in both cases the average is dragged
in both cases the average is dragged
in both cases the average is dragged down by the tail end of this
down by the tail end of this
down by the tail end of this distribution in an unfortunate way and
distribution in an unfortunate way and
distribution in an unfortunate way and because skewness well and that brings me
because skewness well and that brings me
because skewness well and that brings me to the final most important point of my
to the final most important point of my
to the final most important point of my talk namely that the deviation between
talk namely that the deviation between
talk namely that the deviation between truth and the average opinion
truth and the average opinion
truth and the average opinion systematically varies with this
systematically varies with this
systematically varies with this observable skewness and that’s because
observable skewness and that’s because
observable skewness and that’s because it’s observable that we can simply say
it’s observable that we can simply say
it’s observable that we can simply say that skew is a cue for our collective
that skew is a cue for our collective
that skew is a cue for our collective error now you might think measuring the
error now you might think measuring the
error now you might think measuring the weight of a bunch of grapes is using the
weight of a bunch of grapes is using the
weight of a bunch of grapes is using the collective minds of shoppers is a simple
collective minds of shoppers is a simple
collective minds of shoppers is a simple and perhaps impractical example but
and perhaps impractical example but
and perhaps impractical example but looking more broadly using modern
looking more broadly using modern
looking more broadly using modern information technology we can now hope
information technology we can now hope
information technology we can now hope to harness the wisdom of crowds much
to harness the wisdom of crowds much
to harness the wisdom of crowds much more efficiently than we have previously
more efficiently than we have previously
more efficiently than we have previously done now that we know that skew is a cue
done now that we know that skew is a cue
done now that we know that skew is a cue for error
for error
for error indeed doctors managers and even
indeed doctors managers and even
indeed doctors managers and even politicians who care about the facts who
politicians who care about the facts who
politicians who care about the facts who do look do well to look at the entire
do look do well to look at the entire
do look do well to look at the entire distribution judgments because they
distribution judgments because they
distribution judgments because they might just save their patients and them
might just save their patients and them
might just save their patients and them a higher profit or increase the welfare
a higher profit or increase the welfare
a higher profit or increase the welfare of citizens by bringing these important
of citizens by bringing these important
of citizens by bringing these important decision-makers closer to the truth so
decision-makers closer to the truth so
decision-makers closer to the truth so it doesn’t matter if you are buying a
it doesn’t matter if you are buying a
it doesn’t matter if you are buying a bunch of grave so if you’re running a
bunch of grave so if you’re running a
bunch of grave so if you’re running a country if you want to know the truth
country if you want to know the truth
country if you want to know the truth then skew is your cue
then skew is your cue
then skew is your cue [Applause]
[Applause]
[Applause] [Music]
”
Be First to Comment