[Applause] thank you
thank you
thank you I’m collided to be speaking here today
I’m collided to be speaking here today
I’m collided to be speaking here today I’m going to talk about election polling
I’m going to talk about election polling
I’m going to talk about election polling in the 2016 election and my first
in the 2016 election and my first
in the 2016 election and my first thought about this is well what extent
thought about this is well what extent
thought about this is well what extent you you all want to hear about the 2016
you you all want to hear about the 2016
you you all want to hear about the 2016 election again an opinion polling and
election again an opinion polling and
election again an opinion polling and all the missed predictions and so forth
all the missed predictions and so forth
all the missed predictions and so forth on the other hand Donald Trump seems to
on the other hand Donald Trump seems to
on the other hand Donald Trump seems to like to talk about this continually even
like to talk about this continually even
like to talk about this continually even though the election has has long past so
though the election has has long past so
though the election has has long past so what happened in 2016 2016 arguably was
what happened in 2016 2016 arguably was
what happened in 2016 2016 arguably was the biggest debacle of predictions of
the biggest debacle of predictions of
the biggest debacle of predictions of elections of all time in the United
elections of all time in the United
elections of all time in the United States it harkens back to the 1948
States it harkens back to the 1948
States it harkens back to the 1948 election where there was this famous
election where there was this famous
election where there was this famous photo which you’ll see further in a
photo which you’ll see further in a
photo which you’ll see further in a second of the Chicago Tribune reporting
second of the Chicago Tribune reporting
second of the Chicago Tribune reporting that Dewey defeats Truman that is
that Dewey defeats Truman that is
that Dewey defeats Truman that is Republican Thomas Dewey defeated Harry
Republican Thomas Dewey defeated Harry
Republican Thomas Dewey defeated Harry Truman in the 1948 election based on
Truman in the 1948 election based on
Truman in the 1948 election based on pre-election polling and predictions and
pre-election polling and predictions and
pre-election polling and predictions and it’s useful to think back historically
it’s useful to think back historically
it’s useful to think back historically on this in particular I think the best
on this in particular I think the best
on this in particular I think the best analysts of this is the legendary
analysts of this is the legendary
analysts of this is the legendary baseball player and language master Yogi
baseball player and language master Yogi
baseball player and language master Yogi Berra I think he had it right and the
Berra I think he had it right and the
Berra I think he had it right and the terms that his phrases that apply here
terms that his phrases that apply here
terms that his phrases that apply here or in 2016 compared to 1948 was deja vu
or in 2016 compared to 1948 was deja vu
or in 2016 compared to 1948 was deja vu all over again but the similarities are
all over again but the similarities are
all over again but the similarities are different and of course the old
different and of course the old
different and of course the old all-encompassing fallback is hard to
all-encompassing fallback is hard to
all-encompassing fallback is hard to predict especially about the future now
predict especially about the future now
predict especially about the future now to put this in broader historical
to put this in broader historical
to put this in broader historical context I want to go back to 1936 which
context I want to go back to 1936 which
context I want to go back to 1936 which before 1948 was a polling debacle of a
before 1948 was a polling debacle of a
before 1948 was a polling debacle of a particular sort that’s worth reflecting
particular sort that’s worth reflecting
particular sort that’s worth reflecting further on from 1916 to 1936 the
further on from 1916 to 1936 the
further on from 1916 to 1936 the literary digest did public did what they
literary digest did public did what they
literary digest did public did what they called straw polls on presidential
called straw polls on presidential
called straw polls on presidential elections these were polls based on
elections these were polls based on
elections these were polls based on mailing out cards to Ted to basically 10
mailing out cards to Ted to basically 10
mailing out cards to Ted to basically 10 million or more people the people were
million or more people the people were
million or more people the people were selected by their subscriber list
selected by their subscriber list
selected by their subscriber list by automobile registration telephone
by automobile registration telephone
by automobile registration telephone directories and off and also voter lists
directories and off and also voter lists
directories and off and also voter lists now these constituted by current
now these constituted by current
now these constituted by current standards bias samples in the sense that
standards bias samples in the sense that
standards bias samples in the sense that they just sent out cards to as many
they just sent out cards to as many
they just sent out cards to as many people as possible and let people
people as possible and let people
people as possible and let people self-select whether to send in their
self-select whether to send in their
self-select whether to send in their cards and despite that self-selected
cards and despite that self-selected
cards and despite that self-selected kind of sample which arguably was more
kind of sample which arguably was more
kind of sample which arguably was more of an upscale scale sample that is
of an upscale scale sample that is
of an upscale scale sample that is people had telephones people has cars
people had telephones people has cars
people had telephones people has cars and so forth you would think that would
and so forth you would think that would
and so forth you would think that would introduce certain kinds of biases and
introduce certain kinds of biases and
introduce certain kinds of biases and biases of the sort that comport to the
biases of the sort that comport to the
biases of the sort that comport to the stereotype of Republican voters who were
stereotype of Republican voters who were
stereotype of Republican voters who were more at least in those days more of
more at least in those days more of
more at least in those days more of middle middle class and upscale so there
middle middle class and upscale so there
middle middle class and upscale so there were apparent potential biases there but
were apparent potential biases there but
were apparent potential biases there but from 1916 to 1932 the literary digest
from 1916 to 1932 the literary digest
from 1916 to 1932 the literary digest correctly predicted the results of
correctly predicted the results of
correctly predicted the results of national elections so in 1933 when
national elections so in 1933 when
national elections so in 1933 when Roosevelt took off Franklin Roosevelt
Roosevelt took off Franklin Roosevelt
Roosevelt took off Franklin Roosevelt took office he’s on the front page of
took office he’s on the front page of
took office he’s on the front page of the literary digest they got it right
the literary digest they got it right
the literary digest they got it right come 1936 they’re doing their poll once
come 1936 they’re doing their poll once
come 1936 they’re doing their poll once more but but what happens there they
more but but what happens there they
more but but what happens there they predict that Alfred Landon would win the
predict that Alfred Landon would win the
predict that Alfred Landon would win the election
election
election however the upstart George Gallup began
however the upstart George Gallup began
however the upstart George Gallup began doing what he called scientific polling
doing what he called scientific polling
doing what he called scientific polling based on sampling of geographic
based on sampling of geographic
based on sampling of geographic locations and sending interviewers out
locations and sending interviewers out
locations and sending interviewers out to those locations to fill certain kinds
to those locations to fill certain kinds
to those locations to fill certain kinds of demographic quotas based on age and
of demographic quotas based on age and
of demographic quotas based on age and gender and other characteristics and
gender and other characteristics and
gender and other characteristics and Gallup argued that his method was more
Gallup argued that his method was more
Gallup argued that his method was more scientific and that he would his for
scientific and that he would his for
scientific and that he would his for prediction of the election would be
prediction of the election would be
prediction of the election would be right compared to the literary digest
right compared to the literary digest
right compared to the literary digest and he predicted based on his
and he predicted based on his
and he predicted based on his pre-election polling that Roosevelt
pre-election polling that Roosevelt
pre-election polling that Roosevelt would win the election and not not
would win the election and not not
would win the election and not not landed and he consistently had Roosevelt
landed and he consistently had Roosevelt
landed and he consistently had Roosevelt the head head in the polls as she as you
the head head in the polls as she as you
the head head in the polls as she as you see as you see in the chart and at the
see as you see in the chart and at the
see as you see in the chart and at the same time other legendary pollsters
same time other legendary pollsters
same time other legendary pollsters Archibald Crosley and Elmo Roper were
Archibald Crosley and Elmo Roper were
Archibald Crosley and Elmo Roper were entering the polling game using methods
entering the polling game using methods
entering the polling game using methods similar to Gallup and we’re making the
similar to Gallup and we’re making the
similar to Gallup and we’re making the same kinds of predictions what happened
same kinds of predictions what happened
same kinds of predictions what happened as a result of the election George
as a result of the election George
as a result of the election George Gallup became famous he got the election
Gallup became famous he got the election
Gallup became famous he got the election right the literary digest got it wrong
right the literary digest got it wrong
right the literary digest got it wrong what happened afterwards with was that
what happened afterwards with was that
what happened afterwards with was that the literary digest went under in 1938
the literary digest went under in 1938
the literary digest went under in 1938 scientific polling with with the word
scientific polling with with the word
scientific polling with with the word scientific in quotes established Gallup
scientific in quotes established Gallup
scientific in quotes established Gallup as a major figure in public affairs and
as a major figure in public affairs and
as a major figure in public affairs and in polling and so forth but there was
in polling and so forth but there was
in polling and so forth but there was that but the question is well why the
that but the question is well why the
that but the question is well why the deliberate was that was literary digest
deliberate was that was literary digest
deliberate was that was literary digest wrong in 1936 and not earlier and what
wrong in 1936 and not earlier and what
wrong in 1936 and not earlier and what happened was there was a major political
happened was there was a major political
happened was there was a major political realignment such that the kinds of
realignment such that the kinds of
realignment such that the kinds of biases that existed in the polls were
biases that existed in the polls were
biases that existed in the polls were correlated with how people voted and so
correlated with how people voted and so
correlated with how people voted and so what happened was literary digest
what happened was literary digest
what happened was literary digest overestimated the Republican vote what
overestimated the Republican vote what
overestimated the Republican vote what also happened that that that people
also happened that that that people
also happened that that that people don’t fully appreciate is that Gallup
don’t fully appreciate is that Gallup
don’t fully appreciate is that Gallup underestimated Roosevelt’s support by
underestimated Roosevelt’s support by
underestimated Roosevelt’s support by fully five points in the graph that you
fully five points in the graph that you
fully five points in the graph that you saw he was actually five point five
saw he was actually five point five
saw he was actually five point five point short so galip’s poll had those
point short so galip’s poll had those
point short so galip’s poll had those actually similar kinds of biases but he
actually similar kinds of biases but he
actually similar kinds of biases but he got the winner right and he got the
got the winner right and he got the
got the winner right and he got the winner right in 1940 and 1944 but again
winner right in 1940 and 1944 but again
winner right in 1940 and 1944 but again underestimated Roosevelt’s support so
underestimated Roosevelt’s support so
underestimated Roosevelt’s support so there were problems in the polls that
there were problems in the polls that
there were problems in the polls that then came home to roost in 1948 in 1948
then came home to roost in 1948 in 1948
then came home to roost in 1948 in 1948 prior to prior to the election the
prior to prior to the election the
prior to prior to the election the gallop gallop polls had predicted that
gallop gallop polls had predicted that
gallop gallop polls had predicted that Dewey would win by somewhere around five
Dewey would win by somewhere around five
Dewey would win by somewhere around five points in the election and other
points in the election and other
points in the election and other pollsters were getting the same kinds of
pollsters were getting the same kinds of
pollsters were getting the same kinds of results in including academic posters
results in including academic posters
results in including academic posters who were analyzing the election as well
who were analyzing the election as well
who were analyzing the election as well and what happened there was there are
and what happened there was there are
and what happened there was there are obviously problems in the Gallup poll
obviously problems in the Gallup poll
obviously problems in the Gallup poll the uproar was so great there were Wars
the uproar was so great there were Wars
the uproar was so great there were Wars in government there were up growers all
in government there were up growers all
in government there were up growers all throughout Society and it prompted the
throughout Society and it prompted the
throughout Society and it prompted the Social Science Research Council to put
Social Science Research Council to put
Social Science Research Council to put together a committee to investigate the
together a committee to investigate the
together a committee to investigate the pre-election polls of nineteen forty
pre-election polls of nineteen forty
pre-election polls of nineteen forty eight headed by a famous statistician
eight headed by a famous statistician
eight headed by a famous statistician named Frederick Mosteller
named Frederick Mosteller
named Frederick Mosteller so the big question is what did the what
so the big question is what did the what
so the big question is what did the what are the one of the SSRC report find well
are the one of the SSRC report find well
are the one of the SSRC report find well they found that the following things
they found that the following things
they found that the following things the biggest thing problem they found was
the biggest thing problem they found was
the biggest thing problem they found was what they argue were biases in the polls
what they argue were biases in the polls
what they argue were biases in the polls that led to Gallup and others
that led to Gallup and others
that led to Gallup and others interviewing upscale voters more than
interviewing upscale voters more than
interviewing upscale voters more than downscale voters introducing the same
downscale voters introducing the same
downscale voters introducing the same kinds of biases that we we had seen all
kinds of biases that we we had seen all
kinds of biases that we we had seen all along through nineteen nineteen forty
along through nineteen nineteen forty
along through nineteen nineteen forty four and the bias came in not because of
four and the bias came in not because of
four and the bias came in not because of the selection of geographic locations
the selection of geographic locations
the selection of geographic locations but because the interviewers had
but because the interviewers had
but because the interviewers had discretion with regard to what people
discretion with regard to what people
discretion with regard to what people they would interview and talk to and the
they would interview and talk to and the
they would interview and talk to and the more interview friendly looking people
more interview friendly looking people
more interview friendly looking people tended to be more middle-class and
tended to be more middle-class and
tended to be more middle-class and upscale and the interviewers would
upscale and the interviewers would
upscale and the interviewers would gravitate toward those people so that
gravitate toward those people so that
gravitate toward those people so that was the first problem the other problem
was the first problem the other problem
was the first problem the other problem and this
and this
and this thanks to what I call the people like
thanks to what I call the people like
thanks to what I call the people like the lack of humility or to be less
the lack of humility or to be less
the lack of humility or to be less polite the arrogance of the pollsters
polite the arrogance of the pollsters
polite the arrogance of the pollsters and predictors of because of the
and predictors of because of the
and predictors of because of the confidence that they had in their
confidence that they had in their
confidence that they had in their polling methods and the fact that you’ve
polling methods and the fact that you’ve
polling methods and the fact that you’ve gotten prior elections right and the
gotten prior elections right and the
gotten prior elections right and the arrogance was such that they stopped
arrogance was such that they stopped
arrogance was such that they stopped polling two weeks before the election
polling two weeks before the election
polling two weeks before the election and it turned out there was a lot going
and it turned out there was a lot going
and it turned out there was a lot going on in those two weeks in terms of the
on in those two weeks in terms of the
on in those two weeks in terms of the campaign Truman was actively campaigning
campaign Truman was actively campaigning
campaign Truman was actively campaigning there was a sense that the tide was
there was a sense that the tide was
there was a sense that the tide was turning may sound a little familiar to
turning may sound a little familiar to
turning may sound a little familiar to what happened in 2016 but we’ll get
what happened in 2016 but we’ll get
what happened in 2016 but we’ll get there in a second and so the fact that
there in a second and so the fact that
there in a second and so the fact that they stopped polling earlier early that
they stopped polling earlier early that
they stopped polling earlier early that was considered a problem as well further
was considered a problem as well further
was considered a problem as well further there’s always the problem and how to
there’s always the problem and how to
there’s always the problem and how to allocate undecided voters that was a
allocate undecided voters that was a
allocate undecided voters that was a possible major problem and we’ll come
possible major problem and we’ll come
possible major problem and we’ll come back to that today and in addition to
back to that today and in addition to
back to that today and in addition to that it’s always difficult and this
that it’s always difficult and this
that it’s always difficult and this speaks to Leo G barrows comment about
speaks to Leo G barrows comment about
speaks to Leo G barrows comment about its being difficult to predict
its being difficult to predict
its being difficult to predict especially about the future it’s
especially about the future it’s
especially about the future it’s difficult to predict what the electorate
difficult to predict what the electorate
difficult to predict what the electorate on election day is going to look like in
on election day is going to look like in
on election day is going to look like in based on pre-election polls so there was
based on pre-election polls so there was
based on pre-election polls so there was at this time this exaggerated impression
at this time this exaggerated impression
at this time this exaggerated impression of the accuracy of polls and that led to
of the accuracy of polls and that led to
of the accuracy of polls and that led to the reporting and projecting of the
the reporting and projecting of the
the reporting and projecting of the polls and the election to turnout in a
polls and the election to turnout in a
polls and the election to turnout in a certain way and it didn’t so there was a
certain way and it didn’t so there was a
certain way and it didn’t so there was a major problem there Gallup thereafter
major problem there Gallup thereafter
major problem there Gallup thereafter corrected his methods and in subsequent
corrected his methods and in subsequent
corrected his methods and in subsequent years was able to better predict the
years was able to better predict the
years was able to better predict the polls in in the 1958 election he was
polls in in the 1958 election he was
polls in in the 1958 election he was within the Gallup polls were within one
within the Gallup polls were within one
within the Gallup polls were within one point of Dwight Eisenhower’s victory
point of Dwight Eisenhower’s victory
point of Dwight Eisenhower’s victory over at least Stevenson following that
over at least Stevenson following that
over at least Stevenson following that there were advanced further advances in
there were advanced further advances in
there were advanced further advances in Survey Research leading to the
Survey Research leading to the
Survey Research leading to the proliferation of polling the shifting
proliferation of polling the shifting
proliferation of polling the shifting from in-person interviews to telephone
from in-person interviews to telephone
from in-person interviews to telephone surveys the use of random digit dialing
surveys the use of random digit dialing
surveys the use of random digit dialing computer assisted interviewing and so
computer assisted interviewing and so
computer assisted interviewing and so forth and up until a point there were
forth and up until a point there were
forth and up until a point there were sufficiently high response rates so that
sufficiently high response rates so that
sufficiently high response rates so that the quality of polls was good but what
the quality of polls was good but what
the quality of polls was good but what happened afterwards was that because of
happened afterwards was that because of
happened afterwards was that because of the proliferation of polling through the
the proliferation of polling through the
the proliferation of polling through the proliferation of solicitation calls and
proliferation of solicitation calls and
proliferation of solicitation calls and other kinds of calls that people get
other kinds of calls that people get
other kinds of calls that people get there’s been a decline over the years
there’s been a decline over the years
there’s been a decline over the years and response rates to surveys especially
and response rates to surveys especially
and response rates to surveys especially in the modern day with the proliferation
in the modern day with the proliferation
in the modern day with the proliferation of cell phones cell phones in the United
of cell phones cell phones in the United
of cell phones cell phones in the United States today there are more than 50% of
States today there are more than 50% of
States today there are more than 50% of the households in the United States that
the households in the United States that
the households in the United States that are that are cell phone only so that the
are that are cell phone only so that the
are that are cell phone only so that the pollsters had to start calling with
pollsters had to start calling with
pollsters had to start calling with people on cell phones
people on cell phones
people on cell phones people were less likely to talk to
people were less likely to talk to
people were less likely to talk to pollsters on cellphones there other
pollsters on cellphones there other
pollsters on cellphones there other issues involved in cell phones as well
issues involved in cell phones as well
issues involved in cell phones as well so that the response rates began to
so that the response rates began to
so that the response rates began to decline and there were worries about
decline and there were worries about
decline and there were worries about increasing errors in surveys and these
increasing errors in surveys and these
increasing errors in surveys and these were compounded in the 2016 election by
were compounded in the 2016 election by
were compounded in the 2016 election by other factors including especially the
other factors including especially the
other factors including especially the ability of the pollsters to identify
ability of the pollsters to identify
ability of the pollsters to identify that the nature the composition of the
that the nature the composition of the
that the nature the composition of the electorate in the 2016 election the
electorate in the 2016 election the
electorate in the 2016 election the other complication in the 2016 election
other complication in the 2016 election
other complication in the 2016 election was the for lack of a better term again
was the for lack of a better term again
was the for lack of a better term again arrogance associated with projections
arrogance associated with projections
arrogance associated with projections about how the election would occur
about how the election would occur
about how the election would occur because of what was referred to as the
because of what was referred to as the
because of what was referred to as the blue wall that the Democrats had now
blue wall that the Democrats had now
blue wall that the Democrats had now Obama was able to win substantial
Obama was able to win substantial
Obama was able to win substantial electoral college majorities in 2008 and
electoral college majorities in 2008 and
electoral college majorities in 2008 and 2012 he was able to win the state the
2012 he was able to win the state the
2012 he was able to win the state the key states of Ohio and Florida and in
key states of Ohio and Florida and in
key states of Ohio and Florida and in addition he won North Carolina and
addition he won North Carolina and
addition he won North Carolina and Indiana in the 2008 election and so that
Indiana in the 2008 election and so that
Indiana in the 2008 election and so that in the electoral map that you see
in the electoral map that you see
in the electoral map that you see there’s a lot of blue there and the blue
there’s a lot of blue there and the blue
there’s a lot of blue there and the blue there is such that it basically gave
there is such that it basically gave
there is such that it basically gave Obama the Democrats many different ways
Obama the Democrats many different ways
Obama the Democrats many different ways of winning the electoral college if you
of winning the electoral college if you
of winning the electoral college if you go back to the two thousand election
go back to the two thousand election
go back to the two thousand election everybody there’s a big uproar about
everybody there’s a big uproar about
everybody there’s a big uproar about what happened in Florida and have had al
what happened in Florida and have had al
what happened in Florida and have had al gore won Florida he would have won the
gore won Florida he would have won the
gore won Florida he would have won the election but what people forget is that
election but what people forget is that
election but what people forget is that Gore could have won the election without
Gore could have won the election without
Gore could have won the election without Florida if he had taken his home state
Florida if he had taken his home state
Florida if he had taken his home state of Tennessee or if he had taken the
of Tennessee or if he had taken the
of Tennessee or if he had taken the state of New Hampshire
state of New Hampshire
state of New Hampshire so they fast forward to the 2016
so they fast forward to the 2016
so they fast forward to the 2016 election there were great many ways that
election there were great many ways that
election there were great many ways that Hillary Clinton could win the election
Hillary Clinton could win the election
Hillary Clinton could win the election in the blue that you see in in the in
in the blue that you see in in the in
in the blue that you see in in the in the map in front of you you see a lot of
the map in front of you you see a lot of
the map in front of you you see a lot of blue there but if we if we eliminate the
blue there but if we if we eliminate the
blue there but if we if we eliminate the states of Florida Ohio North Carolina
states of Florida Ohio North Carolina
states of Florida Ohio North Carolina and Iowa Hillary Clinton could have
and Iowa Hillary Clinton could have
and Iowa Hillary Clinton could have could have could have still won the one
could have could have still won the one
could have could have still won the one the election the key things being the
the election the key things being the
the election the key things being the key difference between to that the 2000
key difference between to that the 2000
key difference between to that the 2000 election is that the states of Colorado
election is that the states of Colorado
election is that the states of Colorado and Virginia have become democratic
and Virginia have become democratic
and Virginia have become democratic states it was always assumed because of
states it was always assumed because of
states it was always assumed because of decades of election results that
decades of election results that
decades of election results that Michigan Wisconsin and Pennsylvania
Michigan Wisconsin and Pennsylvania
Michigan Wisconsin and Pennsylvania could never go Republican that what that
could never go Republican that what that
could never go Republican that what that was assumed and so the worst-case
was assumed and so the worst-case
was assumed and so the worst-case scenario would be if she if she lost
scenario would be if she if she lost
scenario would be if she if she lost North Carolina floor
North Carolina floor
North Carolina floor and Ohio but as long as she took the
and Ohio but as long as she took the
and Ohio but as long as she took the states and Iowa as long as she took the
states and Iowa as long as she took the
states and Iowa as long as she took the states of New Mexico Nevada and New
states of New Mexico Nevada and New
states of New Mexico Nevada and New Hampshire she could win she could win
Hampshire she could win she could win
Hampshire she could win she could win the election and if you look at what
the election and if you look at what
the election and if you look at what happened in the 2016 election she in
happened in the 2016 election she in
happened in the 2016 election she in fact lost North Carolina Florida Ohio
fact lost North Carolina Florida Ohio
fact lost North Carolina Florida Ohio she won Nevada and New Hampshire and
she won Nevada and New Hampshire and
she won Nevada and New Hampshire and would have won the election except for
would have won the election except for
would have won the election except for one small detail
one small detail
one small detail she lost Pennsylvania Wisconsin in
she lost Pennsylvania Wisconsin in
she lost Pennsylvania Wisconsin in Michigan and that’s that that’s what no
Michigan and that’s that that’s what no
Michigan and that’s that that’s what no one saw coming
one saw coming
one saw coming so what happened in the 2016 election
so what happened in the 2016 election
so what happened in the 2016 election there were all these wild predictions of
there were all these wild predictions of
there were all these wild predictions of the probability of Hillary Clinton
the probability of Hillary Clinton
the probability of Hillary Clinton winning ranging from 71 percent to 99
winning ranging from 71 percent to 99
winning ranging from 71 percent to 99 percent based on simulations of
percent based on simulations of
percent based on simulations of electoral college results based on the
electoral college results based on the
electoral college results based on the pre-election polling the problem was
pre-election polling the problem was
pre-election polling the problem was that the polling in key states was error
that the polling in key states was error
that the polling in key states was error ridden you remember the uproar that was
ridden you remember the uproar that was
ridden you remember the uproar that was that the that the election caused that
that the that the election caused that
that the that the election caused that led of course to a report that was just
led of course to a report that was just
led of course to a report that was just completed by the American Association
completed by the American Association
completed by the American Association for public opinion research a poor had
for public opinion research a poor had
for public opinion research a poor had actually put together a task force to do
actually put together a task force to do
actually put together a task force to do a report on pre on the 2016 election
a report on pre on the 2016 election
a report on pre on the 2016 election polling not expecting any you know
polling not expecting any you know
polling not expecting any you know surprising results but they were
surprising results but they were
surprising results but they were confronted with a major surprise so it
confronted with a major surprise so it
confronted with a major surprise so it basically shifted the focus of the
basically shifted the focus of the
basically shifted the focus of the report that was done by a blue-ribbon
report that was done by a blue-ribbon
report that was done by a blue-ribbon committee of the American Association
committee of the American Association
committee of the American Association for public opinion research as you see
for public opinion research as you see
for public opinion research as you see as you see listed here and the big
as you see listed here and the big
as you see listed here and the big question is what did they find
question is what did they find
question is what did they find well what they found first and foremost
well what they found first and foremost
well what they found first and foremost was that the national polls were
was that the national polls were
was that the national polls were generally generally accurate by
generally generally accurate by
generally generally accurate by historical standards that is Hillary
historical standards that is Hillary
historical standards that is Hillary Clinton won the popular vote by two
Clinton won the popular vote by two
Clinton won the popular vote by two percentage points and the polls had her
percentage points and the polls had her
percentage points and the polls had her winning by three or four points and that
winning by three or four points and that
winning by three or four points and that two point difference was similar to
two point difference was similar to
two point difference was similar to historical averages and actually the
historical averages and actually the
historical averages and actually the polls in 2016 were better than the polls
polls in 2016 were better than the polls
polls in 2016 were better than the polls in 2012 and that and that’s something
in 2012 and that and that’s something
in 2012 and that and that’s something that that got lost in the shuffle the
that that got lost in the shuffle the
that that got lost in the shuffle the problem was is that the states had a bad
problem was is that the states had a bad
problem was is that the states had a bad year and in particular there was a bad
year and in particular there was a bad
year and in particular there was a bad year in the key states of Michigan
year in the key states of Michigan
year in the key states of Michigan Wisconsin and in Pennsylvania now the
Wisconsin and in Pennsylvania now the
Wisconsin and in Pennsylvania now the problems with the polls
particular slide basically shows that
particular slide basically shows that the poles had her winning by an average
the poles had her winning by an average
the poles had her winning by an average of two points based on polls by
of two points based on polls by
of two points based on polls by different by different polling
different by different polling
different by different polling organizations the the error that was
organizations the the error that was
organizations the the error that was found in the polls with the magnitude of
found in the polls with the magnitude of
found in the polls with the magnitude of the error was basically about 2.2
the error was basically about 2.2
the error was basically about 2.2 percentage points with the bias in favor
percentage points with the bias in favor
percentage points with the bias in favor of Hillary Hillary Clinton in the graph
of Hillary Hillary Clinton in the graph
of Hillary Hillary Clinton in the graph that you see that the orange line
that you see that the orange line
that you see that the orange line represents the magnitude of the of the
represents the magnitude of the of the
represents the magnitude of the of the error and the blue bars represent the
error and the blue bars represent the
error and the blue bars represent the bias the direction so was there was an
bias the direction so was there was an
bias the direction so was there was an overestimate of the Hillary Clinton vote
overestimate of the Hillary Clinton vote
overestimate of the Hillary Clinton vote if you go back and look at 1936 by the
if you go back and look at 1936 by the
if you go back and look at 1936 by the way there was a huge underestimate
way there was a huge underestimate
way there was a huge underestimate underestimate of the Democratic vote
underestimate of the Democratic vote
underestimate of the Democratic vote even though in 1936 1940 in 1944 the
even though in 1936 1940 in 1944 the
even though in 1936 1940 in 1944 the Gallup poll got the winner right but got
Gallup poll got the winner right but got
Gallup poll got the winner right but got the winner right but by high margins
the winner right but by high margins
the winner right but by high margins that were that were substantially
that were that were substantially
that were that were substantially substantially off in the states the
substantially off in the states the
substantially off in the states the result the results were different the
result the results were different the
result the results were different the magnitudes were on the order of five
magnitudes were on the order of five
magnitudes were on the order of five percentage points which is a lot and
percentage points which is a lot and
percentage points which is a lot and that’s an average in some states it was
that’s an average in some states it was
that’s an average in some states it was bigger in some states it was smaller and
bigger in some states it was smaller and
bigger in some states it was smaller and the bias was in the direction of of the
the bias was in the direction of of the
the bias was in the direction of of the Democrats that is they underestimated
Democrats that is they underestimated
Democrats that is they underestimated Donald Trump spoke and the problem that
Donald Trump spoke and the problem that
Donald Trump spoke and the problem that occurred has to do with the quality of
occurred has to do with the quality of
occurred has to do with the quality of the polls which for me
the polls which for me
the polls which for me and this isn’t cited in the report is
and this isn’t cited in the report is
and this isn’t cited in the report is indicated by the fact that in the last
indicated by the fact that in the last
indicated by the fact that in the last two days before the election there were
two days before the election there were
two days before the election there were about a there were 11 high quality
about a there were 11 high quality
about a there were 11 high quality public opinion polls that were done
public opinion polls that were done
public opinion polls that were done nationally in contrast in the states in
nationally in contrast in the states in
nationally in contrast in the states in the last week before the election in
the last week before the election in
the last week before the election in Wisconsin Michigan in Pennsylvania there
Wisconsin Michigan in Pennsylvania there
Wisconsin Michigan in Pennsylvania there were just a handful of polls and the
were just a handful of polls and the
were just a handful of polls and the quality of those polls the a poor report
quality of those polls the a poor report
quality of those polls the a poor report called in called into question what what
called in called into question what what
called in called into question what what had happened in the election well there
had happened in the election well there
had happened in the election well there were a lot there were a lot of things at
were a lot there were a lot of things at
were a lot there were a lot of things at the a poor report sites one is voters
the a poor report sites one is voters
the a poor report sites one is voters broke in the last day of week of the
broke in the last day of week of the
broke in the last day of week of the election more toward Donald Trump than
election more toward Donald Trump than
election more toward Donald Trump than Hillary Clinton based on the exit polls
Hillary Clinton based on the exit polls
Hillary Clinton based on the exit polls people who decided late in Wisconsin
people who decided late in Wisconsin
people who decided late in Wisconsin Michigan and Pennsylvania
Michigan and Pennsylvania
Michigan and Pennsylvania disproportionately voted for Donald
disproportionately voted for Donald
disproportionately voted for Donald Trump in addition in addition there was
Trump in addition in addition there was
Trump in addition in addition there was a question of what happened with regard
a question of what happened with regard
a question of what happened with regard to voter turnout in the state and the
to voter turnout in the state and the
to voter turnout in the state and the representativeness of the samples what
representativeness of the samples what
representativeness of the samples what they did find is that one problem in the
they did find is that one problem in the
they did find is that one problem in the polls is that the polls in the states in
polls is that the polls in the states in
polls is that the polls in the states in many cases did not weight the weight and
many cases did not weight the weight and
many cases did not weight the weight and or adjust their data statistically for
or adjust their data statistically for
or adjust their data statistically for education that is white voters without a
education that is white voters without a
education that is white voters without a college degree voted disproportionately
college degree voted disproportionately
college degree voted disproportionately for Trump and
for Trump and
for Trump and and the there their non-response the
and the there their non-response the
and the there their non-response the polls was not compensated for by by any
polls was not compensated for by by any
polls was not compensated for by by any waiting in addition and this isn’t in
waiting in addition and this isn’t in
waiting in addition and this isn’t in the report but based on the latest
the report but based on the latest
the report but based on the latest census data we know that voter turnout
census data we know that voter turnout
census data we know that voter turnout among african-american voters was less
among african-american voters was less
among african-american voters was less and we also know from the exit polls is
and we also know from the exit polls is
and we also know from the exit polls is that African American voters supported
that African American voters supported
that African American voters supported Hillary Clinton at slightly lower rates
Hillary Clinton at slightly lower rates
Hillary Clinton at slightly lower rates and they had supported Barack Obama and
and they had supported Barack Obama and
and they had supported Barack Obama and the same team same thing for Latino
the same team same thing for Latino
the same team same thing for Latino voters and in addition with the aiport
voters and in addition with the aiport
voters and in addition with the aiport cited in some states there were called
cited in some states there were called
cited in some states there were called ballot order effects that is that is
ballot order effects that is that is
ballot order effects that is that is that isn’t there were states in which
that isn’t there were states in which
that isn’t there were states in which Trump was disproportionately listed
Trump was disproportionately listed
Trump was disproportionately listed first on the ballot and that may have
first on the ballot and that may have
first on the ballot and that may have led to some slight biases and then last
led to some slight biases and then last
led to some slight biases and then last but not least the report talks about
but not least the report talks about
but not least the report talks about this but I think under states it the big
this but I think under states it the big
this but I think under states it the big problem was not so much in the polls
problem was not so much in the polls
problem was not so much in the polls themselves but in the reporting and in
themselves but in the reporting and in
themselves but in the reporting and in the exaggerating were the perceived
the exaggerating were the perceived
the exaggerating were the perceived exaggerating of the likelihood that
exaggerating of the likelihood that
exaggerating of the likelihood that Hillary Clinton would win the election
Hillary Clinton would win the election
Hillary Clinton would win the election and the fact that the that people did
and the fact that the that people did
and the fact that the that people did not fully appreciate sampling
not fully appreciate sampling
not fully appreciate sampling variability and probabilities in the
variability and probabilities in the
variability and probabilities in the 1948 report the Mosteller report they
1948 report the Mosteller report they
1948 report the Mosteller report they talked also about the lack of
talked also about the lack of
talked also about the lack of appreciation of variability and survey
appreciation of variability and survey
appreciation of variability and survey error particularly among the reporters
error particularly among the reporters
error particularly among the reporters and predictors of the poll results 71%
and predictors of the poll results 71%
and predictors of the poll results 71% it’s not a prediction that Hillary
it’s not a prediction that Hillary
it’s not a prediction that Hillary Clinton when it would win is not a slam
Clinton when it would win is not a slam
Clinton when it would win is not a slam dunk but if you repeat 71 percent enough
dunk but if you repeat 71 percent enough
dunk but if you repeat 71 percent enough online and on network news and on cable
online and on network news and on cable
online and on network news and on cable news people get the impression that she
news people get the impression that she
news people get the impression that she got the impression that she was a
got the impression that she was a
got the impression that she was a shoo-in and now and that that perception
shoo-in and now and that that perception
shoo-in and now and that that perception was a big part I would say the most
was a big part I would say the most
was a big part I would say the most significant part of the debacle that
significant part of the debacle that
significant part of the debacle that occurred and the final conclusion of the
occurred and the final conclusion of the
occurred and the final conclusion of the a poor report is that election polls are
a poor report is that election polls are
a poor report is that election polls are not all polls what the pollsters all
not all polls what the pollsters all
not all polls what the pollsters all worried about is that when they when
worried about is that when they when
worried about is that when they when they failed to predict elections right
they failed to predict elections right
they failed to predict elections right people lose faith in polling in this in
people lose faith in polling in this in
people lose faith in polling in this in this in this in this election the
this in this in this election the
this in this in this election the national polls were reasonably accurate
national polls were reasonably accurate
national polls were reasonably accurate but more importantly election polls are
but more importantly election polls are
but more importantly election polls are different from all polls in the sense
different from all polls in the sense
different from all polls in the sense they you’re trying to make predictions
they you’re trying to make predictions
they you’re trying to make predictions and wait or adjust your data to an
and wait or adjust your data to an
and wait or adjust your data to an electorate that you don’t know where as
electorate that you don’t know where as
electorate that you don’t know where as normal public opinion polls that are
normal public opinion polls that are
normal public opinion polls that are done between elections and don’t focus
done between elections and don’t focus
done between elections and don’t focus on elections are samples of the adult
on elections are samples of the adult
on elections are samples of the adult population for which we have census data
population for which we have census data
population for which we have census data at the benchmark what
at the benchmark what
at the benchmark what the composition of the public is so
the composition of the public is so
the composition of the public is so there should be much less error in
there should be much less error in
there should be much less error in polling overall compared to election
polling overall compared to election
polling overall compared to election polls what what are predictions about
polls what what are predictions about
polls what what are predictions about the future quality of survey is giving
the future quality of survey is giving
the future quality of survey is giving the even declining response rates and
the even declining response rates and
the even declining response rates and other issues in polling that’s a topic
other issues in polling that’s a topic
other issues in polling that’s a topic for another occasion if you want to read
for another occasion if you want to read
for another occasion if you want to read about this further you should see the
about this further you should see the
about this further you should see the latest Pew Research Center report
latest Pew Research Center report
latest Pew Research Center report entitled what low response rates mean
entitled what low response rates mean
entitled what low response rates mean for telephone surveys the basic finding
for telephone surveys the basic finding
for telephone surveys the basic finding of that study is that for the most part
of that study is that for the most part
of that study is that for the most part good high-quality Poland could be done
good high-quality Poland could be done
good high-quality Poland could be done but there’s certain kinds of problems
but there’s certain kinds of problems
but there’s certain kinds of problems that we need to be cognizant about thank
that we need to be cognizant about thank
that we need to be cognizant about thank [Applause]
”
Be First to Comment