[Music] today we’ll be talking about perception
today we’ll be talking about perception
today we’ll be talking about perception specifically how we the animals perceive
specifically how we the animals perceive
specifically how we the animals perceive the world around us and why it’s
the world around us and why it’s
the world around us and why it’s necessary to bring together computer
necessary to bring together computer
necessary to bring together computer scientists and neuroscientists not just
scientists and neuroscientists not just
scientists and neuroscientists not just understand machine learning but to
understand machine learning but to
understand machine learning but to understand our own brains as well the
understand our own brains as well the
understand our own brains as well the future that’s been promised us by
future that’s been promised us by
future that’s been promised us by science fiction the houses that
science fiction the houses that
science fiction the houses that understand us autonomous cars computers
understand us autonomous cars computers
understand us autonomous cars computers that contribute ideas it all boils down
that contribute ideas it all boils down
that contribute ideas it all boils down to how well a machine can perceive the
to how well a machine can perceive the
to how well a machine can perceive the world around it the key is perception
world around it the key is perception
world around it the key is perception without perception it’s not possible to
without perception it’s not possible to
without perception it’s not possible to make a correct decision whether you’re
make a correct decision whether you’re
make a correct decision whether you’re an animal or a machine but first let’s
an animal or a machine but first let’s
an animal or a machine but first let’s try it everyone in this audience has a
try it everyone in this audience has a
try it everyone in this audience has a fantastically capable sensory system
fantastically capable sensory system
fantastically capable sensory system we’re gonna test it out right now what
we’re gonna test it out right now what
we’re gonna test it out right now what did everyone see on the screen right so
did everyone see on the screen right so
did everyone see on the screen right so we can all agree it’s a fox
we can all agree it’s a fox
we can all agree it’s a fox now that in and of itself is fantastic I
now that in and of itself is fantastic I
now that in and of itself is fantastic I didn’t tell you where to focus your
didn’t tell you where to focus your
didn’t tell you where to focus your attention I didn’t tell you that it was
attention I didn’t tell you that it was
attention I didn’t tell you that it was gonna be an animal I didn’t tell you how
gonna be an animal I didn’t tell you how
gonna be an animal I didn’t tell you how to look or how to respond now for a
to look or how to respond now for a
to look or how to respond now for a computer even locating the screen in the
computer even locating the screen in the
computer even locating the screen in the room presents non-trivial technical
room presents non-trivial technical
room presents non-trivial technical challenges but so with all of us
challenges but so with all of us
challenges but so with all of us agreeing that’s a fox those of you who
agreeing that’s a fox those of you who
agreeing that’s a fox those of you who are familiar with machine learning
are familiar with machine learning
are familiar with machine learning research know that there are algorithms
research know that there are algorithms
research know that there are algorithms out there that exist that can do that
out there that exist that can do that
out there that exist that can do that same type of task that can do it very
same type of task that can do it very
same type of task that can do it very well for example here’s what clarify an
well for example here’s what clarify an
well for example here’s what clarify an online machine learning algorithm has to
online machine learning algorithm has to
online machine learning algorithm has to say about the Fox of course it
say about the Fox of course it
say about the Fox of course it recognizes that as well but here’s the
recognizes that as well but here’s the
recognizes that as well but here’s the question is the Fox cute of course it is
question is the Fox cute of course it is
question is the Fox cute of course it is and the algorithm actually recognizes it
and the algorithm actually recognizes it
and the algorithm actually recognizes it as well but
as well but
as well but here’s the difference that question gets
here’s the difference that question gets
here’s the difference that question gets it the fundamental difference between
it the fundamental difference between
it the fundamental difference between machine learning and actual intelligence
machine learning and actual intelligence
machine learning and actual intelligence yes we can go ahead and train a computer
yes we can go ahead and train a computer
yes we can go ahead and train a computer to identify cute animals or not cute
to identify cute animals or not cute
to identify cute animals or not cute animals by showing it thousands upon
animals by showing it thousands upon
animals by showing it thousands upon thousands of in front of images of cute
thousands of in front of images of cute
thousands of in front of images of cute animals and thousands upon thousands of
animals and thousands upon thousands of
animals and thousands upon thousands of images of less traditionally cute
images of less traditionally cute
images of less traditionally cute animals but that’s not how we decided if
animals but that’s not how we decided if
animals but that’s not how we decided if that fox was cute we learn the concept
that fox was cute we learn the concept
that fox was cute we learn the concept of cute based on a unique combination of
of cute based on a unique combination of
of cute based on a unique combination of personal experience natural instinct
personal experience natural instinct
personal experience natural instinct cultural influences and perceived
cultural influences and perceived
cultural influences and perceived context in the machine learning world we
context in the machine learning world we
context in the machine learning world we call this the ground truth in other
call this the ground truth in other
call this the ground truth in other words it’s the absolute or correct
words it’s the absolute or correct
words it’s the absolute or correct answer as we understand it tell the
answer as we understand it tell the
answer as we understand it tell the machines that it is and the great
machines that it is and the great
machines that it is and the great challenge in machine learning today
challenge in machine learning today
challenge in machine learning today doing now is to program algorithms that
doing now is to program algorithms that
doing now is to program algorithms that agree with your ground truth success in
agree with your ground truth success in
agree with your ground truth success in that case means an algorithm that is as
that case means an algorithm that is as
that case means an algorithm that is as good at a task as you are but that’s not
good at a task as you are but that’s not
good at a task as you are but that’s not the end goal we want to create a future
the end goal we want to create a future
the end goal we want to create a future in which artificial systems extend and
in which artificial systems extend and
in which artificial systems extend and augment our ability and in which
augment our ability and in which
augment our ability and in which artificial systems also can help us to
artificial systems also can help us to
artificial systems also can help us to create and help us to imagine that would
create and help us to imagine that would
create and help us to imagine that would be called artificial intelligence by
be called artificial intelligence by
be called artificial intelligence by many people and to get there we have to
many people and to get there we have to
many people and to get there we have to learn something entirely new and the
learn something entirely new and the
learn something entirely new and the best way to learn something new is to
best way to learn something new is to
best way to learn something new is to watch it being done so this is a photo
watch it being done so this is a photo
watch it being done so this is a photo of my son ari and his cat brother
of my son ari and his cat brother
of my son ari and his cat brother maxwell ari was born very early and he
maxwell ari was born very early and he
maxwell ari was born very early and he spent ten weeks in the neonatal
spent ten weeks in the neonatal
spent ten weeks in the neonatal intensive care unit before he came home
intensive care unit before he came home
intensive care unit before he came home now when he did come home he was a lot
now when he did come home he was a lot
now when he did come home he was a lot like a newborn but he also had life
like a newborn but he also had life
like a newborn but he also had life experience beyond that of a typical term
experience beyond that of a typical term
experience beyond that of a typical term baby so the parents in this audience who
baby so the parents in this audience who
baby so the parents in this audience who have preemies you’ll understand exactly
have preemies you’ll understand exactly
have preemies you’ll understand exactly what I’m talking about for those of you
what I’m talking about for those of you
what I’m talking about for those of you who don’t it’s enough to say that it’s a
who don’t it’s enough to say that it’s a
who don’t it’s enough to say that it’s a unique and complicated experience what
unique and complicated experience what
unique and complicated experience what isn’t complicated was our ease reaction
isn’t complicated was our ease reaction
isn’t complicated was our ease reaction to Maxwell
to Maxwell
to Maxwell all right love Maxwell immediately he
all right love Maxwell immediately he
all right love Maxwell immediately he was reaching out to touch him before he
was reaching out to touch him before he
was reaching out to touch him before he got a smile or crawl Maxwell was
got a smile or crawl Maxwell was
got a smile or crawl Maxwell was friendly to Ari immediately let’s break
friendly to Ari immediately let’s break
friendly to Ari immediately let’s break that down in a little bit of detail our
that down in a little bit of detail our
that down in a little bit of detail our I had seen cats during his lifetime
I had seen cats during his lifetime
I had seen cats during his lifetime zero times maximum rescue he’d seen
zero times maximum rescue he’d seen
zero times maximum rescue he’d seen children before and let’s see how do I
children before and let’s see how do I
children before and let’s see how do I put this nicely
put this nicely
put this nicely he’s domesticated but safe around
he’s domesticated but safe around
he’s domesticated but safe around children definitely not we brought them
children definitely not we brought them
children definitely not we brought them together very carefully after our I had
together very carefully after our I had
together very carefully after our I had been home just a day or two watching
been home just a day or two watching
been home just a day or two watching each of them carefully to see how they
each of them carefully to see how they
each of them carefully to see how they would react and ready to snatch one or
would react and ready to snatch one or
would react and ready to snatch one or the other of them away from either teeth
the other of them away from either teeth
the other of them away from either teeth and claws or the baby hands of doom and
and claws or the baby hands of doom and
and claws or the baby hands of doom and what happened they were fine together
what happened they were fine together
what happened they were fine together somehow
somehow
somehow Maxwell perceived that Ari was different
Maxwell perceived that Ari was different
Maxwell perceived that Ari was different than other humans particularly other
than other humans particularly other
than other humans particularly other children and Ari was not scared of a
children and Ari was not scared of a
children and Ari was not scared of a predator that was three times his body
predator that was three times his body
predator that was three times his body weight at the time somehow the
weight at the time somehow the
weight at the time somehow the algorithms running in their braids
algorithms running in their braids
algorithms running in their braids perceived that they were safe together
perceived that they were safe together
perceived that they were safe together maybe it was the way we were standing we
maybe it was the way we were standing we
maybe it was the way we were standing we brought them together
brought them together
brought them together maybe Maxwell could smell that Ari was
maybe Maxwell could smell that Ari was
maybe Maxwell could smell that Ari was somehow family sadly neither of them
somehow family sadly neither of them
somehow family sadly neither of them were speaking yet so I can’t ask him how
were speaking yet so I can’t ask him how
were speaking yet so I can’t ask him how they figured it out now in the machine
they figured it out now in the machine
they figured it out now in the machine learning world we call this zero shot
learning world we call this zero shot
learning world we call this zero shot learning in other words on the first
learning in other words on the first
learning in other words on the first exposure with no training examples
exposure with no training examples
exposure with no training examples already recognized that maximum was
already recognized that maximum was
already recognized that maximum was something he should reach out and touch
something he should reach out and touch
something he should reach out and touch and Max will recognize that Ari was not
and Max will recognize that Ari was not
and Max will recognize that Ari was not a threat or at least not a threat he
a threat or at least not a threat he
a threat or at least not a threat he should do anything about and this stands
should do anything about and this stands
should do anything about and this stands in sharp contrast to how machine
in sharp contrast to how machine
in sharp contrast to how machine learning is done today today if we want
learning is done today today if we want
learning is done today today if we want to train a machine we show it thousands
to train a machine we show it thousands
to train a machine we show it thousands upon thousands of exactly labeled test
upon thousands of exactly labeled test
upon thousands of exactly labeled test examples we feed them through an
examples we feed them through an
examples we feed them through an algorithm when the algorithm gets one
algorithm when the algorithm gets one
algorithm when the algorithm gets one correct a corrective signal goes back
correct a corrective signal goes back
correct a corrective signal goes back through it adjusts the algorithm and the
through it adjusts the algorithm and the
through it adjusts the algorithm and the next time it sees something that’s
next time it sees something that’s
next time it sees something that’s similar to something that’s seen before
similar to something that’s seen before
similar to something that’s seen before it’s more likely to get it correct now
it’s more likely to get it correct now
it’s more likely to get it correct now you probably expect the rest of this
you probably expect the rest of this
you probably expect the rest of this talk to be me telling you how we
talk to be me telling you how we
talk to be me telling you how we invented some fantastic zero shot
invented some fantastic zero shot
invented some fantastic zero shot algorithm that’s gonna save the world
algorithm that’s gonna save the world
algorithm that’s gonna save the world right it’s not going to be that kind of
right it’s not going to be that kind of
right it’s not going to be that kind of behavior in which a machine is correct
behavior in which a machine is correct
behavior in which a machine is correct based on instinct and context and
based on instinct and context and
based on instinct and context and minimal training that’s the future we’re
minimal training that’s the future we’re
minimal training that’s the future we’re trying to reach so how do we get there
trying to reach so how do we get there
trying to reach so how do we get there well we all know that animals are
well we all know that animals are
well we all know that animals are amazing at handling whatever nature can
amazing at handling whatever nature can
amazing at handling whatever nature can throw at them they can find food and
throw at them they can find food and
throw at them they can find food and mates they can identify friend and foe
mates they can identify friend and foe
mates they can identify friend and foe they do all of this at a complex and
they do all of this at a complex and
they do all of this at a complex and diverse environment
diverse environment
diverse environment animals have to be good at this the
animals have to be good at this the
animals have to be good at this the animals that aren’t good at this get
animals that aren’t good at this get
animals that aren’t good at this get eaten before they reproduce and they
eaten before they reproduce and they
eaten before they reproduce and they don’t get to pass on their genes not a
don’t get to pass on their genes not a
don’t get to pass on their genes not a great strategy for actually continuing
great strategy for actually continuing
great strategy for actually continuing they also have to be efficient to this
they also have to be efficient to this
they also have to be efficient to this if you burn up all your food trying to
if you burn up all your food trying to
if you burn up all your food trying to do mental computations
do mental computations
do mental computations you won’t make it to your next meal so
you won’t make it to your next meal so
you won’t make it to your next meal so to create the algorithms of the future
to create the algorithms of the future
to create the algorithms of the future we are using neuroscience to constrain
we are using neuroscience to constrain
we are using neuroscience to constrain the world of machine learning by working
the world of machine learning by working
the world of machine learning by working in the intersection between the two
in the intersection between the two
in the intersection between the two fields we know that good solutions exist
fields we know that good solutions exist
fields we know that good solutions exist for our problems because we can see
for our problems because we can see
for our problems because we can see nature demonstrating them so here’s the
nature demonstrating them so here’s the
nature demonstrating them so here’s the thought experiment to explain what I
thought experiment to explain what I
thought experiment to explain what I mean
mean
mean if you were running a machine learning
if you were running a machine learning
if you were running a machine learning algorithm in your head how long would it
algorithm in your head how long would it
algorithm in your head how long would it take you to recognize a face well first
take you to recognize a face well first
take you to recognize a face well first look at the person next to you how long
look at the person next to you how long
look at the person next to you how long does it take your thing Hey look at that
does it take your thing Hey look at that
does it take your thing Hey look at that it’s a face well the scientists who do
it’s a face well the scientists who do
it’s a face well the scientists who do this experiment for a living seriously
this experiment for a living seriously
this experiment for a living seriously they tell us it takes about a hundred
they tell us it takes about a hundred
they tell us it takes about a hundred and fifty milliseconds or just a little
and fifty milliseconds or just a little
and fifty milliseconds or just a little bit longer than a tenth of a second now
bit longer than a tenth of a second now
bit longer than a tenth of a second now during that tenth of a second the
during that tenth of a second the
during that tenth of a second the electrical pulses in your brain the
electrical pulses in your brain the
electrical pulses in your brain the spikes after travel from your retina to
spikes after travel from your retina to
spikes after travel from your retina to your deep visual cortex via a series of
your deep visual cortex via a series of
your deep visual cortex via a series of about ten neurons and the gaps between
about ten neurons and the gaps between
about ten neurons and the gaps between them we know from the neuroscience and
them we know from the neuroscience and
them we know from the neuroscience and the anatomy that it takes about ten
the anatomy that it takes about ten
the anatomy that it takes about ten milliseconds for one of those spikes to
milliseconds for one of those spikes to
milliseconds for one of those spikes to cross a gap so if we have ten gaps ten
cross a gap so if we have ten gaps ten
cross a gap so if we have ten gaps ten milliseconds times ten gaps that’s 100
milliseconds times ten gaps that’s 100
milliseconds times ten gaps that’s 100 milliseconds and that tells us that your
milliseconds and that tells us that your
milliseconds and that tells us that your brain is doing that facial recognition
brain is doing that facial recognition
brain is doing that facial recognition tasks just about as fast as you expect
tasks just about as fast as you expect
tasks just about as fast as you expect it to be able to now hold that thought
it to be able to now hold that thought
it to be able to now hold that thought because we’re going to compare this to a
because we’re going to compare this to a
because we’re going to compare this to a machine learning algorithm now modern
machine learning algorithm now modern
machine learning algorithm now modern machine learning algorithms use modeled
machine learning algorithms use modeled
machine learning algorithms use modeled neurons essentially just little
neurons essentially just little
neurons essentially just little calculators in large sets of layers the
calculators in large sets of layers the
calculators in large sets of layers the connections between those layers are
connections between those layers are
connections between those layers are just like the gaps between the neurons
just like the gaps between the neurons
just like the gaps between the neurons that we just talked about but here’s the
that we just talked about but here’s the
that we just talked about but here’s the trick calculators require a number and a
trick calculators require a number and a
trick calculators require a number and a spike isn’t a number to actually get a
spike isn’t a number to actually get a
spike isn’t a number to actually get a good estimate but the number we want to
good estimate but the number we want to
good estimate but the number we want to enter into those calculators we need to
enter into those calculators we need to
enter into those calculators we need to count a few spikes say 10 or so so what
count a few spikes say 10 or so so what
count a few spikes say 10 or so so what does a machine learning algorithm
does a machine learning algorithm
does a machine learning algorithm actually look like here’s an example of
actually look like here’s an example of
actually look like here’s an example of a very popular deep learning algorithm
a very popular deep learning algorithm
a very popular deep learning algorithm this is way way too complicated so we’re
this is way way too complicated so we’re
this is way way too complicated so we’re going to clear away everything
going to clear away everything
going to clear away everything we don’t need to think about right now
we don’t need to think about right now
we don’t need to think about right now so to decide if something is a face or
so to decide if something is a face or
so to decide if something is a face or to decide if it’s not the information
to decide if it’s not the information
to decide if it’s not the information has to flow through about eight of these
has to flow through about eight of these
has to flow through about eight of these calculators across eight gaps now we
calculators across eight gaps now we
calculators across eight gaps now we just said we need to count about ten
just said we need to count about ten
just said we need to count about ten spikes for a calculator so ten spikes
spikes for a calculator so ten spikes
spikes for a calculator so ten spikes times ten milliseconds per spike in your
times ten milliseconds per spike in your
times ten milliseconds per spike in your head that’s a hundred milliseconds times
head that’s a hundred milliseconds times
head that’s a hundred milliseconds times the eight layers that’s 800 milliseconds
the eight layers that’s 800 milliseconds
the eight layers that’s 800 milliseconds that means if you were running an alux
that means if you were running an alux
that means if you were running an alux net in your head it would take you close
net in your head it would take you close
net in your head it would take you close to a second to recognize a face kind of
to a second to recognize a face kind of
to a second to recognize a face kind of awkward at parties
so do we throw away Alec’s debt of
so do we throw away Alec’s debt of course not this is a great way to solve
course not this is a great way to solve
course not this is a great way to solve this problem in artificial systems and
this problem in artificial systems and
this problem in artificial systems and it solves a challenging problem very
it solves a challenging problem very
it solves a challenging problem very well but the thought experiment serves
well but the thought experiment serves
well but the thought experiment serves to show that our brains nature is a very
to show that our brains nature is a very
to show that our brains nature is a very different way
different way
different way to solve this type of problem and that’s
to solve this type of problem and that’s
to solve this type of problem and that’s exactly why we want to bring together
exactly why we want to bring together
exactly why we want to bring together the machine learning experts in the
the machine learning experts in the
the machine learning experts in the neuroscientists experimental observation
neuroscientists experimental observation
neuroscientists experimental observation of nature shows us how good nature could
of nature shows us how good nature could
of nature shows us how good nature could be at the task that we’re interested in
be at the task that we’re interested in
be at the task that we’re interested in solving while the machine learning
solving while the machine learning
solving while the machine learning algorithms show us what can be
algorithms show us what can be
algorithms show us what can be accomplished in artificial systems
accomplished in artificial systems
accomplished in artificial systems bringing them together opens the door to
bringing them together opens the door to
bringing them together opens the door to what I call biologically feasible
what I call biologically feasible
what I call biologically feasible computing so here’s a concrete example
computing so here’s a concrete example
computing so here’s a concrete example at the Space and Naval Warfare Systems
at the Space and Naval Warfare Systems
at the Space and Naval Warfare Systems Center Pacific a Navy research lab where
Center Pacific a Navy research lab where
Center Pacific a Navy research lab where I work we’re very interested in
I work we’re very interested in
I work we’re very interested in understanding the digital language
understanding the digital language
understanding the digital language spoken by radio transmissions now we
spoken by radio transmissions now we
spoken by radio transmissions now we tried regular machine learning to solve
tried regular machine learning to solve
tried regular machine learning to solve that problem and we were about as good
that problem and we were about as good
that problem and we were about as good at solving it as flipping a coin not
at solving it as flipping a coin not
at solving it as flipping a coin not good enough so we drew inspiration from
good enough so we drew inspiration from
good enough so we drew inspiration from the visual system the same one we’ve
the visual system the same one we’ve
the visual system the same one we’ve been talking about today and we
been talking about today and we
been talking about today and we constrained our model to look for the
constrained our model to look for the
constrained our model to look for the same things that your eyes do we then
same things that your eyes do we then
same things that your eyes do we then pointed our newly constrained model at
pointed our newly constrained model at
pointed our newly constrained model at our transmissions and it made
our transmissions and it made
our transmissions and it made essentially no mistakes
essentially no mistakes
essentially no mistakes from that point on we took a machine
from that point on we took a machine
from that point on we took a machine learning model and by constraining it we
learning model and by constraining it we
learning model and by constraining it we made it closer to a biological
made it closer to a biological
made it closer to a biological simulation in other words by better
simulation in other words by better
simulation in other words by better mimicking nature we had a dramatic
mimicking nature we had a dramatic
mimicking nature we had a dramatic performance increase in our algorithm so
performance increase in our algorithm so
performance increase in our algorithm so if simple constraints can reap great
if simple constraints can reap great
if simple constraints can reap great rewards what’s the next step
rewards what’s the next step
rewards what’s the next step places like the University of Manchester
places like the University of Manchester
places like the University of Manchester brain corporation IBM
brain corporation IBM
brain corporation IBM until and others they’re developing
until and others they’re developing
until and others they’re developing special computers that can mimic and
special computers that can mimic and
special computers that can mimic and simulate neurons and biophysical detail
simulate neurons and biophysical detail
simulate neurons and biophysical detail my research team is collaborating with
my research team is collaborating with
my research team is collaborating with those groups with a simple goal in mind
those groups with a simple goal in mind
those groups with a simple goal in mind we want to create algorithms that
we want to create algorithms that
we want to create algorithms that compute in biologically feasible ways in
compute in biologically feasible ways in
compute in biologically feasible ways in other words we want to develop
other words we want to develop
other words we want to develop algorithms that could potentially be
algorithms that could potentially be
algorithms that could potentially be running in your head and if we’re able
running in your head and if we’re able
running in your head and if we’re able to do that we tick off an important
to do that we tick off an important
to do that we tick off an important feedback loop by developing the
feedback loop by developing the
feedback loop by developing the principles for biologically feasible
principles for biologically feasible
principles for biologically feasible computing we demonstrate features that
computing we demonstrate features that
computing we demonstrate features that can be used to develop better brain
can be used to develop better brain
can be used to develop better brain models those better brain models in turn
models those better brain models in turn
models those better brain models in turn lead to better simulations the better
lead to better simulations the better
lead to better simulations the better simulations lead to new neuroscience
simulations lead to new neuroscience
simulations lead to new neuroscience hypotheses new hypotheses lead to new
hypotheses new hypotheses lead to new
hypotheses new hypotheses lead to new experiments and the new experiments
experiments and the new experiments
experiments and the new experiments illuminate new principles that can be
illuminate new principles that can be
illuminate new principles that can be used for biologically feasible computing
used for biologically feasible computing
used for biologically feasible computing by bringing machine learning and
by bringing machine learning and
by bringing machine learning and neuroscience together we make them
neuroscience together we make them
neuroscience together we make them greater than the sum of their parts now
greater than the sum of their parts now
greater than the sum of their parts now I can’t tell you how many times we have
I can’t tell you how many times we have
I can’t tell you how many times we have to go around this loop before our
to go around this loop before our
to go around this loop before our algorithms are as good as our ease or
algorithms are as good as our ease or
algorithms are as good as our ease or maxwell’s but I can tell you that this
maxwell’s but I can tell you that this
maxwell’s but I can tell you that this is the road we take towards creating
is the road we take towards creating
is the road we take towards creating that future thank you
that future thank you
that future thank you [Applause]
”
Be First to Comment