hello I teach the self-driving car engineer nano degree program at Udacity
engineer nano degree program at Udacity
engineer nano degree program at Udacity which is a nine-month program that
which is a nine-month program that
which is a nine-month program that trains software engineers to work on
trains software engineers to work on
trains software engineers to work on autonomous vehicles before I was ever a
autonomous vehicles before I was ever a
autonomous vehicles before I was ever a student at Udacity or an employee at
student at Udacity or an employee at
student at Udacity or an employee at Udacity I was actually a student I was
Udacity I was actually a student I was
Udacity I was actually a student I was for many years a normal Silicon Valley
for many years a normal Silicon Valley
for many years a normal Silicon Valley web software engineer working in Ruby on
web software engineer working in Ruby on
web software engineer working in Ruby on Rails and a few years ago I got really
Rails and a few years ago I got really
Rails and a few years ago I got really excited about self-driving cars but I
excited about self-driving cars but I
excited about self-driving cars but I didn’t have any background in robotics
didn’t have any background in robotics
didn’t have any background in robotics or system software or automotive
or system software or automotive
or system software or automotive software so I started taking classes
software so I started taking classes
software so I started taking classes online at Udacity and at other places to
online at Udacity and at other places to
online at Udacity and at other places to learn how to become an autonomous
learn how to become an autonomous
learn how to become an autonomous vehicle engineer I was hired onto the
vehicle engineer I was hired onto the
vehicle engineer I was hired onto the autonomous vehicle team at Ford Motor
autonomous vehicle team at Ford Motor
autonomous vehicle team at Ford Motor Company at their research and innovation
Company at their research and innovation
Company at their research and innovation center in Palo Alto California and later
center in Palo Alto California and later
center in Palo Alto California and later the team at Udacity asked me to come and
the team at Udacity asked me to come and
the team at Udacity asked me to come and build out a program for people to do
build out a program for people to do
build out a program for people to do essentially what I had done to learn to
essentially what I had done to learn to
essentially what I had done to learn to become autonomous vehicle engineers at
become autonomous vehicle engineers at
become autonomous vehicle engineers at Udacity we have our own self-driving car
Udacity we have our own self-driving car
Udacity we have our own self-driving car her name is Carla and I’d like to tell
her name is Carla and I’d like to tell
her name is Carla and I’d like to tell you a little bit about how Carla works
you a little bit about how Carla works
you a little bit about how Carla works and how most self-driving cars work
and how most self-driving cars work
and how most self-driving cars work through the lens of some of the projects
through the lens of some of the projects
through the lens of some of the projects that our students have done as part of
that our students have done as part of
that our students have done as part of the nanodegree program so I think of
the nanodegree program so I think of
the nanodegree program so I think of self-driving cars as having five core
self-driving cars as having five core
self-driving cars as having five core components computer vision sensor fusion
components computer vision sensor fusion
components computer vision sensor fusion localization path planning and control
localization path planning and control
localization path planning and control computer vision is how we use camera
computer vision is how we use camera
computer vision is how we use camera images to figure out what the world
images to figure out what the world
images to figure out what the world around us looks like and sensor fusion
around us looks like and sensor fusion
around us looks like and sensor fusion is how we incorporate data from other
is how we incorporate data from other
is how we incorporate data from other sensors like lasers and radar to get a
sensors like lasers and radar to get a
sensors like lasers and radar to get a richer understanding of our environment
richer understanding of our environment
richer understanding of our environment once we’ve built this deep understanding
once we’ve built this deep understanding
once we’ve built this deep understanding of what the world around us is we use
of what the world around us is we use
of what the world around us is we use localization to figure out precisely
localization to figure out precisely
localization to figure out precisely where we are in that world and then once
where we are in that world and then once
where we are in that world and then once we figure it out where we are in the
we figure it out where we are in the
we figure it out where we are in the world and what the world looks like we
world and what the world looks like we
world and what the world looks like we use path planning to chart a course
use path planning to chart a course
use path planning to chart a course through the world to get us to where
through the world to get us to where
through the world to get us to where we’d like to go
we’d like to go
we’d like to go and then the final step is control which
and then the final step is control which
and then the final step is control which is how we actually turn the steering
is how we actually turn the steering
is how we actually turn the steering wheel and hit the throttle and hit the
wheel and hit the throttle and hit the
wheel and hit the throttle and hit the brake in order to execute the trajectory
brake in order to execute the trajectory
brake in order to execute the trajectory that we built during path planning so
that we built during path planning so
that we built during path planning so these are the five core components of
these are the five core components of
these are the five core components of self-driving cars and I’d like to
self-driving cars and I’d like to
self-driving cars and I’d like to investigate each of them with you this
investigate each of them with you this
investigate each of them with you this is computer vision this is a video that
is computer vision this is a video that
is computer vision this is a video that Carla took driving up highway 280 in
Carla took driving up highway 280 in
Carla took driving up highway 280 in Silicon Valley and this has work that
Silicon Valley and this has work that
Silicon Valley and this has work that one of our students the Vecchia dab did
one of our students the Vecchia dab did
one of our students the Vecchia dab did vivec is using computer vision so
vivec is using computer vision so
vivec is using computer vision so looking for colors and edges and
looking for colors and edges and
looking for colors and edges and gradients to find the lane on the road
gradients to find the lane on the road
gradients to find the lane on the road and then he’s trained a deep neural
and then he’s trained a deep neural
and then he’s trained a deep neural network to draw bounding boxes around
network to draw bounding boxes around
network to draw bounding boxes around the other vehicles on the road deep
the other vehicles on the road deep
the other vehicles on the road deep neural networks or deep learning is an
neural networks or deep learning is an
neural networks or deep learning is an exciting new part of machine learning
exciting new part of machine learning
exciting new part of machine learning and artificial intelligence and it’s a
and artificial intelligence and it’s a
and artificial intelligence and it’s a way that computers can learn what cars
way that computers can learn what cars
way that computers can learn what cars and other objects look like simply by
and other objects look like simply by
and other objects look like simply by sending them lots and lots of data they
sending them lots and lots of data they
sending them lots and lots of data they see lots of cars and they learn what
see lots of cars and they learn what
see lots of cars and they learn what cars look like and you can see the deep
cars look like and you can see the deep
cars look like and you can see the deep neural network is trying to figure out
neural network is trying to figure out
neural network is trying to figure out is that one car is that two cars and as
is that one car is that two cars and as
is that one car is that two cars and as they separate it sees that it’s a couple
they separate it sees that it’s a couple
they separate it sees that it’s a couple of cars and this is pretty similar to
of cars and this is pretty similar to
of cars and this is pretty similar to what advanced driver assistance systems
what advanced driver assistance systems
what advanced driver assistance systems do on the road today so once we know
do on the road today so once we know
do on the road today so once we know what the world looks like through camera
what the world looks like through camera
what the world looks like through camera images the next step is to augment that
images the next step is to augment that
images the next step is to augment that understanding of the world using other
understanding of the world using other
understanding of the world using other sensors so radar and lasers to get
sensors so radar and lasers to get
sensors so radar and lasers to get measurements that are difficult for a
measurements that are difficult for a
measurements that are difficult for a camera alone to understand so things
camera alone to understand so things
camera alone to understand so things like the distance between us and other
like the distance between us and other
like the distance between us and other cars and how fast other objects in the
cars and how fast other objects in the
cars and how fast other objects in the environment are moving and what you see
environment are moving and what you see
environment are moving and what you see here is data from a trip that Carla took
here is data from a trip that Carla took
here is data from a trip that Carla took up El Camino Real which is the main
up El Camino Real which is the main
up El Camino Real which is the main commercial street in Silicon Valley and
commercial street in Silicon Valley and
commercial street in Silicon Valley and you see on the right the video feed of
you see on the right the video feed of
you see on the right the video feed of what that drive looks like and the main
what that drive looks like and the main
what that drive looks like and the main image here is the laser scan of what the
image here is the laser scan of what the
image here is the laser scan of what the world around Carla looks like and you
world around Carla looks like and you
world around Carla looks like and you can see this lidar which has an array of
can see this lidar which has an array of
can see this lidar which has an array of lasers doing a 360 degree scan of the
lasers doing a 360 degree scan of the
lasers doing a 360 degree scan of the world and seeing what the different
world and seeing what the different
world and seeing what the different objects in that environment look like
objects in that environment look like
objects in that environment look like and how they move so once we understand
and how they move so once we understand
and how they move so once we understand both what the world looks like and how
both what the world looks like and how
both what the world looks like and how to measure it and we incorporate those
to measure it and we incorporate those
to measure it and we incorporate those understandings together to get a rich
understandings together to get a rich
understandings together to get a rich picture of our surrounding environment
picture of our surrounding environment
picture of our surrounding environment the next step is to local
the next step is to local
the next step is to local ourselves in that environment and you
ourselves in that environment and you
ourselves in that environment and you might think this is really
might think this is really
might think this is really straightforward because we all have cell
straightforward because we all have cell
straightforward because we all have cell phones today they all have GPS in them
phones today they all have GPS in them
phones today they all have GPS in them we all know where we are all of the time
we all know where we are all of the time
we all know where we are all of the time but in fact GPS is only accurate to
but in fact GPS is only accurate to
but in fact GPS is only accurate to within about one to two meters and if
within about one to two meters and if
within about one to two meters and if you think about how big one to two
you think about how big one to two
you think about how big one to two meters is if a car is wrong about where
meters is if a car is wrong about where
meters is if a car is wrong about where it is by one to two meters
it is by one to two meters
it is by one to two meters it could be over here on the sidewalk
it could be over here on the sidewalk
it could be over here on the sidewalk hitting things that would be really bad
hitting things that would be really bad
hitting things that would be really bad so we have to use much more
so we have to use much more
so we have to use much more sophisticated mathematical algorithms as
sophisticated mathematical algorithms as
sophisticated mathematical algorithms as well as high-definition maps to localize
well as high-definition maps to localize
well as high-definition maps to localize our vehicle precisely in its environment
our vehicle precisely in its environment
our vehicle precisely in its environment to single-digit centimeter level
to single-digit centimeter level
to single-digit centimeter level accuracy so this is work by one of our
accuracy so this is work by one of our
accuracy so this is work by one of our students his name is Daniel Lopez Madrid
students his name is Daniel Lopez Madrid
students his name is Daniel Lopez Madrid and what Daniel is doing is he is using
and what Daniel is doing is he is using
and what Daniel is doing is he is using a particle filter which is a really
a particle filter which is a really
a particle filter which is a really sophisticated type of triangulation and
sophisticated type of triangulation and
sophisticated type of triangulation and as his blue vehicle moves through the
as his blue vehicle moves through the
as his blue vehicle moves through the simulator it’s measuring its distance
simulator it’s measuring its distance
simulator it’s measuring its distance from various landmarks and it’s figuring
from various landmarks and it’s figuring
from various landmarks and it’s figuring out how far it is from these landmarks
out how far it is from these landmarks
out how far it is from these landmarks and where it sees the landmarks and
and where it sees the landmarks and
and where it sees the landmarks and comparing that to the map and using that
comparing that to the map and using that
comparing that to the map and using that to figure out precisely where it is in
to figure out precisely where it is in
to figure out precisely where it is in the world and this is how self-driving
the world and this is how self-driving
the world and this is how self-driving cars localize in the real world those
cars localize in the real world those
cars localize in the real world those landmarks might be things like street
landmarks might be things like street
landmarks might be things like street lights or traffic signs or mailboxes or
lights or traffic signs or mailboxes or
lights or traffic signs or mailboxes or even manhole covers so once we figured
even manhole covers so once we figured
even manhole covers so once we figured out where we are in the world and what
out where we are in the world and what
out where we are in the world and what the world around us looks like the next
the world around us looks like the next
the world around us looks like the next step is to actually chart a path through
step is to actually chart a path through
step is to actually chart a path through that world to figure out how to get
that world to figure out how to get
that world to figure out how to get where we want to go and this is path
where we want to go and this is path
where we want to go and this is path planning here you can see the work by
planning here you can see the work by
planning here you can see the work by one of our students Kaz a hero Nakagawa
one of our students Kaz a hero Nakagawa
one of our students Kaz a hero Nakagawa and what Kaz a hero has done is in our
and what Kaz a hero has done is in our
and what Kaz a hero has done is in our Udacity highway simulator he’s built a
Udacity highway simulator he’s built a
Udacity highway simulator he’s built a path planner that predicts where the
path planner that predicts where the
path planner that predicts where the other vehicles on the road are going to
other vehicles on the road are going to
other vehicles on the road are going to go and then figures out what maneuver
go and then figures out what maneuver
go and then figures out what maneuver his vehicle should take in response and
his vehicle should take in response and
his vehicle should take in response and then finally builds a series of
then finally builds a series of
then finally builds a series of waypoints those are those Green pearls
waypoints those are those Green pearls
waypoints those are those Green pearls you see in the video for the car to
you see in the video for the car to
you see in the video for the car to drive through that’s the trajectory the
drive through that’s the trajectory the
drive through that’s the trajectory the car should follow and you see when his
car should follow and you see when his
car should follow and you see when his vehicle comes up on other traffic it has
vehicle comes up on other traffic it has
vehicle comes up on other traffic it has to figure out should it slow down and
to figure out should it slow down and
to figure out should it slow down and stay in its lane or should it shift
stay in its lane or should it shift
stay in its lane or should it shift right or shift left and this is in fact
right or shift left and this is in fact
right or shift left and this is in fact the type of decision that real
the type of decision that real
the type of decision that real self-driving cars have to make all the
self-driving cars have to make all the
self-driving cars have to make all the time subject to constraints like speed
time subject to constraints like speed
time subject to constraints like speed limits and acceleration limit so the
limits and acceleration limit so the
limits and acceleration limit so the ride is comfortable and you can imagine
ride is comfortable and you can imagine
ride is comfortable and you can imagine this gets
this gets
this gets more complicated and urban driving where
more complicated and urban driving where
more complicated and urban driving where there are a lot more intersections and
there are a lot more intersections and
there are a lot more intersections and different maneuvers you can make but in
different maneuvers you can make but in
different maneuvers you can make but in principle the decisions are all the same
principle the decisions are all the same
principle the decisions are all the same so once we figured out what the path we
so once we figured out what the path we
so once we figured out what the path we want to take through this world is and
want to take through this world is and
want to take through this world is and we know where we are in the world and we
we know where we are in the world and we
we know where we are in the world and we know what the world looks like the final
know what the world looks like the final
know what the world looks like the final step in the pipeline is control and
step in the pipeline is control and
step in the pipeline is control and control is how we actually turn the
control is how we actually turn the
control is how we actually turn the steering wheel and hit the throttle and
steering wheel and hit the throttle and
steering wheel and hit the throttle and hit the brake
hit the brake
hit the brake in order to execute that trajectory that
in order to execute that trajectory that
in order to execute that trajectory that we built during path planning so this is
we built during path planning so this is
we built during path planning so this is one of our students Emmy wow this is
one of our students Emmy wow this is
one of our students Emmy wow this is Udacity z– racetrack simulator and she
Udacity z– racetrack simulator and she
Udacity z– racetrack simulator and she has a yellow line here that’s the ideal
has a yellow line here that’s the ideal
has a yellow line here that’s the ideal trajectory that her car should follow
trajectory that her car should follow
trajectory that her car should follow and what you see is there’s this green
and what you see is there’s this green
and what you see is there’s this green line layered on top of the yellow line
line layered on top of the yellow line
line layered on top of the yellow line in the Green Line is the trajectory that
in the Green Line is the trajectory that
in the Green Line is the trajectory that her vehicle predicts it’s going to
her vehicle predicts it’s going to
her vehicle predicts it’s going to follow based on the steering wheel and
follow based on the steering wheel and
follow based on the steering wheel and throttle and brake commands that it’s
throttle and brake commands that it’s
throttle and brake commands that it’s sending and what we love is for that
sending and what we love is for that
sending and what we love is for that green line to be 100% aligned with the
green line to be 100% aligned with the
green line to be 100% aligned with the yellow line but of course that’s really
yellow line but of course that’s really
yellow line but of course that’s really difficult and there’s a human driver you
difficult and there’s a human driver you
difficult and there’s a human driver you probably know this it’s really difficult
probably know this it’s really difficult
probably know this it’s really difficult particularly on tight turns to follow
particularly on tight turns to follow
particularly on tight turns to follow the exact line through the middle of the
the exact line through the middle of the
the exact line through the middle of the lane that you’d like to follow but you
lane that you’d like to follow but you
lane that you’d like to follow but you do pretty well with it you managed to
do pretty well with it you managed to
do pretty well with it you managed to drive and in fact computers are really
drive and in fact computers are really
drive and in fact computers are really really good at this and the distance
really good at this and the distance
really good at this and the distance between the trajectory they’d like to
between the trajectory they’d like to
between the trajectory they’d like to follow and the trajectory they actually
follow and the trajectory they actually
follow and the trajectory they actually follow is really really close so that’s
follow is really really close so that’s
follow is really really close so that’s how self-driving cars work at a high
how self-driving cars work at a high
how self-driving cars work at a high level we use computer vision and sensor
level we use computer vision and sensor
level we use computer vision and sensor fusion to get this rich picture of where
fusion to get this rich picture of where
fusion to get this rich picture of where we are in the world we use localization
we are in the world we use localization
we are in the world we use localization to nail down precisely our location in
to nail down precisely our location in
to nail down precisely our location in that rich world we use path planning to
that rich world we use path planning to
that rich world we use path planning to chart the course through the world to
chart the course through the world to
chart the course through the world to get us where we’d like to go and we use
get us where we’d like to go and we use
get us where we’d like to go and we use control to turn the steering wheel and
control to turn the steering wheel and
control to turn the steering wheel and hit the throttle and hit the brake to
hit the throttle and hit the brake to
hit the throttle and hit the brake to execute that trajectory and ultimately
execute that trajectory and ultimately
execute that trajectory and ultimately move the vehicle everything else
move the vehicle everything else
move the vehicle everything else self-driving cars do are essentially
self-driving cars do are essentially
self-driving cars do are essentially more sophisticated implementations of
more sophisticated implementations of
more sophisticated implementations of these key functions and this is how
these key functions and this is how
these key functions and this is how self-driving cars work it is a really
self-driving cars work it is a really
self-driving cars work it is a really exciting time to work on self-driving
exciting time to work on self-driving
exciting time to work on self-driving cars computers are getting better and
cars computers are getting better and
cars computers are getting better and better at these tasks all of the time
better at these tasks all of the time
better at these tasks all of the time and I hope you see self-driving cars on
and I hope you see self-driving cars on
and I hope you see self-driving cars on a road near you soon my name is David
a road near you soon my name is David
a road near you soon my name is David silver I teach self-driving cars at
silver I teach self-driving cars at
silver I teach self-driving cars at Udacity thank you very much
Udacity thank you very much
Udacity thank you very much you
”
Be First to Comment