drone cameras are everywhere in the last few years
few years
few years Hollywood cinematographers and hobbyists
Hollywood cinematographers and hobbyists
Hollywood cinematographers and hobbyists have started to use this new technology
have started to use this new technology
have started to use this new technology to capture shots that would be almost
to capture shots that would be almost
to capture shots that would be almost impossible to get otherwise that’s an
impossible to get otherwise that’s an
impossible to get otherwise that’s an example here’s some pretty dramatic
example here’s some pretty dramatic
example here’s some pretty dramatic drone footage of the Hoover tower on the
drone footage of the Hoover tower on the
drone footage of the Hoover tower on the Stanford campus you tough crowd now what
Stanford campus you tough crowd now what
Stanford campus you tough crowd now what might be less obvious from watching this
might be less obvious from watching this
might be less obvious from watching this clip is that it was captured fully
clip is that it was captured fully
clip is that it was captured fully autonomously with no human pilot to get
autonomously with no human pilot to get
autonomously with no human pilot to get this shot we used an app that I built
this shot we used an app that I built
this shot we used an app that I built with my colleagues in the Stanford
with my colleagues in the Stanford
with my colleagues in the Stanford computer graphics lab the app lets you
computer graphics lab the app lets you
computer graphics lab the app lets you set up shots ahead of time in a virtual
set up shots ahead of time in a virtual
set up shots ahead of time in a virtual environment and then capture them later
environment and then capture them later
environment and then capture them later fully automatically on a real drone so
fully automatically on a real drone so
fully automatically on a real drone so you don’t need to know how to fly a
you don’t need to know how to fly a
you don’t need to know how to fly a drone at all and you can still get the
drone at all and you can still get the
drone at all and you can still get the shots you want I want to share our
shots you want I want to share our
shots you want I want to share our design process with you of how we built
design process with you of how we built
design process with you of how we built the app and also how we can apply the
the app and also how we can apply the
the app and also how we can apply the underlying ideas more broadly to unlock
underlying ideas more broadly to unlock
underlying ideas more broadly to unlock the enormous creative power of drones
but first let’s take a step back to
but first let’s take a step back to understand the thinking that went into
understand the thinking that went into
understand the thinking that went into our app we need to understand this photo
our app we need to understand this photo
our app we need to understand this photo the man in the photo is the actor James
the man in the photo is the actor James
the man in the photo is the actor James Dean and it was taken just before his
Dean and it was taken just before his
Dean and it was taken just before his first movie came out eventually this
first movie came out eventually this
first movie came out eventually this photo went on to become one of the most
photo went on to become one of the most
photo went on to become one of the most iconic images of the 20th century but
iconic images of the 20th century but
iconic images of the 20th century but part of what makes this photo so special
part of what makes this photo so special
part of what makes this photo so special is the technical craftsmanship that went
is the technical craftsmanship that went
is the technical craftsmanship that went into it here we’re seeing the original
into it here we’re seeing the original
into it here we’re seeing the original annotations by Pablo and Ariel the
annotations by Pablo and Ariel the
annotations by Pablo and Ariel the darkroom artist whose job it was to
darkroom artist whose job it was to
darkroom artist whose job it was to develop the photo and in these
develop the photo and in these
develop the photo and in these annotations we see just exquisite
annotations we see just exquisite
annotations we see just exquisite attention to detail where he would use a
attention to detail where he would use a
attention to detail where he would use a different mixture of chemicals and light
different mixture of chemicals and light
different mixture of chemicals and light in each region of the image to highlight
in each region of the image to highlight
in each region of the image to highlight the interesting stuff I mean can you
the interesting stuff I mean can you
the interesting stuff I mean can you imagine putting this much effort into
imagine putting this much effort into
imagine putting this much effort into your most recent brunch selfie
your most recent brunch selfie
your most recent brunch selfie some people are nodding like uh yeah
some people are nodding like uh yeah
some people are nodding like uh yeah definitely my point here is that
definitely my point here is that
definitely my point here is that photography used to be a highly
photography used to be a highly
photography used to be a highly technical endeavor as well as a creative
technical endeavor as well as a creative
technical endeavor as well as a creative one fast forward to today we all have
one fast forward to today we all have
one fast forward to today we all have access to essentially Studio quality
access to essentially Studio quality
access to essentially Studio quality photography with a single button click
photography with a single button click
photography with a single button click on our phones for many of us photography
on our phones for many of us photography
on our phones for many of us photography has become non technical photography has
has become non technical photography has
has become non technical photography has been democratized but drone photography
been democratized but drone photography
been democratized but drone photography has not not yet here we’re seeing a
has not not yet here we’re seeing a
has not not yet here we’re seeing a couple of expert drone pilots as though
couple of expert drone pilots as though
couple of expert drone pilots as though they’re on set capturing footage for a
they’re on set capturing footage for a
they’re on set capturing footage for a music video the guy on the left is
music video the guy on the left is
music video the guy on the left is flying the drone the guy on the right is
flying the drone the guy on the right is
flying the drone the guy on the right is controlling the camera and sometimes
controlling the camera and sometimes
controlling the camera and sometimes this team even works with a third expert
this team even works with a third expert
this team even works with a third expert whose entire job is just to control the
whose entire job is just to control the
whose entire job is just to control the cameras focus and the whole time that
cameras focus and the whole time that
cameras focus and the whole time that they’re doing this they have to use
they’re doing this they have to use
they’re doing this they have to use these remote control helicopter style
these remote control helicopter style
these remote control helicopter style joysticks that can be counterintuitive
joysticks that can be counterintuitive
joysticks that can be counterintuitive to use so in some sense this is the
to use so in some sense this is the
to use so in some sense this is the modern equivalent of spending all day in
modern equivalent of spending all day in
modern equivalent of spending all day in the darkroom painstakingly developing an
the darkroom painstakingly developing an
the darkroom painstakingly developing an analogue photo so we saw how hard it was
analogue photo so we saw how hard it was
analogue photo so we saw how hard it was for people to use drones in these
for people to use drones in these
for people to use drones in these demanding creative scenarios we thought
demanding creative scenarios we thought
demanding creative scenarios we thought what if we built a flight planning app
what if we built a flight planning app
what if we built a flight planning app specifically for drone cinematography
specifically for drone cinematography
specifically for drone cinematography can we keep the creativity but remove
can we keep the creativity but remove
can we keep the creativity but remove some of the technical barriers
some of the technical barriers
some of the technical barriers associated with working with drones it
associated with working with drones it
associated with working with drones it took us a while to wrap our heads around
took us a while to wrap our heads around
took us a while to wrap our heads around what this meant but we eventually
what this meant but we eventually
what this meant but we eventually figured something out so let me show you
figured something out so let me show you
figured something out so let me show you what we came up with the most important
what we came up with the most important
what we came up with the most important thing in our app is the 3d preview
thing in our app is the 3d preview
thing in our app is the 3d preview window the idea here is that you set up
window the idea here is that you set up
window the idea here is that you set up the shots you want by navigating around
the shots you want by navigating around
the shots you want by navigating around in 3d and specifying the still framing
in 3d and specifying the still framing
in 3d and specifying the still framing that you want at specific times then
that you want at specific times then
that you want at specific times then behind the scenes our app synthesizes a
behind the scenes our app synthesizes a
behind the scenes our app synthesizes a smooth drone trajectory that connects
smooth drone trajectory that connects
smooth drone trajectory that connects the still frames and obeys the laws of
the still frames and obeys the laws of
the still frames and obeys the laws of drone physics we also have this 2d map
drone physics we also have this 2d map
drone physics we also have this 2d map where you can get an overview of your
where you can get an overview of your
where you can get an overview of your shot and we have these curves that allow
shot and we have these curves that allow
shot and we have these curves that allow you to control the timing of your shot
you to control the timing of your shot
you to control the timing of your shot now so far this interface is very
now so far this interface is very
now so far this interface is very similar to how an animator at Pixar we
similar to how an animator at Pixar we
similar to how an animator at Pixar we set up a camera path in a virtual
set up a camera path in a virtual
set up a camera path in a virtual environment but we have to do a bit of
environment but we have to do a bit of
environment but we have to do a bit of extra stuff to make sure that our user
extra stuff to make sure that our user
extra stuff to make sure that our user interface works with drone cameras and
interface works with drone cameras and
interface works with drone cameras and that’s because drone cameras are
that’s because drone cameras are
that’s because drone cameras are physical objects if I want to get a
physical objects if I want to get a
physical objects if I want to get a drone to do anything I need to apply
drone to do anything I need to apply
drone to do anything I need to apply just the right thrusts on each of the
just the right thrusts on each of the
just the right thrusts on each of the propellers at just the right times it’s
propellers at just the right times it’s
propellers at just the right times it’s like a whole symphony of gas pedals then
like a whole symphony of gas pedals then
like a whole symphony of gas pedals then I need to play just right to get the
I need to play just right to get the
I need to play just right to get the drone to do what I want and there’s
drone to do what I want and there’s
drone to do what I want and there’s physical limits to the kinds of
physical limits to the kinds of
physical limits to the kinds of maneuvers the drones can actually do so
maneuvers the drones can actually do so
maneuvers the drones can actually do so to account for these limitations we
to account for these limitations we
to account for these limitations we added these feasibility plots to the
added these feasibility plots to the
added these feasibility plots to the side of our app the idea here is that as
side of our app the idea here is that as
side of our app the idea here is that as you’re setting up a shot we compute all
you’re setting up a shot we compute all
you’re setting up a shot we compute all the required thrusts and velocities and
the required thrusts and velocities and
the required thrusts and velocities and that kind of thing we display them on
that kind of thing we display them on
that kind of thing we display them on these plots we also display the drones
these plots we also display the drones
these plots we also display the drones physical limits with these horizontal
physical limits with these horizontal
physical limits with these horizontal red lines so if you’re setting up a shot
red lines so if you’re setting up a shot
red lines so if you’re setting up a shot that exceeds your drones limits if
that exceeds your drones limits if
that exceeds your drones limits if you’re setting up a shot that is
you’re setting up a shot that is
you’re setting up a shot that is infeasible then you get visual feedback
infeasible then you get visual feedback
infeasible then you get visual feedback right away and you can adjust your shot
right away and you can adjust your shot
right away and you can adjust your shot accordingly then at any time during the
accordingly then at any time during the
accordingly then at any time during the design process you can preview what your
design process you can preview what your
design process you can preview what your shots gonna look like and because we’re
shots gonna look like and because we’re
shots gonna look like and because we’re reasoning you know we’re going to some
reasoning you know we’re going to some
reasoning you know we’re going to some lengths to reason about the physics in
lengths to reason about the physics in
lengths to reason about the physics in this app these previews tend to be quite
this app these previews tend to be quite
this app these previews tend to be quite accurate
one of the most inspirational takeaways
one of the most inspirational takeaways from this project was seeing the shots
from this project was seeing the shots
from this project was seeing the shots the total beginners could create this
the total beginners could create this
the total beginners could create this shot was created by one of our beginner
shot was created by one of our beginner
shot was created by one of our beginner users that had never flown a drone
users that had never flown a drone
users that had never flown a drone before we also brought in some super
before we also brought in some super
before we also brought in some super expert users to try out our app
expert users to try out our app
expert users to try out our app including some guys that get paid to
including some guys that get paid to
including some guys that get paid to travel around the world and shoot drone
travel around the world and shoot drone
travel around the world and shoot drone videos but this shot from the total
videos but this shot from the total
videos but this shot from the total beginner ended up being my favorite it
beginner ended up being my favorite it
beginner ended up being my favorite it was almost like the beginner users had
was almost like the beginner users had
was almost like the beginner users had never had their brains polluted by how
never had their brains polluted by how
never had their brains polluted by how hard it was to fly a drone manually and
hard it was to fly a drone manually and
hard it was to fly a drone manually and their creativity could just flow however
their creativity could just flow however
their creativity could just flow however there’s a problem I kind of swept it
there’s a problem I kind of swept it
there’s a problem I kind of swept it under the rug so far so far in what I
under the rug so far so far in what I
under the rug so far so far in what I showed you our our app just notifies you
showed you our our app just notifies you
showed you our our app just notifies you if your shot is infeasible but it
if your shot is infeasible but it
if your shot is infeasible but it doesn’t do anything to help you make it
doesn’t do anything to help you make it
doesn’t do anything to help you make it feasible
feasible
feasible this behavior is kind of like a
this behavior is kind of like a
this behavior is kind of like a spellchecker that can tell you your
spellchecker that can tell you your
spellchecker that can tell you your spelling a word wrong but doesn’t offer
spelling a word wrong but doesn’t offer
spelling a word wrong but doesn’t offer any suggestions for how to spell it
any suggestions for how to spell it
any suggestions for how to spell it correctly now of course in our research
correctly now of course in our research
correctly now of course in our research paper we were like hey that’s not a bug
paper we were like hey that’s not a bug
paper we were like hey that’s not a bug that’s a feature come on it’s more
that’s a feature come on it’s more
that’s a feature come on it’s more expressive that way but honestly I never
expressive that way but honestly I never
expressive that way but honestly I never thought that this was a feature I always
thought that this was a feature I always
thought that this was a feature I always thought that this was a bug and to see
thought that this was a bug and to see
thought that this was a bug and to see why it’s a bug let’s have a look at a
why it’s a bug let’s have a look at a
why it’s a bug let’s have a look at a typical shot that you might set up in
typical shot that you might set up in
typical shot that you might set up in our app and in this example I’m color
our app and in this example I’m color
our app and in this example I’m color coding the shot according to whether or
coding the shot according to whether or
coding the shot according to whether or not it’s feasible where blue is feasible
not it’s feasible where blue is feasible
not it’s feasible where blue is feasible and red is infeasible so notice that
and red is infeasible so notice that
and red is infeasible so notice that this shot is not even close to feasible
this shot is not even close to feasible
this shot is not even close to feasible in the middle of the shot it requires
in the middle of the shot it requires
in the middle of the shot it requires almost three times as much thrust and
almost three times as much thrust and
almost three times as much thrust and three times as much speed as our drone
three times as much speed as our drone
three times as much speed as our drone can deliver if you tried to execute this
can deliver if you tried to execute this
can deliver if you tried to execute this shot on a real drone its behavior would
shot on a real drone its behavior would
shot on a real drone its behavior would be unpredictable it would have to cut
be unpredictable it would have to cut
be unpredictable it would have to cut corners to try to keep up with where
corners to try to keep up with where
corners to try to keep up with where it’s supposed to be and it could easily
it’s supposed to be and it could easily
it’s supposed to be and it could easily crash into this building or something
crash into this building or something
crash into this building or something else in the environment in general
else in the environment in general
else in the environment in general adhering to these physical limits is
adhering to these physical limits is
adhering to these physical limits is important for safety so I really wanted
important for safety so I really wanted
important for safety so I really wanted our app to deal with these physical
our app to deal with these physical
our app to deal with these physical limits automatically the solution we
limits automatically the solution we
limits automatically the solution we came up with is that the app is going to
came up with is that the app is going to
came up with is that the app is going to autocorrect this shot for us but in a
autocorrect this shot for us but in a
autocorrect this shot for us but in a way that respects the drone physics and
way that respects the drone physics and
way that respects the drone physics and under the hood the app is just looking
under the hood the app is just looking
under the hood the app is just looking for opportunities to just slightly bend
for opportunities to just slightly bend
for opportunities to just slightly bend the timing of the shot to make it less
the timing of the shot to make it less
the timing of the shot to make it less aggressive so here’s where we slightly
aggressive so here’s where we slightly
aggressive so here’s where we slightly bend the timing and notice that the auto
bend the timing and notice that the auto
bend the timing and notice that the auto corrected shot is bumping right up
corrected shot is bumping right up
corrected shot is bumping right up against the physical limits but not
against the physical limits but not
against the physical limits but not going past them so we’re keeping as much
going past them so we’re keeping as much
going past them so we’re keeping as much of the excitement and pacing of the
of the excitement and pacing of the
of the excitement and pacing of the original shot as possible this
original shot as possible this
original shot as possible this autocorrect feature expands the creative
autocorrect feature expands the creative
autocorrect feature expands the creative possibilities for beginner users because
possibilities for beginner users because
possibilities for beginner users because it’s easy and safe now for beginners to
it’s easy and safe now for beginners to
it’s easy and safe now for beginners to set up shots that are at the physical
set up shots that are at the physical
set up shots that are at the physical limits of what their drone can do this
limits of what their drone can do this
limits of what their drone can do this is that same shot I showed on the
is that same shot I showed on the
is that same shot I showed on the previous slide after it’s been auto
previous slide after it’s been auto
previous slide after it’s been auto corrected by our app and there’s me and
corrected by our app and there’s me and
corrected by our app and there’s me and a colleague running after the drone
and this is probably the most aggressive
and this is probably the most aggressive drone footage of the Stanford campus
drone footage of the Stanford campus
drone footage of the Stanford campus that’s ever been recorded and it’s worth
that’s ever been recorded and it’s worth
that’s ever been recorded and it’s worth mentioning that I’m not speeding up the
mentioning that I’m not speeding up the
mentioning that I’m not speeding up the video at all so I’m just playing it back
video at all so I’m just playing it back
video at all so I’m just playing it back at 1x speed so we’ve seen how drones can
at 1x speed so we’ve seen how drones can
at 1x speed so we’ve seen how drones can be used to create compelling
be used to create compelling
be used to create compelling cinematography but they can also be used
cinematography but they can also be used
cinematography but they can also be used to create compelling virtual worlds for
to create compelling virtual worlds for
to create compelling virtual worlds for our video games and our VR experiences
our video games and our VR experiences
our video games and our VR experiences and that’s because drones can be used to
and that’s because drones can be used to
and that’s because drones can be used to scan our physical world in 3d to
scan our physical world in 3d to
scan our physical world in 3d to demonstrate this idea let’s try to scan
demonstrate this idea let’s try to scan
demonstrate this idea let’s try to scan this industrial site that’s actually
this industrial site that’s actually
this industrial site that’s actually just down the street from here in dare I
just down the street from here in dare I
just down the street from here in dare I say a slightly less picturesque part of
say a slightly less picturesque part of
say a slightly less picturesque part of Berkeley the standard way that you would
Berkeley the standard way that you would
Berkeley the standard way that you would scan this site with a drone is to fly
scan this site with a drone is to fly
scan this site with a drone is to fly overhead at a safe height above the
overhead at a safe height above the
overhead at a safe height above the scene in kind of a lawnmower shape and
scene in kind of a lawnmower shape and
scene in kind of a lawnmower shape and maybe you’d do an extra loop around the
maybe you’d do an extra loop around the
maybe you’d do an extra loop around the scene if you had enough battery this is
scene if you had enough battery this is
scene if you had enough battery this is what existing flight planning software
what existing flight planning software
what existing flight planning software for 3d scanning would do for you today
for 3d scanning would do for you today
for 3d scanning would do for you today it’s also kind of a microcosm for how
it’s also kind of a microcosm for how
it’s also kind of a microcosm for how Google Earth acquires its 3d data but
Google Earth acquires its 3d data but
Google Earth acquires its 3d data but our approach is gonna be a little bit
our approach is gonna be a little bit
our approach is gonna be a little bit different at first we’re just gonna do
different at first we’re just gonna do
different at first we’re just gonna do that quick loop around the scene like
that quick loop around the scene like
that quick loop around the scene like this and based on this quick loop we’re
this and based on this quick loop we’re
this and based on this quick loop we’re gonna reconstruct approximate scene
gonna reconstruct approximate scene
gonna reconstruct approximate scene geometry and for this step the geometry
geometry and for this step the geometry
geometry and for this step the geometry we reconstruct does not have to be
we reconstruct does not have to be
we reconstruct does not have to be perfect as you can see here this
perfect as you can see here this
perfect as you can see here this geometry is really nothing to write home
geometry is really nothing to write home
geometry is really nothing to write home about but it turns out that it’s going
about but it turns out that it’s going
about but it turns out that it’s going to be really useful in deciding where to
to be really useful in deciding where to
to be really useful in deciding where to fly next so based on that approximate
fly next so based on that approximate
fly next so based on that approximate geometry we generate this scanning
geometry we generate this scanning
geometry we generate this scanning trajectory that’s customized and
trajectory that’s customized and
trajectory that’s customized and tailor-made for this specific scene and
tailor-made for this specific scene and
tailor-made for this specific scene and the idea here is that we’re gonna try to
the idea here is that we’re gonna try to
the idea here is that we’re gonna try to get into all the nooks and crannies of
get into all the nooks and crannies of
get into all the nooks and crannies of the scene and try to observe it from as
the scene and try to observe it from as
the scene and try to observe it from as many different views as possible this
many different views as possible this
many different views as possible this will make it easier for our computer
will make it easier for our computer
will make it easier for our computer vision algorithms to do a more detailed
vision algorithms to do a more detailed
vision algorithms to do a more detailed 3d reconstruction for example this is
3d reconstruction for example this is
3d reconstruction for example this is the best 3d reconstruction that you can
the best 3d reconstruction that you can
the best 3d reconstruction that you can get from doing that standard overhead
get from doing that standard overhead
get from doing that standard overhead lawnmower pattern
lawnmower pattern
lawnmower pattern and this is the 3d reconstruction we get
and this is the 3d reconstruction we get
and this is the 3d reconstruction we get from our method I’ll toggle that one
from our method I’ll toggle that one
from our method I’ll toggle that one more time
more time
more time standard overhead lawnmower ours so
standard overhead lawnmower ours so
standard overhead lawnmower ours so we’re starting to get to the point where
we’re starting to get to the point where
we’re starting to get to the point where our automatically scanned 3d models are
our automatically scanned 3d models are
our automatically scanned 3d models are good enough to use in a video game or a
good enough to use in a video game or a
good enough to use in a video game or a virtual reality experience we also
virtual reality experience we also
virtual reality experience we also repeated this experiment one more time
repeated this experiment one more time
repeated this experiment one more time at this barn and the quality
at this barn and the quality
at this barn and the quality improvements that we observed or even
improvements that we observed or even
improvements that we observed or even more pronounced so here’s the
more pronounced so here’s the
more pronounced so here’s the reconstruction we got from our
reconstruction we got from our
reconstruction we got from our trajectory and here’s the one we got
trajectory and here’s the one we got
trajectory and here’s the one we got from that overhead trajectory I’ll
from that overhead trajectory I’ll
from that overhead trajectory I’ll toggle that again one more time
toggle that again one more time
toggle that again one more time ours overhead so in this talk we’ve seen
ours overhead so in this talk we’ve seen
ours overhead so in this talk we’ve seen a couple of exciting examples of how we
a couple of exciting examples of how we
a couple of exciting examples of how we can use drones to help us in our
can use drones to help us in our
can use drones to help us in our creative endeavors but these are just
creative endeavors but these are just
creative endeavors but these are just baby steps I’m excited about the next
baby steps I’m excited about the next
baby steps I’m excited about the next generation of drone apps that don’t even
generation of drone apps that don’t even
generation of drone apps that don’t even exist yet so I’m excited for a
exist yet so I’m excited for a
exist yet so I’m excited for a journalist to throw a drone in the air
journalist to throw a drone in the air
journalist to throw a drone in the air and immediately start covering a
and immediately start covering a
and immediately start covering a breaking news story or for biologists to
breaking news story or for biologists to
breaking news story or for biologists to start observing wildlife in totally new
start observing wildlife in totally new
start observing wildlife in totally new ways or for archaeologists to launch a
ways or for archaeologists to launch a
ways or for archaeologists to launch a team of drones to scan some big ancient
team of drones to scan some big ancient
team of drones to scan some big ancient monument in other words I don’t just
monument in other words I don’t just
monument in other words I don’t just want to unlock the creative power of
want to unlock the creative power of
want to unlock the creative power of drones
drones
drones I want to unlock the creative power of
I want to unlock the creative power of
I want to unlock the creative power of you thank you
”
Be First to Comment