Press "Enter" to skip to content

Modeling People From Visual Data | Ira Kemelmacher-Shlizerman | TEDxVienna


this is my daughter she was so adorable

this is my daughter she was so adorable when she was born when she was two she

when she was born when she was two she

when she was born when she was two she was all about pianos so everywhere we

was all about pianos so everywhere we

was all about pianos so everywhere we went she wanted to play one when she saw

went she wanted to play one when she saw

went she wanted to play one when she saw one when she got five she forgot about

one when she got five she forgot about

one when she got five she forgot about pianos she was all about makeup and

pianos she was all about makeup and

pianos she was all about makeup and lipsticks to tell arias then later she

lipsticks to tell arias then later she

lipsticks to tell arias then later she was all about horses and now I have two

was all about horses and now I have two

was all about horses and now I have two daughters and the stories continued and

daughters and the stories continued and

daughters and the stories continued and said this is how I like to think about

said this is how I like to think about

said this is how I like to think about my photographs I captured my experiences

my photographs I captured my experiences

my photographs I captured my experiences my memories my photos tell a story yes

my memories my photos tell a story yes

my memories my photos tell a story yes all 35,000 of them except they don’t

all 35,000 of them except they don’t

all 35,000 of them except they don’t really what they actually are 35,000

really what they actually are 35,000

really what they actually are 35,000 thumbnails or my harddrive in my camera

thumbnails or my harddrive in my camera

thumbnails or my harddrive in my camera roll and you guys probably have the same

roll and you guys probably have the same

roll and you guys probably have the same story right what we all have it’s just a

story right what we all have it’s just a

story right what we all have it’s just a digital version of the shoebox our

digital version of the shoebox our

digital version of the shoebox our parents had but in our case it’s a

parents had but in our case it’s a

parents had but in our case it’s a ginormous box so I’m a computer vision

ginormous box so I’m a computer vision

ginormous box so I’m a computer vision researcher and I wanted to see if the

researcher and I wanted to see if the

researcher and I wanted to see if the technology that I’m developing the model

technology that I’m developing the model

technology that I’m developing the model people from big foreign video

people from big foreign video

people from big foreign video collections can be useful to create

collections can be useful to create

collections can be useful to create stories from my own photographs and so I

stories from my own photographs and so I

stories from my own photographs and so I had the following idea I said okay so I

had the following idea I said okay so I

had the following idea I said okay so I have this huge photo collection I’m

have this huge photo collection I’m

have this huge photo collection I’m gonna detect the faces and all my photos

gonna detect the faces and all my photos

gonna detect the faces and all my photos now automatically of course I’m gonna

now automatically of course I’m gonna

now automatically of course I’m gonna recognize all the faces then I’m gonna

recognize all the faces then I’m gonna

recognize all the faces then I’m gonna focus on a single person let’s say my

focus on a single person let’s say my

focus on a single person let’s say my daughter and then what I’ll do I’m gonna

daughter and then what I’ll do I’m gonna

daughter and then what I’ll do I’m gonna create a huge graph out of all these

create a huge graph out of all these

create a huge graph out of all these photographs where each photo is

photographs where each photo is

photographs where each photo is represented by a node in this graph and

represented by a node in this graph and

represented by a node in this graph and the edges represent similarities between

the edges represent similarities between

the edges represent similarities between each pair of photographs similarities in

each pair of photographs similarities in

each pair of photographs similarities in terms of the facial expression and if

terms of the facial expression and if

terms of the facial expression and if she smiles or not in terms of the

she smiles or not in terms of the

she smiles or not in terms of the hairstyle they had posed the time when

hairstyle they had posed the time when

hairstyle they had posed the time when the photo was taken and that once I have

the photo was taken and that once I have

the photo was taken and that once I have this graph is tablished I can teach a

this graph is tablished I can teach a

this graph is tablished I can teach a computer

computer

computer to walk on this graph to create a path

to walk on this graph to create a path

to walk on this graph to create a path or if you will a story so this was

or if you will a story so this was

or if you will a story so this was awesome and I thought okay so I’ve got

awesome and I thought okay so I’ve got

awesome and I thought okay so I’ve got the path now I can do it all right

the path now I can do it all right

the path now I can do it all right automatically the only thing that is

automatically the only thing that is

automatically the only thing that is missing is how to represent this set of

missing is how to represent this set of

missing is how to represent this set of photographs and so I realized that

photographs and so I realized that

photographs and so I realized that before the story part that if I’ll just

before the story part that if I’ll just

before the story part that if I’ll just take the detected faces and I put them

take the detected faces and I put them

take the detected faces and I put them one on top of the other and play them

one on top of the other and play them

one on top of the other and play them chronologically it’s it’s kind of

chronologically it’s it’s kind of

chronologically it’s it’s kind of magical so see I can see how she growled

magical so see I can see how she growled

magical so see I can see how she growled in front of my eyes and what happens now

in front of my eyes and what happens now

in front of my eyes and what happens now is that there’s lots of things while

is that there’s lots of things while

is that there’s lots of things while it’s a cool experience there’s lots of

it’s a cool experience there’s lots of

it’s a cool experience there’s lots of things going on right and so it’s very

things going on right and so it’s very

things going on right and so it’s very long and so the combination of the story

long and so the combination of the story

long and so the combination of the story the big graph is a path plus the video

the big graph is a path plus the video

the big graph is a path plus the video representation creates magic we can even

representation creates magic we can even

representation creates magic we can even summarize twenty years of photographs in

summarize twenty years of photographs in

summarize twenty years of photographs in 30 seconds so let’s watch together the

30 seconds so let’s watch together the

30 seconds so let’s watch together the final result what you will notice is

final result what you will notice is

final result what you will notice is that the person is in the focus the

that the person is in the focus the

that the person is in the focus the transitions between the different

transitions between the different

transitions between the different photographs in the final video are

photographs in the final video are

photographs in the final video are really nice you can see that it’s really

really nice you can see that it’s really

really nice you can see that it’s really transition smoothly between their facial

transition smoothly between their facial

transition smoothly between their facial expressions and the time and so on and

expressions and the time and so on and

expressions and the time and so on and it tells the story

thank you so this is fun I was able to

thank you so this is fun I was able to teach a computer to tell his story from

teach a computer to tell his story from

teach a computer to tell his story from my existing photographs now the next

my existing photographs now the next

my existing photographs now the next question that I asked I thought wow this

question that I asked I thought wow this

question that I asked I thought wow this is kind of cool so can I actually teach

is kind of cool so can I actually teach

is kind of cool so can I actually teach a computer to simulate what my daughter

a computer to simulate what my daughter

a computer to simulate what my daughter will look like she’s now nine what will

will look like she’s now nine what will

will look like she’s now nine what will she look like when she’s 25 this seems

she look like when she’s 25 this seems

she look like when she’s 25 this seems really hard though how would I how would

really hard though how would I how would

really hard though how would I how would I go about it so I thought one idea

I go about it so I thought one idea

I go about it so I thought one idea right so if I do have a pair of

right so if I do have a pair of

right so if I do have a pair of photographs of the same person these are

photographs of the same person these are

photographs of the same person these are two real photos in different ages that

two real photos in different ages that

two real photos in different ages that if I have this transition already I can

if I have this transition already I can

if I have this transition already I can teach a computer to how to learn these

teach a computer to how to learn these

teach a computer to how to learn these transitions but I want to do it for any

transitions but I want to do it for any

transitions but I want to do it for any identity for any person out there in the

identity for any person out there in the

identity for any person out there in the world so what I really need is to be

world so what I really need is to be

world so what I really need is to be able to somehow estimate the average

able to somehow estimate the average

able to somehow estimate the average transformation across all people in the

transformation across all people in the

transformation across all people in the world average aging transformation to be

world average aging transformation to be

world average aging transformation to be able to simulate later so but to get the

able to simulate later so but to get the

able to simulate later so but to get the every transformation and it’s and it’s a

every transformation and it’s and it’s a

every transformation and it’s and it’s a representation of all the people in the

representation of all the people in the

representation of all the people in the world so I had the following idea if I

world so I had the following idea if I

world so I had the following idea if I go to the search engine and just look

go to the search engine and just look

go to the search engine and just look for babies I get lots of photos of

for babies I get lots of photos of

for babies I get lots of photos of babies right here’s an example and then

babies right here’s an example and then

babies right here’s an example and then if I use the same technique as in the

if I use the same technique as in the

if I use the same technique as in the video representation I put all the I

video representation I put all the I

video representation I put all the I detect all the faces and I put them one

detect all the faces and I put them one

detect all the faces and I put them one on top of the other but instead of

on top of the other but instead of

on top of the other but instead of showing them as a video I will Everage

showing them as a video I will Everage

showing them as a video I will Everage them per pixel I will get the following

them per pixel I will get the following

them per pixel I will get the following result so I get an average baby and look

result so I get an average baby and look

result so I get an average baby and look this is not a real photo of a baby this

this is not a real photo of a baby this

this is not a real photo of a baby this is an average baby from the search

is an average baby from the search

is an average baby from the search results I can do it for any age so I can

results I can do it for any age so I can

results I can do it for any age so I can search for 25 year old male and I get a

search for 25 year old male and I get a

search for 25 year old male and I get a ton of photos and then average them

ton of photos and then average them

ton of photos and then average them again

again

again and I get a 25 year old every male so

and I get a 25 year old every male so

and I get a 25 year old every male so far okay cool this is my solution and so

far okay cool this is my solution and so

far okay cool this is my solution and so now I have two photographs if you will

now I have two photographs if you will

now I have two photographs if you will not real but you know and I can teach a

not real but you know and I can teach a

not real but you know and I can teach a computer to estimate the average agent

computer to estimate the average agent

computer to estimate the average agent transformation now the final step is

transformation now the final step is

transformation now the final step is that we want to apply the

that we want to apply the

that we want to apply the every transformation on any photo out

every transformation on any photo out

every transformation on any photo out there even for a three-year-old boy here

there even for a three-year-old boy here

there even for a three-year-old boy here and so the another component that we

and so the another component that we

and so the another component that we developed is that we can apply the

developed is that we can apply the

developed is that we can apply the average transformation on this photo but

average transformation on this photo but

average transformation on this photo but we need to make sure that we keep the

we need to make sure that we keep the

we need to make sure that we keep the identity of the person and so we did

identity of the person and so we did

identity of the person and so we did that and we were able to produce from an

that and we were able to produce from an

that and we were able to produce from an three year old photo of this boy

three year old photo of this boy

three year old photo of this boy let’s say 25 year old version of him

we can even produce an eight-year-old

we can even produce an eight-year-old version of him

version of him

version of him so you’ll notice that somehow the phase

so you’ll notice that somehow the phase

so you’ll notice that somehow the phase changes you see the skin changes the

changes you see the skin changes the

changes you see the skin changes the nose shape changed and so on and all

nose shape changed and so on and all

nose shape changed and so on and all those are real aging changes you will

those are real aging changes you will

those are real aging changes you will notice one thing the milk master state

notice one thing the milk master state

notice one thing the milk master state at eight year old this is because the

at eight year old this is because the

at eight year old this is because the algorithm thinks that it’s part of his

algorithm thinks that it’s part of his

algorithm thinks that it’s part of his identity we can do it for any type of

identity we can do it for any type of

identity we can do it for any type of photographs and equality of photographs

photographs and equality of photographs

photographs and equality of photographs here’s one example and he colored

here’s one example and he colored

here’s one example and he colored females and males of course and we also

females and males of course and we also

females and males of course and we also tested it with ground truth is it is it

tested it with ground truth is it is it

tested it with ground truth is it is it close to reality or not so we had a

close to reality or not so we had a

close to reality or not so we had a photograph of one person where we had

photograph of one person where we had

photograph of one person where we had him at age three and we also had him in

him at age three and we also had him in

him at age three and we also had him in like real photographs in different

like real photographs in different

like real photographs in different different other ages 16 and 16 and then

different other ages 16 and 16 and then

different other ages 16 and 16 and then we applied our algorithm on the

we applied our algorithm on the

we applied our algorithm on the three-year-old for of him just on the

three-year-old for of him just on the

three-year-old for of him just on the face area so we created his synthesized

face area so we created his synthesized

face area so we created his synthesized face and blended it into the head of the

face and blended it into the head of the

face and blended it into the head of the real for us and we got this we did a

real for us and we got this we did a

real for us and we got this we did a human study and people had a really hard

human study and people had a really hard

human study and people had a really hard time to decide which one is real which

time to decide which one is real which

time to decide which one is real which one is not so that was promising so I

one is not so that was promising so I

one is not so that was promising so I talked about Xavier look like in the

talked about Xavier look like in the

talked about Xavier look like in the future but you know some of you may ask

future but you know some of you may ask

future but you know some of you may ask other questions that are related to our

other questions that are related to our

other questions that are related to our appearance and photographs and how we

appearance and photographs and how we

appearance and photographs and how we look and have you may look so I don’t

look and have you may look so I don’t

look and have you may look so I don’t know some of you may ask what if I

know some of you may ask what if I

know some of you may ask what if I exercise more will I look differently do

exercise more will I look differently do

exercise more will I look differently do I look good in hats yeah

I look good in hats yeah

I look good in hats yeah what if I become vegetarian right no I

what if I become vegetarian right no I

what if I become vegetarian right no I look differently maybe not so I wanted

look differently maybe not so I wanted

look differently maybe not so I wanted to ask will black hair feed me okay and

to ask will black hair feed me okay and

to ask will black hair feed me okay and so we all know it is well known that

so we all know it is well known that

so we all know it is well known that when people change their hairstyles or

when people change their hairstyles or

when people change their hairstyles or the hair color they can even become

the hair color they can even become

the hair color they can even become unrecognizable it’s pretty dramatic for

unrecognizable it’s pretty dramatic for

unrecognizable it’s pretty dramatic for our appearance and sometimes research

our appearance and sometimes research

our appearance and sometimes research even showed that sometimes the face is

even showed that sometimes the face is

even showed that sometimes the face is less important than the hair actually

less important than the hair actually

less important than the hair actually because we have sometimes unique

because we have sometimes unique

because we have sometimes unique hairstyles and this is how people

hairstyles and this is how people

hairstyles and this is how people recognize us and so it’s scary to make a

recognize us and so it’s scary to make a

recognize us and so it’s scary to make a big change without knowing if it is

big change without knowing if it is

big change without knowing if it is gonna look nice or not and

gonna look nice or not and

gonna look nice or not and I took a photo of me and I wanted to

I took a photo of me and I wanted to

I took a photo of me and I wanted to answer this question and so of course I

answer this question and so of course I

answer this question and so of course I taught a computer to do it and so I

taught a computer to do it and so I

taught a computer to do it and so I render it myself with black hair started

render it myself with black hair started

render it myself with black hair started to write code to create that

to write code to create that

to write code to create that automatically and that looked fun so I

automatically and that looked fun so I

automatically and that looked fun so I said okay so let’s try to render myself

said okay so let’s try to render myself

said okay so let’s try to render myself with different hairstyles in black hair

with different hairstyles in black hair

with different hairstyles in black hair so I got this and that I said and then I

so I got this and that I said and then I

so I got this and that I said and then I started turning to people and people

started turning to people and people

started turning to people and people were excited like oh yeah that’s so cool

were excited like oh yeah that’s so cool

were excited like oh yeah that’s so cool and so I run myself is curly hair and

and so I run myself is curly hair and

and so I run myself is curly hair and different colors and in fact I started

different colors and in fact I started

different colors and in fact I started to develop it and I built it as a search

to develop it and I built it as a search

to develop it and I built it as a search engine where I could type there anything

engine where I could type there anything

engine where I could type there anything kind of imagine an image search engine

kind of imagine an image search engine

kind of imagine an image search engine so I could type there India and imagine

so I could type there India and imagine

so I could type there India and imagine myself in India so I could see how I

myself in India so I could see how I

myself in India so I could see how I look like an Indian clothing and hair

look like an Indian clothing and hair

look like an Indian clothing and hair and Sun or even go back in time and I

and Sun or even go back in time and I

and Sun or even go back in time and I could write 1930 and see how I would

could write 1930 and see how I would

could write 1930 and see how I would look like in 1930s I tried it on 80

look like in 1930s I tried it on 80

look like in 1930s I tried it on 80 different queries so I could see myself

different queries so I could see myself

different queries so I could see myself with dreadlocks and with different hair

with dreadlocks and with different hair

with dreadlocks and with different hair colors and styles and with shaved hair

colors and styles and with shaved hair

colors and styles and with shaved hair and as a bride and in different

and as a bride and in different

and as a bride and in different traditional clothing it was a lot of fun

traditional clothing it was a lot of fun

traditional clothing it was a lot of fun there are many important applications to

there are many important applications to

there are many important applications to the technology that we’re developing in

the technology that we’re developing in

the technology that we’re developing in modeling people from photos and videos I

modeling people from photos and videos I

modeling people from photos and videos I will talk only about one which is really

will talk only about one which is really

will talk only about one which is really close to my heart there are half a

close to my heart there are half a

close to my heart there are half a million missing children out there

million missing children out there

million missing children out there Justin 2016 Justin you say there are

Justin 2016 Justin you say there are

Justin 2016 Justin you say there are similar numbers in other countries these

similar numbers in other countries these

similar numbers in other countries these numbers are insane it turns out that a

numbers are insane it turns out that a

numbers are insane it turns out that a critical component in the search for

critical component in the search for

critical component in the search for missing children are portraits of how

missing children are portraits of how

missing children are portraits of how they may look like in the future

they may look like in the future

they may look like in the future sometimes people go missing for many

sometimes people go missing for many

sometimes people go missing for many many years for example in this example

many years for example in this example

many years for example in this example this woman is missing since 1963 and

this woman is missing since 1963 and

this woman is missing since 1963 and what you see on the right is a sketch

what you see on the right is a sketch

what you see on the right is a sketch that an artist is making a mate to try

that an artist is making a mate to try

that an artist is making a mate to try to predict how she looks like to be able

to predict how she looks like to be able

to predict how she looks like to be able to find her or here’s another

to find her or here’s another

to find her or here’s another sometimes the artists do photoshop’s and

sometimes the artists do photoshop’s and

sometimes the artists do photoshop’s and try to predict an age progression how so

try to predict an age progression how so

try to predict an age progression how so for example in this case but who knows

for example in this case but who knows

for example in this case but who knows if it is still blonde or if she actually

if it is still blonde or if she actually

if it is still blonde or if she actually looks like this this is based on

looks like this this is based on

looks like this this is based on intuitions of artists it’s also very

intuitions of artists it’s also very

intuitions of artists it’s also very hard to do it for half a million for us

hard to do it for half a million for us

hard to do it for half a million for us because it just manual work in Photoshop

because it just manual work in Photoshop

because it just manual work in Photoshop and so we wanted to combine the two

and so we wanted to combine the two

and so we wanted to combine the two projects one were you imagine how a face

projects one were you imagine how a face

projects one were you imagine how a face of a person looks like in the future and

of a person looks like in the future and

of a person looks like in the future and the other one is how to imagine it

the other one is how to imagine it

the other one is how to imagine it combined with a like a full portrait of

combined with a like a full portrait of

combined with a like a full portrait of the person and so here’s an example

the person and so here’s an example

the person and so here’s an example where we take a baby photo and in the

where we take a baby photo and in the

where we take a baby photo and in the search engine we search for a five year

search engine we search for a five year

search engine we search for a five year old male and we combine the h progressed

old male and we combine the h progressed

old male and we combine the h progressed the face and then create also different

the face and then create also different

the face and then create also different hairstyles to try to imagine what if

hairstyles to try to imagine what if

hairstyles to try to imagine what if somebody am changed the color of the

somebody am changed the color of the

somebody am changed the color of the hair of this person or how he will look

hair of this person or how he will look

hair of this person or how he will look like is different clothing we can

like is different clothing we can

like is different clothing we can imagine how she looks like and as a 20

imagine how she looks like and as a 20

imagine how she looks like and as a 20 year old male as a 30 year old male

year old male as a 30 year old male

year old male as a 30 year old male several examples or even as a 65 year

several examples or even as a 65 year

several examples or even as a 65 year old male I would like to finish with

old male I would like to finish with

old male I would like to finish with saying that computer vision allows us to

saying that computer vision allows us to

saying that computer vision allows us to see ourselves in new lights there are

see ourselves in new lights there are

see ourselves in new lights there are many fun and important applications and

many fun and important applications and

many fun and important applications and I’m so excited about the impact that

I’m so excited about the impact that

I’m so excited about the impact that it’s having and will have in the future

it’s having and will have in the future

it’s having and will have in the future thank you

”

Be First to Comment

Leave a Reply

Your email address will not be published. Required fields are marked *