Human brain mapping and brain decoding. | Jack Gallant | TEDxSanFrancisco

[Music]

[Music] brains a computer I know it seems

brains a computer I know it seems

brains a computer I know it seems strange to think of the human brain as a

strange to think of the human brain as a

strange to think of the human brain as a computer but you know it’s not a

computer but you know it’s not a

computer but you know it’s not a computer like your desktop or like your

computer like your desktop or like your

computer like your desktop or like your cell phone which are architected systems

cell phone which are architected systems

cell phone which are architected systems it’s an evolved system that has

it’s an evolved system that has

it’s an evolved system that has developed over millions of years

developed over millions of years

developed over millions of years continuously building on what was there

continuously building on what was there

continuously building on what was there before but it processes information you

before but it processes information you

before but it processes information you sit here you have sensory experiences

sit here you have sensory experiences

sit here you have sensory experiences you combine those sensory experiences

you combine those sensory experiences

you combine those sensory experiences with prior information prior knowledge

with prior information prior knowledge

with prior information prior knowledge you make decisions and you act in the

you make decisions and you act in the

you make decisions and you act in the world and an information processing

world and an information processing

world and an information processing system like that is a computer so in

system like that is a computer so in

system like that is a computer so in principle we should be able to

principle we should be able to

principle we should be able to understand the human brain as a computer

understand the human brain as a computer

understand the human brain as a computer by reverse engineering it and

by reverse engineering it and

by reverse engineering it and discovering the underlying computation

discovering the underlying computation

discovering the underlying computation algorithms that govern its function and

algorithms that govern its function and

algorithms that govern its function and if we do that we can invert those

if we do that we can invert those

if we do that we can invert those algorithms and create very powerful

algorithms and create very powerful

algorithms and create very powerful brain decoding devices that can probably

brain decoding devices that can probably

brain decoding devices that can probably in the future replace all other kinds of

in the future replace all other kinds of

in the future replace all other kinds of grain machine interfaces and brain

grain machine interfaces and brain

grain machine interfaces and brain computer interfaces now the brain looks

computer interfaces now the brain looks

computer interfaces now the brain looks like a pretty homogeneous organ there’s

like a pretty homogeneous organ there’s

like a pretty homogeneous organ there’s a cerebral cortex this big folded part

a cerebral cortex this big folded part

a cerebral cortex this big folded part on the top and there’s a cerebellum on

on the top and there’s a cerebellum on

on the top and there’s a cerebellum on the bottom underneath the cerebral

the bottom underneath the cerebral

the bottom underneath the cerebral cortex this mantle of tissue there are

cortex this mantle of tissue there are

cortex this mantle of tissue there are many many subcortical nuclei all

many many subcortical nuclei all

many many subcortical nuclei all together though the brain has about some

together though the brain has about some

together though the brain has about some 13 we think on the order of say 500

13 we think on the order of say 500

13 we think on the order of say 500 different parts about a third of those

different parts about a third of those

different parts about a third of those are gonna be subcortical nuclei and

are gonna be subcortical nuclei and

are gonna be subcortical nuclei and two-thirds of them are going to be

two-thirds of them are going to be

two-thirds of them are going to be cortical areas that are functionally and

cortical areas that are functionally and

cortical areas that are functionally and anatomically distinct and these areas

anatomically distinct and these areas

anatomically distinct and these areas are hooked together in very dense

are hooked together in very dense

are hooked together in very dense networks we don’t really know the area’s

networks we don’t really know the area’s

networks we don’t really know the area’s yet for humans or exactly how they’re

yet for humans or exactly how they’re

yet for humans or exactly how they’re hooked together but animal studies

hooked together but animal studies

hooked together but animal studies suggest that the brain is 50% connected

suggest that the brain is 50% connected

suggest that the brain is 50% connected that is for any given location on the

that is for any given location on the

that is for any given location on the cerebral cortex it is connected to 50%

cerebral cortex it is connected to 50%

cerebral cortex it is connected to 50% of all of the other locations and all of

of all of the other locations and all of

of all of the other locations and all of these connections are feed-forward and

these connections are feed-forward and

these connections are feed-forward and feedback so the brain is a giant

feedback so the brain is a giant

feedback so the brain is a giant interconnected recurrent Network and

interconnected recurrent Network and

interconnected recurrent Network and systems like this in science are known

systems like this in science are known

systems like this in science are known as nonlinear dynamical systems and they

as nonlinear dynamical systems and they

as nonlinear dynamical systems and they have interesting properties the property

have interesting properties the property

have interesting properties the property of the system as a whole its function

of the system as a whole its function

of the system as a whole its function cannot necessarily be predicted by

cannot necessarily be predicted by

cannot necessarily be predicted by looking at the individual parts because

looking at the individual parts because

looking at the individual parts because nonlinear dynamical systems have

nonlinear dynamical systems have

nonlinear dynamical systems have emergent properties and in humans of

emergent properties and in humans of

emergent properties and in humans of course one of the most import

course one of the most import

course one of the most import emergent properties is probably

emergent properties is probably

emergent properties is probably consciousness to get a handle on what

consciousness to get a handle on what

consciousness to get a handle on what the human brain is as a computer to

the human brain is as a computer to

the human brain is as a computer to understand its structure we can first of

understand its structure we can first of

understand its structure we can first of all try to map it we can separate the

all try to map it we can separate the

all try to map it we can separate the cerebral cortex into its distinct

cerebral cortex into its distinct

cerebral cortex into its distinct anatomical and functional parts and try

anatomical and functional parts and try

anatomical and functional parts and try to get a handle on them and then look at

to get a handle on them and then look at

to get a handle on them and then look at how they interact now people have known

how they interact now people have known

how they interact now people have known for hundreds of years that the human

for hundreds of years that the human

for hundreds of years that the human brain was divided into different

brain was divided into different

brain was divided into different functional parts and they’ve tried

functional parts and they’ve tried

functional parts and they’ve tried mostly unsuccessfully until recently to

mostly unsuccessfully until recently to

mostly unsuccessfully until recently to really parse laid out – the cerebral

really parse laid out – the cerebral

really parse laid out – the cerebral cortex the main part of the human brain

cortex the main part of the human brain

cortex the main part of the human brain that you see on the in a slide that

that you see on the in a slide that

that you see on the in a slide that distinguishes us mostly from lower

distinguishes us mostly from lower

distinguishes us mostly from lower animals they’ve tried to parse late to

animals they’ve tried to parse late to

animals they’ve tried to parse late to the brain using a variety of methods

the brain using a variety of methods

the brain using a variety of methods that were mostly unsuccessful the most

that were mostly unsuccessful the most

that were mostly unsuccessful the most notorious of these is phrenology a

notorious of these is phrenology a

notorious of these is phrenology a nineteenth-century discredited science

nineteenth-century discredited science

nineteenth-century discredited science in which people tried to look at the

in which people tried to look at the

in which people tried to look at the bumps on the surface of the skull and

bumps on the surface of the skull and

bumps on the surface of the skull and infer from the pattern of bumps what the

infer from the pattern of bumps what the

infer from the pattern of bumps what the underlying function was of the tissue

underlying function was of the tissue

underlying function was of the tissue underneath the bumps today we all laugh

underneath the bumps today we all laugh

underneath the bumps today we all laugh at this it seems completely silly but

at this it seems completely silly but

at this it seems completely silly but the phrenologist weren’t completely

the phrenologist weren’t completely

the phrenologist weren’t completely crazy they were we had very poor methods

crazy they were we had very poor methods

crazy they were we had very poor methods they were 40 poor scientists but they

they were 40 poor scientists but they

they were 40 poor scientists but they did have one good idea which is that the

did have one good idea which is that the

did have one good idea which is that the brain is functionally localized and

brain is functionally localized and

brain is functionally localized and today we can map the functional

today we can map the functional

today we can map the functional localization of the brain using powerful

localization of the brain using powerful

localization of the brain using powerful methods such as functional magnetic

methods such as functional magnetic

methods such as functional magnetic resonance imaging so I’m going to show

resonance imaging so I’m going to show

resonance imaging so I’m going to show you an example of an fMRI experiment

you an example of an fMRI experiment

you an example of an fMRI experiment that kind of gives you a flavor of the

that kind of gives you a flavor of the

that kind of gives you a flavor of the kind of data we can recover from this

kind of data we can recover from this

kind of data we can recover from this method the brain is inconveniently

method the brain is inconveniently

method the brain is inconveniently folded up inside the skull so we

folded up inside the skull so we

folded up inside the skull so we computationally inflate it we put some

computationally inflate it we put some

computationally inflate it we put some slices in it and we flatten it out so

slices in it and we flatten it out so

slices in it and we flatten it out so now we’re looking at the surface of the

now we’re looking at the surface of the

now we’re looking at the surface of the cerebral cortex if we did this to your

cerebral cortex if we did this to your

cerebral cortex if we did this to your brain we would have something about the

brain we would have something about the

brain we would have something about the size of a large pizza there arose a left

size of a large pizza there arose a left

size of a large pizza there arose a left and a right hemisphere the middle of the

and a right hemisphere the middle of the

and a right hemisphere the middle of the screen is showing the visual system at

screen is showing the visual system at

screen is showing the visual system at the far left and the far right as the

the far left and the far right as the

the far left and the far right as the prefrontal cortex

prefrontal cortex

prefrontal cortex now we can paint brain activity on the

now we can paint brain activity on the

now we can paint brain activity on the surface of the cerebral cortex while

surface of the cerebral cortex while

surface of the cerebral cortex while someone watches a movie and they’re

someone watches a movie and they’re

someone watches a movie and they’re simply passively watching this movie and

simply passively watching this movie and

simply passively watching this movie and interpreting what’s going on of course

interpreting what’s going on of course

interpreting what’s going on of course think about what happens to your brain

think about what happens to your brain

think about what happens to your brain when you watch a movie anything that you

when you watch a movie anything that you

when you watch a movie anything that you can conceive of in this movie must be

can conceive of in this movie must be

can conceive of in this movie must be reflected somewhere in the brain in

reflected somewhere in the brain in

reflected somewhere in the brain in terms of patterns of brain activity

terms of patterns of brain activity

terms of patterns of brain activity obviously you can see the structure of

obviously you can see the structure of

obviously you can see the structure of the movie you can see the wall you can

the movie you can see the wall you can

the movie you can see the wall you can see water you can see the edges of the

see water you can see the edges of the

see water you can see the edges of the screen you can see textures you can see

screen you can see textures you can see

screen you can see textures you can see oriented lines those are represented in

oriented lines those are represented in

oriented lines those are represented in one part of the brain other parts of the

one part of the brain other parts of the

one part of the brain other parts of the brain represent information that is

brain represent information that is

brain represent information that is reflects the objects and actions in the

reflects the objects and actions in the

reflects the objects and actions in the movie that there’s a car or a smokestack

movie that there’s a car or a smokestack

movie that there’s a car or a smokestack that there’s smoke that there’s airflow

that there’s smoke that there’s airflow

that there’s smoke that there’s airflow other parts of the brain reflect

other parts of the brain reflect

other parts of the brain reflect information that isn’t directly in the

information that isn’t directly in the

information that isn’t directly in the movie but is associated with it for

movie but is associated with it for

movie but is associated with it for example you might be hungry right now so

example you might be hungry right now so

example you might be hungry right now so you might be thinking about food or you

you might be thinking about food or you

you might be thinking about food or you might be annoyed at the price of gas or

might be annoyed at the price of gas or

might be annoyed at the price of gas or you might want coffee all of these

you might want coffee all of these

you might want coffee all of these things would be also represented in the

things would be also represented in the

things would be also represented in the brain these desires needs wants and

brain these desires needs wants and

brain these desires needs wants and associations all of this stuff must be

associations all of this stuff must be

associations all of this stuff must be represented somewhere that really so you

represented somewhere that really so you

represented somewhere that really so you can think of the problem of brain

can think of the problem of brain

can think of the problem of brain mapping as finding systemic

mapping as finding systemic

mapping as finding systemic relationships between the stuff in the

relationships between the stuff in the

relationships between the stuff in the world in this case the things in the

world in this case the things in the

world in this case the things in the movie and the patterns of brain activity

movie and the patterns of brain activity

movie and the patterns of brain activity those are going to be fairly

those are going to be fairly

those are going to be fairly straightforward relationships to map

straightforward relationships to map

straightforward relationships to map there’s also of course going to be a lot

there’s also of course going to be a lot

there’s also of course going to be a lot of information in the brain that isn’t

of information in the brain that isn’t

of information in the brain that isn’t directly in the movie your implicit

directly in the movie your implicit

directly in the movie your implicit associations your biases your private ik

associations your biases your private ik

associations your biases your private ik prior experiences and those are going to

prior experiences and those are going to

prior experiences and those are going to be a bit more difficult to map but in

be a bit more difficult to map but in

be a bit more difficult to map but in principle we should be able to slowly

principle we should be able to slowly

principle we should be able to slowly over time aggregate a huge amount of

over time aggregate a huge amount of

over time aggregate a huge amount of information about the relationships

information about the relationships

information about the relationships between brain activity and the world in

between brain activity and the world in

between brain activity and the world in order to create functional maps of the

order to create functional maps of the

order to create functional maps of the brain so here are the results of a few

brain so here are the results of a few

brain so here are the results of a few mapping experiments that have been

mapping experiments that have been

mapping experiments that have been aggregated together in these experiments

aggregated together in these experiments

aggregated together in these experiments we had people watch movies or we had

we had people watch movies or we had

we had people watch movies or we had them listen to stories in the MRI

them listen to stories in the MRI

them listen to stories in the MRI machine these experiments probably took

machine these experiments probably took

machine these experiments probably took eight or ten hours in total for one

eight or ten hours in total for one

eight or ten hours in total for one individual but we didn’t do this all at

individual but we didn’t do this all at

individual but we didn’t do this all at once these were experiments that were

once these were experiments that were

once these were experiments that were done across weeks of time and now we can

done across weeks of time and now we can

done across weeks of time and now we can take this information I essentially this

take this information I essentially this

take this information I essentially this is a regression problem where the X

is a regression problem where the X

is a regression problem where the X variables are the world the stories are

variables are the world the stories are

variables are the world the stories are the movies and the Y variables are our

the movies and the Y variables are our

the movies and the Y variables are our brain activity

brain activity

brain activity and we can use machine learning and data

and we can use machine learning and data

and we can use machine learning and data analysis tools to recover functional

analysis tools to recover functional

analysis tools to recover functional maps of brain activity and those are

maps of brain activity and those are

maps of brain activity and those are shown here as colored maps and the

shown here as colored maps and the

shown here as colored maps and the different colors here reflects different

different colors here reflects different

different colors here reflects different kinds of meaning that are represented in

kinds of meaning that are represented in

kinds of meaning that are represented in the patterns of brain activity about the

the patterns of brain activity about the

the patterns of brain activity about the information in the stories and movies

information in the stories and movies

information in the stories and movies now these patterns are fairly

now these patterns are fairly

now these patterns are fairly complicated and rich and this conceptual

complicated and rich and this conceptual

complicated and rich and this conceptual knowledge that we’re mapping this

knowledge that we’re mapping this

knowledge that we’re mapping this particular experiment is distributed

particular experiment is distributed

particular experiment is distributed broadly across the brain this is a lot

broadly across the brain this is a lot

broadly across the brain this is a lot of information to take in we’re making a

of information to take in we’re making a

of information to take in we’re making a hundred thousand spatial measurements

hundred thousand spatial measurements

hundred thousand spatial measurements across these cortical maps and it’s too

across these cortical maps and it’s too

across these cortical maps and it’s too much to think about

much to think about

much to think about so we’ve created an online brain viewer

so we’ve created an online brain viewer

so we’ve created an online brain viewer that you can actually play with on my

that you can actually play with on my

that you can actually play with on my website and this online brain viewer

website and this online brain viewer

website and this online brain viewer allows you to interact with these maps

allows you to interact with these maps

allows you to interact with these maps to determine what kind of conceptual

to determine what kind of conceptual

to determine what kind of conceptual information is represented across the

information is represented across the

information is represented across the brain so this particular dataset comes

brain so this particular dataset comes

brain so this particular dataset comes from a story listening experiment where

from a story listening experiment where

from a story listening experiment where people listened to The Moth Radio Hour

people listened to The Moth Radio Hour

people listened to The Moth Radio Hour from Public Radio International and now

from Public Radio International and now

from Public Radio International and now we can create maps that reveal what kind

we can create maps that reveal what kind

we can create maps that reveal what kind of information is represented at

of information is represented at

of information is represented at different locations in the brain and

different locations in the brain and

different locations in the brain and here I’m clicking on an individual voxel

here I’m clicking on an individual voxel

here I’m clicking on an individual voxel down in the temporal lobe on the side of

down in the temporal lobe on the side of

down in the temporal lobe on the side of the head that represents social

the head that represents social

the head that represents social information families mothers fathers

information families mothers fathers

information families mothers fathers daughters things that happen to families

daughters things that happen to families

daughters things that happen to families weddings births deaths and so on there

weddings births deaths and so on there

weddings births deaths and so on there are many locations to represent social

are many locations to represent social

are many locations to represent social information here’s one in the lateral

information here’s one in the lateral

information here’s one in the lateral prefrontal cortex that also represents

prefrontal cortex that also represents

prefrontal cortex that also represents social information in fact all of the

social information in fact all of the

social information in fact all of the red spots on this map represents social

red spots on this map represents social

red spots on this map represents social information there are other networks

information there are other networks

information there are other networks that represent other kinds of

that represent other kinds of

that represent other kinds of information for example here’s a spot in

information for example here’s a spot in

information for example here’s a spot in the posterior parietal cortex that

the posterior parietal cortex that

the posterior parietal cortex that represents a numerical information

represents a numerical information

represents a numerical information numbers time money mathematical

numbers time money mathematical

numbers time money mathematical equations you might be processing in

equations you might be processing in

equations you might be processing in your brain anything related to numbers

your brain anything related to numbers

your brain anything related to numbers is represented this network of green

is represented this network of green

is represented this network of green areas in fact there are networks for

areas in fact there are networks for

areas in fact there are networks for hundreds of different kinds of

hundreds of different kinds of

hundreds of different kinds of conceptual information every piece of

conceptual information every piece of

conceptual information every piece of conceptual information truth dreaming

conceptual information truth dreaming

conceptual information truth dreaming sculptures anything you can think of

sculptures anything you can think of

sculptures anything you can think of these concept concepts are represented

these concept concepts are represented

these concept concepts are represented in multiple locations across the surface

in multiple locations across the surface

in multiple locations across the surface of cortex and each potential each

of cortex and each potential each

of cortex and each potential each location on the cortex represents

location on the cortex represents

location on the cortex represents multiple kinds of information now of

multiple kinds of information now of

multiple kinds of information now of course brains aren’t all the same I you

course brains aren’t all the same I you

course brains aren’t all the same I you know everyone has the same ears everyone

know everyone has the same ears everyone

know everyone has the same ears everyone hears the same

hears the same

hears the same your bits but everybody’s ears look

your bits but everybody’s ears look

your bits but everybody’s ears look slightly different and it’s the same for

slightly different and it’s the same for

slightly different and it’s the same for brains

brains

brains everyone’s brain is essentially the same

everyone’s brain is essentially the same

everyone’s brain is essentially the same in its general form but all individual

in its general form but all individual

in its general form but all individual brains are different and we can

brains are different and we can

brains are different and we can aggregate data across different brains

aggregate data across different brains

aggregate data across different brains like we see here we can see that about

like we see here we can see that about

like we see here we can see that about 1/3 of the functional activity and

1/3 of the functional activity and

1/3 of the functional activity and brains is shared across individuals and

brains is shared across individuals and

brains is shared across individuals and about 2/3 reflects individual

about 2/3 reflects individual

about 2/3 reflects individual differences which are both developmental

differences which are both developmental

differences which are both developmental and experiential differences that are

and experiential differences that are

and experiential differences that are reflected in function now even though

reflected in function now even though

reflected in function now even though there are individual differences in the

there are individual differences in the

there are individual differences in the brain because the brain is essentially a

brain because the brain is essentially a

brain because the brain is essentially a collection of wires which are the nerves

collection of wires which are the nerves

collection of wires which are the nerves that connect different areas together

that connect different areas together

that connect different areas together there’s a very strong relationship

there’s a very strong relationship

there’s a very strong relationship between the structure of the brain and

between the structure of the brain and

between the structure of the brain and the function of the brain and by

the function of the brain and by

the function of the brain and by aggregating data across many subjects

aggregating data across many subjects

aggregating data across many subjects and using computational modeling methods

and using computational modeling methods

and using computational modeling methods we can create a mathematical model that

we can create a mathematical model that

we can create a mathematical model that links an individual brain anatomy to the

links an individual brain anatomy to the

links an individual brain anatomy to the distribution of function across that

distribution of function across that

distribution of function across that brain here we show one particular

brain here we show one particular

brain here we show one particular functional map for one individual who’s

functional map for one individual who’s

functional map for one individual who’s listening to stories and this is a

listening to stories and this is a

listening to stories and this is a conceptual map this is mapping to

conceptual map this is mapping to

conceptual map this is mapping to thousand different kinds of objects and

thousand different kinds of objects and

thousand different kinds of objects and actions that can occur in stories across

actions that can occur in stories across

actions that can occur in stories across one individual brain they’re about a

one individual brain they’re about a

one individual brain they’re about a hundred and sixty different brain areas

hundred and sixty different brain areas

hundred and sixty different brain areas that represent conceptual knowledge in

that represent conceptual knowledge in

that represent conceptual knowledge in these stories and we can do this for any

these stories and we can do this for any

these stories and we can do this for any individual now up to now I’ve suggested

individual now up to now I’ve suggested

individual now up to now I’ve suggested to you that the brain is static that

to you that the brain is static that

to you that the brain is static that different places in the brain represent

different places in the brain represent

different places in the brain represent a certain small constellation of

a certain small constellation of

a certain small constellation of quantities and that those

quantities and that those

quantities and that those representations are static in fact

representations are static in fact

representations are static in fact they’re not the way the brain is

they’re not the way the brain is

they’re not the way the brain is designed it there are emergent

designed it there are emergent

designed it there are emergent properties that arise as a consequence

properties that arise as a consequence

properties that arise as a consequence of the task that you are trying to

of the task that you are trying to

of the task that you are trying to perform that actually change the way

perform that actually change the way

perform that actually change the way that information is represented in the

that information is represented in the

that information is represented in the brain so this is data from a simple

brain so this is data from a simple

brain so this is data from a simple experiment where we have people

experiment where we have people

experiment where we have people passively watching movies versus

passively watching movies versus

passively watching movies versus attending to humans in the movies versus

attending to humans in the movies versus

attending to humans in the movies versus attending to vehicles in the movies and

attending to vehicles in the movies and

attending to vehicles in the movies and you can see that the conceptual maps in

you can see that the conceptual maps in

you can see that the conceptual maps in these three different cases are

these three different cases are

these three different cases are different and actually the maps vary

different and actually the maps vary

different and actually the maps vary this least in the visual system these

this least in the visual system these

this least in the visual system these functional maps very enormous lis in

functional maps very enormous lis in

functional maps very enormous lis in prefrontal cortex because prefrontal

prefrontal cortex because prefrontal

prefrontal cortex because prefrontal cortex represents abstract thought plans

cortex represents abstract thought plans

cortex represents abstract thought plans goals console things like that and as

goals console things like that and as

goals console things like that and as you change your task the activity in

you change your task the activity in

you change your task the activity in cortex actually represents different

cortex actually represents different

cortex actually represents different kinds of quantities so for example if

kinds of quantities so for example if

kinds of quantities so for example if you lose your cat your prefrontal cortex

you lose your cat your prefrontal cortex

you lose your cat your prefrontal cortex becomes a giant cat detector and all

becomes a giant cat detector and all

becomes a giant cat detector and all other functions are suppressed and you

other functions are suppressed and you

other functions are suppressed and you everything is trying to represent cats

everything is trying to represent cats

everything is trying to represent cats as much as it can and that is true for

as much as it can and that is true for

as much as it can and that is true for pretty much any tasks you perform now

pretty much any tasks you perform now

pretty much any tasks you perform now that one of the interesting things about

that one of the interesting things about

that one of the interesting things about these encoding models if you are just a

these encoding models if you are just a

these encoding models if you are just a civilian or not a neuroscientist is that

civilian or not a neuroscientist is that

civilian or not a neuroscientist is that once you have a computational model of

once you have a computational model of

once you have a computational model of the brain that relates the world to

the brain that relates the world to

the brain that relates the world to brain activity you can invert that model

brain activity you can invert that model

brain activity you can invert that model and you can convert your encoding model

and you can convert your encoding model

and you can convert your encoding model into a decoding model and can use it to

into a decoding model and can use it to

into a decoding model and can use it to decode brain activity and in fact

decode brain activity and in fact

decode brain activity and in fact there’s an inevitable symmetry between

there’s an inevitable symmetry between

there’s an inevitable symmetry between brain and coding and brain decoding such

brain and coding and brain decoding such

brain and coding and brain decoding such that whenever we can create a better

that whenever we can create a better

that whenever we can create a better computational model of the brain we can

computational model of the brain we can

computational model of the brain we can always by definition build a better

always by definition build a better

always by definition build a better brain decoder and oftentimes it’s a

brain decoder and oftentimes it’s a

brain decoder and oftentimes it’s a little easier to understand what kind of

little easier to understand what kind of

little easier to understand what kind of information we can recover from current

information we can recover from current

information we can recover from current brain measurements by looking at decoder

brain measurements by looking at decoder

brain measurements by looking at decoder so I’d like to show you a few of these

so I’d like to show you a few of these

so I’d like to show you a few of these as examples here’s one brain decoder

as examples here’s one brain decoder

as examples here’s one brain decoder that’s only recovering information from

that’s only recovering information from

that’s only recovering information from primary visual cortex a very early

primary visual cortex a very early

primary visual cortex a very early visual area that represents very simple

visual area that represents very simple

visual area that represents very simple aspects of the scene like edges and

aspects of the scene like edges and

aspects of the scene like edges and textures and colors

textures and colors

textures and colors on the upper left is the movie we showed

on the upper left is the movie we showed

on the upper left is the movie we showed two subjects on the upper right is

two subjects on the upper right is

two subjects on the upper right is information we’re decoding from primary

information we’re decoding from primary

information we’re decoding from primary visual cortex and the bottom two photos

visual cortex and the bottom two photos

visual cortex and the bottom two photos show you the edge maps these should

show you the edge maps these should

show you the edge maps these should correspond well because we know that

correspond well because we know that

correspond well because we know that this part of the brain represents edges

this part of the brain represents edges

this part of the brain represents edges here’s a decoder that is recovering

here’s a decoder that is recovering

here’s a decoder that is recovering information and decoding it from

information and decoding it from

information and decoding it from high-level visual areas these visual

high-level visual areas these visual

high-level visual areas these visual areas they don’t really carry a lot of

areas they don’t really carry a lot of

areas they don’t really carry a lot of information about this specific

information about this specific

information about this specific structure of a scene the distribution of

structure of a scene the distribution of

structure of a scene the distribution of surfaces and colors and textures they

surfaces and colors and textures they

surfaces and colors and textures they carry information about the content of

carry information about the content of

carry information about the content of the scene what people were present were

the scene what people were present were

the scene what people were present were their objects present the the semantic

their objects present the the semantic

their objects present the the semantic information the scene and you can see

information the scene and you can see

information the scene and you can see that this decoder also works fairly well

that this decoder also works fairly well

that this decoder also works fairly well it doesn’t do well when the scene

it doesn’t do well when the scene

it doesn’t do well when the scene changes very rapidly like right now

changes very rapidly like right now

changes very rapidly like right now because these MRI changes that we are

because these MRI changes that we are

because these MRI changes that we are measuring our metabolic not neural and

measuring our metabolic not neural and

measuring our metabolic not neural and they are fairly slow but when the scene

they are fairly slow but when the scene

they are fairly slow but when the scene changes slowly the decoder does quite a

changes slowly the decoder does quite a

changes slowly the decoder does quite a good job of recovering information about

good job of recovering information about

good job of recovering information about the semantic content of the scene so

the semantic content of the scene so

the semantic content of the scene so where is this technology going

where is this technology going

where is this technology going well the analogy I like to make is with

well the analogy I like to make is with

well the analogy I like to make is with early photography this is the very first

early photography this is the very first

early photography this is the very first photograph that was taken by Joseph

photograph that was taken by Joseph

photograph that was taken by Joseph Nietzsche in the South of France in the

Nietzsche in the South of France in the

Nietzsche in the South of France in the early 19th century and this is a bad

early 19th century and this is a bad

early 19th century and this is a bad photograph right no one would pay for

photograph right no one would pay for

photograph right no one would pay for this photograph but you can see that it

this photograph but you can see that it

this photograph but you can see that it is a photo there are walls there’s a

is a photo there are walls there’s a

is a photo there are walls there’s a horizon there’s a roof you can tell it’s

horizon there’s a roof you can tell it’s

horizon there’s a roof you can tell it’s maybe some sort of farm scene or a

maybe some sort of farm scene or a

maybe some sort of farm scene or a village scene it is a photograph and

village scene it is a photograph and

village scene it is a photograph and this photograph launched an entire

this photograph launched an entire

this photograph launched an entire industry that sought to develop

industry that sought to develop

industry that sought to develop photography into what we have today

photography into what we have today

photography into what we have today where photography is so ubiquitous and

where photography is so ubiquitous and

where photography is so ubiquitous and of such high quality that no one even

of such high quality that no one even

of such high quality that no one even wants to pay for it they expect to just

wants to pay for it they expect to just

wants to pay for it they expect to just get it for free on their cell phones and

get it for free on their cell phones and

get it for free on their cell phones and brain decoding we’ll pursue this same

brain decoding we’ll pursue this same

brain decoding we’ll pursue this same trajectory as long as the government is

trajectory as long as the government is

trajectory as long as the government is investing in neuroscience and

investing in neuroscience and

investing in neuroscience and neuroscientists are working to

neuroscientists are working to

neuroscientists are working to understand the brain they’ll be

understand the brain they’ll be

understand the brain they’ll be developing new methods of measurement

developing new methods of measurement

developing new methods of measurement those new brain measurement methods will

those new brain measurement methods will

those new brain measurement methods will allow us to create more sophisticated

allow us to create more sophisticated

allow us to create more sophisticated computational models of the brain and

computational models of the brain and

computational models of the brain and those computational models will allow us

those computational models will allow us

those computational models will allow us to decode brain activity with much

to decode brain activity with much

to decode brain activity with much higher fidelity so at some point in the

higher fidelity so at some point in the

higher fidelity so at some point in the future we’re going to have very cheap

future we’re going to have very cheap

future we’re going to have very cheap very powerful portable brain decoding

very powerful portable brain decoding

very powerful portable brain decoding devices that we’re not there yet as I

devices that we’re not there yet as I

devices that we’re not there yet as I mentioned at the beginning of this talk

mentioned at the beginning of this talk

mentioned at the beginning of this talk the brain is a nonlinear dynamical

the brain is a nonlinear dynamical

the brain is a nonlinear dynamical system and one of the important

system and one of the important

system and one of the important properties of these systems is that in

properties of these systems is that in

properties of these systems is that in order to understand the entire dynamical

order to understand the entire dynamical

order to understand the entire dynamical surface the sort of dynamical landscape

surface the sort of dynamical landscape

surface the sort of dynamical landscape you need to record in both space and

you need to record in both space and

you need to record in both space and time at very high resolution

time at very high resolution

time at very high resolution simultaneously but right now we don’t

simultaneously but right now we don’t

simultaneously but right now we don’t have any methods that allow us to record

have any methods that allow us to record

have any methods that allow us to record in both space and time all the methods

in both space and time all the methods

in both space and time all the methods we have either sacrificed time or they

we have either sacrificed time or they

we have either sacrificed time or they sacrifice space so for example

sacrifice space so for example

sacrifice space so for example functional MRI the method that I showed

functional MRI the method that I showed

functional MRI the method that I showed you earlier in the talk it has fantastic

you earlier in the talk it has fantastic

you earlier in the talk it has fantastic spatial resolution but very poor

spatial resolution but very poor

spatial resolution but very poor temporal resolution because it measures

temporal resolution because it measures

temporal resolution because it measures metabolic signals changes in blood flow

metabolic signals changes in blood flow

metabolic signals changes in blood flow that are consequent to neural activity

that are consequent to neural activity

that are consequent to neural activity it doesn’t measure changes in neural

it doesn’t measure changes in neural

it doesn’t measure changes in neural activity itself so when we’re I cannot

activity itself so when we’re I cannot

activity itself so when we’re I cannot really recover any dynamic information a

really recover any dynamic information a

really recover any dynamic information a very common consumer method that is used

very common consumer method that is used

very common consumer method that is used to measure and decode brain activity oh

to measure and decode brain activity oh

to measure and decode brain activity oh today is electroencephalography EEG and

today is electroencephalography EEG and

today is electroencephalography EEG and the EEG has fantastic temporal

the EEG has fantastic temporal

the EEG has fantastic temporal resolution but it has horrible spatial

resolution but it has horrible spatial

resolution but it has horrible spatial resolution so although it can measure

resolution so although it can measure

resolution so although it can measure happening in time it does not know where

happening in time it does not know where

happening in time it does not know where those events came from in the brain so

those events came from in the brain so

those events came from in the brain so all the different maps that occur in the

all the different maps that occur in the

all the different maps that occur in the brain get all essentially mixed together

brain get all essentially mixed together

brain get all essentially mixed together and this is reduces the amount of

and this is reduces the amount of

and this is reduces the amount of information we can decode from EEG to

information we can decode from EEG to

information we can decode from EEG to just a few bits but we do know that

just a few bits but we do know that

just a few bits but we do know that brain measurement improves all the time

brain measurement improves all the time

brain measurement improves all the time because neuroscience community is very

because neuroscience community is very

because neuroscience community is very focused on improving measurement since

focused on improving measurement since

focused on improving measurement since measurement limitations are the main

measurement limitations are the main

measurement limitations are the main factor we have limiting our progress in

factor we have limiting our progress in

factor we have limiting our progress in neuroscience today so in the future we

neuroscience today so in the future we

neuroscience today so in the future we are going to end up with better brain

are going to end up with better brain

are going to end up with better brain measurement methods that will recover

measurement methods that will recover

measurement methods that will recover information in both space and time and

information in both space and time and

information in both space and time and at high resolution we don’t know when

at high resolution we don’t know when

at high resolution we don’t know when this is going to happen it could be a

this is going to happen it could be a

this is going to happen it could be a decade it could be 20 years it could be

decade it could be 20 years it could be

decade it could be 20 years it could be 50 years down the road but at some point

50 years down the road but at some point

50 years down the road but at some point probably in the lifetimes of young

probably in the lifetimes of young

probably in the lifetimes of young people here there will be a brain

people here there will be a brain

people here there will be a brain decoding method that is cheap portable

decoding method that is cheap portable

decoding method that is cheap portable very powerful and that will become

very powerful and that will become

very powerful and that will become ubiquitous people will you know be able

ubiquitous people will you know be able

ubiquitous people will you know be able to just wear a brain hat around and have

to just wear a brain hat around and have

to just wear a brain hat around and have their internal thoughts decoded all the

their internal thoughts decoded all the

their internal thoughts decoded all the time

time

time and this would have a very profound

and this would have a very profound

and this would have a very profound effect on society after all how do I

effect on society after all how do I

effect on society after all how do I interact with my computer today I used

interact with my computer today I used

interact with my computer today I used these meaty thumbs that were evolved

these meaty thumbs that were evolved

these meaty thumbs that were evolved billions of years ago millions of years

billions of years ago millions of years

billions of years ago millions of years ago and it’s kind of insane for me to

ago and it’s kind of insane for me to

ago and it’s kind of insane for me to interact with my cellphone with my

interact with my cellphone with my

interact with my cellphone with my thumbs when I have a very good internal

thumbs when I have a very good internal

thumbs when I have a very good internal speech a little man in my head that’s

speech a little man in my head that’s

speech a little man in my head that’s talking to me continuously and it could

talking to me continuously and it could

talking to me continuously and it could just interact with my cellphone directly

just interact with my cellphone directly

just interact with my cellphone directly in fact anything that is going on in

in fact anything that is going on in

in fact anything that is going on in your head is potentially decodable all

your head is potentially decodable all

your head is potentially decodable all of your inner thoughts your intentions

of your inner thoughts your intentions

of your inner thoughts your intentions desires your attitudes in fact things

desires your attitudes in fact things

desires your attitudes in fact things you haven’t even that haven’t even

you haven’t even that haven’t even

you haven’t even that haven’t even reached conscious awareness yet are

reached conscious awareness yet are

reached conscious awareness yet are potentially decodable and all that

potentially decodable and all that

potentially decodable and all that information will be decoded using future

information will be decoded using future

information will be decoded using future brain decoding technology now that’s

brain decoding technology now that’s

brain decoding technology now that’s both exciting and frankly quite scary

both exciting and frankly quite scary

both exciting and frankly quite scary because you can imagine all the ethical

because you can imagine all the ethical

because you can imagine all the ethical issues that this will open up after all

issues that this will open up after all

issues that this will open up after all Americans in particular but humans in

Americans in particular but humans in

Americans in particular but humans in general are pretty bad about guarding

general are pretty bad about guarding

general are pretty bad about guarding their privacy and there are real

their privacy and there are real

their privacy and there are real interesting privacy and ethical issues

interesting privacy and ethical issues

interesting privacy and ethical issues are going to be opened up when Google

are going to be opened up when Google

are going to be opened up when Google and Facebook have access to all of your

and Facebook have access to all of your

and Facebook have access to all of your innermost thoughts with no filter from

innermost thoughts with no filter from

innermost thoughts with no filter from your thumbs right that’s going to be a

your thumbs right that’s going to be a

your thumbs right that’s going to be a very strange world but it is the world

very strange world but it is the world

very strange world but it is the world we’re going to live in very soon thank

we’re going to live in very soon thank

we’re going to live in very soon thank you for your time