Detecting Multiple Sclerosis with Machine Learning | Grey Kuling | TEDxUofTSalon

it wasn’t until a conversation with the

it wasn’t until a conversation with the childhood friend

childhood friend

childhood friend that I realized the truth

that I realized the truth

that I realized the truth of my research and here’s what she said

of my research and here’s what she said

of my research and here’s what she said I’ve been very fortunate but it’s

I’ve been very fortunate but it’s

I’ve been very fortunate but it’s definitely because I tried to exercise

definitely because I tried to exercise

definitely because I tried to exercise and eat well but it hasn’t always been

and eat well but it hasn’t always been

and eat well but it hasn’t always been this good I’ve lost my vision in the

this good I’ve lost my vision in the

this good I’ve lost my vision in the past and my ability to walk but

past and my ability to walk but

past and my ability to walk but obviously I’ve got it back

obviously I’ve got it back

obviously I’ve got it back I have relapse remitting multiple

I have relapse remitting multiple

I have relapse remitting multiple sclerosis so I’m fine

sclerosis so I’m fine

sclerosis so I’m fine 90% of the time but yeah that other 10%

multiple sclerosis is an autoimmune

multiple sclerosis is an autoimmune disease for the immune system attacks

disease for the immune system attacks

disease for the immune system attacks the healthy neurons of the patient’s

the healthy neurons of the patient’s

the healthy neurons of the patient’s central nervous system this renders the

central nervous system this renders the

central nervous system this renders the neuron incapable of communicating

neuron incapable of communicating

neuron incapable of communicating signals to the rest of the body

signals to the rest of the body

signals to the rest of the body the damaged in scarred tissue of the

the damaged in scarred tissue of the

the damaged in scarred tissue of the neuron is called an MS lesion and we can

neuron is called an MS lesion and we can

neuron is called an MS lesion and we can visualize these using magnetic resonance

visualize these using magnetic resonance

visualize these using magnetic resonance imaging or MRI in analyzing different

imaging or MRI in analyzing different

imaging or MRI in analyzing different scans we see a very distinct pattern

scans we see a very distinct pattern

scans we see a very distinct pattern between the lesions and the healthy

between the lesions and the healthy

between the lesions and the healthy tissue and currently radiologists use

tissue and currently radiologists use

tissue and currently radiologists use this pattern to outline the lesions

this pattern to outline the lesions

this pattern to outline the lesions manually calling it a segmentation now

manually calling it a segmentation now

manually calling it a segmentation now this is important in all stages of an MS

this is important in all stages of an MS

this is important in all stages of an MS such as initial diagnosis evaluation of

such as initial diagnosis evaluation of

such as initial diagnosis evaluation of disease progression and determining

disease progression and determining

disease progression and determining treatment efficacy

treatment efficacy

treatment efficacy however manual segmentation is difficult

however manual segmentation is difficult

however manual segmentation is difficult and often variable to physicians may not

and often variable to physicians may not

and often variable to physicians may not agree on the same segmentation as well a

agree on the same segmentation as well a

agree on the same segmentation as well a physician may not segment the lesion the

physician may not segment the lesion the

physician may not segment the lesion the same way twice and this results in human

same way twice and this results in human

same way twice and this results in human error therefore it has been proposed to

error therefore it has been proposed to

error therefore it has been proposed to segment lesions automatically using

segment lesions automatically using

segment lesions automatically using machine learning now

machine learning now

machine learning now machine learning has become a hot topic

machine learning has become a hot topic

machine learning has become a hot topic in recent years it’s literally

in recent years it’s literally

in recent years it’s literally everywhere recognizing and learning

everywhere recognizing and learning

everywhere recognizing and learning patterns and information around us

patterns and information around us

patterns and information around us Google Facebook and now Apple’s own

Google Facebook and now Apple’s own

Google Facebook and now Apple’s own iPhone uses machine

iPhone uses machine

iPhone uses machine learning for facial recognition as well

learning for facial recognition as well

learning for facial recognition as well our email inboxes use machine learning

our email inboxes use machine learning

our email inboxes use machine learning to filter out spam we could use a

to filter out spam we could use a

to filter out spam we could use a machine learning algorithm to learn the

machine learning algorithm to learn the

machine learning algorithm to learn the pattern of lesions in these four scans

pattern of lesions in these four scans

pattern of lesions in these four scans but first what information are we

but first what information are we

but first what information are we looking at in these images

looking at in these images

looking at in these images what is this data exactly well a picture

what is this data exactly well a picture

what is this data exactly well a picture is a matrix of pixel intensities at a

is a matrix of pixel intensities at a

is a matrix of pixel intensities at a given x and y coordinate we then can

given x and y coordinate we then can

given x and y coordinate we then can stack the images on top of each other

stack the images on top of each other

stack the images on top of each other making a third dimension so we can

making a third dimension so we can

making a third dimension so we can visualize the data and tissue in 3d this

visualize the data and tissue in 3d this

visualize the data and tissue in 3d this then turns the pixel intensity into a

then turns the pixel intensity into a

then turns the pixel intensity into a volumetric intensity or a voxel in a

volumetric intensity or a voxel in a

volumetric intensity or a voxel in a given X Y and Zed coordinate so if we

given X Y and Zed coordinate so if we

given X Y and Zed coordinate so if we were looking at these voxels ourselves

were looking at these voxels ourselves

were looking at these voxels ourselves and we wanted to try and identify what

and we wanted to try and identify what

and we wanted to try and identify what tissue were looking at how would we

tissue were looking at how would we

tissue were looking at how would we figure that it well let’s make it a

figure that it well let’s make it a

figure that it well let’s make it a little bit simpler if we were looking at

little bit simpler if we were looking at

little bit simpler if we were looking at any kind of object how would we

any kind of object how would we

any kind of object how would we recognize what we’re looking at say for

recognize what we’re looking at say for

recognize what we’re looking at say for example animals we could build a

example animals we could build a

example animals we could build a dictionary of examples in our memory

dictionary of examples in our memory

dictionary of examples in our memory that we could reference and it turns out

that we could reference and it turns out

that we could reference and it turns out that a computer can do the same so that

that a computer can do the same so that

that a computer can do the same so that it could identify a gorilla from a lion

it could identify a gorilla from a lion

it could identify a gorilla from a lion in dictionary learning we take a

in dictionary learning we take a

in dictionary learning we take a collection of image examples and we take

collection of image examples and we take

collection of image examples and we take the pixel intensities and we rearrange

the pixel intensities and we rearrange

the pixel intensities and we rearrange them into a vector we then place those

them into a vector we then place those

them into a vector we then place those vector examples side-by-side into a new

vector examples side-by-side into a new

vector examples side-by-side into a new matrix that we call a dictionary then we

matrix that we call a dictionary then we

matrix that we call a dictionary then we can identify a test subject by using

can identify a test subject by using

can identify a test subject by using regression analysis now this just means

regression analysis now this just means

regression analysis now this just means that we try to recreate the test subject

that we try to recreate the test subject

that we try to recreate the test subject by using a select amount of dictionary

by using a select amount of dictionary

by using a select amount of dictionary examples so if a dictionary of lions

examples so if a dictionary of lions

examples so if a dictionary of lions tried to identify a gorilla it wouldn’t

tried to identify a gorilla it wouldn’t

tried to identify a gorilla it wouldn’t do a very good job

do a very good job

do a very good job and the error would be large but if it

and the error would be large but if it

and the error would be large but if it tried to identify a lion the error would

tried to identify a lion the error would

tried to identify a lion the error would be small declaring that it is indeed

be small declaring that it is indeed

be small declaring that it is indeed perceiving a lion in the case of MS we

perceiving a lion in the case of MS we

perceiving a lion in the case of MS we want to look at the individual voxels

want to look at the individual voxels

want to look at the individual voxels and determine what kind of tissue were

and determine what kind of tissue were

and determine what kind of tissue were looking at specifically gray matter

looking at specifically gray matter

looking at specifically gray matter white matter cerebral spinal fluid and

white matter cerebral spinal fluid and

white matter cerebral spinal fluid and ms lesions so we could build the

ms lesions so we could build the

ms lesions so we could build the dictionary for all four the question now

dictionary for all four the question now

dictionary for all four the question now is how will we characterize each

is how will we characterize each

is how will we characterize each individual voxel to build examples in

individual voxel to build examples in

individual voxel to build examples in our dictionary we’re not looking at a

our dictionary we’re not looking at a

our dictionary we’re not looking at a whole picture anymore we’re just looking

whole picture anymore we’re just looking

whole picture anymore we’re just looking at a small portion of it well

at a small portion of it well

at a small portion of it well classically researchers have taken a

classically researchers have taken a

classically researchers have taken a large spatial patch of the voxel

large spatial patch of the voxel

large spatial patch of the voxel intensities that surround the voxel of

intensities that surround the voxel of

intensities that surround the voxel of interest so the voxel of interest is in

interest so the voxel of interest is in

interest so the voxel of interest is in the center and then we take all the

the center and then we take all the

the center and then we take all the information around them we would

information around them we would

information around them we would rearrange that information into a vector

rearrange that information into a vector

rearrange that information into a vector and then build our dictionaries so let’s

and then build our dictionaries so let’s

and then build our dictionaries so let’s see how well this turns out this is a

see how well this turns out this is a

see how well this turns out this is a picture of the ground truth this is the

picture of the ground truth this is the

picture of the ground truth this is the correct answer this is what we want to

correct answer this is what we want to

correct answer this is what we want to see the algorithm recreate and these are

see the algorithm recreate and these are

see the algorithm recreate and these are the results we get it turns out it

the results we get it turns out it

the results we get it turns out it doesn’t do a very good job there’s a lot

doesn’t do a very good job there’s a lot

doesn’t do a very good job there’s a lot of over segmentation and a lot of false

of over segmentation and a lot of false

of over segmentation and a lot of false positives my supervisor dr. hamaji and I

positives my supervisor dr. hamaji and I

positives my supervisor dr. hamaji and I looked at this critically and thought it

looked at this critically and thought it

looked at this critically and thought it must be looking at too much information

must be looking at too much information

must be looking at too much information at once resulting in too many confusing

at once resulting in too many confusing

at once resulting in too many confusing factors so we thought why not just keep

factors so we thought why not just keep

factors so we thought why not just keep it simple let’s just use the voxel in

it simple let’s just use the voxel in

it simple let’s just use the voxel in question and not a spatial patch so we

question and not a spatial patch so we

question and not a spatial patch so we tried this out and it’s segmented the

tried this out and it’s segmented the

tried this out and it’s segmented the lesions perfectly now we were a little

lesions perfectly now we were a little

lesions perfectly now we were a little skeptical about these results because we

skeptical about these results because we

skeptical about these results because we actually had pre-processed our data to

actually had pre-processed our data to

actually had pre-processed our data to be ideal we took out any noise or

be ideal we took out any noise or

be ideal we took out any noise or distortions in the images that may make

distortions in the images that may make

distortions in the images that may make the information confusing to the

the information confusing to the

the information confusing to the computer so we thought why not

computer so we thought why not

computer so we thought why not bit up without their level make it

bit up without their level make it

bit up without their level make it harder and try and segment images with

harder and try and segment images with

harder and try and segment images with three percent noise added and it

three percent noise added and it

three percent noise added and it couldn’t handle the noise and our false

couldn’t handle the noise and our false

couldn’t handle the noise and our false positives returned to solve this problem

positives returned to solve this problem

positives returned to solve this problem we asked ourselves what is the noise

we asked ourselves what is the noise

we asked ourselves what is the noise changing in these photos if we take a

changing in these photos if we take a

changing in these photos if we take a look at them side by side we can see

look at them side by side we can see

look at them side by side we can see that the lines are a little bit blurry

that the lines are a little bit blurry

that the lines are a little bit blurry but the biggest difference is between

but the biggest difference is between

but the biggest difference is between the lines we can see that the white

the lines we can see that the white

the lines we can see that the white matter and cerebral spinal fluid have

matter and cerebral spinal fluid have

matter and cerebral spinal fluid have gone from one solid color to a fuzzy

gone from one solid color to a fuzzy

gone from one solid color to a fuzzy pattern so if you were looking at this

pattern so if you were looking at this

pattern so if you were looking at this fuzzy pattern how would you describe it

fuzzy pattern how would you describe it

fuzzy pattern how would you describe it I’ll make it a little bit simpler how

I’ll make it a little bit simpler how

I’ll make it a little bit simpler how would you describe the difference

would you describe the difference

would you describe the difference between these images each panel has a

between these images each panel has a

between these images each panel has a very distinct texture the texture of

very distinct texture the texture of

very distinct texture the texture of ashphalt is contrasting ly different

ashphalt is contrasting ly different

ashphalt is contrasting ly different than the texture of a brick wall

than the texture of a brick wall

than the texture of a brick wall therefore our brains understand what

therefore our brains understand what

therefore our brains understand what we’re looking at we subconsciously

we’re looking at we subconsciously

we’re looking at we subconsciously analyze the texture and it turns out

analyze the texture and it turns out

analyze the texture and it turns out again a computer can do the same by

again a computer can do the same by

again a computer can do the same by looking at the pixel intensities we can

looking at the pixel intensities we can

looking at the pixel intensities we can count how often a certain intensity

count how often a certain intensity

count how often a certain intensity level shows up in the picture by

level shows up in the picture by

level shows up in the picture by Counting the frequency of each intensity

Counting the frequency of each intensity

Counting the frequency of each intensity level we get a probability distribution

level we get a probability distribution

level we get a probability distribution then we can calculate the mean the

then we can calculate the mean the

then we can calculate the mean the standard deviation the entropy and all

standard deviation the entropy and all

standard deviation the entropy and all sorts of information we wanted to build

sorts of information we wanted to build

sorts of information we wanted to build a dictionary for all the tissues that

a dictionary for all the tissues that

a dictionary for all the tissues that uses the voxel intensity in question and

uses the voxel intensity in question and

uses the voxel intensity in question and a texture quantification calculated

a texture quantification calculated

a texture quantification calculated statistically from the adjacent voxels

statistically from the adjacent voxels

statistically from the adjacent voxels to the voxel of interest now as it turns

to the voxel of interest now as it turns

to the voxel of interest now as it turns out as you increase the noise the

out as you increase the noise the

out as you increase the noise the standard deviation will increase but the

standard deviation will increase but the

standard deviation will increase but the mean will stay relatively the same since

mean will stay relatively the same since

mean will stay relatively the same since the standard deviation is so sensitive

the standard deviation is so sensitive

the standard deviation is so sensitive to the noise we coupled it we

to the noise we coupled it we

to the noise we coupled it we the voxel intensity and rebuilt our

the voxel intensity and rebuilt our

the voxel intensity and rebuilt our dictionaries and it solved the problem

dictionaries and it solved the problem

dictionaries and it solved the problem it could even segment small lesions that

it could even segment small lesions that

it could even segment small lesions that are hard to find now these are very

are hard to find now these are very

are hard to find now these are very exciting results but unfortunately it

exciting results but unfortunately it

exciting results but unfortunately it might be a while before this becomes

might be a while before this becomes

might be a while before this becomes clinical use the results that I’ve shown

clinical use the results that I’ve shown

clinical use the results that I’ve shown you today were calculated on a simulated

you today were calculated on a simulated

you today were calculated on a simulated brain scan from McGill University called

brain scan from McGill University called

brain scan from McGill University called brain web and papers that have made the

brain web and papers that have made the

brain web and papers that have made the jump from simulated data to real patient

jump from simulated data to real patient

jump from simulated data to real patient data have dropped in accuracy by almost

data have dropped in accuracy by almost

data have dropped in accuracy by almost 50% I currently at this moment do not

50% I currently at this moment do not

50% I currently at this moment do not have a database of real patient data to

have a database of real patient data to

have a database of real patient data to test on and the approval of a clinical

test on and the approval of a clinical

test on and the approval of a clinical trial to acquire such data set depends

trial to acquire such data set depends

trial to acquire such data set depends on one imminent issue accountability

on one imminent issue accountability

on one imminent issue accountability when a scan is segmented manually for

when a scan is segmented manually for

when a scan is segmented manually for lesions a physician is diagnosing the

lesions a physician is diagnosing the

lesions a physician is diagnosing the lesions if the segmentation was done

lesions if the segmentation was done

lesions if the segmentation was done automatically who is doing the

automatically who is doing the

automatically who is doing the diagnosing now this is the big issue

diagnosing now this is the big issue

diagnosing now this is the big issue with introducing machine learning to

with introducing machine learning to

with introducing machine learning to medicine now in the case of dictionary

medicine now in the case of dictionary

medicine now in the case of dictionary learning the machine makes a decision

learning the machine makes a decision

learning the machine makes a decision based off of the dictionary it was

based off of the dictionary it was

based off of the dictionary it was provided with therefore it’s like I’ve

provided with therefore it’s like I’ve

provided with therefore it’s like I’ve trained assistant but it’s the doctor

trained assistant but it’s the doctor

trained assistant but it’s the doctor who needs to make the final decision

who needs to make the final decision

who needs to make the final decision it’s not a question of will the machine

it’s not a question of will the machine

it’s not a question of will the machine make the wrong choice but will the

make the wrong choice but will the

make the wrong choice but will the doctor allow the machine to make the

doctor allow the machine to make the

doctor allow the machine to make the wrong choice thank you

wrong choice thank you

wrong choice thank you you