So when I got the confirmation to talk at this TEDx event,
the first thing I did was calling my brother
to tell him that I would talk about research methods and statistics
at this event.
And his response was very cute.
He said to me, “You know, I love you.
And you know, I believe in you, and I’ve seen you do incredible things,
but do you really think that this is a good idea?”
Then he said, “You know, nobody can talk about this topic
without using a lot of math, being extremely boring,
and scaring everybody off.”
And as you can see, I haven’t changed my topic.
Not because I’m a really stubborn little sister,
but because I love challenges, and I think that I can talk about this topic
without math, without being boring, without scaring you off.
But the reaction of my brother was the common reaction I got
from anybody I told what I would do here today.
That made me reflect on emotion we have towards research methods and statistics,
and hearing just the term seems to bring up
all the negative emotions and associations we have towards this topic.
Mostly, there are two emotions which I found quite visible,
which were boredom and fear.
If you try to put these two together, you realize how weird this is,
and I actually tried it.
Because these two can be located
at the extreme ends of a continuum of arousal.
Here you can see what it might look like if you try to put both of them together.
This shows how weird emotions are
we have towards research methods and statistics.
However, despite their disparity, they share a common reaction in people.
Because when we are afraid,
we have learned that the only right way to behave
is run away as fast as we can and hide.
When we’re bored,
things don’t even cross the cut-off line to make us care at all.
So, both of these emotions lead to avoidance.
But avoidance leads to more ignorance,
to more disbelief, to more doubt.
We fear how research methods and statistics might trick or deceive us,
and this is how vicious cycle is born.
Now whenever somebody tells me
about the torture of being sent to a research methods and statistics class,
or when somebody is complaining about some form of information
presented too geeky,
I always realize this person is trapped in this vicious cycle.
This always makes me really, really sad.
Because I think little that you know how much you’re missing.
You might tell me now, “Come on, this is research methods and statistics!
What could we be possibly missing out?”
And I tell you: how about feeling powerful,
feeling in charge, feeling free, independent,
and incredibly proud?
At least these were my feelings, the feelings I had back then,
in second semester of my undergrad studies,
when I conducted my first own little research project.
That was the moment why I became aware of the power of methods,
and the empowerment given to the ones who know how to properly use them.
In that moment, I had mastered methods instead of been mastered by methods.
To all of you who think that individual empowerment
is not enough motivation to overthink our attitude
towards research methods and statistics,
I would like to elaborate more on why it made me feel this way.
To the ones who ask me, “Why should we care?”
I tell you, I’ll show you what happens if we don’t.
Before I now go on with the next part of my presentation,
I’d like to introduce it with a quote that many of you might know,
that fits, I think, quite well the context of my presentation.
Because “Fear is the path to the dark side.
Fear leads to anger.
Anger leads to hate.
Hate leads to suffering.”
This is a quote made by Yoda.
But that doesn’t make it less true,
and that’s what I will show you now in the following.
Others before me have envisioned worlds with wizards, with vampires, with hobbits,
and today I’ll try to be as courageously creative
and imagine a world without method.
We’re now abandoning the conviction of the exclusive video ranking view
and now not assuming anymore that scientifically produced knowledge
is the only or the best kind of knowledge.
I would like to describe a situation, a little utopia to you.
Today is your annual assessment.
You’re entering your boss’s office, and he sits there behind his big desk,
but instead of a big pile of papers, he suddenly takes out a dice.
Maybe a smaller one than this one, depending on his ego problems.
Like I’m doing now, he rolls it.
It’s three.
Then he looks at you and says,
“We’re really sorry. We’re really, really sorry.
We have to cut your payment by 20 percent.
We’ve done that to all the people who received the three.
This is the fairest procedure we can think of.
We have financial restrictions from the top.
So, I’m sorry about that.”
Yeah.
Surprisingly, a few weeks later,
he sees a huge decrease in employee satisfaction.
What a surprise.
Now he decides to, once again, go back to the scientific paradigm,
he sends out a survey link to ask the employees about their opinion.
The funny thing is that you participate,
you say how really angry you are about this treatment,
but a few weeks later, you see the results, and they’re quite OK.
Nothing really to bother about,
so your boss keeps on doing this procedure
because he sees no reason in changing that.
I would like to know, who of you thinks that this was a fair treatment?
When I asked people this and told them this little story,
their reaction was mostly: this is unfair, this is arbitrary, totally subjective
and poor decision making in every way.
Why do we feel this way?
To answer this, I’ll have a look at three questions.
The first is: what happens if you don’t use methods at all?
The little utopia I presented here was based on the rationale
of on purpose violating every form of quality criterion
we have developed for our knowledge production process in our society.
The first thing would be to ask yourself:
is the measurement really independent of the person assessing you?
The answer is no because your boss was the only one
who decided to change to this new and weird procedure,
so this is lacking objectivity.
The next question: is this a reliable measurement?
Every morning, when I go on my scale and I stand on it,
it shows me how much I weigh whether I like it or not.
But it shows me this reliably.
In this case, rolling a dice is influenced by a lot of confining variables,
mostly by chance.
So, this is not a reliable measurement.
That brings us to the third aspect: is this a valid measurement?
The question we’re asking in this case is:
are we really measuring what we want to measure?
I hugely question whether we’re measuring anything at all by rolling a dice.
At least we’re not measuring what we want to measure
if we want to link this to performance because there’s literally no relationship
between your performance and rolling a dice.
You might say now, “Okay, yeah, great, you’ve created this great utopia,
everything works out fine, so why should we bother?”
I admit, I developed this to increase the effect size,
to make visible what might be overlooked in the small.
I’ve never met anybody who told me, “Yes, when I make decisions,
I think about these three concepts every time.”
We don’t do this.
We leave this to researchers, to statisticians
because we don’t think this is important.
But by doing this, we willingly accept
to chose the wrong candidate for a job position,
to punish our children more severely than would be fair,
and in general make our life so much harder by making bad decisions.
So to all of the people who are now in research,
we should feel obliged to this quality criteria,
and they’re not only quality criteria for quantitative research
but also for qualitative research.
We should prove ourselves worthy with the great trust
society has given upon us to produce good and sound knowledge.
So now, what happens if we use methods, but we use them poorly?
Before we do that, I’d like to ask you three questions to reflect on,
which is – the first, who of you values free speech?
Please, I know everybody’s tired, but raise your hands.
Yes. There are a lot of people who value free speech. Amazing.
Who of you cares about standard deviations? Whoo!
No? Okay.
Oh, yeah. One person. Thank you very much.
Who of you thinks that there is a relationship
between free speech and standard deviations?
Okay, I think I’m the only one.
But I’ll explain why I think there is a relation.
Let’s look at the survey results.
Here you can see what the results of the survey of your boss were.
Nothing really surprising, everything is fine.
I understand why he keeps on doing what he’s started to do.
But now I’ll show you a different picture.
I put in here a graphical visualization of standard deviations.
As you can see –
Okay, standard deviations mean and show
how much the individual rating differs from the mean.
Now you can see that not everybody was so satisfied.
Actually, there are huge differences.
I’ll start with one explanation why that might be the case.
Imagine that there were three departments
and only the last two received the dice treatment.
But because nobody bothered to report any information
on standard deviation, variance, inclusion or exclusion criteria,
you have just this one mean average,
and you, who were really dissatisfied with the treatment, you were silenced.
Nobody knew what you expressed as an opinion
because somebody thought it was enough to report a mean.
This goes not only for answering questionnaires.
When you think about sampling,
you see that we should make sure that we ask the ones
we really are interested in.
When you’re interested in the effect of a treatment,
ask the ones who are affected by it.
Whenever you think “this never happens” because this is so obviously wrong
that nobody would do this,
I ask you: how often do we talk to the ones who were there
instead of searching for the ones we ought to talk to?
How often do we listen to the ones who scream to our faces,
instead of making the ones who don’t speak up?
Talk to us.
This is a big thing – whenever you do study,
you’re giving somebody the opportunity to speak up,
to say what they think about the world.
This is very important.
Make sure you talk to the right people.
You give them the chance to express what they think about the world.
Even when they’re silent or invisible, make them visible.
This is how you secure that minorities are heard,
that we value pluralism, and have diversity in our society.
I now want to talk to people who are asked to participate in surveys,
refuse and don’t think that this is a problem.
Imagine our situation
when all the departments receive the dice treatment.
Why is the first department still so much better off,
or why do they have so high satisfaction rates?
I give you now the number of respondents for each of the departments.
As you can see, in department one only four people participated.
Imagine that they benefited from the treatment, maybe.
They got a pay rise. Surely, they like the new treatment.
This is what happens when only a few people participate.
Extreme results get more likely.
So bear this in mind,
that whenever you’re asked to participate in a survey
and you trust research to on purpose select people they want to speak up,
that you have the chance to say what you think about the world,
and that is hugely important
because otherwise we base our decisions on just the opinions of a few.
Now, the last question.
What happens if we don’t really know how to work
with scientifically produced knowledge?
Last semester I brought a bike to one of my lecture session.
I asked one of the students to cycle with the bike
from one side to the other side of the lecture hall.
As you can imagine, he did this quite successfully.
Then I told him this is how it looks like to use knowledge.
You have a clear task,
you have the knowledge produced to exactly fulfill this task,
and you use the knowledge to fulfill it, quite easy.
Then I gave them a second task and said,
“You see, you have here now a square on the floor.
Imagine this is the trunk of my car.
Please put the bike – which was obviously too big – in my trunk”.
After a few unsuccessful trials,
I resolved the case and just removed he front wheel.
Then I told them,
“This is what it looks like to work with knowledge.”
We’re living in a complex world,
where we face with problems we haven’t encountered before,
with even wicked problems we don’t know how to solve, probably,
and we have to be creative with knowledge.
But you can’t be just creative.
There are parts which are fixed,
and there are parts which are free, which you can move,
and you have to know which parts you’re allowed to move.
So I hope that with this presentation
I showed you that methods are definitely more than math –
they don’t need to be boring, and they’re nothing to be afraid of,
and that anybody who cares for this is not a weird nerd,
but actually a warrior.
A warrior for a more just place, a fairer place,
a place where we value pluralism, free speech,
where we include minorities, secure diversity,
and give a voice to the unheard.
The mastery of methods will lead to your empowerment.
Thank you very much.
(Applause)
”
Be First to Comment