More on so called tests of EI - May 2004 letter to David Caruso

I have been thinking about the MSCEIT test and what Roberts and Matthews etc. said that it might miss an emotional genius. Could you take a look at this and give me your thoughts? I am working on an update to my page on EI tests.

Your 6 year old daughter starts to cry because her 16 year old brother doesn't want her tagging along when he goes out with his friends. As the parent, what is the most effective response?

A. Tell her to go to her room until she has stopped crying.

B. Tell your son to stop being so selfish and invite her along.

C. Invite her to help you in the kitchen. Tell her you could really use her help.

D. Tell her that her older brother really loves her but sometimes he needs to spend time with his friends.

So which one of these is the best answer? First, what do we mean by effective? Do we mean

Depending on how you define the word "effective", certain answers are "better" than others. The other problem is that there are lots more possible answers. Almost an infinite amount of answers really. This is not at all like a math test, or even a reading comprehension test.

But anyhow, the way I would define "effective" would be a combination of several of these possible choices. I would want her to feel validated and cared about. I would also want to build the relationship between the two of them. I would also want to help them learn to work things out between themselves so I didn't have to be involved and no one would be upset, crying or resentful. But the main thing would be to help her feel validated and cared about. I would evaluate the "best" answer according to how well it resulted in her feeling validated and cared about. Remember, she is just six years old. There will be time for more theoretical emotional education latter. This is the time for a literally "hands on" response. In other words, I would say she needs a hug. But that is not an option. And I would suspect many PhDs would not notice that it was not an option. They tend to think in cognitive responses to problems. (But I would guess that a lot of 14 year old females would say "a hug" should be part of the "best" answer.)

So I would say none of the above four choices is the best answer. But on the MSCEIT test, "none of the above" is not an option. And if it were, how would you know what the person thought the best answer was, unless you allowed for written mini-essays? In other words, how would you know what other ideas the test taker might have? There might be some other fantastic answer, or several other fantastic answers. But you will never know what they are on a multiple choice test.

For example, in this situation, I would say it would be better to validate her feelings. And I would definitely say it would be better to offer her a hug. And I would ask her how she is feeling. (I suspect not many PhD's, not even psychologists, or perhaps especially not psychologists or psychiatrists, would think of doing this either!) Then I would ask the brother how he is feeling, then I would try to guide them to a solution which both feel good about.

But this isn't an answer. And a few years ago, I would have never thought of this as a possible test answer. It was not until I had many new life experiences and spent a lot of time on independent research and self-study and self-growth, and a lot of time talking to teenage females and reading their journal entries, that I began to think of answers like this.

And I am not sure how many of the "experts" would think of my answer as a possible answer. Nor am I sure how many would agree with me that it is the "best" answer, even if it were one of the options.

1. The four possible answers in a multiple choice question are limited to the imagination, knowledge, wisdom, experience, etc. of the test authors.

2. The possibility of someone offering a better answer does not exist because the choices are limited by the test design.

3. Tests of emotional problem solving are not like math test where you can say "What is 2+2?" or "What is the 7,836 divided by 2.89?" The "best" answer will always be subject to debate.

4. There are at least two ways the "best" answer will always be subject to debate. First, the selection the "best" answer among those offered by the test authors can be debated by the "experts" who are selecting the "best" answers. It is possible, even likely that sometimes the "experts" will disagree with the test authors themselves! For example, when the test authors designed the test, they might have been thinking "A" was the best answer. But when the "experts" looked at the test the "consensus" might be that "B" or "C" was the best answer. The test authors and the "experts" could then easily enter into a heated and lengthy debate about why "their" answer is the "best" answer.

Second, the test taker could debate about what the right answer is. Let's say the experts choose "B". And let's say there was a division between the "experts." Maybe 49% of them chose answer "A" along with test authors themselves. Then when the test taker selects "A", he has a very strong case that his answer is the "right" answer. (Of course we don't know when the test authors and the "experts" might have disagreed, but I plan to ask one of the authors to see if they will comment on this!)

It may also be that the 49% of the "experts" who were in the majority, were actually the most emotionally intelligent of the group. They just happen to be somewhat of non-conformists compared to the norm. I am definitely not satisfied that the "norm" is the best indication of emotional intelligence. In fact, this leads to another problem, which I discuss next. (By the way, let's look at the elections in the USA in 2004 - would it be fair to say the most intelligent people were in the majority when the country voted for George Bush? What about the most emotionally intelligent?)

5. By it's design, a bell curve type distribution of intelligence tells us that the most intelligent people will be in the minority. The person of average intelligence will be in the middle of the curve. But those of high intelligence will be the farthest off to the right side of the curve. The further they are to the right, the more intelligent they are. Thus, it would seem that the most intelligent people would be less likely to select the same answer as those in the "norm." So it is possible, then ,that the test authors themselves are more emotionally intelligent than the "experts" who have been selected to chose the best answers. In this case, the results of the MSCEIT will be misleading as to who really is the most emotionally intelligent among us. (Maybe it would be good to remind you how the MSCEIT test is "scored", in other words, how the best answers to the multiple test questions are selected. If you want to check this, click here.)

6. A final problem I see with the way the MSCEIT test is scored is that even if you ask people you believe are "experts", and you get a consensus, you are still dependent upon their level of imagination, knowledge, experience etc. If they were all trained in similar ways, if they had similar educational experiences, it is likely they will tend to think in similar ways. And in fact, I would say that the experts were indeed trained in similar ways.

I would say they read many of the same books and learned to conduct their research, and learned to think in very similar ways. In order to get their PhD's they had to have certain things in common. For example, they had to be people who fit into the world of academia, with all its rules, procedures, bureaucracy, policies, and politics.

I would suggest, that this world of academia is not made for everyone. I go further and suggest it is not made for someone who places a very high value on feelings. I would suggest this is because feelings are not highly valued in such a world. What is more highly valued are thoughts and facts and research. It is not a world which would be likely to develop one's innate emotional intelligence. In fact, I would say quite the opposite occurs. Those who enter with a high level of emotional sensitivity, which certainly has to be one of the foundations of emotional intelligence, find that emotional sensitivity is not of much benefit to them in this world. It can be expected, then, that this sensitivity, as well as any kind of processing and use of feelings, will tend to atrophy the longer they stay in the system. It is a fundamenal law of nature to shed something which is not being used.

So what you are left with, perhaps, is a group of people who a) do not place a high value on feelings and b) who were all trained in a similar way. It follows that such a group can be expected to tend to give similar, and possibly somewhat biased, answers to subjective questions. But no matter how similar the answers are, they still may not be the "best" ways of solving the problem. A group of 14 year old females who would prefer to sit around and talk about their feelings rather than read journal articles about them, might very well be able to come up with better answers (assuming we could ever agree on what "better" even means!).

A group of 21(?) members of the xxx Association were given the MSCEIT test. The answers most often picked by them as the "correct" answer, were then used by Mayer, Salovey and Caruso as the "best" and "right" answer. My understanding is that this would be true even if Mayer, Salovey and Caruso (MSC) disagreed with the "experts" popular choice. MSC have presented research to support their decision to score the test this way, but I am still not fully satisfied.

Hi Steve,

One of the weaknesses of the MSCEIT is that it is not open-ended. In
addition, there are probably some items where we were not always
specific about the outcome or goal for managing emotions.

In our new test, the YV, we are quite clear about the goal. For example, consider
an item like this:

Debbie comes back from a great, relaxing vacation. She wants this
relaxing feeling to continue. How effective would each of these
strategies be in maintaining her relaxed feelings?

What makes it great? Doesn't this depend on how you define "great"? Also, most people here in peru don't go on vacations. The idea of "going on a great, relaxing vacation is literally foreign to them. Their employers don't give them vacation time. They work all year round. By traveling around the world I have learned things like this which I never thought of while I lived in the USA. It was a very sheltered, artificial life. Your test reflects too many American, Western ideas, values. That is one reason I cannot endorse it.

Also, why does she need to relax? This implies to me she has a stressful life, a stressful job. I don't like the system we have in much of the fast-paced world. Work hard, do things you don't feel good about, live under a lot of stress and pressure, transfer this stress and pressure to your kids, but make a lot of money so you can buy a big house, two cars, and take an expensive vacation for a few weeks a year.

I'd say it would be better for humanity if Debbie didn't just "relax" during her vacation, but instead, really thought about what she would rather be doing to earn a living, what could be more useful to the world, and what would make her feel more fulfilled.

a) Make a list of the things she needs to do.
b) ignore the feeling as it will go away anyway.
c) Reflect on the best parts of the vacation.

We should not use words such as effective or successful - we should use
words or phrases that stress adaptiveness.

Adaptiveness? This reminds me of the well-adusted slave question. "What do we call a well-adjusted slave?" I don't want people to "adapt" to the shitty life our fathers and grandfathers have created for us. I want us to change things.

Maybe Debbie 'should' make that list, but we are asking about effectiveness of emotion management
strategies. And in this test item, we are clear about the emotional outcome we are looking for. This is critical.

When you say "should" you put it in quotes. This tells me that you realize it is a very subjective word. What Debbie "should" do depends on who is doing the "shoulding." Her boss would want her to make a list of the things to do. But I might want her to tell her boss to go fuck himself. And Debbie might really want to do that, too. So what "should" Debbie do?

You have decided you want Debbie to say she wants this relaxing feeling to continue. Then you give three possible answers. But what about this possible answer.. what if she were to call her boss, tell him to fuck off, then call the airline and book another ticket to go back to the beach or wherever she was? Would you say that would be more effective than any of the answers you have provided? Or I guess I should say, "Any of the answers the emotional experts" have provided. And let me say right now, I don't believe these people are "emotional experts." I am not convinced. I want to know who they are. I want to ask them some questions myself. I want to know more about them. I want to know more about their families and personal lives. What I would really like is to talk to their children. When you say "emotional experts" I need you to prove to me they are experts in practice, not just in theory. If you can't prove this I will not take your word for it that they are "experts."

Also, what if I disagree with your choice of the "emotional outcome?" What if I don't want Debbie to continue feeling relaxed? What if I want her to feel sad that she is working in a job she doesn't really love? A job that is doing nothing for humanity?

To me, a test of emotional intelligence should always keep in mind what is good for humanity. Your isolated test questions miss this completely, I'd say. Emotional intelligence isn't like math. 2+2 is always 4, no matter who is doing the math. It doesn't matter if a person is using math to create something which will help humanity or something which kill thousands of people. 2=2 is still 4 for that person, just like it is for me and you.

Then this leads to what I call the "dark side" of emotional intelligence. We have to admit that a person who is emotionally intelligent can use this intelligence in destructive, hurtful ways. So we really are talking about two very different things.

One is raw emotional intelligence. The other is what I would call applied emotional intelligence. You and Jack have not done a good job of making this distinction. You have hardly ever even mentioned it, as far as I recall. Correct me if I am wrong.

I'd say we need two kinds of EI tests. One testing raw EI and one testing applied. What the MSCEIT test does is it tries to combine both. So I'd say, seperate them. Make two tests. And make it clear what you are trying to measure.

A test of raw emotional intelligence would tell us who has the potential. A test of applied EI would tell us what they are likely to do with that potential.

Take 16 year old Sarah. I have been talking to her since she was not quite 12 years old. I'd say she is an emotional genius, or very close to it. But as she lives in her emotionally dysfunctional home and goes to her emotionally dysfunctional school and lives in the emotionally dysfunctional USA, she becomes more dangerous to society each year. Or at the least, she becomes less useful. She has been in so much emotional pain that pretty much all she does now is try to find ways to stop or numb her pain. I have watched her change. Later I will probably show this change on my site so others can learn from it. The whole world has lost the special gift that Sarah had to offer us. I don't feel optimistic that Sarah will ever make a valuable contribution to humanity now. She may well end up killing herself. If she has children, she will use them emotionally to try to fill her own unmet emotional needs, just like my mother used me. In her relationships she will be so needy she will drive people away. Then she will feel rejected over and over and become more and more bitter. No one in Sarah's home or school or community saw the potential she had. They just wanted her to be like everyone else. Your country, the one you are so proud of, is forcing her to stay in the school system till she is 18. The law used to be 16. They have changed it recently. And the Americans are so brainwashed to think "education is good" that no one probably even objected to the new law. And they also have a new law saying they can put someone like Sarah in jail for missing more than 5 days of school, or even part of five days. (reference) So would you say there is more freedom now in the land of the free, or less? But I have digressed.

And such strategies are more or less effective. There is excellent research and data in this area, it is not merely an opinion. We created MSCEIT items by referring to such strategies but we - the authors - did not create the answer key. That was up to the emotions experts. (Several, open-ended IQ test tasks were developed in a similar way.)

Your example item about the 6 year old is a good example. You have to
specify the desired emotional outcome. If the outcome is to have the 6
year old feel understood, then one of the strategies you list later
would be quite effective. This is YOUR criterion of effectiveness! I
wonder whether listening and a hug would result in an outcome where the
6 year old feels not just validated, but where her feelings of
loneliness and isolation are directly addressed. What if the 16 year old
promised to play with her, but failed to do so? What would be the most
effective strategy in that case? It has to, largely, depend upon the
desired outcome.

- - - -

Ideally, you would individually administer an EI test, like an IQ test,
and record the answer. The answer would be evaluated based on how
effective it would be in achieving the stated outcome. To do this
'right' BTW, is a million dollar project. We know this as we worked with
a test publisher with such expertise, and were turned down by them after
nearly two years of study! Then, to train people to give the test is a
whole other issue.

- - - -

Unlike IQ tests, the MSCEIT can, and should, have multiple correct
answers. This is exactly your argument about the 49% / 51% situation.
It is not always black and white, and the scoring allows for this - I
view that as a good thing.

- - - -

Here is a creative poem:

Ibid et al non sibis.
You are a mon libis.
Ibid et al domin.
You are a non lomin.

Creative in that it is unique, but not very good or effective.
Emotional genius is communicating an emotion in a way that is unique,
original and powerful. But since emotions have meaning, a real
emotional genius has to be more than different or weird - they must
understand other people and grab other people in a unique and a
meaningful way.

- - -

When you talk about best answers to subjective questions, I completely
agree! "What's your favorite color?" is a subjective question.
However, "What emotion helps people to generate a large number of
ideas?" is a MSCEIT type question. And, as terrible as it sounds to
some, there is a better answer to this sort of question.

- - -

Remember that smart people don't always act smart. High EI people have
the potential to be emotionally smart - but they don't always show it.
As I tell my kids and my wife all the time, imagine what a knucklehead I
would be if I did not have at least a minimal level of EI skills! It is
these skills that allows me to rise above any of the negative influences
of my own upbringing as well as the negative personality traits I have.

In other words, an anxious person with low EI is probably a lot worse
off than an anxious person with high EI. The latter person can develop
compensatory strategies.

- - -

What drives me crazy is to see the MSCEIT, or any test, used in a manner
that suggests "this is your intelligence", or "The test says that you
are ...". A good test, and I feel the MSCEIT is a good test, should
generate hypotheses about a person that you need to take a look at.

We should say, "This result suggests that you may not be all that
accurate in identifying other's emotions. You may attend to emotion, but
there is a chance that you don't always read others accurately. That is
what the test result suggests, but let's consider whether it really is a
valid measure of your actual ability or not".

And when you do that, there should be times when you discover that the
test was wrong. With a good test, it should be more right than wrong.

David

Hi David,

Glad to see that is how you handled it. I guess I was counting on something like that!

I have been thinking about it too and will give you more to ponder soon probably!

You probably know that I really believe in the idea of EI and I want the world to take it more seriously. This is one reason I am so passionate about everything related to it. This really is a tricky thing to define and measure. I really hope that you each continue to stay open to suggestions, criticisms. That is how you will keep improving the tests and stay the leaders in the field.

To me it seems to be fair to say that we wouldn't feel defensive if there were not some truth in what someone said. Like if you say you think my hair is too green, for example, it doesn't have much emotional effect one me. What do you think?

Also, another question for you. Did I really "make" you feel defensive?!

What do you think about this? Does one person "make" another one feel a certain way?

I am glad that my questions make you think! It gives me a little more sense of value. Sometimes I really get depressed and even suicidal, as you know. So you are helping me too.

Eventually I'd like to get my site back on the top spot. It is number two this week and was number three for a long time when I wasn't writing much. And eventually I would like to have it taken more seriously by the acadmics and be seen more as a reference point for what is legit and what isn't in the field. No one else is doing this yet, so there is an opportunity, and a need.

Anyhow, thanks for being open to my thoughts even when they sting a little. BTW, I didn't realize that you and Jack didn't come up with the answers. In a way, this scares me even more though! I think I would trust the two of you more than the "experts" in that group, the name of which I can't remember now!

How many of these people in that org. do you actually know personally anyhow?! I am pretty skeptical of them, as you might have guessed!

Which reminds me, will you think about the comments I made that they may well have very similar beliefs, backgrounds, educational levels, training etc. This concerns me. I wasn't just being facetious when I said a group of 14 year olds might come up with better reponses. Some of these teenagers I talk to are extremely intuitive when it comes to solving emotional problems.

I was thinking that one way of approaching a test would be to first identify people who you just "know" are emotionally smart. Then using them to check the test results. For example, I have two teenage friends in particular. Sarah in the USA, who I talked about at the workshop two years ago, or whenever it was, and Anna R in England. I was thinking that if the test didn't show these people to be high in EI, then the test needs improvement. From talking to them for a long time I feel very convinced these are close to emotional geniuses. So what about that idea? After knowing someone well, seeing them in real live, and determining that they are emotionally smart, then checking a test against them, using them as benchmarks....

What do you think? The more I think about an EI test, the harder it seems to come up with one.

On the other hand, another thought. You said an emotionally intelligent person would come up with a unique response. But maybe not. Maybe it is actually more like math than we think. Maybe if the two or three high EI people had all the facts, they would come with a very similar answer. Having all the facts with emotional issues is harder though than having all the facts in a math problem. A person's whole life is part of the equation. What would work for person A, might not work for person B. Agree?

Yet, if two high EI people knew each person completely, they might
come up with exactly the same "best" response. So maybe there really is a "right" answer to emotional problems. The answer that best leads to the survival of the species. This, to me, is the ultimate goal. But again this is subject to debate. The goals, objectives and the values play a huge role in determining what we call the best answer. Or that is how it seems to me.

Anyhow,

Tttyl (talk to you later)

Steve

ps - while I'd like my site to be #1, I'd like yours to be # 2! And even if that were reversed, I feel content!