This article craeted quite some interest and discussion. See the response from BELBIN below.
I like psychometric tests. They are fun, can be insightful and provide a starting point for discussing my relationships with others. Used as a tool for provoking some introspection giving rise to greater self-awareness they provide a launching pad for improved emotional intelligence.
What bothers me about psychometric tests is the way in which they are sometimes used. Or should I say misused.
Take this excerpt from an HR fraternity website (Note: bold emphasis is mine).
- ” Psychometrics are used within the following contexts:
- SelectionHR professionals have realized that experience and professional qualifications are not sufficient as indicators of future performance. As a selection tool, psychometrics shed light on behavior, an element which is usually difficult to measure during interviews especially as candidates are keen to mask their true personality and to present themselves in a favorable light. Psychometrics will allow you to understand your candidates’ behavioral patterns and make better selection decisions for optimum “job fit”.
- DevelopmentEmployees benefit from receiving objective feedback on their behavior, which they more readily accept as unbiased since reports are computer generated.
- For Career/Promotion DecisionsEmployees who excel in one job will not necessarily perform adequately in another. When selecting people for promotion, otherwise excellent employees have too often been miscast into roles they could not perform satisfactorily. Psychometrics enable you to predict the employee’s behavior within a different role before making the promotion decision.
The implication from this excerpt is that psychometrics make it difficult for people to mask their true behaviour or personality, that they are unbiased because they are generally computer generated and predict an employee’s behaviour in a future role.
Research shows each of these implied assertions to be untrue in many cases. For example, one of the most popular psychometric tests, the Myers-Briggs Type Indicator has no faking scale. When candidates complete a recruitment questionnaire, their answers can be influenced by what is quaintly known as “motivational distortion”. Consciously or subconsciously they may recognise the types of behaviours that the organisation is looking for and their answers reflect that perception rather than their own personality. When this happens, the results of the questionnaire may be unreliable or even misleading.
Good recruitment questionnaires contain faking scale, which help identify the extent to which responses have been influenced by motivational distortion. The MBTI does not have a faking scale, so you don’t know whether the result is a true reflection of the individual, or an image that is projected for the benefit of the interviewer. Bias is not only possible with psychometric tests, it is probable, whether the tests are computer generated or not. Factors affecting test experience and the capacity for bias include:
- The nature of the test and its supporting information e.g. language, instructions, structure, medium, timescales
- The nature of the person e.g. their experience, confidence, emotion at the time of the test, motivation, memory, culture
- The nature of the environment during the test e.g. Light, heat, humidity, noise, distractions, test administrator, on-line or off-line
- Aspects of time e.g. time of day, age of test (vernacular used), time available to the person completing the test (stress)
Psychometric tests, in general, have low levels of predictability of behaviour in given situations. Correlations of 0.25 and above between a test and performance in a given situation are considered good. A correlation coefficient of 0.25 indicates a 6% covariance between the characteristic and performance.
Some psychometric tests, particularly cognitive tests, have shown a high degree of predictive ability scoring correlations of 0.6 and above (predicts 36% or more of performance). Other well known tests, such as the Myers-Briggs Type Indicator, are known to have little or no predictive ability.
Yet there is an industry worth millions, if not billions, of dollars in “predicting” the behaviour of people through psychometric tests. [Two and a half million Americans a year take the Myers-Briggs. Eighty-nine companies out of the US Fortune 100 make use of it for recruitment and selection or to help employees understand themselves or their co-workers.] Much of this industry is franchised with people certified to interpret the results of psychometric tests after one day of training. That in itself is not a problem. General interpretations of many tests can be learnt in a single day. The problem is how the psychometric tests interpreted by inexperienced people are used.
If they are used by these “certified” assessors as a means of beginning a discussion that helps self-awareness and thus with some effort, self-regulation at a personal or team level, they are of benefit. Cognitive tests and other tests with predictive correlation coefficients in excess of 0.5 are useful in themselves as they do provide a reasonable guide to future behaviour.
However, undisciplined, uniformed use of psychometric tests that pigeon-holes people into “types” of behaviour need to be questioned.
Most of us have been on a course which uses a psychometric test that creates a label for us. For example, I am a “Completer-Finisher” or I am an “Implementer” in Belbin’s Team Roles. When these personality profiles are used as labels there is a danger of the interactions between people becoming guided by the label they wear. For example, “Oh, you are a completer-finisher and this role requires a shaper, so you won’t fit the role in the project team”. At best, questionnaires such as Belbin’s give us a view of the preferences of the individual and not their actual skill or knowledge.
To continue this example, I have known many people who have a preference for completing projects and making sure that there are no loose ends who have been fine in the role of a challenger of the status quo (a shaper in Belbin terms).
To get the best out of psychometric tests other than as a provocation of self reflection choose tests which have high reliability (repeatability of the test) and validity (predictor of behaviour). Look for reliability correlations of 0.7 or more and validity of 0.4 or more. The single best predictors are cognitive ability tests such as verbal and numerical reasoning. Select tools which are designed for the performance areas you want to evaluate. Do not, for example use the Myers-Briggs Type Indicator for recruitment.
Also ask what norm group a test is used for. A test designed for a norm group in the US would be unlikely to be suitable for a test group in India. A test designed for a norm group of managers may not be suitable for a group which consists of blue collar workers.
One final word of warning is to view the results of psychometric tests as an hypothesis to be confirmed or disproved by actual results. If your psychometric tests are not discriminating in performance, then discontinue them and save yourself some money or seek out a better test that does discriminate. If you do not measure how well your psychometric tests predict performance, then start measuring now.
Dear Kevin,
I was really interested to read your article on psychometrics and their potential for being used in an undisciplined manner. I would very much agree that questions should always be asked about the reliability of a subjective measure. In fact, questions might also be asked about reliability scores in general, when these are generally achieved by a high repetition of the same question in order to give internal reliability, rather than by repeated testing.
I just thought it worth adding a few things about Belbin Team Roles, which might potentially be useful. I know that they are often represented in a somewhat strange manner on courses, as you say in your article, so hopefully the below will be useful!
The first point is that the Belbin SPI is actually not a psychometric (though it is frequently wrongly labelled as such). The test purely relates to workplace behaviour, and not to inner motivations. Furhtermore, the test is intended not as a standalone, but to have input from observers. As you point out, answering deceptively can be a problem, as can a general lack of self-awareness. The idea of observers is to see exactly how that individual is coming across in the workplace, and drawing comparisons between them.
The second would be that the Team Roles are not intended to be used categorically. We would not say that someone “is a Shaper,” but that they have tendencies towards those sorts of behaviour. What is particularly important in Belbin is that people have a whole range of Team Roles they can play in different circumstances. The SPI just tells the strength of the tendencies. When giving people a “fingerprint” of themselves, the Roles are charted in terms of “preferred,” “manageable,” and “least preferred.” Most people have several of each. And yes, as you so rightly say, a strong Completer-Finisher might well also be entirely capable of playing the Shaper role. It might simply be one that they do not wish to play all the time, or perhaps is simply another preferred role that they can pick up and put down as needed.
Finally, when being used in recruitment, we are always very clear on what the Belbin SPI does and doesn’t differentiate between. It is designed to be used to help get to know what the candidate is generally perceived (by themselves and others) to be like, prior to interview. Many other features inevitably have to be taken into account. The only thing it might be worth adding is that, following decades spent in recruitment and the placement of high level personnel, Meredith Belbin did find that those with inappropriate qualifications, but who clearly possessed the right roles for a job, ended up being far more successful than those who were correctly qualified but had the wrong roles, and tended to stay much longer as well.
I wondered if you had been given a proper online Belbin test or simply a paper version with no advice? The full reports online are very different, and much subtler. If you’d like to have a look, do let me know and I will send you a free link.
Kind regards,
Gytha Lodge
Publishing Manager
BELBIN
