How To (and How Not To) Assess the Integrity of Managers (19 pages)

Document rating:
  3.0 out of 5
Vote this doc:
background image

Finally, we evaluated the degree of interrater reliability and agreement to justify aggregating

ratings across subordinates for each focal manager. We used the r

wg

statistic to determine the degree

of interrater agreement (do raters of the same manager provide the same, or very similar, level of
ratings?; James, Demaree, & Wolf, 1993) and intraclass correlations (ICC) to determine the degree
of interrater reliability (do raters provide ratings that produce the same, or very similar, rank
orderings of target managers?). We used ICC (1,1) to estimate the interrater reliability of the results
from a single rater, and ICC (1,5) to estimate the stability of the mean of five raters (Shrout & Fleiss,
1979), since five was the median number of raters per target in our sample. The results are presented
in Table 1, and indicate a relatively high degree of interrater agreement (average mean r

wg

ϭ .81)

and reliability [mean ICC (1,5)

ϭ .53] compared to published samples of subordinate ratings (cf.

LeBreton, Burgess, Kaiser, Atchley, & James, 2003). Therefore, we computed the average rating
across all subordinates for each item and then computed scale scores for each of the five compe-
tencies. The internal consistency reliabilities for these aggregate scale scores were high; Cronbach’s
Coefficient alpha values were .87 for Vision, .88 for Execution, .84 for Managerial Courage, .88 for
Building Talent, and .85 for Integrity.

In sum, the foregoing analyses revealed very good psychometric properties for the competency

measures and rule out the possibility that poor measurement could complicate the interpretation of
our findings.

Criterion measure.

We also had an independent measure of overall performance for the

directors and VPs. Several months after the competency ratings were made, each director and VP
was placed in one of three categories by his or her boss—“highest performing,” “solidly perform-
ing,” and “lowest performing.” The organization required a forced distribution so that each of the
bosses assigned 30% of their direct reports to the “highest performing” category, 65% to the “solidly
performing” category, and 5% to the “lowest performing category.” Most of the bosses categorized
between eight and 12 direct reports. The bosses then met with their peers to make a case for their
classifications. The peers were allowed to challenge the bosses’ classifications using facts and
specific examples. After a discussion, the peer groups voted on whether to reclassify the subordinate
in question. As a result of this challenge process, approximately 10% of managers were moved into
different categories; the majority of these changes moved managers into the “solidly performing”
category.

The final data set included 28.3% in the highest performing group, 67.3% in the solidly

performing group, and 4.3% in the lowest performing group. This forced distribution is arbitrary; it
likely produced a restricted range compared to the true underlying performance continuum.
Undoubtedly distinctions are lost among the two thirds of the sample in the “solidly performing
group.” However, the distribution probably accentuates the difference between the highest and the
lowest performing groups.

Table 1
Interrater Agreement and Reliability for Subordinate Ratings of Competency Scales

Competency scale

Interrater Agreement

r

wg

Interrater Reliability

ICC

Median

M

SD

(1, 1)

(1, 5)

Vision

.86

.83

.15

.20

.55

Execution

.88

.83

.16

.20

.55

Managerial Courage

.86

.82

.17

.17

.50

Building Talent

.82

.77

.21

.15

.47

Integrity

.84

.79

.19

.22

.58

Note.

r

wg

values were computed for each focal manager first, and then descriptive statistics were computed

across the sample. ICC (1,1) represents the reliability of ratings from a single rater; ICC (1,5) represents the
stability of the mean of five raters, which is the unit of analysis in subsequent statistical tests.

223

SPECIAL ISSUE: HOW TO ASSESS INTEGRITY

This

document

is

copyrighted

by

the

American

Psychological

Association

or

one

of

its

allied

publishers.

This

article

is

intended

solely

for

the

personal

use

of

the

individual

user

and

is

not

to

be

disseminated

broadly.

Document rating:
  3.0 out of 5
Vote this doc: