R'B-56-5
~! [I
S [ f3
A U
~ l
OBJECTIVE DESCRIPTION AND CLA.SSIFICA']'ION OF ENGINEERnm ~TOBS)
n:
FACTOR AN.A.LYSIS OF TEE KEY GROUP DATA FORM
Cl
1-1 [
T
David R. Saunders
I
N --------This Bulletin is a circulation.
dl~qft
for interoffice
Corrections and suggestions
for revision are solicited.
The Bulletin
should not be cited as a reference without the specific permission of the author. is automatically superseded upon formal publication of the material.
Educational Testing Se~vice Princeton, NEM Jersey April 1956
It
OBJECTIVE .DESCRIPrION AND CLASSIFICATION OF ENGINEERING JOBS, II: FACTOR ANALYSIS OF THE KEY GROUP DATA FORM Abstract Factor analysis of the Key Group Data Form for engineers yields five factors, which represent clusters of engineering activities and are interpretively lab~Iled as; A:
Employs basic scientific skills.
B:
Applies developed technical information.
C:
Provides supportive technical
D:
Prepares detailed plans for future use.
E:
Employs various
non-techn~cal
ser~ices.
skills.
Each of 23 engineering jobs studied may be described in terms of the relative importance these factors are judged to have in producing success.
Other engineering jobs may be readily incorporated and described
in these same terms. The factorial results are able to explain many difficulties that were encountered in attempts to cross-validate predictions of success made within certain "Fu.Tlctional Groups" of engineers, when the grouping was based on a less arduous analysis of the Key Group Data Form (1). The new results are judged to provide an improved basis for job description and classification.
OBJECTIVE DESCRIPl'IONAND CLASSIFICATION
OF ENGINEERING JOBS, II: FACTOR ANALYSIS OF TEEKEY' GROUP DATA FORM Table of Contents Page
Introduction
1
The Factor Analysis
5
Interpretation of the Factors
7
Discussion
13
Conclusions
15
References
18
******
Key Group Data Form
19
Table 2
Matrix of Job-Description Correlations
20
Table 3
Unrotated Centroid Factor Matrix
21
Table 4
.Matrix of Fifth Factor Residual
22
Table 5
Orthogonally Rotated Factor Matrix
23
Table 6
Pattern of High Factor Loadings
24
.Table 1
Objective Description and Classification of Engineering Jobs, II: Factor Analysis of the Key Group Data Form Introduction Since April 1954, Educational Testing .Service,has been engaged ina large-scale research project designed to find kinds of information that might be useful in the more efficient guidance, selection, and placement of engineering graduates into jobs wherein they are most likely to be successful.
An essential ftrst step is the recognition and objective defini-
tionof an appropriate series of distinct engineering jobs.
The value of
4'proposed system of job classification may be determined by several criteria,. such as: (1) Is it comprehensive?
Does it COYer both jobs that exist within
anyone organization and jobs that exist in various organizations, including companies, schools and "private" practice? (2) Is it siIrq;lle'l
Does it require the understanding and application
of a minimum number of concepts?
Are these concepts readil.v comprehendible?
(3) Does it have psychological utilit;y'l
By psychological utility we
refer to the ease and extent to which the job groupings can be related to other things, such as the validities of tests for predicting job placement, job success or job satisfaction.
(4) Does it provide for fine differentiation? We refer here to the precision with which it is able to pin-point a particular job as being like a relatively small group of jobs.
-2-
A start was made in the di.rection of these goa.ls 'with the development of the "Key Group Data Form" (KGDF).
This form cO:'ls:Lsts of a list of 24
activities which any engineer may engage in to some degree as a part of his job.
In order to describe a particular engineering job, it is only
necessary to put the 24 activities into a rank-order of "lim., crucial it is for the engineer in this job to perform this activity adequately.tl ("Table 111 reproduces the entire KGDF.)
Job descriptions obtained in
this manner have the advantage over conventional job descriptions of being readily subjected to statistical analyses and comparisons) includ':' iug analyses bearing on the four criteria listed above. A preliminary analysis reported previously (1) was sufficient to support two conclusions, based on data for 23 engineering jobs in five companies. * First, for engineers having five to seven years of postgraduation experience, the differences among engineering jobs are much greater within companies than between.
Second, the differences as ob-
served in one company tend to parallel the differences observed in other companies, although not every job grouping is represented in every compan;)'. In addition, on the basis of an inspectional analysis of the same data, a seven-category classification of the 23 engineering jobs was evolved, and a brief description of each functional job category was presented in
.lE-
The five companies whose data are used for this report are the Detroit Edison Company, the B. F. Goodrich Company, the International Business Machines Corporation, the Westinghouse Electric Corporation, and certain companies of the Bell SJrstem includil1..g operating Bell Telephone Companies, the Western Electric Company, and the Bell Telephone Laboratories; the latter are treated as one company for purposes of this report. The companies' Advisory Committee representatiYes are Dr. G. M. Worbois, Dr. E. L. Stromberg, Dr. H. J. McNamara, Mr. A. J. Murphy, Jr.) and IIJ1r. D. S. Bridgman, respectively. Educational Testing Service wishes to record its appreciation of the cooperation afforded by these companies and their representatives'in a research program on the placement of engineering college graduates.
-3terms of its most crucial activities.
The seven categories were labelled,
for convenience, as follows: I
II
Research Engineering Applications Engineering
III
Design Engineering
IV
Product Engineering
V
VI VII
Operations Engineering Supervisor.! Engineering
1
( J
]
Technical Engineering
Administrative Engineering
Sales Engineering
This seven-category classification has since been used as a basis for several analyses of the validities of a battery of 56 experimental tests. Efforts have been made to isolate and cross-validate particular tests capable of the concurrent prediction of placement and/or success of engineers in these groups (2,
3).
While these efforts ,vere generally successful.,. the
results are not so clean-cut as
desired~
Prediction of placement was notably
easiest for Groups 1, VII" VI, and II. , Initially it appeared easiest to pred:i.ct- success within Groups I, VI, II, V, a..l1d Dr, but only Groups II and
Dr held up satisfactorily on cross-validation. It was felt that the shrinkage of the success predictors on crossvalidation was greater than could be attributed to chance factors alone; therefore, possible sources of systematic effects were sought.
For all
of the groups except II end IV, it '..Tas found. that the available cross-validation samples differed. significantly from the corresponding analysis samples
-~.-
with respect to the distribution of scores for one or more of the pertinent predictors. * One kind of difference between the analysis and cross-validation samples was fOlll1d in the proportional representation of the va.rious sources of engineers thought to belong in each .functional group.
This difference
is particularly striking for Group I, where one company contributed
50%
of the analysis sample and only 15% of the cross-validation sample, while a second company contributed 13% of the analysis sample and none of the
cross-validation sample.
If unique factors associated with one or both
of these companies' data. are largely responsible for the original validity, then a severe shrinkage in cross-validation should follow.
(The inspec-
tional procedure that was used to sort·the 24 various jobs into groups provided some control over suchan outcome, but by no means precluded it. The dividing line between Group
r
and Group II, for example, was particu-
larly arbitrary, and could have been placed higher or lower in Table 1 of (1).) The result just cited suggests that the inspectionally-derived classification of engineering jobs does not adequately meet the criterion of psychological utility, mentioned at the beginning of this report .. Other evidence casting doubt on the adequacy with which this criterion is met has
*Without
exception, the most significant of these differences were associated with two test scores that were non-objective, that is, could not be scored by a rigorous set of rules, and require the intervention of a human scorer's judgment; although the same individuals performed the scoring for both the analysis and cross-validation samples, the two samples were scored independently under different conditions of recent practice and motiva.tion, and the emergence of differences in results should not be surprising.. This explanation can be applied to Groups V and VI; however 1 this cannot fully aceoUn't for the observed shrinkage in Group I, since all its tests are objective.
-5also come to light.
From a number of viewpoints} group V appears to be
a particularly non-homogeneous group of jobs; the relative position of Groups VI and VII in the series can be brought into question (2); the single job classed in Group III is not really as unique as i t appears} since its description has appreciable correlations. with several other KGDF descriptions in Table 2. In the face of mounting evidence suggesting inadequacies of the inspectional approach} the decision was made to perform the more expensive factor analysis of the .KGDF job-description data.
It was expected
that this would contribute. a better understanding .of' the several variables involved in the interrelations of the jobs, and :it
'Ha.S
hoped that it might
be possible both to simplify the job classification scheme·to be recommended and to improve its psychological utility.
This report describes. such a
factor analysis, and discusses the results from the point of view of the four criteria established at the beginning of this introduction. The Factor Analysis The basic data for the factor analysis were taken from Table 1 of our earlier report (I), containing the intercorrelations of jobs in five companies.
d~tafor
23
Replicated data for seven bf the jobs W'ere in-
cluded in the previous report as a means of communicating the degree of stability of the data.,
Since this information is also conveyed by the
communalities in a factor analysis} and since we did not wish to force the occurrence of doublet factors based only on these replications, one member of each pair of variables was chosen at random and dropped from the matrix.
This resulted. in the eliJnination of variables 2b, 3b,
4a, Bb,
-69b,171>, and 20b.
The matrix of correlations formed by the remaining
variables is given as Table 2 of this report, After inserting estimated communalities. on the
di~onal,
the 23 x 23
matrix of Table 2 was factored * by Thurstonera complete centroid method. Residual communalities were re-estimated after the extraction of each factor, and were taken to be eq,ual to the highest absolute value of any residual.. correlation involving the variable.. which are shown in Table 3.
Five factors were extracted,
It is interesting to note that the last fac-
tor extracted accounts for appreciably more variance than the fourth.
The
existence of a sixth .or addit1.onal important factor is possible} but does not seem probable from an over-all inspection of the resj.dualcorrelations shown in Table
4. The largest residual correlation is only .11, between
variables 22 and 23.
No rigorous test for the number of factors to be
extracted.can be made in connection with the centroid method of analysis. Orthogonal rotations were conducted by the single plane method, with the objective of finding positive manifold and simple structure.
Until
the general nature of the structure was well-established, all decisions concerning the direction of rotation were made from plots on which the variables were anonymous
in ignorance of the identification of any of
them.' Once the general structure had emerged interpretation was attempted, and further rotations served
emp~tic •.
tion might have been applied; but
wa~
(An objective procedure.for rota-
not deemed necessary vTith so few
factors and variables,. and following use. of a method . of factoring that was already somewhat suboptimal.)
*All
The resulting simple structure is very
of the computations associated with the factor analysis and rotation were efficiently performed by 1~. Carl S. Helm.
-7clear, and would almost certainly be the one found by an objective method of rotation. The results of the rotation are shown in Table 5,. along with the communality of each job-description variable.
The factors have been
arranged in order to facilitate the subsequent discussion.
The impor-
tance of eacb in accounting for the variety of job-descriptions.enalyzed is shown by the sum of squared factor loadings at t.he bottom of each column.
All of the factors except "D Il account for similar proportions of
the total variance. In view of the small number of activities
(24)
and small number of
jobs (23) included in the data on which this factor analysis· is based, it will be obvious that the results may be muchin:fluenced. by random or chance fluctuations in the data.
To the extent that the results
appear to be useful, it will be appropriate to attempt to verif;yr them in new data in order to be sure of their stability.
At the same time,
it will be appropriate to apply more rigorous computational procedures) so that true statistical significance tests may be applied. Interpretation of the Factor& Factor A:
'While this factor is the most ilnportant of the five in
accounting for variance} it only accounts for about one-sixth.
Variables
3, l,and 2 have their highest loadings on this factor, and do not load any other factor.
Variables 4, 5, 6, 7, and 8. also have high loadings
on this factor, as well as substantial loadings on Factor B.
Variables
9,11,12, and 14:have moderate loadings on this factor, but have higher loadings on Factor C.
The remaining variables. have unimportant loadings
on this factor, but still show a notable relationship in the sense that
-8the lower-numbered variables have algebraically higher loadings.
Over-all,
there is a rank-order correlation of +.79 between the priority of listing of the variables and the magnitude of their loading on Factor A.
This im....
plies that this factor was implicitly given major weight in determining the rank-order of the jobs Ln the previous inspectional classification. Factor A appears to be due to a cluster of activities that are uniquely important in jobs commonly called I1Research, It of considerable importance in jobs known asllDevelopment l1 or fl.Product Improvement," and of less importance in other engineering jobs.
Activities from the KGDF which follow this pat-
tern are: "A:
Completing experimental or pilot projects.
liD:
Developing and testing usefuJ. hypotheses or generalizations.
Ifp:
Originating technical ideas."
While there is no necessary similarity in these activities, it may be convenient to refer to them collectively as !1Factor A: . Employs Basic Scientific Skills. /1 I
At this point, it should be strongly emphasi.zed that any interpretation that may be made of
an~r
based on the numerical data.
of the five factors is merely an inference Such interpretations should be made cautiously,
and with full regard to the possibility of alternative interpretations, such as will be specifically mentioned again under Factor E.
There is some risk
invoJ.ved in encouraging premature crystallization of our interpretations of the factors.
However, with these cautions in mind, there are advantages
in providing some speculations concerning the factors, if fqr no other reason than that others cannot be stopped from doing it for themselves.
-9In the case of Factor A .it is quite possible to identify a number of KGDF activities that are
~important
-- that is" relatively unimportant.
These naturally include many of the activities which seem to be important for other factors, as well as some that never achieve much importance. For this reason) and from the point of view of our' ow~ value syst~, it is tempting to label this factor as
t~Protected
Research," and to summarize
its activities in the action-verb, r'Originates. '1
:I{owever) the element of
tl'protectionll is clearly spurious, and may result simply from the impossibility of assigning a high rank simultaneously to every activity in the KGDF.
In other respects, the notion of a Eesearch function, with responsi.
bilityto
Originat~J
Factor B:
I
is in accord with the natural expectation.
This factor accounts for about 10% of the total variance,
and bas moderately large loadings on about 10 of the 23.variables.
However,
only one of these does not also have a substantial l0a4ing on some other factor -- a fact which makes interpretation relatively more difficult than it \Vas for Factor A.
The pure factor variable is 23, representing a job
called "Purchasing and Sales." loadings on Factor A.
Variables 4, 5, 6, 7, and 8 all have other
Variables 15 and 22 have other loadings on Factor E,
while variable 16 has another loading on Factor C. with Factor D.)
(There is no oYerlap
Almost the entire gamut of job titIes
;~fr·
used for these
jobs, but we note an emphasis on Development, Production, and Sales. No individual activities listed in the KGDF conform perfectly to the patterns of 10adings ..P!'ovided by this
factor~
but the folloWing ones come
relatively close: "J:
Keeping informed of improved materials., designs) methods, processes, products) equipment.
-10-
"H:
Evaluating ideas.
"U:
Selling ideas to people. tT
It may be possible to think of these together under the rubric "Factor B: Applies
develo~d
technical information. to major current problems."
In Factor B "Yre seem to have a pretty clear picture. of the Consultant
function, whose responsibility it is to AdVise.
The engineer whose job
is described by Factor B is expert in the use of existinltin:formation, and concerns himself with dispensing it where it is needed. We note that Factor B is the only one providing high loadings for both
However) in view
of the failure of the rotation to provide the expected separate factor for Sales, and in view of the fact that the largest residual correlation in Table 4 is between the two Sales jobs, it may still be possible that SB..les is a separate factor that has been missed.
A close eXB.l11 ination of
Table 4 tends to support this hypothesis, since most ot: the residuals between variables 22 and 23, and other variables loaded in Factor Bare negative.
It can now be seen that extraction of a sixth centroid,factor
would probably have led to the expected result, but that the resulting "Factor F" would have comparatively little variance and be of only tenuous significance.
At the same time, the relation of variables 22a.nd 23
to Factor B ivould be weakened, whi1e the other loadings on Factor B wbuld tend to be increased. Factor ,C:
This factor accounts for one-seventh of the total variance;
and has seven substantial loadings.
Variables 11, 12, 13, and 14 all have
high lbadings here, and all of them were preViously placed in the catego.ry
-llbt',UProduct Engineering."
Variable 9 has a similar but less clear-cut
loading pattern, and could have been put in the same group.
Variables
16 and 18 have loa-dings here, but also load other factors, as do variables 11, 12, and
14.
Several activities·come close to the profile of factor loadings, as follows:
"X:
Writing technical articles, correspondence, instructions, manuals, patent disclosures, reports, specifications.
n·E:
Developing and/or maintaining technical records.
"VI:
Trouble-shooting and/or meeting emergencies.
"I:
Evaluating performance of present materials, designs, methods, processes, products, equipment."
These activities taken together provide a meaningful cluster which we will label as "Factor C: .Provides various supportive technical services"Jl In Factor C we seem to see a Sta£f Engineering function, whope responsibilit;i.es center about the action-verb, Supports.
The appropriate support
includes both clerical and technical functions, and is typically performed in connection with activities grouped under one or another of the Factors
A, 'B, D, and E. Factor D:
This factor is very small, accounting for only
variance and prOViding only three loadings of significance. which has no other loading, is called "Design. It
4%
of the
Variable 10,
(But note that Variable 8,
also called "Design, r~ has a different .profile of loadings.)
Variable 17,
-12-
which has its highest loading on this factor, also loads Factor E; engineersholding this job are responsible for such things· as planning the best location for future telephone exchanges.
Variable 18 also has its
highest loading here, but had an almost equal loading. on Factor C.
All
three of the jobs loaded here involve detailed planning of future activities. Activities from the KGDF. that are ratedhigh.for all three jobs include: "R:
Plarming best use of equipment or material.
IfL:
Making decisions on the basis of data.
liN:
Making preliminary sketches. 11
Actually only the first of these is at all Clear-cut, so we are left with "'Factor D:
Pre~e6
detailed plans for future use."
The general picture in Factor D continues to emphasize the Design
function, with responsibility to
~.in
the broad sense of that term.
The tilne-factor comes into this picture very directly, and shows that this function looks to the future more than others do. Factor E:
This factor also accounts for about one-seventh of the
total variance, and has large loadings on seven variables.
Variables 19,
20, and 21 have loadings only on this factor, and correspond perfectly
to the inspectionally-derived job cluster called Supervisory Engineering. Variables 14, 15, 17, and 22 all have other loadings, on Factor B or C or D.· Several activities from the KWF correspond roughly to the pattern of loadings, and these are: IlC:
Controlling expenses.
IfS:
Planning best use of personnel.
"U:
Selling ideas to people."
-13Since these activities appear to be psychologically heterogeneous, it seems safest to label this as "Factor E: accounting skills."
Employs human-relations and
However, other non-technical skills, such as public
relations, may also properly belong in this· cluster, and we might label it "Factor E:
Provides various non-technical functions. 1I
In accordance with the pattern of our description and interpretation of the previous factors, we may even be able to recognize this factor as a picture of the Engineering.Executive,whose responsibility it is to Administer.
This use of the term "executive" may be rather
loose, since many of the individuals in the jobs described do not stand above the first line of supervision.
However, as long as we are willing
to consider alternative interpretations of the factors, we rnay as well include mention of this possibility for Factor E. Discussion The use of five factors instead of seven categories as a basis for describing engineering jobs represents an important reduction in the number of distinct concepts that must be held in mind.
At the same time
it allows for a much greater variety of descriptions, to fit the possible variety of jobs, because the factors need not be treated on an all-or-none basis.
Jobs such as are described by variables 1, 2, 3, 10, 11, 12, 13,
19, 20, 21, and 23 can be substantially covered by use of a single factor. Most of the other jobs in this study can be well described as combinations of some two of the five factors. Given the factorial results,. it is relatively easy to hypothesize the f'actorial descriptions of other jobs not even included in the study.
For
example, Engineering Teaching looks like a combination of Factors A and E;
-14Consultation looks like a pure form. of Factor B; and Field Engineering looks like a broad combination of Factors B, C, D, and E. Sticking, to the jobs that were included in the study, what would now appear to be the best grouping of them? these groups.)
(Table 6 may help to identify
Jobs 1, 2, and 3 form. one Group, which load only Factor A.
Jobs 4, 5, 6, 7, and 8 form a second Group, loading both Factors A and B. Jobs 10, 17, and 18 form a third, with loadings primarily on Factor D. Jobs 9, 11, 12, 13, and 14 forma fourth, with loadings primarily on Factor C.
Jobs 15, 17, 19, 20, and 21 form a fifth, with loadings pri*
marily on Factor E.
The remaining jobs -- 16, 22, and 23 -- form a
sixth Group, .with loadings primarily on Factor B l:!Jld no loadings on Factor A.
These six groups may be seen to correspond pretty well with
our previous categories I, II, III, IV, VI, and VII, respectively.
How-
ever, no grouping of this kind can be entirely satisfactory, since several of the jobs will fit t,-ro groups almost equally ,vell. Our previous category V is not read~y matched by any grouping based on the factor analysis and is shown to be both heterogeneous and diffuse. The jobs that were once grouped· as V tend to have loadings. of .25 or more on most of the factors, whereas most of the other jobs show no more tha,n one or two such loadings.
Also, four of the five factors load at least
two of the four jobs which were put into this group by the inspectional procedure. The job which previously stood alone to represent category III is still unique; it is the only job with a pure loading on Factor D.
Row-
ever, two other ,jobs that are related. through their loading on this factor have been located.
-15Evidence is found which supports the notion that our previous categories II" IV,," VI" and VII are relatively homogeneous, but that there is a possible uncontrolled variable that affects our validation-study results 1.
~or
category
Thus, the job described in variable 3 contributed 5010 .o~ the analysis
sample" and only 13% of the cross-validation sample.. This situation vas originally permitted because
o~
the knoym high reliability of the avail-
able success criterion for this job. to obtain
~resh
In the
~uture,
it will be desirable
cross-validation data" and to restrict attention only to
those jobs which load Factor A but not Factor B. Conclusions (1) It is possible to describe the relationships between the activities listed in the Key Group Data Form and a representative sample of engineering jobs by employing five factors. of its contribution to each or alternatively
de~ined
o~
Each factor maJT be
de~ined
the jobs studied, as displayed in Table
~actor's
5,
in terms of its dependence "on a particular cluster
of job activities, which have been tentatively listed. strictly, each
either in terms
(Speaking very
alternative definition is a particular rank-ordering
of all 24 of the actiVities; however, these are relatively laborious to compute and have not been worked out.
The listed clusters of activities are
judged as most likely to be among the first in the complete rank-order.) (2) It is possible to give meaningful interpretations/to the five ~actors,
and these have been offered in a preceding section.
It must be
emphasized that these interpretations are not the definitions of the factors, but are inferences based upon the way in which the factors interact with the various jobs and activities.
However, these interpretations do
-16provide a most convenient conceptual framework and may be expected to facilitate thinking about the pro;)lem of classifying engineering jobs. (3)
Any particular engineering job may be described and, in a sense,
classified by determining its dependence on each of the five factors. descriptions of the jobs studied are already provided in Tables
Such
5 and 6.
In order to describe a new job in this manner it is only necessary to obtaina typical KGDF description, and correlate the obtained. rank-order of activities with each of the fivefactorially-perfect rank-orders referred to under conclusion 1.
The completeness of the description obtained in
this manner can be assessed by summing the squares of these five correlations, and dividing this sum by the reliability of the typical KGDF description that was employed. (4) Even jobs that are not similar to any of the jobs initially studied may be effective.ly described, if they depend on a new combination of the five factors.
Hypotheses as to appropriate descriptions for teaching, consulting,
and field engineering have been offered above, which illustrate this principle.
(5) Some descriptions that may be obtained from the five factors probably do not apply to jobs in existence anywhere.
This does not mean that
such jobs could not be created; perhaps some of them should be created as a means of providing jobs in which some engineers can be more successful than in any present job; resulting in better utilization of engineering talent.
(6) Conclusions 3, 4, and 5 indicate that a system of job classification based on these five factors can be very comprehensive. (7) Since the nature of only five factors needs to be understood, instead of seven functional job groups, or some very large'number of
-17particular job titles., such a system of classification is distinctly easier to grasp.
This, together with conclusion 2, demonstrates the
plicity of such
relativ~
a system.
-
sim*
~8) Evidence on the psychological utility of this system is implicit
in the entire Section headed "Discussion." *
(9) Since any job description using this system must specify a degree of' relationship to each of five factors,there are literally an inf'inite number of possi.ble job descriptions.
This certainly attests to the
ness of discrimination that may theoretically be achieved.
Of
fine~
course, in
practice it will be inadvisable to distinguish between jobs With closely similar sets of coefficients, out of respect for inescapable experimental error.
*Additional
evidence believed to bear on this point is being incorporated into two separate reports (2, 3).
-18References 1.
Saunders, David R.
Use of an objective method to determine engineerI
ing job fa.Dlilies that will apply in several companies.
ETS
Research Bulletin 54-26, September 1954. 2.
Saunders, David R.
Concurrent validation of Engineering Placement
Tests against placement criteria for experienced engineers. ETS Research Bulletin (in preparation).
3.
Saunders, David R.. Concurrent validation of Engineering Placement Tests against effectiveness ratings on experienced engineers, for various functional groups. preparation) .
ETS Research Bulletin (in
-19TAllLE 1
Key Group Data Form Name
---------------
Job Rated
7(;;Nam-e-o-:f~K;-e-y--:::G-ro-u-p""')-
For this list please (a) indicate in the space to the left of each activity the approximate percentage of time that typical members.of this group spend in this kind of activity. These percentages will not generally total exactly 100%, since some kinds of activity should be included under more than one item on this list, and other activities are omitted. (b) indicate by rank in the column at the right how crucial it is for members of this group to perform adequately in these activities. For example, if "Evaluating ideas" is the most crucial aspect of this key-group's work) you will want to ',rrite "H II in space 1 to show this. Pay no attention to the percentages you have just written. Pay no attention to hO'o' hard it may be to find men who can perform the activities. Rank all 24 of the activities. ACTIVITY
(J)
~~~ (M) (N)
~~~
(Q) (R)
(S) (T) (U)
(V)
(W) (X)
RANK
Completing experimental or pilot projects Conducting negotiations Controlling expenses Developing and testing useful hypotheses or generalizations Developing and/or maintaining technical records Developing people toward promotion Estimating budgets, costs, or prices Evaluating ideas Evaluating performance of present materials, designs, methods, processes, products, equipment Keeping informed of improved materials, designs, methods, processes, products) equipment Making and/or checking complex calculations Making decisions on the basis of data Making detailed draWings Making preliminary sketches Non-routine assembling of equipment Originating t~chnical ideas Participating in technical sooiety or community activities Planning best use of equipment or materials Planning best use of personnel Preparing and making technical recommendations or proposals Selling ideas to people Sizing-up people Trouble shooting and/or meeting emergencies Writing technical articles, correspondence) instructions, manuals, patent disclosures, r':!ports, specj.fications
LETl'ER
1
(most crucial) 2
(next most crucial)
3 4 "
5 6
-----
7 8 9 10 11
12
13 14 15 16 17 18 19 20 21
22
2; (next least crucial)
24 (least crucial)
CHECK TO BE SURE THAT YOU HAVE USED EACH LE'ITER EXACTLY ONCE. PLACE ANY COMMENTS ON BACK OF THIS SHEET.
*** Confidential. All rights reserved. Unauthorized reproduction or use prohibited. Educational Testing Service. Princeton, New Jersey, and Los Angeles, California.
.33
.43
.14
.26
.17
.32
.14
.17
-.05
.12
.04
.17
.04 -.13
11
12
13
14
15
16
17
-.15
.14
-.09
-.06
22
23
-.30
-.17
21
-.09
.04
20
-.09 -.22
.26
.12
.05
10
19
.38
.33
.16
9
-.02
.53
.51
.28
8
~.01
·36
·51
·31
7
18
.42
.44
.26
6
.09
-.05
-.08
.14
.19
.01
.06
·31
-.08
-.01
.03
.11
.12
-.14
.33
.20
.17
.14
.45
.18
.08
.16
.21
-.01
-.16
-.14
.04 -.24
.25
.18 -.13
-.01
-.06 -.01 -.12
.20
.14 -.07
.44
.36
.09
.29
.16
.28
.33
.37
.54 .49
.14
.34
·56
.20
.55
.76
·59
.49
.45
·59
.48
.36
.51
.31
.66
.42
.48
·50
.43
.41
.06
.25
.56
.28.
·30
.18
.33
·57
.22
.63
.49
.49
.48
·72
.45
.54
.55
.66
.54
.62
.44
.26
.40
.10
.09
.14
.02
.25
.20
.54
.44
.41
.29
.44
.15
.20
.22
.19
.48
.30
.5~
.46
.52
.41
.56
.64
.5'4 .46
.43
.64
.34
.55
.33
.63
.38
.33
.16
9
.38
.64
.56
.76
.. 49
.72
.53
.51
.28
678
.19
.23
.32
.12
.37
.36
.21
·55
·54
·30
5
.62
.41
4
.54
.54
.60
.55
3
.54
.60
.51
.30
.41
.55
·51
2
1
5
4
123
.09
-.04
.09
.01
.01
.31
.22
·33
.21
.24'
.17
.22
.25
.43
.38
.14
.20
.18
·29
.08
.06
.16
.12
·33
.14
.41
.45
.'{o
.44
.64
.25
·31
.03
.09
.18
.08
·30
·33
.16
.20
.17
.31
·37
.28
.22
.46 .16
.28
.41
.38
.56
.42
.44
.64 .42
.17
.41
.22
.56
.64
.29
.14
.33 .46
.28
.49
.06
.28
.56 .25
.12
.14
-.05
13
.37
.43
.17
12
.54
·37
.54
·30
·57
.36
.21 .22
·33
.14
II
.12
.05
10
.28
.17
.24
.44
·35
·36
·30
·32
.48
.41
.56
·70
.36
.34
.40
.41
.45
.25
·30
·30
.48
.28
.38
.45
.21
.46
.44
.U.l
.24
.18
-.01
·51
.27
.18
.20
.03
.46
.35
.30
·32
.22
.46
.41
.33
.54
.54
.33
.48
·33
.40
.51
.59
.46
.52
.35
.30
.30
.28
.16
.14
.22
.30
.20
-.14
19
21
22
.il
.14
.16
.03
.45
.35
.31
.08
.12
.01
.19
.02
.20
.41
.44
.17
.18
.16
.01
.22
.14
-.24 -.14
.04
-.12 -.01
-.01
-.06
23
.15
.10
-.01
.18
.01
.06
-.05
.18
.40
.24
.20
.09
.06
.27
.34
.17
.16
.03
.08
.09 -.04
.20
.09
-.16
.08
-.08
.03
-.08
·51
.36
.28
.33
·31
.29
.09
.44
.40
.21
.45
.19
.31
.09
.14
.04 -.17 -.09 -.06
20
·33
.21
.37
.29
.22
.16
.34
.54
~61
.22
·33
.47
.54
.61
.29
·27
.44
.54
.54
.37
·33
.16
·33
.57
.57
.44.27
.47
.34
.21
.52.46.59.51.40·33
.46
.25
.36
·37
.30
.33
.31
.48
.25
.12
.25
.20
.14
.09
.44
.50
.26
.19 -.07 -.13
.20
·52
18
.04 -.01 -.09
17
.32 -.13 -.02 -.22 -.09 -.30 -.15
.17
16
.29
.43
.23
.17
.04
15
.17
·36
.16
.41
·32
.26
.12
14
Matrix of Job-Description Correlations
TABLE 2
I
oI
{\)
-21TABLE 3 Unrotated Centroid Factor Matrix II
III
IV
.477 .673 .633 .643 .470 .484 .309 .454 ·332 .342 .561
.402 .601 .446 .415 .454 .268 ·502 .342 .090 .087 .135 .151 -.172 -.135 -.212 .057 ";'·521 -.298 -.649 -.593 -.630 -·509 -.237
-·395 -.145 -·388 .... 129 -.298 .057 .088 .049 .232 .182 ·311 .207 .250 .107 -.107 .191 -.077 ·355 -.216 -·327 -.105 .150
... 123 .107 -.086 .010 .158 .070 .102 .004 -.064 -.367 .163 .096 .140 .015 .211 .... 177 -·391 -·350 .007 -.001 -.077 .003 .023
.038 .187 -.025 ••040 -.182 -.085 .123 .409 ·399
12·301
-.007
-.120
-·527
.024
7·115
3·539
1.127
.626
1.002
I
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 ~ai
2
~a. 1.
.295 .454 ·533 .745 .444 ·735 .451 ·773 .776 ·363 .696
.658
-.11.3
V
-.175 -.095 -.167 .030 .150 .275 .204 .195 ....076 -.040 -.268 -.231 ... 194
-.406
-.035
-.065
.038
.030
.011
-.024
.058
-.002
.025
-.020
.005
-.067
.023
10 1-.072
II 1-.024
12 1-.032
131-.040
-.051
151-.071
.044
17 I .028
18 I .057
19 1-.037
20 I .000
1-.044
14
22 1.040
23
21
.085
-.043
9 1-·035
.002
-.021
8 1-.031
I
.025
7 I .058
16
-.028
.020
5 1-.086
.015
-.005
4 1-.021
6
-.033
.020
.074
.024
.020
-.003
.021
-.049
-.034
.004
··.018
.000
•. 065
.033
-.058
.012
.006
.005
.011
.037
-.064
-.032
.043
.056
-.033
.000
-.017
-.028
.015
6
-.001
-.002
-.010
.028
.030
-.006
.021
-.022 -.027
.007
-.019
-.041
-.018
.027
.049
-.006 -.042 -.010
.001
.018
.034
.029
-.030
.013
-.004 ··.010
-.030
.037
.097
.035
-.024
.041
-.033
-.024
.034
.020
-.086
5
.009
.040
.042
-.060
.047
.003
-.060
.000
-.024
-.006
-.005
-.021
4
.033
.019
.002
.019
.010
··.002
.017
-.027
... 017
.03 4
-.006
-.033
3
2
3 I .020
2 I .074
1
_ 1
-.021
.017
-.001
-.001
.001
.064
-.039
-.026
.001
.006
.014
-.023
.001
-.038
-.054
-.005
.056
.041
... 060
-.027
.025
.058
7
-.038
-.065
.030
.023
-.012
.028
-.009
.045
.048 .024
.023
-.027
-.005
.029
... 045
-.009
-.030
.025
-.022
.020
.010
.005
-.024
.006
-.008 -.066
.072
.070
.013
-.096
-.082
.020
-.047
-.Oll
-.034
-.036
-.001
.097
-.023
.010
-.04~
-.022
.072
-.038
-.064
.097
-.060
.010
-.012
.055
-.008
.003
.010
-.03l
-.036
-.025
.047
.105
-.021
.020
-.022
.006
-.008
.001
.037
-.042
.022
-.021
.010
.035
-.001
.024
.019
-.023
-.023
-.023
.049
.055
.004
-,055
-.083
-.047
.049
-.054
-.042
.058
.017
.016
.049
•. 0117
-.013
.057
-.035
.099
-.006
-.033
-.044
.01f9
.006
.047
.097
.029
.020
.001
.012
.034
.020
.105
-.023
-.045
.010
.006
.006
.013
.029
-.030
-.004 -.010
-.024
-.071
15
.011
-.051
14
-.006 -.006
-.083
.067
.040
.035
-.064
.006 -.054
.020
.022
.020
-.044
-.009
.005
-.066 -.024
.014
.005
.009
.033
.019
.030
-.040
13
-.023
.Oll
-.030
.040
.042 .037
.002
.038
-.032
12
.019
-,065
-.024
-.072 -.035
11
10
-.054
-.032
.035
.047
-.C02
-.043
-.035
9
.070
.013
-.005
.043
-.024
.003
.017
-.021
-.031
8
Matrix of Fifth Factor Residuals
TABLE 4
.064
.025
-.014
.020
-.055
.043
.028
-.044
-.027
-.023
-.009
.040
-.042
.027
-.041
-.050
.037
.006
-.051
.027
.043
.028
-.001
.006
.040
.020
-.035
.055
-.083
.019
.003
-.047
-.009
.024
-.001
.027
-.027
-.022
.021
.005
.000
20
-.062
-.046
.018
.011
.029- -.008
-.001
-.051
-.042
-.055
.099
-.006 -.033
.067
-.023
.010
-.011
.045
.048
.001
.049
-.001
.007
-.049
-.020
-.037
19
.004
-.083
-.0~7
.040
-.023
-.031
-.034
.023
-.030
.064
-.010
-.042
-.006
-.034
.025
.057
18
-.055
.035
-.02~
-.036
-.036
-.027
.025
-.039
.033
.000
.001
.004
-.002
.028
17
-.064
.049
-.025
-.001
-.005
-.022
-.026
-.058
-.065
.018
-.018
.058
.044
16
-.097
-.056
.011
-.008
.11'
- .056
-.046
-.050
-.023
.025
-.013
.049
.017
-.001
.055
-.082
-.012
-.065
.017
-.041
-.006
-.010
.020
.023
.040
22
.029
.037
-.009
-.014
.057
-.006
-.006
.024
-.008
.020
.028
.030
-.001
-.018
.021
.028
-.003
-.067
-.0440
21
. III
-.097
.018
-.062
-.041
-.027
.064
-.04'7
.016
.058
.035
-.012
-.096
.023
-.038
-.021
-.019
.030
-.C02
.024
.085
.002
23
I
I
rI.) rI.)
...23TABLE 5 Orthogonally Rotated Factor Matrix Factor
Factor
Factor
Job
A
J3
.C
1
.65 ·72 .80
.... 08 .12 .04 .35 ·33 .58 .38 .51 ·30 .05
·53 .63
"".11 .16 .02 .26 .... 0)+ .32 .21 .34 ·59 .26 ·72 .60 .54 .60 .29 .36 .16 .48, .12 .07 .07 -.04 .26
2.32
2.89
2 3
4 5 6 7 8 9 10 II
12 13 14 15 16 17 18 19 20 21 22 23
·73 .64 .48 .47 ·57 .39 .17 .38 .42 .08 .34
·27 .25 -.03 -.09 -.08 .06 -.19 -.17 -.01
4.15
.15 ,,14 .. 12 .00 ·39 .43 .121.15 . .03 .16 .28
Factor :p
Factor E
h
.10 -.03 .13 .15 -.07 .15
.45 .61
.04
.02 -.21 .06 .00 -.09 ... ~03 -.31
.21
.... 05
·27
~13
.46 .02 .06 -.01
-.01
.11
.~·O
.... 06 .37
.4.4
.1+8
.50 .02 .06 .16 .12 .19
1.12
.07 .08 .24
.06 .62 .26 ·76 .80 .63
2
.67 .74 .54 ·70 ·51 .75 .67 ·31 ·70 ·56 .38 .65 ·50 ·52 .65
.57
.48
.60 .67 .54 .56
.23
·55
2·93
13·40
0 0
0 0 0 0
0 0 0 0 L
0
L M
0 0 0 L L M
L M
L
H
H
H
H
H
M
M
M
L
2
3
4
5 6
7
8
9
10
Westinghou:s~
Bell Laboratories
Research Development
0
0
IBM'
Westinghouse IBM
Westinghouse Western Electric Detroit Edison Goodrich Detroit Edison Western Electric
Development Development Product Improvement Design Manufacturing Design Technical Service Equipment Testing Equipment
0 0
M M
0
0 0
H
0
L 0 M
0
0
0 0
0 0
L L 0 0
0
M H
0
L
0
0
0
0
0
0
0
0
0
13
14 15 16
17
18
19
20
21
22
23
0 0
L
M
0
L
12
0
0
0
R
0
L
11
0
0
0
0
vie stinghouse Detroit Edison Bell System Detroit Edison Goodrich Bell System Goodrich Westinghouse Detroit Edison
Manufacturing Project Operations Planning Plant Operations Supervision Production Supervision Technical Sales Purchasing and Sales
L
H
0
H H
0
M
H
Detroit Edison Production
L
0
0
H
0
0
0
0
-L
0
0
Goodrich
Product Development
0
M
0
Company IBM
Job . . Title Research
0
0
0
0
0
E
Factor
0
0
D
Factor
1
C
B
A
Factor
Factor
Factor
Job Variable
Pattern of High Factor Loadings
TABLE 6
I I
+:-
f\)
