Empirically Mapping the Subspecialties of Cardiovascular-Interventional Technology

The practice of cardiovascular interventional technology (CVIT) has roots traceable to the late 1950’s, when the first selective catheterization of diseased coronary arteries was performed.^1-3 In the early 1960s, radiologists were performing selective procedures on organ systems other than the heart.^4-6 In 1976, the introduction of preformed coronary catheters facilitated studies of left ventricles and coronary arteries.^7-9 Selective studies of the arteries in the neck, brain, thoracic cavity, abdominal cavity, and pelvic cavity and peripheral runoff studies of the upper and lower extremities were done.^10,11 Percutaneous transluminal coronary angioplasty first was performed in 1978.^12,13 This procedure altered the management of severely diseased coronary arteries from coronary bypass surgery to percutaneous methods. It used a tiny inflatable balloon that compressed obliterating luminal cholesterol plaque into the walls of a coronary artery.^10,12 This technique has made the need to open arteries in the lower extremities obsolete¹⁴ and has become the gold standard for opening occluded arteries not only in the heart, but also in other major organs of the body. In the late 1980s, balloon expandable intracoronary stents first were used to maintain the patency of arteries that had been opened with the angioplasty balloons.¹⁵ As the medical side of CVIT evolved, the roles of the allied health professionals who assisted with these procedures also grew. In particular, the responsibilities of radiologic technologists expanded beyond activities related specifically to medical imaging. Radiologic technologists were assisting physicians with the procedures, assessing and monitoring patients, recording hemodynamic data such as pressures and electrocardiogram waveforms, and administering medications under the supervision of physicians.^10,15 Recognizing these new roles for technologists, the American Registry of Radiologic Technologists (ARRT) initiated a comprehensive job analysis in 1986 to determine if CVIT warranted a specialty certification program.¹⁶ Job analysis, also called practice analysis, is the generally accepted procedure for documenting job responsibilities and for identifying the knowledge and skills required to carry out those responsibilities.^17,18 The results of the practice analysis guided the development of a certification examination in CVIT, which was administered first in 1991. A notable feature of the certification program was that a single examination was offered, which covered diagnostic and interventional procedures, and included all applicable organ systems (e.g., cardiac, neurologic). After administering the first examination, the ARRT received many inquiries regarding its broad scope. ¹⁹ Some technologists working in general vascular suites believed that the cardiac section of the examination was not important because they did not perform cardiac procedures. Likewise, technologists working exclusively in cardiac catheterization laboratories noted that most noncardiac sections of the examination were not relevant to their practices. The ARRT’s public response advocated an examination that was broad in scope, based on the premise that a set of core competencies unites all CVIT procedures into a single discipline. A technologist who had mastered these core competencies and who had knowledge of specific procedures across all organ systems would be better qualified for professional practice in CVIT.¹⁹ This article reports on a follow-up study on CVIT specialty practice and its implications for professional certification. The ARRT completed another practice analysis project to update the certification requirements and content for the examination in CVIT.²⁰ As part of that project, data were collected with the intent of learning more about specialization within CVIT. The purpose of the present investigation was to determine empirically the number of viable CVIT subspecialties that exist and the extent to which each subspecialty is similar to and different from others. The methodology is similar to that used by large organizations to classify specific jobs into more general job families.²¹ Data were obtained from a detailed task inventory questionnaire completed by a national sample of CVIT radiologic technologists. Responses to the task inventory provided the basis for a practice profile, which summarizes the clinical activities each technologist performs. Practice profiles were subjected to cluster analysis to classify technologists into homogeneous groups based on the similarities of their practice profiles. It is anticipated that the groups will correspond to the major subspecialties within CVIT. Methods Questionnaire Design. A practice analysis questionnaire was developed by the CVIT examination committee and ARRT staff (see acknowledgments at end of article). The task inventory portion of the questionnaire consisted of 137 practice activities and procedures. The first 53 task statements consisted of general activities thought to be performed by most CVIT technologists. General activities included statements such as the following: evaluate the operation of the automatic film programmer and monitor patient’s vital signs and ECG readings. The remaining 84 task statements corresponded to specific cardiovascular procedures performed on patients and included activities such as neurologic angioplasty and coronary angiography. The questionnaire included a rating scale for respondents to indicate the frequency with which they performed each of the 137 activities and procedures. The rating scale had six response categories: (1) not responsible for performing, (2) yearly, (3) monthly, (4) weekly, (5) daily, (6) and several times daily. The task inventory portion of the survey was followed by several demographic questions related to education, experience, and practice setting. It is believed that responses provided a reliable measure of the frequency with which respondents actually perform each activity. Although empirical studies on the testretest reliability of task inventory ratings are impractical, the few studies that have been conducted indicate that responses are stable over time, with testretest reliability coefficients ranging from the 0.80s to the low 0.90s.^22,23 Study Participants. Names and addresses for study participants were obtained from the database of registered radiologic technologists maintained by the ARRT. The population of interest included approximately 4,100 radiologic technologists certified in CVIT and working in the United States. Because the study focused on clinical practice, individuals working primarily in administrative positions or as educators were excluded from the sample. The questionnaire was mailed in February 2000 to a random sample of 1,200 CVIT technologists. A three-stage mailing was employed, which consisted of an initial mailing, a reminder postcard, and a follow-up questionnaire to persons who did not respond after the first two mailings. A total of 933 questionnaires was returned within a 6-week period for an overall response rate of 78%. Data were screened to exclude respondents with excessive missing data, respondents who did not spend at least 50% of their time working in CVIT, and those who were not full-time radiologic technologists. The final sample included 848 surveys yielding a usable response rate of 71%. Although the response rate was good, there is always the potential that respondents represent a biased sample. We were able to verify that the sample was representative of the population in terms of years of experience and geographic distribution; however, the sample may not have been representative in terms of other characteristics. Given the high response rate and the nature of this investigation, any bias due to nonresponse is likely to be minimal and should not have a substantial impact on the results. Analyses. Data analyses required developing a profile for each technologist to indicate his or her involvement in different areas (or domains) of practice. To construct the profiles, the 137 activities were organized into 19 relatively homogeneous domains of practice. The 19 domains are based on earlier work by the ARRT and generally correspond to the manner in which the field of CVIT is organized in other contexts. For example, textbooks typically include a chapter on common medications, and one of the domains corresponded to assisting with medication administration. Similarly, practice settings often are organized according to body systems (e.g., cardiac catheterization laboratory), and many of the 19 domains corresponded to organs or anatomic systems. Table 1 identifies the 19 domains of practice that served as the basis for the practice profiles. All data analyses were completed using the SPSS statistical software package.²⁴ The first step of the analysis involved computing a score for each technologist on the 19 practice domains. A score for a domain conveys the extent to which a technologist is responsible for the activities that compose the domain. Assume that the following three procedures belong to a domain called cardiac diagnostic procedures: coronary angiography, left ventriculography, and ventricular volume measurement. If a technologist indicated that he or she participated in the first two procedures but not the third, the score would be 0.67 for that domain. A complete profile for a technologist consists of scores on each of the 19 performance domains. Classification schemes such as this are most effective when each domain conveys unique information and is relatively independent of other domains.^21,25 If each domain consisted of a random collection of activities, the profiles would be relatively flat, and there would be no discernible pattern to the relationships among the domains. To evaluate the extent to which the 19 categories are empirically meaningful, descriptive statistics for each of the 19 domains were obtained, along with the 19x19 matrix of correlations. The correlations also would be helpful for interpreting the results of subsequent analyses. After the profiles were created and evaluated, cluster analysis was used to identify groups of technologists with similar profiles. Cluster analysis is a method for classifying entities into groups when the group structure and rules for classification are unknown and is common in the behavioral and biologic sciences.^21,25-27 The data for the present study were subjected to a clustering algorithm described by Ward.²⁶ Cluster analysis is an iterative procedure requiring numerous passes through data. At the beginning of the first pass, each technologist belongs to his or her own cluster, so there are 848 clusters. The algorithm searches for the two technologists who have the most similar profiles (i.e., least distance between their profiles) and classifies them into the same group. Although it would be possible to stop at this point, little would have been gained. The algorithm completes a second pass through the data and continues in this manner until all technologists have been clustered into one large group. The challenge with cluster analysis is to determine when clustering should stop. If clustering is stopped early, the solution has too many groups that differ in trivial ways. If clustering continues all the way down to just one or two groups, the persons within those groups may have very different profiles. One strategy for determining an optimal number of clusters is to evaluate a distance index produced at each stage of the clustering. The distance index generally increases in a fairly smooth manner at each successive stage of the analysis. When the index exhibits a sharp increase, it means that two quite different groups were forced together. It is common to accept the clustering solution that occurs just before the index sharply increases. Another strategy for evaluating clusters is to inspect the mean profile for each group formed by the cluster analysis, which is analogous to examining the profile for an individual technologist over the practice domains. The mean scores over the 19 domains inform us about the practice responsibilities of the technologists classified into a given cluster and can be interpreted as the profile for that cluster. If the cluster analysis is worthwhile, each profile should correspond to a recognizable subspecialty. To aid in interpreting the clusters, the profiles were subjected to multidimensional scaling (MDS) analysis.^27,28 Similar to cluster analysis, MDS uses similarities or distances as input. For the present study, distances based on differences between all pairs of profiles were used as input. The goal of MDS is to create a map depicting the similarities and differences among the entities being studied. The output of primary interest is a graph showing the location of each cluster relative to all others. Multi-dimensional scaling analysis is similar to factor analysis in that the results lend themselves to a spatial or geometric representation. This representation may be helpful for identifying the attributes responsible for the similarities and differences among groups.^27,28 An MDS analysis can require just one dimension to model the data but usually requires two or more dimensions. An important part of the analysis is to determine how may dimensions are required. If the MDS model fits the data well, the distances among entities in the MDS map correspond closely to the original distances. The two common indices of model fit are R2 and stress. The R2 in MDS is interpreted as in most statistical analysesas a measure of proportion of variance accounted for by the model, with values closer to 1 indicating better fit. The stress index is a nonlinear measure of model fit specific to MDS.²⁷ It is inversely related to R2 , with values closer to 0 indicating better fit. Results Evaluation of Score Profiles. Each technologist was assigned a score (ranging from 0 to 1) to indicate the proportion of practice activities within each of the 19 domains that he or she performs. The scores over all 19 domains represent the profile for a technologist. Before subjecting the profiles to cluster analysis, we obtained correlations among the practice domains to evaluate their empirical properties. Table 2 presents the correlation matrix; it has been partitioned into zones for ease of interpretation. The correlations in zone 1 of Table 2 show the relationships among the domains that comprise the support activities. Most of these correlations are small. The correlation of 0.09 between physiologic monitoring and equipment quality control indicates that these two domains are unrelated. Knowing if a technologist is involved in patient physiologic monitoring tells us little about the probability of involvement in equipment quality control. The highest correlations in zone 1 are between physiologic monitoring and medications (r = 0.40) and between physiologic monitoring and postprocedure care (r = 0.39). The moderate degree of overlap between these two sets of domains is expected. In general, most correlations in zone 1 are low, indicating that these practice domains represent distinct areas of practice. The correlations in zone 2 depict the relationships between support activities and diagnostic procedures, whereas the correlations in zone 3 show the relationships between support activities and interventional procedures. One feature of these values is that the imaging domain is moderately correlated with all diagnostic procedures (except cardiac), confirming that imaging has some role in most cardiovascular diagnostic studies. Cardiac procedures (diagnostic and interventional) have negative correlations with many of the support activities except for physiologic monitoring. This suggests that physiologic monitoring is more central to technologists who participate in cardiac procedures and that cardiac procedures are different from the other procedures in terms of the performance of other support activities. The correlations in zones 4 and 6 shed light on the relationships among the various diagnostic procedures and, separately, among the interventional procedures. Zone 4 shows the correlations just among diagnostic procedures. The most notable feature is the negative correlation between cardiac procedures and other procedures. Also of interest are the positive correlations among the remaining noncardiac procedures, suggesting substantial overlap among them. Although there is only moderate overlap between neurologic and pulmonary procedures (r = 0.34), there is substantial overlap between neurologic, genitourinary/gastrointestinal (GU/GI), and peripheral vascular diagnostic procedures. Zone 6 contains correlations just among interventional procedures. These correlations generally are similar to the patterns in zone 4. Taken as a whole, the values in zone 4 and zone 6 indicate that (1) cardiac procedures are quite unique, and (2) substantial overlap exists among the noncardiac procedures. Zone 5 provides information concerning the overlap between diagnostic procedures on one hand and interventional procedures on the other hand. The underscored correlations answer questions such as, Are technologists who perform neurologic diagnostic procedures also likely to perform neurologic interventional procedures? For neurologic and pulmonary procedures, there is only minimal crossover from the diagnostic domain to the interventional domain (r = 0.32, 0.28). For GU/GI, peripheral, and cardiac procedures, the crossover from diagnostic to interventional seems to be substantial (r = 0.83, 0.75, 0.88). Technologists who perform diagnostic GU/GI procedures also are likely to perform interventional GU/GI procedures. The same can be said for peripheral and cardiac procedures. Other correlations in zones 4, 5, and 6 are of interest but are not discussed here. Cluster Analysis and Multi-dimensional Scaling. Cluster analysis was done to classify technologists into homogeneous subspecialty groups based on their scores on the 19 practice domains. Table 3 shows the distance statistics for the last 15 iterations of the cluster analysis. The last column indicates the increase in distance over the previous stage. The first large jump in distance occurs when going from six to five clusters. Another notable jump occurs in the transition from four clusters to three, and a third large increase occurs when merging two clusters into one. These results suggest that solutions with six clusters, four clusters, and two clusters might prove useful. The following discussion starts with six clusters, then addresses the four-cluster and two-cluster solutions. Table 4 presents the profiles for the six-cluster solution. The columns correspond to the different clusters or subspecialties, whereas the rows list the 19 performance domains. The values in Table 4 represent the mean for a cluster on each domain. More precisely, the values indicate the mean proportion of activities for which technologists in a given cluster are responsible. The high and low values in each column indicate which procedures technologists in that cluster do and do not perform. Although there is some ambiguity in the data, most clusters were easy to label after studying the rows and columns. Table 5 provides tentative labels and summary descriptions for each subspecialty. The first and largest subspecialty cluster consists of technologists who engage in most patient care activities (except medications) and who are involved in all diagnostic procedures except cardiac. This group also performs some interventional procedures, but apparently only the most routine ones. As indicated in Table 5, technologists in this group worked in a variety of settings. This group was labeled VDc1, to designate that they perform primarily diagnostic procedures. We recognize the arbitrary nature of these labels and offer them as a temporary communication heuristic. The second cluster is similar to cluster 1, in that all of its members perform most noncardiac diagnostic procedures and are involved in a limited number of interventional procedures. Technologists in cluster 2 are different from technologists in cluster 1 because of their limited involvement in pulmonary diagnostic and interventional procedures. They were assigned the label of VDc2 , because they were the second primarily diagnostic group to emerge from the analysis. Cluster 3 corresponds to the cardiovascular generalists. The technologists in this cluster perform procedures related to all organ systems. However, they perform primarily diagnostic procedures, with limited involvement in a few interventional techniques. As might be expected, most of these technologists work in small and moderatesized community hospitals. Only 6% work in large institutions (>500 beds), and 5% are located at university medical centers. Clusters 4 and 5 correspond to the practice of vascular interventional technology. The technologists in these groups (VI1 and VI2 ) perform a wide range of diagnostic and interventional procedures; their label uses the letter I for interventional. The major difference between these two subspecialties is that technologists in VI1 are not involved in cardiac procedures, whereas technologists in VI2 perform cardiac diagnostic and cardiac interventional procedures. Although VI2 may be the most broadly skilled group, they also are the smallest cluster (n = 62) and constitute only 7% of the entire sample. Technologists in these two clusters tend to be in larger hospitals; 33% of technologists in VI1 are employed by university medical centers, whereas 23% in VI2 are employed by university medical centers. Finally, cluster 6 comprises technologists who specialize primarily in cardiac diagnostic and interventional procedures. This is the second largest of the six clusters (n = 159). Although the mean for the cardiac interventional domain (0.68) seems to be low, this domain includes some infrequently performed procedures (e.g., cardiac ablation). This cluster has the highest mean on the physiologic monitoring, and the lowest means on the two contrast domains. The cardiac specialists can be found in all sizes and types of institutions. The four-group solution resulted when the clustering algorithm combined VDc1 and VDc2 into one cluster, then combined VI1 and VI2 into another cluster. A four-group solution suggests the existence of the following subspecialties: (1) diagnostic procedures only; (2) diagnostic and interventional procedures for most organ systems; (3) diagnostic procedures for all organ systems, with limited interventional procedures (i.e., the small hospital generalist); and (4) cardiac diagnostic and interventional procedures. The two-group solution resulted when all clusters except for the cardiac group were combined into a single cluster. To illustrate how the six profiles in Table 4 eventually distill into fewer clusters, the profiles were subjected to MDS. Figure 1 illustrates the two-dimensional solution. A high R2 (0.98) and low stress (0.05) indicate that the two-dimensional solution adequately models the profiles in Table 4. The final stages of the cluster analysis also are shown in Figure 1. The two large ovals show how the six clusters were condensed into four, and the lines connecting the ovals depict how the five clusters on the right eventually merge. This figure shows that of all the subspecialties that exist within CVIT, cardiac is least similar to the others. The spatial maps produced by MDS also are helpful for identifying the attributes that seem to be responsible for the similarities and differences among groups.^27,28 The subspecialties involved primarily with diagnostic procedures are located toward the top of Figure 1, whereas subspecialties located toward the bottom are involved in diagnostic and interventional practice. That is, subspecialties are located on a vertical dimension according to whether they are diagnostic only or diagnostic and interventional. Figure 1 also lends itself to a horizontal interpretation. Subspecialties on the left are involved with fewer body systems (cardiac on far left), and movement to the right denotes involvement with progressively more systems. Although this trend is general and imperfect, it provides a useful way to conceptualize the field of cardiovascular interventional technology. In short, it seems as if the subspecialties can be characterized by a two-dimensional model: (1) the extent to which the subspecialties are diagnostic or interventional and (2) the number or types of organ systems involved. Discussion The most salient outcome of this research is the empirical verification that many technologists perform only cardiac procedures, and their practice profiles are substantially different from the profiles of other CVIT technologists. This result suggests the need to divide the current cardiovascular interventional examination into a cardiac interventional examination and a noncardiac interventional examination. The ARRT made the decision to make such a change in 2001, and the new examinations first were offered in January 2003.²⁰ The data also indicate that the major noncardiac subspecialty could be subdivided into several clusters, but that some of these clusters did not consist of many technologists. Although there is clear evidence of increasing levels of specialization to the point where true generalists are becoming less common, a substantial proportion of broadly skilled technologists working in CVIT still remains (VI2 and GENDx in Tables 4 and 5). Although the few technologists in VI2 perform the full range of diagnostic and interventional procedures, the many technologists in the GENDx cluster perform primarily diagnostic procedures along with some of the more common interventional procedures. This latter group is sure to retain a crucial role in small and moderate sized hospitals, particularly facilities located some distance from urban medical centers. The results of this study may have implications for organizations other than certification boards. Educational institutions that cover the full range of procedures may choose to take a fresh look at the scope of their programs and evaluate whether they want to focus on one or more of the subspecialties identified in Table 4. When decisions about program scope are made, the 19 performance domains in Table 1 might guide the curriculum development so that students proceed from the diagnostic focus to the interventional tracks. The data in Table 2 might help determine the emphasis that certain tracks receive in various topics. The correlations suggest that technologists who practice in cardiac suites are required to have medical knowledge different from their radiology-based peers (e.g., more medications and physiologic monitoring, but less contrast medium). Study results also may be useful for administrators who develop staff training or who are considering the implementation of job rotation or multiskills programs. The data can help inform decisions about sequencing technologists from one set of procedures to the next. In Table 2, high correlations between any two areas (peripheral and GU/GI) suggest substantial overlap between these two areas. The results of studies such as this frequently are useful for performance evaluation.^21,22 Job descriptions and annual evaluations might be based on the domains of practice. The correlations in Table 2 would be of use for this application as well. Domains that are highly correlated might be combined, whereas domains with moderate and low correlations should remain separate because those domains are relatively independent in the practice setting. Finally, professional societies might look to the results of this study as they evolve to represent the interests of their members. The challenge for organizations is to be responsive to diverse interests, while remaining sensitive to the elements of practice and the professional issues that unite the various subspecialties. Studies such as this one help identify those common elements. This study showed that practice analysis, when used in conjunction with certain multivariate procedures, can provide information to guide decisions about specialty practice. The results of this study indicate that CVIT consists of two relatively distinct major specialties: cardiac and all other organ systems. The term ‘relatively distinct’ is used to emphasize the fact that there are technologists whose practice does cross the two specialties. The ARRT’s response to this finding was to divide a single certification program in CVIT into two separate programs corresponding to the two specialties.²⁰ The data also suggest the presence of other, smaller, subspecialties based on organ system and on whether the practice involves diagnostic or interventional procedures. The pattern of correlations in Table 2 (zones 4, 5, and 6), when considered in light of the two-dimensional model, suggest that the diagnostic versus interventional dichotomy is as important as organ system in shaping scopes of practice. Although it would be important to monitor the CVIT subspecialties as they continue to evolve, the decision to offer certification programs corresponding to emerging areas would need to consider factors such as number of practitioners, demand for staff who work in the subspecialties, congruence with training programs, impact on the profession, and effects on patient care and public access to qualified technologists. Practice analyses to be completed by ARRT in the future will provide a mechanism to monitor the continued evolution of CVIT. Acknowledgements Members of the Cardiovascular Interventional Examination Committee and ARRT staff assisted with survey development and interpretation of data. The authors gratefully acknowledge the assistance of Dan Anderson, MS, Betty Ashdown, RT(R)(CV), RCIS, William Bedford, RT(R)(CV), Kevin Dickey, MD, Donna Davis, RT(R)(CV), and Jennifer Levesque, RT(R)(CV). The authors can be contacted at: mark.raymond@arrt.org or codywms@msn.com This study was supported by the American Registry of Radiologic Technologists; the authors are solely responsible for its content. This article is reprinted with permission from the J Allied Health 2004;33(2):95-103.

1. Sones FM, Shirey EK: Cine coronary arteriography. Mod Concepts Cardiovas Dis 1962;31:735-738.

2. Hurst JW: History of cardiac catheterization. In King SB III, Douglas JS Jr (eds): Coronary Arteriography and Angioplasty. New York: McGraw-Hill, 1985:1-9.

3. Heupler F Jr: Coronary arteriography and left ventriculography: Sones technique. In King SB III, Douglas JS Jr (eds): Coronary Arteriography and Angioplasty. New York: McGraw-Hill, 1985:137-181.

4. Viamonte M Jr, Gusselin AJ, Sommer LS: Coronary arteriography: Some observations in techniques and interpretation. AJR Am J Roentgenol 1964;92:872-876.

5. Viamonte M Jr, Warren WD, Fomon JJ: Liver panangiography in the assessment of portal hypertension in liver cirrhosis. Radiol Clin North Am 1970;8:147-167.

6. White RI Jr: Fundamentals of Vascular Radiology. Philadelphia: Lea & Febiger, 1976.

7. Judkins MP: Selective coronary arteriography: part I. a percutaneous transfemoral technique. Radiology 1967;89:815-824.

8. Judkins MP: Percutaneous transfemoral selective coronary arteriography. Radiol Clin North Am 1968;6:467-492.

9. Judkins MP, Judkins E: Coronary arteriography and left ventriculography: Judkins technique. In King SB III, Douglas JS Jr (ed): Coronary Arteriography and Angioplasty. New York: McGraw-Hill, 1985; 182-238.

10. Bruckner MG: Circulatory system. In Ballinger PW, Frank ED (eds): Merrill’s Atlas of Radiographic Positions and Radiologic Procedures, 9th ed. St. Loui: Mosby, 1999;26:19-85.

11. Koolpe HA, Embil W, Koolpe L, et al: Hemodynamic guidelines for surgical therapy of portal hypertension. Ann Surg 1981;194: 553-561.

12. Gruentzig AR, Senning A, Siegenthaler WE: Nonoperative dilatation of coronary artery stenosis. N Engl J Med 1979;301:61-68.

13. King SB III, Douglas JS Jr, Gruentzig, AR: Percutaneous transluminal coronary angioplasty. In King SB III, Douglas JS Jr (eds): Coronary Arteriography and Angioplasty. New York: McGraw- Hill, 1985:17:433-460.

14. Dotter CT, Judkins MP: Transluminal treatment of atherosclerotic obstruction: description of a new technic and a preliminary report of its application. Circulation 1964;30:654-670.

15. Huff JA: Cardiac catheterization: interventional procedures of the vascular system: Adults. In Ballinger PW, Frank ED (eds): Merrill’s Atlas of Radiographic Positions and Radiologic Procedures, 9th ed. St. Louis: Mosby, 1999:239-260.

16. American Registry of Radiologic Technologists: The History of the American Registry of Radiologic Technologists. St. Paul, MN: ARRT, 1996:111-112.

17. Miller GE: The orthopaedic training study. 17. Miller GE: The orthopaedic training study. JAMA 1968;206: 601-606.

18. Raymond MR: Job analysis and the specifications of content for licensure and certification examination. Appl Meas Educ 2001; 14:369-415.

19. American Registry of Radiologic Technologists: Open letter to cardiovascular- interventional technology community. In: Annual Report to Registered Technologists. St. Paul, MN: ARRT, 1992: 9-10.

20. American Registry of Radiologic Technologists: Cardiovascular-interventional exam to split into cardiac, vascular. In: Annual Report to Registered Technologists. St Paul, MN: ARRT, 2001:11.

21. Fleishman MT, Quaintance MK: Methodological issues in developing and evaluating classificatory systems. In: Taxonomies of human performance: Description of Human Tasks. New York: Academic Press, 1984:42-61.

22. Wilson MA, Harvey RJ, Macy BA: Repeating items to estimate the test-retest reliability of task inventory ratings. J Appl Psych 1990; 75:158-163.

23. Kane MT, Miller T, Trine M, et al: The precision of practice analysis results in the professions. Eval Health Prof 1995;18:29-50.

24. Norusis MJ: SPSS for Windows: Professional Statistics. Release 6.0. Chicago: SPSS, 1993.

25. Sokal R: Classification: purpose, principles, progress. Science 1974; 185:1115-1123.

26. Ward JH Jr: Hierarchical grouping to optimize an objective function. J Am Stat Assoc 1963;58:236-244.

27. Shepard RN: Multidimensional scaling, tree-fitting, and clustering. Science 1980;210:390-398.

28. Raymond MR: Applications of multidimensional scaling analysis to research in the health professions. Eval Health Prof 1989;12: 379-408.