An analysis of bibliometric indicators, National Institutes of Health funding, and faculty size at Association of American Medical Colleges medical schools, 1997–2007^EC

Gathering Data

Employing raw citation data gathered and synthesized from ISI's online citation index, Web of Science, this study intended to provide measures that might describe research productivity of US medical schools that were AAMC members (n = 128). Web of Science contains bibliographic and bibliometric data from approximately 8,700 journals. In searching for research from specific medical schools, the author used broad searches in the Web of Science address field and limited to the date range, 1997–2007. Because the author used the search term, “Med*,” in most of the searches, he took precautions to eliminate articles dealing with veterinary medicine and media studies. The author subsequently refined the results by institution using the Analyze Results feature in Web of Science. Exercising due diligence in capturing all possible name variants of a medical school in Web of Science, the author linked proper names of medical schools (Weill Medical College) and institutional names (Cornell University) with a Boolean “OR” operator. Furthermore, Web of Science attributes the same status to all institutions listed in the address field regardless of primary authorship or total number of authors. The author did not use proportional or fractional attribution methods [35]. For example, an article with the primary author and nine authors from medical school A and one author from medical school B treated both medical schools the same in the article and citation counts.

For each medical school search, a citation report was run in Web of Science on the refined results set to gather five measures:

• the total number of published articles per medical school (a)

• the total number of citations to published articles per medical school (c)

• the average citations per article (a/c)

• the institutional h-index (h)

• the number of articles with zero citations per medical school

From these measures, the author calculated:

• institutional impact indices (i)

• institutional percentages of articles with zero citations (z)

The citation report feature in Web of Science limits users to results sets less than 10,000 articles. Given that numerous medical schools exceeded 10,000 published articles in medicine and biomedical sciences, the author ran multiple citation reports for the most prolific institutions. Articles and the accompanying citation data from these institutions were harvested through refining the data by individual year and totaling the sums. These data were searched during the week of March 16, 2008, through March 22, 2008.

Using faculty information from AAMC [36] and institutional grant information from NIH [37], the following measures were retrieved or calculated:

• total amount of NIH funding, 1997–2005, per medical school (t)

• annual average number of faculty per medical school, 1997–2007 (f₂₀₀₇)

• annual average number of faculty per medical school, 1997–2005 (f₂₀₀₅)

• average amount of NIH grant money awarded in 1997–2005 per faculty member (t/f₂₀₀₅)

• average number of articles per faculty member, 1997–2007 (a/f₂₀₀₇)

• average number of citations per faculty member, 1997–2007 (c/f₂₀₀₇)

Two caveats must be mentioned in regard to the NIH data. Because NIH had only published NIH institutional funding data through 2005 by March 2008, the author calculated the average amount of NIH grant money awarded per faculty member (t/f₂₀₀₅) for only the nine-year period of 1997–2005. Secondly, NIH did not publish funding figures for 5 AAMC medical schools: Florida International University College of Medicine, Florida State University College of Medicine, Mayo Medical School, San Juan Bautista School of Medicine, and University of Central Florida College of Medicine. These 5 institutions were eliminated from the statistical analyses, resulting in 123 medical schools studied. Isolated data points or groups of points, such as the data from the aforementioned medical schools, exert powerful influence over variable correlations and can camouflage fundamental relationships that may exist.

Calculating Impact Index

Hirsch developed the h-index (h) originally to measure the quantitative and qualitative impact of the individual scholar's research [24]. The h-index is an indicator that is defined as the number of published papers, h, each of which has been cited by others at least h times. Universities that produce larger numbers of publications will generally command higher h-indices, demonstrating the metric's inherent size dependence. In this study, size-dependent variables included total number of publications (a), total number of citations (c), faculty size (f₂₀₀₇), and total NIH funding (t), which all favored larger medical schools. To allow comparative equity among variously sized medical schools, size-independent variables—such as average citations per article (a/c), institutional impact indices (i), institutional percentages of articles with zero citations (z), average amount of NIH grant money per faculty member (t/f₂₀₀₅), average number of articles per faculty member (a/f₂₀₀₇), and average number of citations per faculty member (c/f₂₀₀₇)—were necessary for faculty-level productivity and research impact analyses.

To mitigate the influence of the number of articles published, Molinari and Molinari devised a methodology to extend the utility of the h-index with a new metric, the impact index (i), to evaluate the research productivity of large groupings. This study applied Molinari and Molinari's methodology to retrieve and calculate size-independent bibliometric data. In their paper, the impact index (i) was calculated by dividing a set of published articles' h-index (h) by a factor (m) of the number of published articles (a). Molinari and Molinari computed 0.4 as the power law exponent, or the master curve (m), as it has shown to be the universal growth rate for citations over time for large numbers of papers [38]. Thus, impact index was calculated to be:

This equation characterizes the number of relatively high-impact articles at an institution in relation to all articles published at the same institution, producing a metric that is able to compare publication sets of differing sizes [38]. Recently, a similar study in the Proceedings of the National Academy of Sciences also applied Molinari and Molinari's methodology, to calculate h-indices and impact indices of non-biomedical research for selected universities, laboratories, and government agencies in the United States [39].

Analyzing Data

The author entered the collected and synthesized data into SPSS v.15. Because the ten indicators were dependent and interval variables, the author employed factor analysis to examine the relationships between variables. A multivariate statistical technique, factor analysis, can ascertain the underlying structure of the variables' relationships by reducing them to a smaller number of hidden variables, known as factors. These factors, in turn, provide insight to how variables cluster together [40–42].

Problems arise with factor analysis when variables in the correlation matrix exhibit very high correlations, also known as multicollinearity. The author assumed multicollinearity existed in the data due to the fact that size-independent variables were synthesized from size-dependent variables and the concomitant presence of several significant bivariate linear correlations. SPSS calculated the Pearson's correlation coefficients (r) and the coefficients of determination (R²). Furthermore, statisticians suggest a very large sample size when conducting factor analysis. This study's sample size of 123 medical schools was a poor-to-fair sample size for common factor analysis [42]. SPSS was unable to calculate factors using principal axis, or common, factor analysis, so another method of factor analysis, principal components analysis, was utilized to extract factors. Analogous to common factor analysis, principal components analysis uncovers how many factors describe the set of variables, what the factors themselves are, and which variables form independent coherent groupings [41].

After extracting the factor coefficients, or loadings, both orthogonal (using varimax technique) and oblique (using promax technique) rotations were performed to provide breadth to the principal component analysis. Rotational methods simplify the data interpretation by diversifying the variable loadings, thus showing which variables cluster together more clearly. Orthogonal rotations assume no correlations exist among factors, whereas oblique rotations presume that correlations do exist. When applied, the two types of rotations reap different results.

Collected and Synthesized Data

Table 1 provides the total number of published articles from 1997–2007 per medical school (a), total number of citations to those published articles per medical school (c), average number of citations per article (a/c), institutional impact index (i), and institutional percentage of articles with no citations (z) from the top quartile of medical schools in total published articles (n = 31). From 1997–2007, Harvard University Medical School published more than twice as many articles as any other medical school and garnered almost three times as many citations. Two of the three size-independent bibliometric measures, the institutional impact index (i) and the percentage of articles with no citations (z), were topped by the Mount Sinai School of Medicine of New York University. The University of California, San Francisco, School of Medicin, had the highest average number of citations per article (a/c) with 24.91 citations per article. Additionally, dividing many of the collected metrics, such as publications (a) and citations (c), with the annual average of total faculty members (f₂₀₀₇) provided useful size-independent measures. Table 1 also illustrates these measures among the top quartile of the medical schools in total articles (a) (n = 31). Stanford University School of Medicine ranked highest in publications (35.25) and citations (853.75) per faculty member over 1997–2007.

The 1997–2005 NIH funding and AAMC statistics from the top quartile of medical schools in total NIH funding (n = 31) are displayed in Table 2. From 1997–2005, Johns Hopkins Medical School garnered the most NIH funding with $3,032,629,799 total, and at $2,631,130 per faculty member, Stanford University Medical School faculty were awarded the most NIH funding on average. Harvard University Medical School annually averaged the highest number of faculty from 1997–2005 at 5,217 members. (Bibliometric, NIH, and AAMC statistics for all 123 medical schools for the full data sets are available with the online version of this journal.)

Descriptive Statistics

To demonstrate scale and variance of the data, Table 3 displays descriptive statistics such as the means, medians, standard deviations, and coefficients of variance for bibliometric, NIH, and AAMC data. The publication and citation histories across the studied medical schools showed a tremendous amount of variance for size-dependent variables. The standard deviations of published articles (a), citations to those articles (c), and total amounts of NIH funding (t) exceeded 100% of their means. The size-independent variables—average citations per article (a/c), impact index (i), and percentage of articles with no citations (z)—showed a smaller amount of variance with their relative standard deviations amounting to 32%, 20% and 20% of their means, respectively.

Principal Components Analysis

The relationships of the 10 studied indicators were analyzed via principal components analysis for 123 medical schools. For this data set, SPSS extracted 3 factors that explain 91% of the total explained variance. Table 4 illustrates the role of the variables in the composition of the factors rotated orthogonally using the varimax technique. The first factor, which accounted for 31% of the rotated variance, was highly associated with size-dependent measures—total articles (a), total citations (c), total number of faculty (f₂₀₀₇), and to a lesser degree, total NIH funding (t). Responsible for 30% of the rotated variance, the second factor grouped citations per article (a/c), impact index (i), and percentage of articles with no citations (z), which were all size-independent measures, and to a lesser extent, total NIH funding (t). Finally, the third factor, influenced heavily by size-independent measures—average number of published articles per faculty member (a/f₂₀₀₇), average number of citations per faculty member (c/f₂₀₀₇), and average NIH funding per faculty member (t/f₂₀₀₅)—also accounted for 30% of the rotated variance. Total NIH funding (t) was most significantly connected with this final grouping.

The oblique rotation offered a slightly different interpretation of the data. Table 5 displays the pattern matrix extracted from an oblique rotation using the promax technique. Theoretically, the extracted pattern matrix approximated the orthogonally rotated factor matrix (Table 4) as it measured one linear factor-variable relationship, the uniqueness of the relationship. The oblique pattern matrix revealed many of the same structural patterns and groupings modeled in the orthogonal factors matrix. However, the one major difference was that the total NIH funding (t) factor loadings only showed a significant association with the third grouping of variables. Oblique rotation also revealed that the three extracted factors correlated highly with each other.

Using both rotational analyses in tandem, the data summary provided three distinct clusters of variables. The first grouping of variables, labeled gross research productivity, comprised total articles (a), total citations (c), and total number of faculty (f₂₀₀₇). The second cluster reflected the general influence or impact of research and included citations per article (a/c), impact index (i), and percentage of articles with no citations (z). The variable, percentage of articles with no citations (z), displayed a negative correlation; thus, it decreased when other variables demonstrated positive growth. The third grouping described the size-independent variables that reflected research productivity on an individual level: average number of published articles per faculty member (a/f₂₀₀₇), average number of citations per faculty member (c/f₂₀₀₇), and average NIH funding per faculty member (t/f₂₀₀₅). Interestingly, total NIH funding (t) demonstrated a significant relationship with all three factors when the data were rotated orthogonally, but only with the individual productivity group with oblique rotation. Counterintuitively, the relationship between gross research productivity and total NIH funding (t) was the weakest relationship. Practically speaking, these data might show that the influence of research carried more weight than volume of research in NIH-funded awards to medical schools.

Analysis of Collected and Synthesized Data

Despite methodological and data-specific limitations, the compiled data sets presented in Table 1 and Table 2 broadly characterize the research productivity and impact at AAMC medical schools from 1997–2007. Each metric characterizes research output distinctively. Size-dependent measures may quantify aspects of overall institutional productivity, while size-independent measures may describe a medical school's impact in the research community or productivity of individual faculty members. The principal components analysis combined with a perusal of the data reveal a general sense of the most productive or influential research faculties on an institutional and individual basis. Due to scalability and generalizability problems, synthesizing a single metric based on the studied indicators does not honestly assess the multifaceted nature of medical school research.

The most productive medical schools on an institutional level—Johns Hopkins University School of Medicine; Harvard Medical School; University of Pennsylvania School of Medicine; Washington University in St. Louis School of Medicine; University of California, Los Angeles, David Geffen School of Medicine; Yale University School of Medicine; and Stanford University School of Medicine—demonstrated high rankings in total number of published articles (a), total number of citations (c), total number of faculty (f₂₀₀₇), and total NIH funding (t).

Stanford University School of Medicine; Yale University School of Medicine; Washington University in St. Louis School of Medicine; University of California, San Diego, School of Medicine; University of California, Los Angeles, David Geffen School of Medicine; Johns Hopkins University School of Medicine;, and Duke University School of Medicine appeared to have faculties who were among the most productive per person based on the data calculated for average number of published articles per faculty member (a/f₂₀₀₇), average number of citations per faculty member (c/f₂₀₀₇), average NIH funding per faculty member (t/f₂₀₀₅), and total NIH funding (t).

Size-independent measures, average citations per article (a/c) and impact index (i) percentage of articles not cited (z), and size-dependent measure, total NIH funding (t), might best describe the influence or impact of a medical school's research. Mount Sinai School of Medicine of New York University; University of Texas Medical School at San Antonio; University of California, San Francisco, School of Medicine; University of California, San Diego, School of Medicine; and University of Chicago Pritzker School of Medicine had faculties who produced very influential research, according to this study's data. It must be noted that the above-mentioned medical schools were not ranked or listed in any particular order.

Comparisons with Other Studies

Earlier analyses of biomedical and medical research typically focused on specific disciplines [34, 43–64], journals and the influence of journal impact factors [47,48,51,52,55,57,58,61,64,65], and national research performance of countries [44, 46–51, 53–55, 59–63,66,67]. Few studies concentrated on medical school research productivity and impact on an institutional level. The methods used by the investigators of these studies differed, so comparisons of data sets were complicated. Nevertheless, discrete correlations found in certain studies offered some points of comparisons with this research.

In 1983, McAllister and Narin found a much more significant linear correlation between total number of articles and NIH funding than this study (r = 0.95 versus r = 0.69, P<0.001). Changes in publishing behaviors, such as the enormous growth in the volume of scholarship, and grant funding might explain this. They also discovered interconnectedness between impact metrics, size-dependent productivity metrics, and faculty perceptions of institutional quality [31]. More recently, a study on peritoneal dialysis publications significantly correlated articles and citation counts (r = 0.63, P = 0.039) [46], though the significance appeared somewhat weaker than this study's findings (r = 0.98, P<0.001). In concordance with this study's findings, British researchers concluded that research impact correlated positively to funding [68,69]. A recent JAMA study corroborated this causal relationship, discovering that funding over $20,000 generally indicated higher-quality medical education research [70]. Lewison also arrived at the same conclusion as this study: an assortment of statistical indicators is essential in institutional analyses as each indicator paints a very different picture of research [34].

Though not focused exclusively on medical schools, van Raan conducted a comprehensive institutional analysis using statistics from the 100 largest European universities. His research also found a strong correlative relationship between average citations per article (a/c) and percentage of articles that are not cited (z) (r = -0.92), which bolstered the current study's principal components analysis and basic linear correlation (r = −0.74, P<0.001). Other metrics from the current study exhibited similar associations with van Raan's power law analysis, including strong correlations between total articles (a) and total citations (c) (r = 0.98, P<0.001; van Raan, r = 0.89) and a negligible correlation between total articles (a) and percentage of articles that were not cited (z) (r = −0.255, P = 0.002; van Raan, r = 0.35) [71].

Implications for Medical Librarianship

This study's results can inform the provision of bibliometric services in health sciences libraries. Primarily, medical librarians may become aware of the nature of their institution's unique research profile in regard to gross productivity, individual productivity, research impact, and funding. Medical librarians may support administrative assessment efforts by providing benchmarks or preliminary data that can be used in intra-campus departmental analyses or larger intercampus comparisons. Furthermore, familiarity with the various evaluative methodologies of research metrics permits health sciences librarians to better evaluate their collections—particularly regarding resources dealing with the assessment of universities or medical schools. Beyond simple impact factors and citation counts, librarians may assist health sciences faculty, administrators, and departments with tenure and promotion dossiers and their analyses and provide educational offerings on the basics of bibliometrics. At the author's home institution, the health sciences library offers a workshop, “Tenure Metrics,” and a support wiki that covers cited reference searching for ten databases [72], among other services.

Methodological Limitations

Due to the nature of Web of Science's data set and capabilities, limitations to this methodology exist. For example, research linked to institution names that were misspelled or used unfamiliar variants in the address field were not retrieved. Moreover, citation errors occur in all bibliographic fields in Web of Science, thus, these citations were not accounted for under this methodology. In fact, Moed estimated that 7% of citations from ISI databases contain errors [21]. As stated in the methodology section, no fraction or proportional attribution techniques were applied in the case of multiple authors from different medical schools. Selective in its coverage, Web of Science does not track the citation histories of thousands of journals, proceedings, technical reports, and patents. It must be noted that the collected Web of Science data include self-citations, which may skew the resulting data. Other bibliometric studies demonstrate that including self-citations proves insignificant to the overall results in macro-level studies [73,74].

NIH was the only source of grant funding data used in this study. Monies from private foundations, nongovernmental organizations, and other government departments were not taken into account in this study. NIH grants used for other programs apart from research (i.e., training) were included in this study. In regard to AAMC data, the association stresses that the numbers from their faculty roster are not the official numbers reported to the Liaison Committee on Medical Education, and their ongoing data collection methodology may report different faculty numbers [36].

Due to the outliers at both extremes, the variances of many metrics were high. The most notable outliers were data from the top-ranking medical schools for particular variables. For example, Harvard Medical School exceeded 3 standard deviations from the mean in total articles (a), total citations (c), and total number of faculty (f₂₀₀₇). Stanford University School of Medicine also surpassed 3 standard deviations from the mean in average NIH funding per faculty member (t/f₂₀₀₅), articles per faculty member (a/f₂₀₀₇), and citations per faculty member (c/f₂₀₀₇). Six schools exceeded 3 standard deviations from the mean in at least 1 indicator, and 20 schools were 2 standard deviations from the mean in 1 or more indicators. The data analysis was notably affected by the presence of these medical schools. By eliminating the 20 schools that exceeded 2 standard deviations from the mean in 1 indicator, SPSS was able to run a principal axis factor analysis on the remaining 103 medical schools. In this type of analysis, only 2 factors were extracted. From these analyses, the author observed variant patterns of variable clustering where total articles (a), total citations (c), average number of articles per faculty member (a/f₂₀₀₇), and average number of citations per faculty member (c/f₂₀₀₇) associated with the first factor, and average citations per article (a/c), impact index (i), percentage of article with no citations (z), total NIH funding (t), and average NIH funding per faculty member (t/f₂₀₀₅) associated with the second factor.

Given this study's intention of conducting a macro-analysis of bibliometric statistics, NIH funding data, and AAMC faculty data, results are necessarily broad. Furthermore, every discipline has a unique culture and variant citation rates [46], so the data cannot be extrapolated to specific departments at the medical schools studied. This is also evident when calculating impact indices, where the 0.4 m-curve may not exactly represent the universal growth rate for all biomedical disciplines. The use of indicator averages negates the study of temporal qualities of the data, so potentially illustrative statistics, such as growth over time, have not been described.

Further Refinement and Study

To get a more focused picture of research productivity, the author suggests refining broad bibliometric data in various ways. Specifically, researchers could conduct bibliometric analyses at the school, departmental, or laboratory level. For instance, the general citation behaviors of pediatricians may be much different than those of cell biologists. A great deal of research delves into the bibliometrics of medical specialties and subspecialties [6, 34, 43–64]. Furthermore, the author believes the high data variance, especially in size-dependent measures, requires researchers to comparatively study medical schools among similar sized institutions to generate valid associative assessments. This is evident in the differences of analyses when a group of the highest and lowest performing medical schools were removed from the population.

Not taken into account in this study, metrics synthesized from journal prestige information, such as ISI's impact factors and EigenFactor's article influence [75], can also shed a different light on research productivity. Mathematically, modeling bibliometric data according to a power law distribution [76] may reveal another view of the data. Commercially, ISI provides citation analysis products, the university science indicators and essential science indicators, which allow users to customize and mine their extensive data sets. Their web-based newsletter, ScienceWatch, discusses several bibliometric-related topics such as emerging research fronts, top researchers in specific fields, and “hottest” papers [77]. Other resources—such as Google Scholar, Scopus, Citebase, and other subject-specific databases—can also provide supplementary citation information mined from the web, journals not indexed by ISI, patents, and gray literature.