Notes on Methodology | 2022
The BRIMR rankings are derived each year from data compiled and released by the National Institutes of Health shortly after the federal fiscal year closes. The NIH posts these data on its Research Portfolio Online Reporting Tool (RePORT) website in the form of a master Excel spreadsheet file called “Worldwide”. For fiscal 2022, which began on 1 October 2021 and ended on 30 September 2022, this file was posted in late December 2022. It contained tabulated information on more than $36 billion in funding through 65,307 extramural NIH grants or contracts awarded to more than 43,500 Principal Investigators (PIs) that year.
BRIMR begins by making a large number of modifications to the NIH master file, most aimed at further standardizing the terminology used. For example, BRIMR combines
various names used for a given grantee organization in the NIH file (under column A) into a single listing where appropriate, so that any grants attributed to ACME UNIVERSITY COLLEGE OF MEDICINE, or ACME UNIVERSITY MEDICAL CENTER, or ACME UNIVERSITY HEALTH CENTER INC, might all be credited simply to ACME
UNIVERSITY. Similarly, the NIH’s designations SCHOOL OF MEDICINE & DENTISTRY or OVERALL MEDICAL are each replaced by SCHOOLS OF MEDICINE in column K, and city names are fully capitalized (e.g., BOSTON) in column P. We also continue to move toward a more uniform format in naming grantee organizations, such as the various University of Texas or University of California campuses.
Some of BRIMR’s revisions are particularly significant. For example, the Mayo Clinic Rochester is not classified as a medical school in the NIH master file, but BRIMR has always designated it as such because it has been granting medical degrees since 1972. Commencing in 2021, on the other hand, BRIMR classifies MD Anderson as a hospital rather than as a medical school, in keeping with the longtime practices of NIH and of MD Anderson itself. In recent years, the Case Western Reserve University and Cleveland Clinic Lerner medical schools (which share faculty and facilities) have been combined by BRIMR under one listing, as have several teaching or research entities affiliated with Rutgers University that were listed separately in the past. All such changes are intended to facilitate computerized analysis and to make the BRIMR database as valid, intuitive, and user-friendly as possible.
Those changes give rise to a modified Worldwide file that we post in draft form on the BRIMR webpage for a 4-week period of public review, typically in January-February. Grantee institutions and individual PIs are invited to review, supply missing data, or propose corrections to their own awards during that time. All requests for changes must be received by BRIMR’s stated deadline, which this year fell on 5 February 2023, and are subject to review and implementation at BRIMR’s sole discretion. Once finalized, the modified file serves as the basis for all other BRIMR lists and rankings that year.
Important aspects of our rankings are dictated by NIH policy and practice, as reflected in the data posted on NIH RePORT. We conform to the NIH’s longstanding policy of crediting only one lead PI and one institution for any given award, even in the case of large consortia, program projects, subcontracts, or multi-PI awards. Grants to faculty at some medical schools are credited by NIH to affiliated teaching hospitals, which tends to understate the funding and rankings of those schools; examples can be found in Harvard Medical School’s affiliations with Massachusetts General Hospital and the Brigham and Women’s Hospital, as well as in those of other medical schools with research-intensive cancer or neuropsychiatric institutes or children’s hospitals.
In its Worldwide file (under column J), the NIH assigns departments into 44 standard categories. BRIMR ranks departments within 27 of those NIH-designated categories. Only medical-school departments are ranked. The NIH taxonomy combines certain disciplines, such as diagnostic and therapeutic radiology, into a single category even if they reside in separate departments at a given institution, and it likewise assigns departments with hybrid names (such as “PHYSIOLOGY AND PHARMACOLOGY”) to one category or the other. BRIMR typically adopts those assignments made by NIH. We also accept NIH’s practice of crediting a few medical schools with grants to non-traditional departments such as biology, chemistry, psychology, or physics.
Although R&D contracts constitute about 10% of all NIH extramural funding, they are not considered in most BRIMR rankings because the NIH master file often does not
ascribe them to a specific school (such as a School of Medicine) or department within an institution. Contracts are, however, listed as a stand-alone file by institution and are also ranked by PI.
BRIMR received requests for corrections to the 2022 dataset from more than 100 grantee organizations or individuals concerning more than 1,100 awards. Most involved grants thought to have been credited to an incorrect department, to no department at all, or to the wrong School within an institution. Some involved PIs whose grants were administered by more than one department or had been ascribed to a center or institute rather than to the PI’s home department. Of note, BRIMR rarely, if ever, approves requests to change the grantee organization (column A) or dollar amount (column N) of a listed award, and does not add awards not listed in the NIH’s Worldwide file.
These corrections and all other aspects of the rankings remain transparently accessible for public review by comparing BRIMR’s Worldwide file to other files on the BRIMR website and to the Worldwide file on NIH RePORT for the corresponding year. To download the NIH file, go to the “Awards by Location” page in RePORT, select the fiscal year and “Data” tabs, and then press “EXPORT TABLE”.
It is important to emphasize that NIH RePORT is not the same as NIH RePORTER. The latter is a separate search tool that often captures data from more than one fiscal year, no-cost extensions, subcontracts, or fund carry-forwards, as well as funding from other agencies, such as CDC, FDA, HRSA, and the VA, that are not included in NIH’s Worldwide file nor considered in BRIMR’s rankings. BRIMR draws all of its data from NIH RePORT – we do not use data from NIH RePORTER unless it is also listed in NIH RePORT.
We regret that no revisions will be applied retroactively to prior years, and that none will be communicated by BRIMR to NIH. If you find what you think may be a discrepancy or error, please check whether it is present in the original Worldwide file on NIH RePORT; if so, you may wish to contact the NIH to have it corrected there.
~ Robert Roskoski Jr. and Tristram G. Parslow