Notes on Methodology | 2023
The BRIMR rankings are derived each year from data compiled and released by the National Institutes of Health shortly after the federal fiscal year closes. The NIH posts these data on its Research Portfolio Online Reporting Tool (RePORT) website in the form of a master Excel spreadsheet file called “Worldwide”, accessible under the “Awards by Location” tab. For fiscal 2023, which began on 1 October 2022 and ended on 30 September 2023, this file was posted on 10 January 2024. It contained tabulated information on more than $38.1 billion in funding through 66,147 extramural NIH grants or contracts awarded to nearly 45,000 Principal Investigators (PIs) that year. Schools of Medicine garnered $18.9 billion of that total (excluding R&D contracts), or about 50% of NIH extramural funding.
BRIMR begins by making a large number of modifications to the NIH master file, most aimed at further standardizing the terminology used. For example, BRIMR combines various names used for a given grantee organization in the NIH file (under column A) into a single listing where appropriate, so that any grants attributed to ACME UNIVERSITY COLLEGE OF MEDICINE, or ACME UNIVERSITY MEDICAL CENTER, or ACME UNIVERSITY HEALTH CENTER INC, might all be credited simply to ACME UNIVERSITY. Similarly, the NIH’s designations SCHOOL OF MEDICINE & DENTISTRY or OVERALL MEDICAL are each replaced by SCHOOLS OF MEDICINE in column K, and city names are standardized as appropriate (e.g., NEW YORK CITY is converted to NEW YORK) in column P. We also continue to implement a more uniform format in naming grantee organizations, such as the various University of Texas or University of California campuses.
Some of BRIMR’s revisions are particularly significant. For example, the Mayo Clinic Rochester is not classified as a medical school in the NIH master file, but BRIMR has always designated it as such because it has been granting medical degrees since 1972. In recent years, the Case Western Reserve University and Cleveland Clinic Lerner medical schools (which share faculty and facilities) have been combined by BRIMR under one listing, as have several teaching or research entities affiliated with Rutgers University that were listed separately in the past. An initial set of new awards submitted jointly by the Henry Ford Health System and Michigan State University has been credited to the University for 2023 ranking purposes as those two institutions move to forge a closer research alliance in the years ahead. All such changes are intended to facilitate computerized analysis and to make the BRIMR database as valid, intuitive, and user-friendly as possible.
Those changes give rise to a modified Worldwide file that we post in draft form on the BRIMR webpage for a 4-week period of public review, typically in January-February. Grantee institutions and individual PIs are invited to review, supply missing data, or propose corrections to their own awards during that time. All requests for changes must be received by BRIMR’s stated deadline, which this year fell on 2 February 2024, and are subject to review and implementation at BRIMR’s sole discretion. Once finalized, the modified file serves as the basis for all other BRIMR lists and rankings that year.
Important aspects of our rankings are dictated by NIH policy and practice, as reflected in the data posted on NIH RePORT. We conform to the NIH’s longstanding policy of crediting only one lead PI and one institution for any given award, even in the case of large consortia, program projects, subcontracts, or multi-PI awards. Grants to faculty at some medical schools are credited by NIH to affiliated teaching hospitals, which tends to understate the funding and rankings of those schools; examples can be found in
Harvard Medical School’s affiliations with Massachusetts General Hospital and the Brigham and Women’s Hospital, as well as in those of other medical schools with research-intensive cancer or neuropsychiatric institutes or children’s hospitals.
In its Worldwide file (under column J), the NIH assigns departments into 44 standard categories. BRIMR ranks departments within 27 of those NIH-designated categories. Only medical-school departments are ranked. The NIH taxonomy combines certain disciplines, such as diagnostic and therapeutic radiology, into a single category even if they reside in separate departments at a given institution, and it likewise assigns departments with hybrid names (such as “PHYSIOLOGY AND PHARMACOLOGY”) to one category or the other. BRIMR typically adopts those assignments made by NIH. We also accept NIH’s practice of crediting a few medical schools with grants to departments such as biology, physics, or veterinary sciences. PIs are ranked in comparison to others who hold awards in the same departmental categories, irrespective of their individual training or specialty.
BRIMR received requests for corrections to the 2023 dataset from nearly 100 grantee organizations or individuals concerning more than 1,200 awards. Most involved grants thought to have been credited to an incorrect department, to the wrong School within an institution, or to no School or department at all. Some involved PIs whose grants were administered by more than one department or had been ascribed to a center or institute rather than to the PI’s home department. Because NIH’s listings for R&D contracts do not credit a specific school or department, BRIMR never includes such contracts in any of our rankings, even when grantees provide that information to us. Of note, BRIMR rarely, if ever, approves requests to change the grantee organization (column A) or dollar amount (column N) of a listed award, and does not add awards not listed in the NIH’s Worldwide file.
These corrections and all other aspects of the rankings remain transparently accessible for public review by comparing BRIMR’s Worldwide file to other files on the BRIMR website and to the Worldwide file on NIH RePORT for the corresponding year. To download the NIH file, go to the “Awards by Location” page in RePORT, select the fiscal year, and press “RESUBMIT QUERY”; once the new dataset has loaded, press the “Data” tab, and then press “EXPORT TABLE”.
It is important to emphasize that NIH RePORT is not the same as NIH RePORTER. The latter is a separate search tool that often captures data from more than one fiscal year, no-cost extensions, subcontracts, or fund carry-forwards, as well as funding from other agencies, such as CDC, FDA, HRSA, and the VA, that are not included in NIH’s Worldwide file nor considered in BRIMR’s rankings. BRIMR draws all of its data from NIH RePORT – we do not use data from NIH RePORTER unless it is also listed in NIH RePORT.
We regret that no revisions will be applied retroactively to prior years, and that none will be communicated by BRIMR to NIH. If you find what you think may be a discrepancy or error, please check whether it is present in the original Worldwide file on NIH RePORT; if so, you may wish to contact the NIH to have it corrected in their records.
~ Robert Roskoski Jr. and Tristram G. Parslow