🔬 research

U.S. research & development statistics from NSF NCSES: national R&D expenditure by performing sector and funding source (National Patterns), federal R&D obligations by agency and performer (Federal Funds), and Higher Education R&D (HERD) expenditure by institution, field, and funding source. R&D is the innovation input; joined to patents (the innovation output) on (state_fips, year) it yields patents-per-R&D-dollar productivity, and to geo/edu for place- and institution-level analysis.

6 datasets · 41 columns

nsf_national_rd · table

NSF NCSES National Patterns of R&D — the top-line macro view of U.S. research spending, from Table 1 (1953→latest). One row per year × decomposition. The all/all total row carries total R&D in current and constant (2017-chained) USD millions plus R&D-as-%-of-GDP; the per- performing-sector (Business / Federal government / Higher education / Other) and per-funding-source (Business / Federal / Other) rows carry R&D-as-%-of-GDP only — Table 1 publishes those breakdowns as shares of GDP, not dollars. Semantically joins econ.fred_indicators (R&D-share-of-GDP) and is the national context for nsf_herd_by_institution.

Column	Type	Null	Description
year	integer	no	Statistics year (partition column; emitted from the Table 1 row)
performing_sector	string	yes	Business / Federal government / Higher education / Other; "All performers" on source-decomposition and total rows
funding_source	string	yes	Business / Federal / Other; "All sources" on performer-decomposition and total rows
rd_type	string	yes	R&D type; Table 1 reports total R&D only ("Total")
rd_expenditure_usd_million	double	yes	Total R&D expenditure in current USD millions; populated only on the all/all total row (Table 1 gives per-sector breakdowns as % of GDP, not dollars)
rd_expenditure_constant_usd_million	double	yes	Total R&D expenditure in constant 2017 USD millions; populated only on the all/all total row
pct_of_gdp	double	yes	R&D as a share of GDP (percent) — total on the all/all row, and by performing sector / funding source on the breakdown rows

nsf_federal_rd_obligations · table

NSF NCSES Survey of Federal Funds for R&D — federal R&D obligations by funding agency × performing sector for the latest fiscal year in the publication (Table 7). Values in current USD millions. rd_field is "All fields" and rd_type is "Total R&D" (Table 7 does not break these out). The reference fiscal year is read from the table title. Shows which federal agencies fund research and where it flows; the agency name aligns with fiscal.usaspending agency budgets for the R&D slice.

Column	Type	Null	Description
year	integer	no	Federal fiscal year (partition column; read from the Table 7 title)
funding_agency	string	yes	Federal agency (or sub-agency) obligating funds; "All agencies" for the national total row
performing_sector	string	yes	Intramural — Federal agencies / Intramural — FFRDCs / Businesses / Higher education / Nonprofit organizations / State and local government / Non-U.S. performers
rd_field	string	yes	S&E field; "All fields" (Table 7 aggregates over field)
rd_type	string	yes	R&D type; "Total R&D" (Table 7 aggregates over type)
obligations_usd_million	double	yes	Federal R&D obligations in current USD millions ("*" in source rounds to 0.0)

nsf_rd_by_field · table

NSF NCSES Survey of Federal Funds for R&D — total federal obligations for RESEARCH (basic + applied, all funding agencies, all performers combined) broken out by detailed field of R&D, FYs 2016-2025. Values in current USD millions. Closes the all-agency/all-performer field-of-science gap that nsf_federal_rd_obligations (rd_field hard-coded to "All fields") and nsf_herd_by_institution (real rd_field column, but scoped to university-performed R&D only) each leave open on their own. Does not cover experimental development obligations by field — NCSES's own field taxonomy was historically collected for research only, and the field breakdown for development obligations is not published as its own crosscut in this table. field_level encodes the row's place in NCSES's own field hierarchy (read from the source XLSX cell's indentation, not inferred): 0 = grand total ("All fields"), 1 = a broad field of R&D, 2 = a detailed sub-field nested under the preceding level-1 row. Level-1 rows sum to the level-0 total; level-2 rows sum to their enclosing level-1 row. Pick one level to aggregate on — summing across levels double-counts, the same rollup-vs-leaf discipline as energy.eia_electricity_generation's sector_is_rollup/fuel_is_rollup flags. A field's taxonomy label can differ across the 2021 field-of-R&D revision (e.g. "Computer sciences and mathematics" pre-2021 vs. "Computer and information sciences" post-2021); both eras' labels are preserved as-is from the source. A field/year cell marked "NA" in the source is dropped — no row is emitted for it at all, not a row with a null value — because NCSES uses NA for two different reasons this table can't tell apart: a field that didn't exist under that year's taxonomy, or a "(preliminary)" most-recent year published only at the level-0/level-1 grain (NCSES withholds level-2 sub-field detail until the following year's edition finalizes it), which is why the newest year always has far fewer rows than the years before it. Re-running this table's full-refresh fetch after NCSES publishes the next edition picks up the finalized detail automatically — a low row count for only the single most recent year is expected, not a defect. Externally-compiled "federal R&D by field" series (e.g. AAAS, secondary policy trackers) report R&D total (research + development), not research alone, and will differ from this table by the development share of a given field — larger for fields with a substantial development component (life sciences via NIH, defense- adjacent engineering), smaller elsewhere. NCSES does not publish "development by detailed field" as its own crosscut — Table 83 breaks development out only by funding agency, never by field of science — so any "R&D by field" figure necessarily combines Table 91's research-by-field with an outside-NCSES allocation of development spending across fields under each compiler's own methodology. Both are correct at what they claim to measure; use this table for research-only field breakdowns, and NCSES agency- level tables (82, 83) or a third-party compiler for R&D total.

Column	Type	Null	Description
year	integer	no	Federal fiscal year
rd_field	string	no	Field of R&D label, as published (taxonomy differs pre/post the 2021 revision)
field_level	integer	no	0 = grand total, 1 = broad field, 2 = detailed sub-field of the preceding level-1 row
obligations_usd_million	double	yes	Federal research obligations in current USD millions ("*" in source rounds to 0.0; null where NA under that year's taxonomy)

nih_award_projects · table

NIH award-level grant microdata from RePORTER's POST /v2/projects/search API — one row per funded project per fiscal year, with award_amount and activity_code (mechanism, e.g. R01/F32), closing the gap that nsf_federal_rd_obligations/nsf_herd_by_institution leave open: those are agency/institution TOTALS, not individual-award size or count. Answers "is award size/count shifting toward fewer-bigger or more-smaller grants" directly. RePORTER caps every query at offset<=14999 (confirmed live — a higher offset returns a plain error, not an empty page), and a single fiscal year runs ~85,000 awards nationally, well over that. NihReporterAwardsProvider slices by agency_ic (NIH Institute/Center) to stay under the cap per (year, IC) query — the largest IC (NCI) was ~13,200 awards in a single year when this was checked live, versus the ~15,000 ceiling — and walks each slice's own pages to exhaustion, so the (year, IC) union is the true complete population for that year, not a sampled prefix. Not NCSES data (unlike the rest of this schema — see the schema-level header note); NIH itself, added because this schema is where funding- input data already lives. Scoped to a recent-years trend window rather than full history by deliberate choice — RePORTER's data goes back to FY1985. The window starts at FY2022 and follows the calendar forward; lower minYear on the year dimension to reach further back.

Column	Type	Null	Description
appl_id	long	no	RePORTER application id — PK (one per funded project per fiscal year; a renewal gets a new appl_id each year)
fiscal_year	integer	no	NIH fiscal year
project_num	string	yes	Full project number (grant number), stable across a project's renewal years
agency_ic	string	yes	Administering NIH Institute/Center abbreviation, from the response (not just the query filter)
activity_code	string	yes	NIH activity/mechanism code (e.g. R01, F32, U01)
award_amount	double	yes	Total award amount for this fiscal year (USD)
is_active	boolean	yes
contact_pi_name	string	yes
org_name	string	yes	Recipient institution name
org_city	string	yes
org_state	string	yes	FK to geo.state_ref.state_abbr (nullable — foreign recipients have no US state)
org_country	string	yes

nsf_herd_by_institution · table

NSF NCSES Higher Education R&D (HERD) survey — R&D expenditure by academic institution × S&E field × funding source × year, from the per-year institution microdata (herd{year}.csv, a tall EAV file). Projects the two R&D-expenditure question families: "Federal expenditures by field and agency" (funding_source=Federal, federal_agency=column) and "Nonfederal expenditures by field and source" (funding_source=column). 2008-2009 use NCSES's predecessor questionnaire (different question text, no per-field nonfederal-by-source cross-tab); NsfHerdTransformer maps both vintages into this same shape (see its class javadoc). Values in USD thousands. Carries the IPEDS UnitID crosswalk into edu (2010+ only — not published pre-2010) and state_fips into geo; the finest-grained NCSES table (institution grain). For the state×year rollup used in the patents-per-R&D-dollar productivity join, see research_herd_rd_by_state_year. For 2010+ rows, prefer joining ipeds_unitid directly to edu.ipeds_institutions — it is already on this row. For pre-2010 rows (no ipeds_unitid) or a cross-schema identity beyond edu, join ref.canonical_org_entity on nsf_herd_inst_id = inst_id instead of matching the institution name as text.

Column	Type	Null	Description
year	integer	no	Fiscal year of expenditure (partition column; from the HERD year field)
inst_id	string	yes	Unique campus identifier (1:1 with the institution, survives name changes); the institution key in the PK
ncses_inst_id	string	yes	NCSES parent/system id — coarser than inst_id (one id can span several campuses); for system-level rollups
institution	string	yes	Institution legal name (inst_name_long); descriptive
ipeds_unitid	int	yes	IPEDS UnitID — FK to edu.ipeds_institutions.unitid; null when NCSES did not match the institution
state_fips	string	yes	2-digit state FIPS derived from inst_state_code (FK to geo.state_ref)
county_fips	string	yes	County FIPS — not published in HERD microdata (always null)
control	string	yes	Institutional control (Public / Private) — not resolved from microdata (null)
rd_field	string	yes	S&E field (row label of the expenditure question) — a two-level rollup: 'All' is the institution grand total, then a broad field (e.g. 'Life sciences, all') and its detailed sub-fields (e.g. 'Life sciences, health sciences') both appear as separate rows. Summing across rd_field for one institution/funding_source without filtering to a single level multiplies the total; filter to rd_field='All' for an institution-level total.
funding_source	string	yes	Federal for the federal-agency question; otherwise the nonfederal source (Institution funds / State and local government / Business / Nonprofit organziations / All other sources), or 'Total nonfederal' — NCSES's own sum of those five nonfederal sources, which does NOT include Federal. The true grand total per institution/rd_field is 'Total nonfederal' plus the row where funding_source='Federal' AND federal_agency='Total'.
federal_agency	string	yes	Sponsoring federal agency (HHS/NIH, DOD, NSF, DOE, NASA, …); null unless funding_source = Federal
rd_expenditure_usd_thousand	double	yes	R&D expenditure in USD thousands

research_herd_rd_by_state_year · view

HERD R&D expenditure rolled up from nsf_herd_by_institution (institution grain) to state × year: total rd_expenditure_usd_thousand, the federal-funded subset (federal_rd_usd_thousand), and a distinct institution_count, for rows with a resolved state_fips. This is the R&D-spend half of the patents-per-R&D-dollar productivity ratio — join to patents.patent_assignees (grant counts by state_fips × grant_year) to compute innovation input ÷ output. Sums only each institution's two true totals — funding_source='Federal' AND federal_agency='Total', plus funding_source='Total nonfederal' — filtered to rd_field='All', since the base table's rd_field and funding_source columns are both rollup hierarchies (see nsf_herd_by_institution's own comments); summing every row would multiply the total by the field/source fan-out.

View — columns are resolved by the query engine at runtime.