🔬 research¶
U.S. research & development statistics from NSF NCSES: national R&D expenditure by performing sector and funding source (National Patterns), federal R&D obligations by agency and performer (Federal Funds), and Higher Education R&D (HERD) expenditure by institution, field, and funding source. R&D is the innovation input; joined to patents (the innovation output) on (state_fips, year) it yields patents-per-R&D-dollar productivity, and to geo/edu for place- and institution-level analysis.
9 datasets · 72 columns
nsf_national_rd · table¶
NSF NCSES National Patterns of R&D — the top-line macro view of U.S. research spending, from Table 1 (1953→latest). One row per year × decomposition. The all/all total row carries total R&D in current and constant (2017-chained) USD millions plus R&D-as-%-of-GDP; the per- performing-sector (Business / Federal government / Higher education / Other) and per-funding-source (Business / Federal / Other) rows carry R&D-as-%-of-GDP only — Table 1 publishes those breakdowns as shares of GDP, not dollars. Semantically joins econ.fred_indicators (R&D-share-of-GDP) and is the national context for nsf_herd_by_institution.
| Column | Type | Null | Description |
|---|---|---|---|
year |
integer | no | Statistics year (partition column; emitted from the Table 1 row) |
performing_ |
string | yes | Business / Federal government / Higher education / Other; "All performers" on source-decomposition and total rows |
funding_ |
string | yes | Business / Federal / Other; "All sources" on performer-decomposition and total rows |
rd_ |
string | yes | R&D type; Table 1 reports total R&D only ("Total") |
rd_ |
double | yes | Total R&D expenditure in current USD millions; populated only on the all/all total row (Table 1 gives per-sector breakdowns as % of GDP, not dollars) |
rd_ |
double | yes | Total R&D expenditure in constant 2017 USD millions; populated only on the all/all total row |
pct_ |
double | yes | R&D as a share of GDP (percent) — total on the all/all row, and by performing sector / funding source on the breakdown rows |
nsf_federal_rd_obligations · table¶
NSF NCSES Survey of Federal Funds for R&D — federal R&D obligations by funding agency × performing sector for the latest fiscal year in the publication (Table 7). Values in current USD millions. rd_field is "All fields" and rd_type is "Total R&D" (Table 7 does not break these out). The reference fiscal year is read from the table title. Shows which federal agencies fund research and where it flows; the agency name aligns with fiscal.usaspending agency budgets for the R&D slice.
| Column | Type | Null | Description |
|---|---|---|---|
year |
integer | no | Federal fiscal year (partition column; read from the Table 7 title) |
funding_ |
string | yes | Federal agency (or sub-agency) obligating funds; "All agencies" for the national total row |
performing_ |
string | yes | Intramural — Federal agencies / Intramural — FFRDCs / Businesses / Higher education / Nonprofit organizations / State and local government / Non-U.S. performers |
rd_ |
string | yes | S&E field; "All fields" (Table 7 aggregates over field) |
rd_ |
string | yes | R&D type; "Total R&D" (Table 7 aggregates over type) |
obligations_ |
double | yes | Federal R&D obligations in current USD millions ("*" in source rounds to 0.0) |
nsf_federal_rd_by_state · table¶
NSF NCSES Survey of Federal Funds for R&D — federal R&D obligations by state or location x funding agency for the latest fiscal year in the publication (Table 60). Values in current USD thousands (note: Table 7's nsf_federal_rd_obligations reports USD millions — different unit, same survey). funding_agency names are normalized to the full department/agency name nsf_federal_rd_obligations already uses (e.g. Table 60's "DOD" header becomes "Department of Defense") so the two tables' agency values line up for a join. Answers "which places get the most federal (and defense) R&D money, by agency" at state grain — the DOD column is this table's state-level view of defense R&D spending; NCSES's own DOD-specific RDT&E table (Table 63, by performer and RDT&E sub-category) is a narrower, DOD-only cut not carried here. Table 60's per-location "Total" column (sum across the twelve agencies) is not emitted as its own row — same rollup-omission convention as nsf_federal_rd_obligations dropping Table 7's per-agency total column: summing the twelve funding_agency rows for a state recovers it. state_fips is populated only for the 50 states + DC (FK to geo.state_ref); it is null for "All locations" (the national total row, this table's equivalent of nsf_federal_rd_obligations' "All agencies" row), "Outlying areas" (a territories subtotal), the five individual territories, "Other areas", "Offices abroad", and "Undistributed" — none of those resolve to a real state FIPS code, the same best-effort-FK limitation nsf_herd_by_institution.state_fips already documents for territory institutions. "Undistributed" carries a value for only DOD and the grand total; every other agency column is NA there and is not emitted as a row. Only the publication's latest fiscal year is available at this state x agency grain — like nsf_federal_rd_obligations, each past edition only ever published its own latest FY in this table, so there is no bulk historical backfill of state x agency detail; history accumulates one year per edition going forward. NCSES's own Table 97 (state x year, FYs 2016-24) covers the historical trend at state grain but carries only the all-agency total, no agency split — considered and rejected as a separate table here since it would add a year dimension with no agency detail to join against this table's agency rows.
| Column | Type | Null | Description |
|---|---|---|---|
year |
integer | no | Federal fiscal year (partition column; read from the Table 60 title) |
state_ |
string | no | State or location label as published (state name, "All locations", a territory, or a non-state aggregate row — see table comment) |
state_ |
string | yes | 2-digit state FIPS (FK to geo.state_ref); null for the national total, territories, and other non-state aggregate rows |
funding_ |
string | no | Federal department/agency obligating funds, normalized to match nsf_federal_rd_obligations.funding_agency |
obligations_ |
double | yes | Federal R&D obligations in current USD thousands ("*" in source rounds to 0.0) |
nsf_rd_by_field · table¶
NSF NCSES Survey of Federal Funds for R&D — total federal obligations for RESEARCH (basic + applied, all funding agencies, all performers combined) broken out by detailed field of R&D, FYs 1993-2025 (FYs 1993-2003 and 2004-14 from NCSES's own long-run historical tables in publication nsf14316, FY2015 from NCSES's Data Explorer-era table 123 filtered to that single year, FYs 2016-2025 from the current edition — a contiguous FY1993-2025 run, closing the single-year gap tracked in govdata-ops#257/#145/#325). Values in current USD millions. Closes the all-agency/all-performer field-of-science gap that nsf_federal_rd_obligations (rd_field hard-coded to "All fields") and nsf_herd_by_institution (real rd_field column, but scoped to university-performed R&D only) each leave open on their own. Does not cover experimental development obligations by field — NCSES's own field taxonomy was historically collected for research only, and the field breakdown for development obligations is not published as its own crosscut in this table. field_level encodes the row's place in NCSES's own field hierarchy: 0 = grand total ("All fields"), 1 = a broad field of R&D, 2 = a detailed sub-field nested under the preceding level-1 row. Level-1 rows sum to the level-0 total; level-2 rows sum to their enclosing level-1 row. Pick one level to aggregate on — summing across levels double-counts, the same rollup-vs-leaf discipline as energy.eia_electricity_generation's sector_is_rollup/fuel_is_rollup flags. The FYs 2016-2025 source encodes this via each field-name cell's own style indent; the FYs 1993-2014 historical tables (a different NCSES publication era) instead encode it via 4 leading non-breaking-space characters per level baked into the field-name string itself — confirmed live 2026-09-13 against table 134 (FYs 2004-14): 0/4/8 leading NBSPs map to level 0/1/2 exactly. The FY2015 source (NCSES's intermediate "Data Explorer" era) uses a real style indent like the current edition, but at double the step size (2/4 for level 1/2, not 1/2) — confirmed live 2026-09-17. NsfRdByFieldTransformer normalizes the style indent to the file's own smallest positive indent before dividing, and falls back to counting NBSPs only when the style carries none, so all three vintages populate field_level identically. A field's taxonomy label can differ across the 2021 field-of-R&D revision (e.g. "Computer sciences and mathematics" pre-2021 vs. "Computer and information sciences" post-2021); both eras' labels are preserved as-is from the source. A field/year cell marked "NA" in the source is dropped — no row is emitted for it at all, not a row with a null value — because NCSES uses NA for two different reasons this table can't tell apart: a field that didn't exist under that year's taxonomy, or a "(preliminary)" most-recent year published only at the level-0/level-1 grain (NCSES withholds level-2 sub-field detail until the following year's edition finalizes it), which is why the newest year always has far fewer rows than the years before it. Re-running this table's full-refresh fetch after NCSES publishes the next edition picks up the finalized detail automatically — a low row count for only the single most recent year is expected, not a defect. The same mechanism explains why FYs 2013-14 carry only 9 rows apiece (level 0-1 only) against 41 for every other year in the FYs 1993-2014 range — confirmed live 2026-09-13, they were the newest, still-preliminary years as of the nsf14316 (FYs 2012-14) edition this table's historical rows are sourced from, and no later edition's equivalent long-run table has been located yet to backfill their level-2 detail (see govdata-ops#257). FY2015 itself is fully finalized (41 rows, full level-2 detail) — it was not the preliminary year of the edition it is sourced from (see below). The FYs 1993-2014 historical vintages are sourced from NCSES publication nsf14316 (FYs 2012-14 edition) via its own long-run tables 133 and 134. The original nsf.gov/statistics/nsf14316/ URLs are dead — NSF now serves an "Archive Goodbye" redirect for that whole publication era with no static-file replacement — so these two are fetched from the Wayback Machine's mirror instead (re-verify-if-404s risk, same pattern already accepted for other individually-hosted sources in this codebase; a Wayback snapshot URL is otherwise permanent). FY2015 is sourced from a third NCSES vintage: table 123 of the ncsesdata.nsf.gov "Data Explorer" era (edition fedfunds/2017, itself spanning FYs 2009-18 in one file — confirmed live 2026-09-17). Only FY2015 is kept from it; FYs 2009-14 and 2016-18 in that same file are discarded because the hist_2004_2014 and current vintages already source those years, and keeping them too would emit duplicate (year, rd_field) rows. The original ncsesdata.nsf.gov URL is dead (404, no redirect) — fetched via its Wayback Machine mirror instead, same re-verify-if-404s risk as the two nsf14316 vintages above. Externally-compiled "federal R&D by field" series (e.g. AAAS, secondary policy trackers) report R&D total (research + development), not research alone, and will differ from this table by the development share of a given field — larger for fields with a substantial development component (life sciences via NIH, defense- adjacent engineering), smaller elsewhere. NCSES does not publish "development by detailed field" as its own crosscut — Table 83 breaks development out only by funding agency, never by field of science — so any "R&D by field" figure necessarily combines Table 91's research-by-field with an outside-NCSES allocation of development spending across fields under each compiler's own methodology. Both are correct at what they claim to measure; use this table for research-only field breakdowns, and NCSES agency- level tables (82, 83) or a third-party compiler for R&D total.
| Column | Type | Null | Description |
|---|---|---|---|
year |
integer | no | Federal fiscal year |
rd_ |
string | no | Field of R&D label, as published (taxonomy differs pre/post the 2021 revision) |
field_ |
integer | no | 0 = grand total, 1 = broad field, 2 = detailed sub-field of the preceding level-1 row |
obligations_ |
double | yes | Federal research obligations in current USD millions ("*" in source rounds to 0.0; null where NA under that year's taxonomy) |
nih_award_projects · table¶
NIH award-level grant microdata from RePORTER's POST /v2/projects/search API — one row per funded project per fiscal year, with award_amount and activity_code (mechanism, e.g. R01/F32), closing the gap that nsf_federal_rd_obligations/nsf_herd_by_institution leave open: those are agency/institution TOTALS, not individual-award size or count. Answers "is award size/count shifting toward fewer-bigger or more-smaller grants" directly. RePORTER caps every query at offset<=14999 (confirmed live — a higher offset returns a plain error, not an empty page), and a single fiscal year runs ~85,000 awards nationally, well over that. NihReporterAwardsProvider slices by agency_ic (NIH Institute/Center) to stay under the cap per (year, IC) query — the largest IC (NCI) was ~13,200 awards in a single year when this was checked live, versus the ~15,000 ceiling — and walks each slice's own pages to exhaustion, so the (year, IC) union is the true complete population for that year, not a sampled prefix. Not NCSES data (unlike the rest of this schema — see the schema-level header note); NIH itself, added because this schema is where funding- input data already lives. Coverage starts at FY2010 and follows the calendar forward, so an award-size trend spans more than a decade of NIH funding. RePORTER's data goes back to FY1985; lower minYear on the year dimension to reach further back. NSF's counterpart at the same grain is nsf_award_projects; the two share fiscal_year and a recipient state, and together cover the bulk of federal research grants.
| Column | Type | Null | Description |
|---|---|---|---|
appl_ |
long | no | RePORTER application id — PK (one per funded project per fiscal year; a renewal gets a new appl_id each year) |
fiscal_ |
integer | no | NIH fiscal year |
project_ |
string | yes | Full project number (grant number), stable across a project's renewal years |
agency_ |
string | yes | Administering NIH Institute/Center abbreviation, from the response (not just the query filter) |
activity_ |
string | yes | NIH activity/mechanism code (e.g. R01, F32, U01) |
award_ |
double | yes | Total award amount for this fiscal year (USD) |
is_ |
boolean | yes | |
contact_ |
string | yes | |
org_ |
string | yes | Recipient institution name |
org_ |
string | yes | |
org_ |
string | yes | FK to geo.state_ref.state_abbr (nullable — foreign recipients have no US state) |
org_ |
string | yes |
nsf_award_projects · table¶
NSF award-level grant microdata from the NSF Award Search API (api.nsf.gov/services/v1/awards.json) — one row per award, dated by the award's initial award date and assigned to the federal fiscal year (October-September) containing it. Carries funds_obligated_amt (dollars obligated to date) and estimated_total_amt (the award's full expected amount over its life), the recipient (awardee_name, awardee_uei, city, state, congressional district), the NSF directorate and division that made the award, the funding program, and transaction_type (Standard Grant, Continuing Grant, Cooperative Agreement, fellowship ...). Answers "are federal research awards getting bigger or more numerous" for NSF at award grain, where nsf_federal_rd_obligations and nsf_herd_by_institution carry only agency and institution TOTALS; nih_award_projects is the NIH counterpart at the same grain. Both amounts are as of the latest refresh, not as of the award date: a multi-year continuing grant's funds_obligated_amt grows as later years are funded, so a recent fiscal year's obligated total is understated relative to what the same cohort will show once its increments are made; estimated_total_amt is the stable size measure for an award-size distribution. Dollars are nominal (not inflation-adjusted). Not every NSF row is a research grant: education, infrastructure, fellowships and cooperative agreements are all here, so filter on transaction_type or directorate when the question is specifically about research project grants. NSF only — other agencies' awards are not in this table. The API truncates any query at 10,000 results, so the provider fetches one calendar month of award dates at a time (the busiest month on record is ~3,600 awards) and rejects a month that reports the cap instead of loading a truncated prefix. Coverage starts at FY2010 and follows the calendar forward; lower minYear on the year dimension to reach further back.
| Column | Type | Null | Description |
|---|---|---|---|
award_ |
string | no | NSF award number (7-digit id) — PK; one row per award |
fiscal_ |
integer | no | Federal fiscal year (October-September) containing award_date |
award_ |
date | yes | Date of the award's initial funding action; later amendments do not move it |
start_ |
date | yes | Start of the award's period of performance (can be after award_date) |
exp_ |
date | yes | End of the award's period of performance as currently amended |
title |
string | yes | Award title |
transaction_ |
string | yes | Award instrument, e.g. Standard Grant, Continuing Grant, Cooperative Agreement, Fellowship |
fund_ |
string | yes | NSF funding program that made the award |
cfda_ |
string | yes | Assistance Listing (CFDA) number, e.g. 47.076 (Education and Human Resources) |
directorate |
string | yes | NSF directorate abbreviation (e.g. EDU, ENG, MPS, BIO, GEO, CSE) |
division |
string | yes | NSF division abbreviation within the directorate (e.g. DGE, CBET) |
funds_ |
double | yes | Dollars obligated to the award to date (USD) — grows as continuing-grant increments are funded |
estimated_ |
double | yes | Expected total amount over the award's life (USD), including increments not yet obligated |
awardee_ |
string | yes | Recipient institution name |
awardee_ |
string | yes | Recipient Unique Entity Identifier (SAM.gov UEI) |
awardee_ |
string | yes | |
awardee_ |
string | yes | FK to geo.state_ref.state_abbr (nullable — foreign recipients have no US state) |
awardee_ |
string | yes | |
awardee_ |
string | yes | Recipient's congressional district code (state abbreviation + district, e.g. NH01) |
nih_publications · table¶
Long-format (appl_id, pmid) publication links from RePORTER's POST /v2/publications/search API — one row per funded-project/publication pair per fiscal year, joinable back to nih_award_projects.appl_id. Closes the funding-to-output gap: nih_award_projects, nsf_federal_rd_obligations, and nsf_herd_by_institution all carry funding totals or award-level funding size, but none carries an output measure to compute a genuine agency-funding-volatility-vs-research-output relationship. RePORTER's publications endpoint takes a batch of appl_ids as its criteria rather than a year/agency filter directly, so this table's provider first re-walks /v2/projects/search for the same (year, agency_ic) slice nih_award_projects uses (fetching only appl_id, not the full award record) to get that slice's appl_id population, then batches those appl_ids (500 per call, RePORTER's own page-size convention) into /v2/publications/search calls. Cross-linked appl_id filtering: the publications response's applid field is not necessarily one of the appl_ids just queried — confirmed live, a batch of FY2023/FIC appl_ids surfaced a linked FY2025/NHGRI appl_id in the same response (a renewal-year continuation of the same underlying project, credited on the same publication). Rows are kept only when applid is a member of the batch queried, so every row's fiscal_year/slice is guaranteed accurate; the cross-linked appl_id is captured correctly when its own (year, agency_ic) slice is walked instead. Two different years: the /v2/publications/search response carries no year field, so fiscal_year is the funding fiscal year of the (year, agency_ic) slice being walked, i.e. when the underlying grant was active. The paper's own year is pub_year, looked up from NIH iCite (/api/pubs?pmids=, field year); it is null for a pmid iCite does not index. A paper appears against every fiscal_year its funding project spanned, and RePORTER credits a renewed project's whole publication history to each of its appl_ids — so counting links by fiscal_year measures accumulated portfolio output, not output produced that year. For a publication-output series count DISTINCT pmid grouped by pub_year (and by agency_ic via nih_award_projects.appl_id when institute attribution is needed). Scope: NIH only. NSF's publication linkage is an inline field (publicationResearch) on its own Award API record rather than a separate paginated endpoint and needs its own extraction pass — tracked as follow-on work, not this table. Patent linkage (NIH-award-to-patent) has no public RePORTER/iCite endpoint at all; USPTO PatentsView plus a government-interest-clause parse would be a genuinely separate, harder build. Scoped to a recent-years trend window (FY2022+) rather than RePORTER's full FY1985+ history — lower minYear on the year dimension to reach further back.
| Column | Type | Null | Description |
|---|---|---|---|
appl_ |
long | no | RePORTER application id — FK to nih_award_projects.appl_id |
fiscal_ |
integer | no | Funding fiscal year of the (year, agency_ic) slice this link was collected under — NOT the publication's own year (see pub_year) |
core_ |
string | yes | Full project number, stable across a project's renewal years |
pmid |
long | yes | PubMed ID of the linked publication; null only on a record_status=invalid_source_record row |
pub_ |
integer | yes | Calendar year the paper was published, from NIH iCite; null when iCite does not index the pmid |
record_ |
string | no | valid for a normal appl_id/pmid link; invalid_source_record when RePORTER's publications API kept returning HTTP 5xx for this appl_id alone (that row carries pmid NULL and the appl_id's publications are unknown, not zero) |
record_ |
string | yes | Why the row is invalid, e.g. the HTTP status and attempt count; null when record_status is valid |
nsf_herd_by_institution · table¶
NSF NCSES Higher Education R&D (HERD) survey — R&D expenditure by academic institution × S&E field × funding source × year, from the per-year institution microdata (herd{year}.csv, a tall EAV file). Projects the two R&D-expenditure question families: "Federal expenditures by field and agency" (funding_source=Federal, federal_agency=column) and "Nonfederal expenditures by field and source" (funding_source=column). 2008-2009 use NCSES's predecessor questionnaire (different question text, no per-field nonfederal-by-source cross-tab); NsfHerdTransformer maps both vintages into this same shape (see its class javadoc). Values in USD thousands. Carries the IPEDS UnitID crosswalk into edu (2010+ only — not published pre-2010) and state_fips into geo; the finest-grained NCSES table (institution grain). For the state×year rollup used in the patents-per-R&D-dollar productivity join, see research_herd_rd_by_state_year. For 2010+ rows, prefer joining ipeds_unitid directly to edu.ipeds_institutions — it is already on this row. For pre-2010 rows (no ipeds_unitid) or a cross-schema identity beyond edu, join ref.canonical_org_entity on nsf_herd_inst_id = inst_id instead of matching the institution name as text.
| Column | Type | Null | Description |
|---|---|---|---|
year |
integer | no | Fiscal year of expenditure (partition column; from the HERD year field) |
inst_ |
string | yes | Unique campus identifier (1:1 with the institution, survives name changes); the institution key in the PK |
ncses_ |
string | yes | NCSES parent/system id — coarser than inst_id (one id can span several campuses); for system-level rollups |
institution |
string | yes | Institution legal name (inst_name_long); descriptive |
ipeds_ |
int | yes | IPEDS UnitID — FK to edu.ipeds_institutions.unitid; null when NCSES did not match the institution |
state_ |
string | yes | 2-digit state FIPS derived from inst_state_code (FK to geo.state_ref) |
county_ |
string | yes | County FIPS — not published in HERD microdata (always null) |
control |
string | yes | Institutional control (Public / Private) — not resolved from microdata (null) |
rd_ |
string | yes | S&E field (row label of the expenditure question) — a two-level rollup: 'All' is the institution grand total, then a broad field (e.g. 'Life sciences, all') and its detailed sub-fields (e.g. 'Life sciences, health sciences') both appear as separate rows. Summing across rd_field for one institution/funding_source without filtering to a single level multiplies the total; filter to rd_field='All' for an institution-level total. |
funding_ |
string | yes | Federal for the federal-agency question; otherwise the nonfederal source (Institution funds / State and local government / Business / Nonprofit organziations / All other sources), or 'Total nonfederal' — NCSES's own sum of those five nonfederal sources, which does NOT include Federal. The true grand total per institution/rd_field is 'Total nonfederal' plus the row where funding_source='Federal' AND federal_agency='Total'. |
federal_ |
string | yes | Sponsoring federal agency (HHS/NIH, DOD, NSF, DOE, NASA, …); null unless funding_source = Federal |
rd_ |
double | yes | R&D expenditure in USD thousands |
research_herd_rd_by_state_year · view¶
HERD R&D expenditure rolled up from nsf_herd_by_institution (institution grain) to state × year: total rd_expenditure_usd_thousand, the federal-funded subset (federal_rd_usd_thousand), and a distinct institution_count, for rows with a resolved state_fips. This is the R&D-spend half of the patents-per-R&D-dollar productivity ratio — join to patents.patent_assignees (grant counts by state_fips × grant_year) to compute innovation input ÷ output. Sums only each institution's two true totals — funding_source='Federal' AND federal_agency='Total', plus funding_source='Total nonfederal' — filtered to rd_field='All', since the base table's rd_field and funding_source columns are both rollup hierarchies (see nsf_herd_by_institution's own comments); summing every row would multiply the total by the field/source fan-out.
View — columns are resolved by the query engine at runtime.