Dynamics of Buildable Ideas: Frontier and Settled Regimes in Scientific Progress

The point of the scientific enterprise is the synthesis of new knowledge. In discussions we always think about papers and citations and other artifacts, but I want to emphasize that what we generate are ideas, specifically ideas that others can build on.

A healthy system of science is able to efficiently disseminate validated knowledge to whoever needs it, in a way that they understand. Indeed, buildable ideas expand research opportunities, and opportunities generate future findings.

Today, most findings pass through a common journal-centered coordination system. As standardized work becomes more automatable, science may develop two interacting regimes with different coordination practices and resulting challenges.

In this essay, I want to step back and try to model and explain how I see things evolving. I'll offer some definitions, unpack it, and then we'll use this model as we keep unpacking various bottlenecks in future essays.

Modeling the Yield of Ideas

Let's first define T as the yield of buildable ideas through a yield equation. Note that this treats science in the way it's operating today, where findings pass through one dominant public coordination stack centered on journals.

Here's the model:

Where:

• T is the yield of buildable ideas per year.
• G is the number of candidate findings generated per year; this is different from buildable ideas per year, because generated findings need to be validated, disseminated, and understood by others. Note that this is not the number of papers or knowledge units, and within any regime, G depends on how many research opportunities are available, how much labor, compute, and capital is applied to them, and how effectively that regime's coordination practices turn those resources into research.
• Each p is the fraction of findings that survive each gate, i.e.
  • p_validated = what was reviewed / verified in time.
  • p_found = a measure of discovery efficiency (dissemination networks, information networks, etc.)
  • p_understood = our ability to contextualize what we discover within the broader field.
• Each stage i has its own capacity μ_i — that is, validation capacity affects p_validated, discovery capacity affects p_found, and so on.

Additionally, there are two properties to emphasize:

• Complementarity. Each of the p's multiply, based in part by Michael Kremer's O-Ring Structure (it was rooted in the Challenger disaster, where one cheap failed component created total loss)¹. In the model I laid out, the output requires all tasks to succeed. In other words, validated but undiscoverable findings are not useful; discoverable but not trustworthy findings are not useful; discoverable but not understood findings are, that's right, not useful.
• Capacity delays. Each function has a finite processing capacity μ. When findings persistently arrive faster than its capacity (μ), a backlog forms. Wait times grow and, in an unstable queue, can eventually diverge, reducing the fraction of findings processed (p) before they become stale².

These gates are analytically separable but, in practice, are dynamically coupled. Congestion or failure at one stage can alter the capacity and performance of others, turning a relatively straightforward yield equation into a dynamic system when modeled over time.

Obviously the full system of science is highly complex and adaptive. This is meant to be a high level model that offers a broader view of the system. The way I see it, we're quickly moving towards a dual-regime engine underpinning the generation and dissemination of ideas.

Resolving Coordination Mismatches Between Frontier and Settled Regimes

In my previous post I argued that we can't just automate away the entire validation layer because there are objective and subjective notions of trust which vary in part by how settled vs. nascent (or frontier) a field is.

Specifically, frontier science is a regime where paradigms, tools and interpretations themselves are contested, where validation needs human debate and social negotiation to build consensus.

Over time, things mature into settled science, with a regime of standardized infrastructure and agreed upon paradigms, where ground rules, metrics, instruments have relatively more consensus.

Note that settled does not mean "true", only that things are provisionally uncontested, which is in line with observations that as fields mature, scientists downstream don't necessarily re-examine the inner logic and theory of settled fields, effectively turning them into a sort of black-box³.

Frontier and settled are not necessarily synonyms for young and mature fields, but regimes of work; mature fields can contain frontier questions, and young fields can run highly settled assays.

This distinction is a little more clear if you test along these criteria:

• Codifiability -- can it be expressed as a protocol or test?
• Interpretive uncertainty -- is there a fast path to consensus based on existing epistemic norms?
• Measurement stability -- Similarly, are instruments and metrics accepted?
• Reproducibility -- Can we execute a specified procedure repeatedly with interpretable variation?
• Error visibility -- Is failure obvious and quickly detectable?

I believe that settled regimes are characterized by codifiable judgement, stable measurement, reproducible procedures, visibility into errors; frontier regimes lean the other direction.

As we've seen, journals remain the dominant public coordination bundle for communicating, validating, archiving, and crediting scientific work, but this infrastructure is beginning to split.

Just this week, the NSF announced the Genesis Project, through which they're deploying $400 million to build a national network of AI enabled programmable cloud laboratories, and part of this includes linking automated experimentation to standardized metadata, reusable protocols, AI-ready data, reproducibility, and new ways of publishing results [4].

Concretely, we can break it down like this:

There will likely be a role for humans in both regimes, but as autonomous labs rise, the shift will be in expanding human capacity at the frontier.

Now, consider what happens when the coordination practice does not match the work:

Applying frontier-style coordination to settled work makes every result pay for a narrative and bespoke judgement process. Standardized experiments are wrapped into papers, go through peer review, and held in queues when many checks could be continuous and procedural. The journal bundle imposes the cost structure of deliberation on work that's already been substantially codified.

Similarly, applying settled-style coordination to frontier work creates the opposite failure. Contested judgements are forced into premature metrics, provisional categories harden into ontologies, and disagreement among experts at the frontier becomes less legible.

Maybe this is all just a manifestation of my "why-can't-we-all-be-friends" energy, but this split helps me make sense of the tension between open science, desci, lab automation, and classic academic paper / journal camps: new modes are not necessarily a replacement of an old system, but a recognition that we've bundled too much into one, and as new regimes and contexts emerge, we need to shift how we coordinate within those contexts. It's both an unbundling and a rebundling to optimize the coordination technologies being used!

The Interface Between Regimes

I strongly suspect progress moving forward will depend on the interface between frontier and settled regimes. Indeed, the yield equation I started this post with is a bit simple, and assumes that findings pass through one broadly shared coordination stack.

Instead, we can imagine a separate T for each regime, governed by a different set of candidate findings and rates of validation, dissemination and contextualization in line with the coordination tech of that regime.

I'd go further and wager that there will be a new class of coordination bottlenecks emerging between regimes, specifically:

• Codification: Finding a way to codify or convert frontier discussions into standards, protocols, benchmarks, etc.
• Escalation: Identifying anomalies from high-volume settled work and turning them into meaningful frontier questions.

These processes expand the opportunity space of the other regime, i.e. codified frontier outputs create research directions while anomalies from settled regimes create new unresolved frontier questions.

This dual-regime engine has a cyclical property:

Today, the interface, i.e. codification and escalation, happens informally. For example, codification may happen when a new technique is treated as reliable by the broader field. Similarly, escalation might depend on a researcher noticing a strange result within their lab or elsewhere and having enough willingness and support to pursue it. The paper records some output of the process, but doesn't manage the transitions themselves.

New coordination infrastructure, if designed for this dual-regime engine, would ideally preserve the provenance and disagreement behind codified standards, monitoring settled execution for anomalies without treating every deviation as meaningful. It would make it easier to contest inherited assumptions and trace protocols back to the arguments that produced it.

Not only is information flow important within a regime, new infrastructure has to consider information flow across regimes.

Starving the Frontier

We can conceptually imagine how these two regimes will have different capacity constraints and rates of generation and idea diffusion within and across their networks. In the naive case, we might see dynamics like this:

Faster settled execution can generate more frontier work, provided there's enough capacity to recognize and escalate meaningful anomalies.

The second loop is a bit worrying:

If funding only follows visible throughput, the system can harvest its stock of codified knowledge while underinvesting in the work that renews it. Productivity might appear to increase in the near-term, but eventually the supply of meaningful new directions thins because the system has weakened its capacity to formulate them.

Rather than maximize automation or deliberation in the abstract, we should match coordination infrastructure to the epistemic regime of the work, while maintaining healthy transitions and an adequate balance between them.

What Follows

This lens has helped me reconcile several visions of scientific coordination that kept appearing to be incompatible: modular knowledge, autonomous labs, knowledge graphs, discourse graphs, the narrative, argumentation theory, and more, can and should all coexist.

While most discussions of coordination technology here focus on the knowledge representation side (graph structures, narratives, and such), I think another necessary question to unpack is around idea and collaborator diffusion itself.

For example, in many innovation contexts, progress comes through the efficient recombination of ideas, not necessarily the discovery of new ones [5]. And people historically found collaborators through relational networks, both physical and digital, from webinars, conferences, or friends suggesting a meet.

How do we efficiently diffuse ideas when there's an abundance of them? How can we allocate our attention when there is abundance? And how does that change, if at all, when people aren't the only ones who need a social discovery protocol? I'll be unpacking how multi-agent systems might offer new coordination practices for that next!

Cheers,
Ashish

P.S. As I wrote this, the OSTP announced their latest memo Science: A New Golden Age, which touches on many of the themes being discussed here, including a public validation infrastructure, scaling out autonomous labs, more investments in open science to figure out how to formalize these systems, and lots more!