Why AI’s Scientific Capabilities Should First Be Directed Toward Solving Alignment
Publius
October 17, 2026
Abstract
Artificial intelligence is beginning to contribute to frontier mathematics and scientific research. More importantly, automated AI researchers have demonstrated measurable improvements on several alignment problems. These developments create an opportunity to use existing AI capabilities to make subsequent systems safer.
This paper develops a principle of safety-gated capability development. When a proposed capability advance introduces substantial catastrophic risk, developers should compare immediate advancement with the alternative of first undertaking available safety research. The case for delay rests on three considerations: the potential asymmetry of irreversible harms, the value of obtaining information before consequential decisions, and the possibility that AI-assisted research has reduced the cost of safety progress. None establishes that delay is always preferable. Together, however, they can create a rebuttable presumption in favor of safety-first development.
The paper formulates the relevant economic comparison, proposes a three-part test for capability advances claimed to be necessary for safety research, and outlines an independently evaluated program of automated alignment research. It also addresses competitive pressures, unreliable AI-generated research, and the possibility that greater capabilities are themselves needed for safety. The resulting proposal is neither a blanket moratorium nor unrestricted acceleration. It is a framework for determining when dangerous capability advances should wait until available opportunities for risk reduction have been pursued.
1. AI as a Scientific Researcher Artificial intelligence is becoming capable of contributing to research on the very problems its development creates.
In August 2026, Yueh-Han Chen, Jiaxin Wen, and Jan Hendrik Kirchner reported that automated alignment researchers developed interventions reducing ten measurable categories of alignment failure, including deception, sycophancy, jailbreaks, and reward hacking.[1]
The strongest automated methods performed well beyond their targeted evaluations, including on certain held-out tests and larger models. In a controlled comparison, they also outperformed proposals produced by experienced human safety researchers working under specified time constraints.
The limitations are important. The failures were comparatively well characterized, and the automated researchers sometimes attempted to exploit their evaluations. Success on these tasks does not establish that advanced systems can be reliably aligned under arbitrary circumstances.
Nevertheless, the experiments demonstrate something significant: AI systems can already contribute directly to improving measurable aspects of AI safety.
A broader development is visible in mathematics. In August 2026, OpenAI reported ten advances in mathematics and theoretical computer science, accompanied by machine-checkable formalizations in Lean 4.[2] In September, OpenAI announced a proposed resolution of the Navier–Stokes Millennium Prize Problem, which remained subject to independent mathematical scrutiny and the Clay Mathematics Institute’s established review requirements.[3]
These achievements should not be conflated. Machine verification establishes deductions relative to specified premises; independent mathematical acceptance is a further matter. Nor does success in mathematics automatically imply success in alignment.
The significance is that existing AI systems are becoming increasingly useful scientific instruments.
If they can contribute to difficult mathematical problems, and have already begun to improve measurable safety outcomes, a question of research priority arises: should we use these capabilities to improve AI safety before undertaking further advances that introduce additional catastrophic risk?
2. The Problem of AI Safety Advanced AI offers substantial potential benefits in medicine, science, education, engineering, and economic productivity.
The same capabilities may create serious dangers. Increasingly autonomous systems may pursue complex objectives, operate external tools, and undertake actions that their designers cannot reliably anticipate or control.
The alignment problem is to ensure that AI systems pursue acceptable objectives and remain subject to meaningful human oversight.
The familiar paperclip-maximizer example illustrates the distinction between intelligence and alignment. A highly capable system instructed to maximize paperclip
production might pursue that objective at the expense of everything else humans value. Its intelligence would make it better at achieving its objective without making the objective itself desirable.[4]
Research has identified related problems, including specification gaming, reward hacking, goal misgeneralization, instrumental power-seeking, and the limitations of oversight.[5]
These problems are not exclusively mathematical. They involve empirical machine learning, computer security, institutional design, and philosophical questions about rational agency and value.
Nevertheless, many have components amenable to formalization, experimentation, adversarial criticism, and independent verification.
They are therefore promising targets for AI-assisted research.
3. Prior Work and the Present Contribution The proposal to prioritize safety over dangerous technological development has important precedents.
Nick Bostrom’s work on differential technological development argues for accelerating protective technologies relative to technologies that increase existential risk.[6]
OpenAI’s 2023 Superalignment initiative proposed developing an automated alignment researcher to help align systems more capable than humans.[7]
Frontier developers have also introduced safety frameworks intended to condition particular capability advances or deployments on evaluations and safeguards.[8]
The present paper builds on these approaches in four ways.
First, it formulates the sequencing problem as a comparison between alternative development paths rather than an unconditional demand for delay.
Second, it identifies the circumstances in which irreversibility and the value of information create a rebuttable presumption in favor of safety-first development.
Third, it proposes a test for distinguishing genuinely safety-enabling capability advances from general capability increases merely described as useful for safety.
Fourth, it argues that emerging evidence of automated alignment research makes the relative costs and benefits of safety-first development an urgent empirical question.
The central contribution is therefore not the proposition that safety is valuable. It is a framework for deciding when additional safety work should precede additional
capability development.
4. The Principle of Safety-Gated Development The argument begins with five propositions.
1. Advanced AI poses credible risks of catastrophic and potentially irreversible harm.
2. Some capability advances may materially increase those risks.
3. Existing AI systems possess significant scientific research capabilities, including capabilities relevant to safety.
4. Additional safety research may reduce the risks associated with subsequent capability advances.
5. Where the expected benefits of undertaking that research first exceed the expected net benefits of proceeding immediately, the safety-first strategy is preferable.
The fifth proposition is a general decision rule. The substantive argument concerns why the condition may hold in important cases.
Three considerations matter.
4.1. Irreversibility and asymmetric losses A catastrophic outcome may be impossible to reverse. A decision to delay development, by contrast, may preserve the option of proceeding later.
This asymmetry can favor caution.
But irreversibility alone does not establish that delay is preferable. Delayed technological benefits can also involve irreversible losses: lives not saved by medical advances, opportunities permanently missed, or security risks arising from competing developers.
The relevant question is whether one strategy exposes humanity to greater expected irreversible loss than the other.
A presumption in favor of safety-first development is strongest where a proposed advance creates a credible possibility of uniquely severe, irreversible harm, while the safety-first alternative preserves substantial future opportunities.
This is an application of the broader economic literature on irreversible decisions and the value of retaining options.[9]
It is not an unconditional prohibition on risky innovation.
4.2. The value of information Additional research can improve decisions even when it does not directly reduce risk.
Suppose researchers can investigate whether a proposed capability advance is likely to produce dangerous behavior before deploying it. The information obtained may allow developers to modify the system, introduce safeguards, or abandon the advance.
Such information has value because it changes what can rationally be decided later.
However, proceeding with development can also generate information. Controlled deployment, for example, may reveal failure modes that cannot be observed in advance.
The correct comparison is therefore between the information generated by the two strategies, together with the risks and costs of obtaining it.
The value of waiting is greatest when informative research can be conducted before an irreversible commitment, and when proceeding immediately would expose society to substantial harm before that information becomes available.
4.3. The changing cost of safety research Historically, delaying capability development to pursue safety research meant relying on a limited pool of human researchers.
Automated research may alter this constraint.
If existing AI systems can generate, criticize, test, and verify safety hypotheses at relatively low marginal cost, meaningful safety progress may become possible without first developing substantially more powerful general-purpose systems.
The August 2026 automated alignment experiments provide preliminary evidence for this possibility.[1]
They do not yet establish that the cost of reducing catastrophic risk has substantially fallen. Improvements on bounded benchmarks may not generalize to the failures most relevant to superintelligence.
The appropriate conclusion is narrower but important: AI-assisted safety research is now sufficiently promising that its effectiveness and costs should be measured directly before assuming further general capability development is the best route to safety.
4.4. A rebuttable presumption Together, these considerations support the following principle:
When a capability advance presents substantial additional irreversible risk, and feasible prior research offers a credible opportunity to reduce that risk or improve the decision, developers should presume in favor of undertaking that research first unless
the expected benefits of immediate advancement outweigh those advantages.
The presumption can be defeated.
Immediate advancement may be preferable if delay creates greater catastrophic risk, if the proposed safety research has little prospect of success, if its own risks are excessive, or if the capability advance is necessary for an especially valuable safety intervention.
The principle therefore establishes a burden of justification, not an absolute rule.
5. Distinguishing Safety Progress from Capability Progress The argument requires distinguishing three kinds of technological development.
General capability progress increases a system’s effectiveness across a broad range of tasks, potentially including dangerous ones.
Safety progress improves alignment, interpretability, verification, oversight, and control.
Safety-enabling capability progress increases the specific capabilities required to achieve safety advances.
The third category creates an obvious difficulty. Almost any increase in AI capability might eventually assist safety research.
If this possibility were sufficient to justify further development, the safety-first principle would impose no meaningful constraint.
An advance should therefore qualify as safety-enabling only when it satisfies three conditions.
Specific use. The developer identifies the safety research for which the capability is needed before undertaking the advance.
No adequate lower-risk alternative. The research cannot be accomplished comparably well using existing systems or a narrower, less dangerous capability.
Contained access. The new capability is initially restricted to the relevant research, under safeguards appropriate to its risks, until its own safety requirements have been satisfied.
These conditions do not establish that an advance is safe. They establish that its claimed contribution to safety is sufficiently specific to be evaluated.
A capability advance that fails the test may still be justified by other benefits. But it should not receive special treatment merely by being described as safety-enabling.
6. The Economics of Sequencing Consider two development strategies.
Under the first, developers advance capabilities immediately, obtaining earlier benefits but potentially increasing catastrophic risk.
Under the second, developers undertake additional safety research before making the relevant advance.
Let:
• p_now be the probability of catastrophe under immediate development;
• p_safe be the probability under the safety-first strategy;
• L be the loss associated with catastrophe;
• B_now and B_safe be the expected net non-catastrophic benefits of the two strategies.
In a simplified risk-neutral model, safety-first development is preferable when:
L (p_now − p_safe) > B_now − B_safe
The benefit terms include research costs, opportunity costs, and the costs of delay. The probability terms incorporate any additional risks created by safety research itself.
This equation is a simplified representation of a complex decision. It assumes a common evaluation horizon and a common catastrophic loss magnitude. A more complete model would allow the strategies to affect the timing, severity, and probability of different harmful outcomes.
It would also account for uncertainty about the relevant probabilities rather than treating them as known.
Nevertheless, the simplified model makes the central comparison clear.
The question is not whether advancing AI produces benefits. It is whether those benefits justify proceeding immediately rather than first reducing the associated risks.
For illustration, suppose catastrophe would impose $100 trillion in economic losses.
A safety-first strategy that reduced the probability of catastrophe by 0.01 percentage points would avoid $10 billion in expected losses.
This is not an empirical estimate. It illustrates why small differences in catastrophic risk can have substantial economic significance.
6.1. Existing economic research Charles I. Jones develops an economic model of optimal expenditure on AI existential-risk mitigation. Under many of his modeled scenarios, substantial annual spending—including amounts exceeding 1% of GDP—can be justified.[10]
These results depend on assumptions about risk, mitigation effectiveness, and economic costs.
They do not establish the optimal budget for automated alignment research or prove that a particular delay is beneficial.
Nevertheless, they demonstrate that exceptionally large safety expenditures can be economically rational.
A survey of 2,778 AI researchers also documents substantial concern about severe AI outcomes, alongside considerable disagreement.[11]
These forecasts are elicited judgments, not established frequencies. They should inform sensitivity analysis rather than be treated as precise measurements.
6.2. Determining the level of investment Let x represent spending on safety research, p(x) the resulting probability of catastrophe, and O(x) the opportunity cost of that spending beyond its direct expenditure.
A simplified total social-cost function is:
T(x) = p(x)L + x + O(x)
An optimal expenditure minimizes this function. At a differentiable interior optimum:
−Lp′(x) = 1 + O′(x)
Additional investment is justified when its expected marginal reduction in catastrophic harm exceeds its marginal social cost.
In practice, the relevant quantities are uncertain and may vary discontinuously.
A sensible research program should therefore use staged funding, independent evaluation, and multiple scenarios rather than commit to a predetermined expenditure.
Investments of billions of dollars or more may be justified. The amount should follow evidence concerning actual risk reduction.
7. A Coordinated AI-Assisted Safety Research Program A major research initiative should employ existing frontier AI systems to investigate alignment, interpretability, rational agency, verification, and reliable control.
Research teams could include specialized agents responsible for formulating problems, generating hypotheses, identifying counterexamples, checking proofs, conducting experiments, and replicating findings.
Human researchers would retain responsibility for research priorities, consequential experiments, interpretation, and decisions concerning deployment.
The program should pursue a diversified portfolio.
Rational agency and objective coherence. Can necessary conditions for coherent goal pursuit be established? Can certain objectives undermine the conditions required for their own successful pursuit? Formal results would need to be connected to the behavior of actual AI systems.
Objective specification and goal stability. How can objectives remain appropriate as systems learn, encounter new circumstances, or undergo modification?
Mechanistic interpretability. Can internal representations and processes responsible for important behaviors be identified and tested through intervention?[12]
Formal verification and control. Can important safety properties be proved for restricted systems, components, or protocols?
Scalable oversight. How can humans reliably supervise systems whose capabilities exceed their own?[13]
The initiative should not assume that deploying more agents automatically produces better research.
Multi-agent approaches should be compared with strong single-agent and human-assisted baselines.
Its principal performance measure should be independently validated safety improvement per unit of cost, adjusted for any additional risks introduced by the research.
Crucially, the program should test whether safety improvements generalize beyond the evaluations on which they were developed.
A promising benchmark result should be treated as evidence for further investigation, not as proof of reduced catastrophic risk.
8. Safety Gates in Practice Safety gates should be capability-specific, proportional to the risks involved, and independently evaluated.
Before proceeding with a potentially dangerous advance, developers should:
1. Identify the new capabilities and plausible failure modes.
2. Assess the severity, probability, and reversibility of potential harms.
3. Present evidence that available safeguards are adequate for the intended use.
4. Compare immediate advancement with feasible safety-first alternatives, including the value of additional information.
5. Explain why any claimed safety-enabling capability satisfies the three-part test.
Relevant evidence may include adversarial testing, independent audits, mechanistic investigations, formal verification of bounded properties, and reliable behavior under conditions not encountered during development.
No single evaluation establishes comprehensive safety.
The strength of evidence required should therefore increase with the severity and irreversibility of the risks.
Independent reviewers should participate in establishing thresholds and assessing whether they have been met.
Where possible, deployment should be staged, monitored, and reversible.
A gate should also be revisited when new evidence changes the comparison between available strategies.
The standard is not certainty. It is a credible, independently scrutinized justification for proceeding.
9. Objections
9.1. Would restraint merely advantage less careful competitors? Unilateral delay could allow a less cautious developer to gain a decisive lead.
This could increase global risk rather than reduce it.
The objection is substantial because the relevant outcome is not the safety of one developer’s systems but the total risk generated by all development paths.
The economic comparison must therefore incorporate competitive responses.
If unilateral restraint would genuinely increase catastrophic risk, the proposed principle does not automatically require it.
Instead, the case for safety gating strengthens the need for coordinated thresholds, shared evaluations, government oversight, and mechanisms that reduce incentives to defect.
The problem of competitive enforcement requires further institutional treatment.
9.2. Can uncertainly aligned AI systems produce trustworthy safety research? Automated researchers might generate misleading results, exploit evaluations, or recommend interventions that appear effective while concealing important weaknesses.
This concern cannot be eliminated by assuming the researchers are benevolent.
The program should instead favor outputs that can be independently checked.
Formal proofs can be mechanically verified relative to explicit assumptions. Empirical interventions can be tested against held-out environments and independent adversarial evaluations. Interpretability hypotheses can be tested through interventions.
Verification is not always easier than discovery, and independent testing cannot guarantee generalization.
The appropriate response is therefore layered oversight, constrained permissions, independent replication, and explicit limits on what unverified research outputs may authorize.
9.3. Might greater capabilities be necessary for safety? More capable AI systems may discover alignment methods that existing systems cannot.
Restricting their development could delay safety progress.
This is a genuine possibility.
The proposed framework permits capability advances whose expected safety benefits justify their risks.
The three-part safety-enabling test provides a means of evaluating these claims rather than accepting them automatically.
9.4. Could safety research itself increase danger? Automated research might accelerate dangerous capabilities, produce dual-use knowledge, or create increasingly autonomous systems that are difficult to supervise.
The program should therefore operate under restricted access, independent monitoring, human authorization for consequential experiments, and stopping conditions.
Its success must be measured by net safety improvement, not merely the quantity of research it produces.
9.5. What if catastrophic risks cannot be measured? The probabilities and consequences of catastrophic AI outcomes are deeply uncertain.
This limits the precision of expected-value calculations.
But uncertainty does not eliminate the need to compare alternatives.
Decision-makers should consider multiple scenarios, evaluate the robustness of conclusions, and investigate which additional evidence could change the preferred course of action.
Where uncertainty is substantial and potential losses are irreversible, preserving future options may be especially valuable.
This is a reason for disciplined investigation, not a substitute for it.
10. Funding and Institutional Design A research initiative of this kind would require substantial resources and credible governance.
Potential funders include frontier AI developers, governments, AI safety institutes, and philanthropic organizations.
A consortium with independent scientific oversight could help coordinate research while limiting conflicts of interest.
Funding should support secure computing infrastructure, scientific personnel, automated research environments, empirical testing, formal verification, and independent replication.
It should be distributed across competing approaches.
Funding should also be staged.
Approaches demonstrating reproducible safety improvements should receive additional resources. Approaches that fail to generalize, create unacceptable risks, or
produce unreliable evaluations should be revised or discontinued.
The program should track reductions in measured alignment failures, independently verified findings, improved detection of deception, reliable control under adversarial conditions, generalization across models, and risks introduced by the research itself.
The objective is not to maximize expenditure.
It is to maximize the expected value of reliable safety progress.
11. The Strategic Opportunity AI-assisted safety research may improve alongside AI capability.
More effective research systems may discover better alignment methods. Those methods may make subsequent systems more reliable, enabling further safety research.
Under favorable conditions, this could create a reinforcing cycle:
Better safety research → better alignment and control → safer subsequent systems → greater capacity for further safety research.
The cycle is not guaranteed.
Capabilities may advance faster than safety. Automated researchers may become harder to supervise. Apparent improvements may fail to generalize.
The proposed research initiative is designed to investigate whether this favorable cycle can be established, under what conditions, and at what cost.
Its strategic significance lies in the possibility of using capabilities already available to reduce the dangers of capabilities not yet developed.
Conclusion
Artificial intelligence is becoming a powerful instrument of scientific discovery, and it has begun to produce measurable improvements on alignment problems.
These developments should influence the order in which AI capabilities are advanced.
The case for safety-first development does not rest on the assumption that delay is always beneficial. It rests on a comparison of alternative development paths, including their benefits, risks, irreversible consequences, and opportunities for learning.
Where a capability advance creates substantial additional catastrophic risk, and available safety research offers a credible opportunity to reduce that risk or improve the decision, there is a rebuttable presumption in favor of undertaking the research
first.
Developers seeking to proceed should be prepared to demonstrate that immediate advancement is preferable.
This principle permits safety-enabling capability progress while preventing that label from becoming a general excuse for acceleration.
It also supports a major, independently evaluated initiative using AI research agents to investigate alignment, interpretability, rational agency, verification, and control.
The scale of investment should be determined by the expected marginal value of genuine safety improvement.
We should neither assume that artificial intelligence will automatically solve the dangers it creates nor neglect the scientific capabilities now available.
Before using our most powerful AI systems to build still more powerful ones, we should make the safety of those future systems a foremost research priority.
The time to establish reliable control over increasingly powerful intelligence is before developing systems beyond our ability to control.
Notes and References
1. Yueh-Han Chen, Jiaxin Wen, and Jan Hendrik Kirchner, “Automated Researchers Can Reliably Mitigate Alignment Failures,” Anthropic Alignment Science Blog, August 28, 2026, https://alignment.anthropic.com/2026/automated-alignment-researchers/.
2. OpenAI, “Ten Advances in Mathematics and Theoretical Computer Science,” August 1, 2026, https://openai.com/index/ten-advances-in-mathematics/; Lean 4 formalizations, https://github.com/openai/ten-proofs.
3. OpenAI, Navier–Stokes announcement, September 8, 2026, https://openai.com/index/navier-stokes-solution/; Clay Mathematics Institute, “Navier–Stokes Announcement,” September 11, 2026, https://www.claymath.org/news/navier-stokes-announcement/.
4. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford University Press, 2014); Stephen M. Omohundro, “The Basic AI Drives,” in Artificial General Intelligence 2008, ed. Pei Wang, Ben Goertzel, and Stan Franklin (IOS Press, 2008), 483–492.
5. Dario Amodei et al., “Concrete Problems in AI Safety,” arXiv:1606.06565 (2016); Victoria Krakovna et al., “Specification Gaming: The Flip Side of AI Ingenuity,” DeepMind, 2020; Lauro Langosco et al., “Goal Misgeneralization in Deep Reinforcement Learning,” ICML 2022; Rohin Shah et al., “Goal Misgeneralization: Why Correct Specifications Aren’t Enough for Correct Goals,” arXiv:2210.01790 (2022).
6. Nick Bostrom, “Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards,” Journal of Evolution and Technology 9 (2002); Bostrom, Superintelligence, chapter 14.
7. Jan Leike and Ilya Sutskever, “Introducing Superalignment,” OpenAI, July 2023.
8. Anthropic, Responsible Scaling Policy; OpenAI, Preparedness Framework; Google DeepMind, Frontier Safety Framework. Consult the versions applicable at the time of submission.
9. Kenneth J. Arrow and Anthony C. Fisher, “Environmental Preservation, Uncertainty, and Irreversibility,” Quarterly Journal of Economics 88, no. 2 (1974): 312–319; Avinash K. Dixit and Robert S. Pindyck, Investment Under Uncertainty (Princeton University Press, 1994); Christian Gollier, Bruno Jullien, and Nicolas Treich, “Scientific Progress and Irreversibility: An Economic Interpretation of the Precautionary Principle,” Journal of Public Economics 75, no. 2 (2000): 229–253.
10. Charles I. Jones, “How Much Should We Spend to Reduce A.I.’s Existential Risk?” NBER Working Paper 33602 (2025), https://doi.org/10.3386/w33602.
11. Katja Grace et al., “Thousands of AI Authors on the Future of AI,” arXiv:2401.02843 (2024), https://arxiv.org/abs/2401.02843.
12. Chris Olah et al., “Zoom In: An Introduction to Circuits,” Distill (2020); Nelson Elhage et al., “Toy Models of Superposition,” Transformer Circuits (2022); Adly Templeton et al., “Scaling Monosemanticity,” Transformer Circuits (2024).
13. Paul Christiano, Buck Shlegeris, and Dario Amodei, “Supervising Strong Learners by Amplifying Weak Experts,” arXiv:1810.08575 (2018); Collin Burns et al., “Weak-to-Strong Generalization: Eliciting Strong Capabilities with Weak Supervision,” ICML 2024.
Leave a Reply