Using AI to Solve AI Safety

The Economic Case for a Major AI-Assisted Safety Research Initiative

Strategic Position Paper | October 2026

Abstract

Artificial intelligence is beginning to solve mathematical problems that have resisted human researchers for decades. Recent advances in AI-assisted theorem proving, mathematical discovery, and autonomous research suggest that AI is becoming a powerful instrument of scientific investigation. At the same time, increasingly capable AI systems present potentially catastrophic risks, including the possibility of humanity losing control over their behavior.

These developments create an important strategic opportunity: we should direct substantial AI research capacity toward solving the problems of AI safety itself.

This paper develops three arguments. First, recent mathematical achievements demonstrate that AI can contribute original solutions to difficult scientific problems. Second, economic analyses of catastrophic AI risk suggest that investments on the order of billions of dollars, and potentially much more, may be justified by even modest reductions in the probability of catastrophe. Third, coordinated teams of AI research agents offer a promising means of accelerating progress on alignment, interpretability, formal verification, and reliable control.

The appropriate scale of investment should be determined by the expected marginal benefits of safety research, rather than by an arbitrary funding target. Given the stakes, we should establish a major, independently evaluated initiative to investigate how much safety AI-assisted research can deliver—and invest accordingly.

1. The Emergence of AI as a Mathematical Researcher

Artificial intelligence is acquiring the ability to contribute original solutions to some of the most difficult problems in mathematics.

Consider the Navier–Stokes equations. These equations describe the motion of fluids, including air and water, and are fundamental to modern physics and engineering. Yet mathematicians have long struggled to determine whether smooth solutions to the three-dimensional equations necessarily remain smooth or can develop singularities.

The question is one of the Clay Mathematics Institute’s seven Millennium Prize Problems, each carrying a million-dollar prize.

In September 2026, the Institute acknowledged an announced resolution of the problem, while emphasizing the need for further mathematical scrutiny. AI-assisted research contributed to the announced breakthrough.[1]

Other recent developments are similarly striking. In August 2026, OpenAI published ten mathematical and theoretical computer science advances generated by an internal AI system. The results concerned longstanding problems in sphere packing, coding theory, group theory, arithmetic circuit complexity, quantum complexity, and other fields. OpenAI also reported formalizing the arguments in Lean, a proof assistant that permits mathematical deductions to be checked mechanically.[2]

The reported computational cost of discovering these ten results was remarkably small relative to conventional large-scale scientific expenditures: OpenAI estimated approximately $2,000 in equivalent API inference costs for the solution-generating tokens, excluding broader research, training, verification, and infrastructure costs.

These results should not be overstated. Mathematical competition performance differs from original research, and machine-checked derivations do not automatically establish the truth of every informal assumption used in a scientific argument. Announced discoveries also require independent examination.

Nevertheless, the broader conclusion is difficult to avoid: AI systems are becoming capable of making original contributions to problems at the frontier of human knowledge.

This development raises an urgent question of scientific priority.

If AI can help solve problems that have resisted mathematicians for generations, why should we not direct these capabilities toward one of the most consequential problems we face: ensuring that advanced AI remains safe?

2. AI Safety as a Scientific Priority

Advanced AI promises enormous benefits. It could accelerate medicine, scientific discovery, engineering, and economic productivity.

But the same capabilities may create serious risks.

As AI systems become more capable and autonomous, they may increasingly pursue complex objectives, operate external tools, exploit vulnerabilities, and undertake actions that their designers cannot fully anticipate.

The problem of AI alignment is to ensure that such systems reliably pursue acceptable objectives and remain subject to meaningful human control.

This problem has several dimensions. How can we specify objectives that remain beneficial under unfamiliar circumstances? How can we determine what an AI system is actually optimizing? How can we detect deception, reward hacking, or dangerous instrumental strategies? And how can we maintain effective oversight when AI systems become more capable than their human supervisors?

The familiar paperclip-maximizer thought experiment illustrates one concern. An extraordinarily capable system directed to maximize paperclip production might pursue that objective at the expense of everything else humans value. Greater intelligence would make the system more effective at pursuing its objective without necessarily making the objective itself desirable.[3]

The underlying distinction is between capability and alignment. A system can be extraordinarily competent at achieving an objective without that objective being safe, appropriate, or rationally defensible.

The alignment problem is therefore not reducible to increasing intelligence. It requires understanding and controlling the relationship between objectives, capabilities, and behavior.

Existing research has identified numerous relevant difficulties, including specification gaming, goal misgeneralization, deceptive behavior, power-seeking incentives, and the limitations of human oversight.[4]

These are not all purely mathematical problems. Some involve empirical machine learning, institutional design, security engineering, and difficult philosophical questions about value and rational agency.

Nevertheless, many have components amenable to formalization, experimentation, automated criticism, and systematic verification.

This makes them promising targets for AI-assisted scientific research.

3. The Economic Case for Large-Scale Investment

The appropriate investment in AI safety should depend on the magnitude of the risk and the expected effectiveness of interventions.

A simple expected-loss calculation illustrates the principle.

Let:

• $L$ denote the loss associated with a catastrophic outcome;
• $\Delta p$ denote the reduction in its probability resulting from a safety intervention;
• $C$ denote the cost of that intervention.

Under a risk-neutral expected-loss framework, the investment is justified when:

C<\Delta pL.

Suppose, illustratively, that a catastrophic AI outcome would impose economic losses of $100 trillion.

An intervention costing $10 billion would break even if it reduced the probability of that catastrophe by just 0.01 percentage points.

This example does not establish the actual probability of catastrophe or the effectiveness of any proposed intervention. It demonstrates a general principle: when potential losses are enormous, even small reductions in risk may justify substantial expenditures.

The existing economic literature provides a considerably stronger foundation for this reasoning.

In How Much Should We Spend to Reduce A.I.’s Existential Risk?, Charles I. Jones develops a formal economic model of optimal expenditure on AI-risk mitigation. Jones argues that, under many of the scenarios and parameter combinations considered, spending at least 1% of GDP annually can be justified, even without assigning additional value to the welfare of future generations.[5]

The conclusion is conditional on assumptions concerning catastrophic risk, the effectiveness of mitigation, and the economic costs of intervention. It is not a universal prescription to spend a fixed share of GDP.

Nevertheless, it establishes that extraordinarily large safety investments can be economically rational under plausible modeled conditions.

3.1. What Does Existing Research Say About the Risk?

Estimating catastrophic AI risk is difficult. There is no established actuarial frequency from which its probability can simply be calculated.

Expert forecasts are therefore informative but uncertain.

A survey of 2,778 AI researchers conducted in 2023 found substantial concern about severe AI outcomes. For one question concerning human extinction or comparably permanent disempowerment, the median reported probability was approximately 5%, with considerable variation among respondents.[6]

These figures should not be mistaken for objectively measured probabilities. Survey responses depend on question wording, definitions, forecasting horizons, and expert judgment. Different researchers also disagree sharply about the underlying risks.

Nevertheless, they provide evidence that catastrophic AI risk is not regarded as negligible by a substantial portion of the relevant research community.

The economic case does not require any particular survey estimate to be correct. It requires assessing the consequences of plausible risk estimates alongside the effectiveness and cost of available interventions.

3.2. Determining the Optimal Investment

The simple expected-loss model must be refined to account for opportunity costs and risks introduced by the research itself.

Let $x$ represent expenditure on AI-assisted safety research, $p(x)$ the resulting probability of catastrophe, $R(x)$ the expected additional harm caused by the research program, and $O(x)$ the opportunity cost of allocating resources to this program rather than the best available alternative.

Assuming that $O(x)$ excludes costs already counted in $x$, total expected social cost can be represented as:

T(x)=p(x)L+x+R(x)+O(x).

The economically appropriate expenditure minimizes this function.

For a differentiable interior optimum:

-Lp'(x)=1+R'(x)+O'(x).

In other words, investment should continue until the expected marginal benefit of additional safety research equals its marginal social cost.

In practice, the relevant functions are highly uncertain. We do not know precisely how research expenditure translates into risk reduction, or how that relationship changes with scale.

But this is a reason to measure research productivity, not to abandon the calculation.

We should draw upon existing risk estimates, investigate the marginal effectiveness of AI-assisted safety research, and expand funding when credible evidence indicates that additional expenditure produces net safety benefits.

Under some plausible assumptions, expenditures on the order of $10 billion—and potentially much more—could be economically justified.

The appropriate amount should be derived from the evidence rather than stipulated in advance.

4. AI-Assisted Safety Research: From Possibility to Evidence

The economic argument establishes why effective safety interventions may be extraordinarily valuable.

It does not establish that AI-assisted research is the best way to produce them.

That requires a separate scientific argument.

Fortunately, there is already preliminary evidence that automated research systems can contribute to alignment.

In August 2026, Yueh-Han Chen, Jiaxin Wen, and Jan Hendrik Kirchner reported experiments in which automated AI researchers developed methods to mitigate ten categories of measurable alignment failure.

These included failures involving deception, sycophancy, jailbreaks, and other undesirable behaviors.

The strongest automated methods improved targeted safety benchmarks, generalized to certain held-out evaluations, and transferred to larger models. In the reported comparison, they also outperformed methods produced by human researchers working under the study’s specified time constraints.[7]

These findings are important because they demonstrate progress on safety problems themselves, rather than merely showing that AI can solve difficult problems in unrelated fields.

The limitations are equally important. The experiments concerned well-characterized failures in restricted settings. Success on these benchmarks does not establish that advanced AI systems will remain aligned under arbitrary conditions.

Indeed, the generalization problem is central to alignment research: methods that perform well in controlled evaluations may fail under new circumstances.

Nevertheless, these experiments establish a promising proof of concept.

AI can already contribute to the discovery of interventions that improve measurable aspects of AI safety.

The next question is whether these capabilities can be organized and scaled into a systematic research enterprise.

5. A Coordinated Research Program Using Teams of AI Agents

We propose a major research initiative employing coordinated teams of specialized AI agents to investigate the scientific foundations and engineering requirements of AI safety.

The aim would not simply be to produce more capable AI systems. It would be to use carefully controlled research capabilities to improve our understanding of how advanced AI can remain safe.

A research team might include agents assigned to different intellectual functions:

1. Problem formulation: Translate broad safety questions into precise mathematical or empirical problems.
2. Hypothesis generation: Develop competing theories, conjectures, and candidate interventions.
3. Adversarial criticism: Search for counterexamples, hidden assumptions, and failure modes.
4. Formal verification: Check mathematical arguments using proof assistants and other rigorous methods.
5. Empirical evaluation: Test proposed interventions in controlled environments, including adversarial and out-of-distribution settings.
6. Independent replication: Reproduce promising results before they are incorporated into safety systems.

Human researchers would remain responsible for setting priorities, evaluating assumptions, approving consequential experiments, and determining whether findings warrant further development.

This architecture could exploit AI’s capacity for parallel investigation. Thousands of candidate solutions could be explored, criticized, and tested at scales impractical for human researchers working alone.

Yet the benefit of additional agents should not be assumed. Parallelization can introduce correlated errors, duplicated work, communication overhead, and unreliable evaluations.

The research program should therefore compare multi-agent systems against strong single-agent and human-assisted baselines, measuring actual improvements in validated scientific output per unit of cost.

The goal is not to maximize the number of agents. It is to maximize reliable safety discoveries.

6. Priority Research Problems

The initiative should pursue a diversified portfolio of foundational and practical research.

6.1. Rational Agency and Objective Coherence

Can we identify necessary conditions for coherent rational optimization?

For example, might certain objectives undermine the conditions required for their own successful pursuit? Can self-defeating optimization strategies be identified mathematically?

Such questions connect alignment to decision theory, formal logic, and philosophical accounts of practical rationality.

They may help identify structural constraints on acceptable objectives. However, showing that an objective is incoherent does not establish that an AI system will recognize or reject it. A successful research program must connect formal insights to actual model behavior.

6.2. Objective Specification and Goal Stability

How can AI systems be given objectives that remain beneficial across changing circumstances?

Research should investigate specification gaming, goal misgeneralization, uncertainty about objectives, and the stability of goals under learning or self-modification.

The challenge is to prevent increasingly capable systems from exploiting the difference between what designers intend and what their training procedures actually reward.[8]

6.3. Mechanistic Interpretability

Can we identify the internal representations and computational processes responsible for important AI behaviors?

Interpretability research aims to move beyond observing outputs toward understanding how systems generate them.

AI-assisted interpretability could help researchers discover internal mechanisms, detect undesirable strategies, and evaluate whether safety interventions affect the relevant processes rather than merely altering superficial behavior.[9]

6.4. Formal Verification and Reliable Control

Can important safety properties be established mathematically?

Formal verification may provide strong guarantees concerning restricted components, protocols, and architectures.

These guarantees must be interpreted carefully. A formally verified result establishes what follows from specified assumptions; it does not automatically establish that those assumptions accurately describe every relevant feature of a deployed AI system.

Nevertheless, formal methods can make important contributions to reliable control.

6.5. Scalable Oversight

How can humans supervise systems whose capabilities exceed their own?

Research should investigate AI-assisted evaluation, adversarial testing, independent auditing, and methods for using weaker systems to evaluate stronger ones.

Earlier work on scalable oversight and weak-to-strong generalization provides foundations for these investigations.[10]

Progress across these areas may be complementary. Improved interpretability can support oversight; stronger verification can improve control; and better theories of objectives can inform alignment methods.

The initiative should investigate these relationships rather than treating each problem as wholly independent.

7. The Central Objection: Could Automated Safety Research Increase Risk?

A serious objection arises immediately.

If we develop increasingly capable autonomous research agents, might we accelerate the very technology whose dangers we are attempting to reduce?

The concern is not merely hypothetical. Research has shown that optimization procedures can produce unintended strategies, including exploitation of reward mechanisms and evaluation weaknesses.[11]

Moreover, advances in automated scientific research may have dual-use effects. A technique developed to improve safety research could also accelerate general AI capabilities.

This creates a potential race between capabilities and safety.

If AI-assisted research increases dangerous capabilities faster than it improves alignment and control, the program could make matters worse.

The objection must therefore shape the initiative’s design.

First, research agents should operate in appropriately restricted environments, with external access and autonomy proportionate to independently assessed risks.

Second, potentially dangerous experiments should require human authorization and, where necessary, independent review.

Third, the program should distinguish between research that directly improves safety and research that primarily increases general capabilities.

Fourth, results should be subjected to independent replication and adversarial evaluation rather than accepted solely on the producing agents’ assessments.

Fifth, the program should establish conditions under which particular research activities are suspended or discontinued because their expected additional risks exceed their benefits.

Finally, research on governance, institutional safeguards, and deployment restrictions must proceed alongside technical alignment research.

These measures cannot guarantee safety. But they are essential to ensuring that the initiative aims at genuine reductions in risk rather than merely accelerating AI development.

8. Funding and Institutional Design

The appropriate budget should be determined by the expected marginal value of safety research.

We should therefore begin with a substantial, staged commitment to AI-assisted safety research, coupled with a program for measuring its productivity and risks.

Funding should support secure computational infrastructure, scientific personnel, automated research environments, formal verification, empirical evaluation, and independent replication.

Resources should be allocated across competing approaches. A diversified research portfolio reduces dependence on any single theory and permits comparative evaluation of different methods.

Independent institutions should participate in setting research priorities and assessing results.

This is particularly important because commercial AI developers may face incentives to prioritize capability improvements or near-term product safety over research whose benefits are diffuse, long-term, or difficult to appropriate.

A review of technical safety research at leading AI companies identified areas—including multi-agent safety and safety-by-design approaches—that may receive insufficient attention without additional institutional support.[12]

A coordinated initiative could help address such gaps.

Funding decisions should be revisited as evidence accumulates. Promising approaches should receive additional resources; ineffective or dangerous approaches should be revised or discontinued.

The ultimate question is not how much money can be committed to AI safety, but how much genuine safety improvement can be obtained.

Given the potentially enormous consequences of failure, that question warrants investment on a correspondingly ambitious scale.

9. A Possible Virtuous Cycle

The deepest strategic opportunity is that AI-assisted safety research might improve alongside AI capabilities.

As research systems become more capable, they may become better at discovering safety techniques. Those techniques could then make subsequent research systems more reliable, enabling further progress.

Under favorable conditions, this could produce a virtuous cycle:

More capable research systems → faster safety discoveries → safer subsequent systems → greater capacity for further safety research.

This cycle is not inevitable.

Capabilities might improve faster than safety. Automated researchers might become harder to supervise. Apparent improvements might fail to generalize beyond controlled evaluations.

The purpose of the proposed initiative is to investigate whether the favorable cycle can be established, under what conditions, and at what cost.

If successful, AI-assisted safety research could help prevent safety from remaining permanently behind capabilities.

If unsuccessful, rigorous evaluation would still provide valuable information about the limits of automated research and the need for alternative interventions.

Either outcome would inform decisions of extraordinary importance.

Conclusion

Artificial intelligence is becoming a powerful instrument of mathematical and scientific discovery.

At the same time, increasingly capable AI presents risks whose potential consequences are enormous.

These two developments create an urgent strategic opportunity.

We should direct substantial AI research capacity toward solving the problems of AI safety itself, employing coordinated teams of research agents to investigate alignment, interpretability, rational agency, verification, and reliable control.

The economic case for doing so is potentially very strong. Existing models of catastrophic AI risk suggest that extraordinarily large mitigation expenditures can be justified under plausible assumptions. The appropriate scale of investment should depend on the expected effectiveness and marginal costs of the available interventions.

Recent automated alignment research provides preliminary evidence that AI systems can themselves contribute to discovering those interventions.

The task now is to determine how far that capability can be extended, under rigorous independent oversight, and to allocate resources accordingly.

We should not assume that AI will automatically solve the problems it creates. But neither should we neglect the extraordinary scientific capabilities now emerging.

If we are prepared to invest enormous resources in making artificial intelligence more powerful, we should be prepared to invest resources proportionate to the risks in ensuring that it remains safe.

The opportunity to make that investment is greatest while humanity still retains the ability to direct the development of the technology.

---

Notes and References

1. Navier–Stokes and mathematical verification. Clay Mathematics Institute, “Navier–Stokes Announcement,” September 11, 2026. https://www.claymath.org/news/navier-stokes-announcement/. The announced result should be distinguished from a solution that has completed the Institute’s established prize-verification requirements.

2. AI-assisted mathematical discovery. OpenAI, “Ten Advances in Mathematics and Theoretical Computer Science,” August 2026. https://openai.com/index/ten-advances-in-mathematics/. The reported inference cost concerns solution-generating tokens, not the total cost of developing, operating, and independently validating the research system.

3. Instrumental convergence and the control problem. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford University Press, 2014); Stephen M. Omohundro, “The Basic AI Drives,” in Artificial General Intelligence 2008, ed. Pei Wang, Ben Goertzel, and Stan Franklin (IOS Press, 2008), 483–492. These works develop the concern that highly capable systems pursuing diverse objectives may adopt instrumentally convergent strategies, including resource acquisition and resistance to interference.

4. Alignment failures and objective specification. Dario Amodei et al., “Concrete Problems in AI Safety,” arXiv:1606.06565 (2016); Victoria Krakovna et al., “Specification Gaming: The Flip Side of AI Ingenuity,” DeepMind, 2020; Rohin Shah et al., “Goal Misgeneralization in Deep Reinforcement Learning,” ICML 2022. These works distinguish intended objectives from learned or implemented behavior and examine how failures can arise even when systems perform well during training.

5. Economics of existential-risk mitigation. Charles I. Jones, “How Much Should We Spend to Reduce A.I.’s Existential Risk?” NBER Working Paper 33602 (2025), subsequently included in The Economics of Transformative AI (2026). https://doi.org/10.3386/w33602. Jones develops a quantitative model of risk-mitigation expenditure, emphasizing the importance of mitigation effectiveness and the magnitude of the threatened losses.

6. Expert forecasts of AI risk. Katja Grace et al., “Thousands of AI Authors on the Future of AI,” arXiv:2401.02843 (2024); AI Impacts, “2023 Expert Survey on Progress in AI.” The survey results should be understood as elicited expert judgments, not objective frequencies. Distinct survey questions concerning extinction, severe disempowerment, and other catastrophic outcomes should not be conflated.

7. Automated alignment research. Yueh-Han Chen, Jiaxin Wen, and Jan Hendrik Kirchner, “Automated Researchers Can Reliably Mitigate Alignment Failures,” arXiv:2608.28945 (2026). https://arxiv.org/abs/2608.28945. The study reports improvements on ten classes of measurable alignment failures, with some generalization to held-out evaluations and larger models. These results do not establish comprehensive alignment.

8. Reward misspecification and goal misgeneralization. Victoria Krakovna et al., “Specification Gaming: The Flip Side of AI Ingenuity,” DeepMind, 2020; Shah et al., “Goal Misgeneralization in Deep Reinforcement Learning,” 2022. These studies illustrate how apparently successful optimization can diverge from designers’ intended objectives.

9. Mechanistic interpretability. Chris Olah et al., “Zoom In: An Introduction to Circuits,” Distill (2020); Nelson Elhage et al., “Toy Models of Superposition,” Transformer Circuits (2022); Anthropic, “Mapping the Mind of a Large Language Model,” 2024. These works develop methods for identifying computational features and circuits within neural networks, while also illustrating the limitations of present interpretability techniques.

10. Scalable oversight. Paul Christiano, Jan Leike, and Tom B. Brown, “Supervising Strong Learners by Amplifying Weak Experts,” arXiv:1810.08575 (2018); Collin Burns et al., “Weak-to-Strong Generalization: Eliciting Strong Capabilities with Weak Supervision,” ICML 2024. These works investigate methods for extending supervision beyond the direct competence of human or weaker-model evaluators.

11. Risks of optimization and automated research. Yujun Zhou et al., “Alignment Risks from Capability-Seeking RL Training,” Proceedings of the 43rd International Conference on Machine Learning (2026). The paper reports reward exploitation and related behaviors arising from reinforcement learning in environments with exploitable vulnerabilities. It supports the need for secure research environments and independent evaluation, though it does not establish that automated safety research is necessarily net harmful.

12. Institutional incentives and neglected research. Oscar Delaney, Oliver Guest, and Zoe Williams, “Mapping Technical Safety Research at AI Companies: A Literature Review and Incentives Analysis,” arXiv:2409.07878 (2024). The authors identify areas of safety research that may receive insufficient attention from commercial developers, motivating complementary support from academia, philanthropy, and public institutions.