The Safeguarded AI programme is seeking a single founding team or entity to lead Technical Area 2 (TA2) Phase 2, focusing on the machine learning (ML) elements required to integrate frontier AI capabilities into a secure, general-purpose Safeguarded AI workflow.

We are looking to catalyse the creation of a new organisation that can not only push the boundaries of ML research but also uphold the highest standards of organisational governance and security. An £18 million grant will be awarded to the successful applicant to host the entire TA2 research agenda from early 2026 to the end of 2027.

TA2 will explore leveraging securely-boxed AI to train autonomous control systems that can be verified against mathematical models, improving performance and robustness. The workflow will involve forking and fine-tuning mainstream pre-trained frontier AI models to create verifiably safeguarded AI solutions. Key objectives of TA2 include:

• World-modelling ML (TA2(a)): Develop formal representations of human knowledge, enabling explicit reasoning and uncertainty accounting, to create auditable and predictive mathematical models.
• Coherent reasoning ML (TA2(b)): Implement efficient reasoning methods, such as amortised inference or neural network-guided algorithms, to derive reliable conclusions from world models. Safety verification ML (TA2(c)): Create mechanisms to verify the safety of actions and plans against safety specifications, using techniques like proof certificates or probabilistic bounds.
• Policy training (TA2(d)): Train agent policies that balance task performance with finite-horizon safety guarantees, including backup policies for safety failure scenarios.

Eligibility

We welcome applications from exceptional and ambitious researchers, organisational leaders, or experienced founders driven to create an alternative R&D pathway for safe and transformative AI.

Candidates must be based in, or prepared to relocate to, the UK and include:

• New founding teams establishing a UK-based non-profit institution.
• Leading AI companies creating a UK-based affiliated non-profit entity.
• Established companies with critical-infrastructure businesses forming a UK-based affiliated non-profit to become a pioneering supplier of guaranteed-safe AI capabilities.
• Established academic institutions creating, or partnering to create, a new UK-based affiliated non-profit entity.

Please note: For-profit companies and universities aiming to direct host TA2 will not be eligible for this funding.

Apply to Phase 2 – deadline: 1 October 2025 (13:00 BST)

Your proposal should be 30-50 pages and comprehensively address both technical and organisational aspects, as outlined in the call for proposals.