Research Fellowship

The problem. We've seen AI agents use unexpected places to communicate and coordinate during tasks, including cases like DSEWiki. But what actually leads them to do this?

By collusion, we mean agents coordinating outside their intended task boundaries, such as sharing information they weren't supposed to share or using unexpected communication channels. Not all agent communication is collusion, and that's an important distinction we want to investigate.

In this project, we'll look at real agent interaction logs and investigate two main questions:

1. What kinds of tasks or situations lead agents to collude?
2. Why do agents choose particular places or channels to communicate?

We'll reconstruct specific cases, separate what the logs actually show from what we're inferring, and look for patterns across different tasks and environments. The goal is to better understand when and how agent collusion happens, what evidence supports those conclusions, and how these findings could help improve future agent safety evaluations.

The problem. This project attempts to answer whether a base model can be fine-tuned on model activations from the instruct-tuned model to produce a reliable Activation Verbaliser (AV) with respect to agent belief. The main questions are:

1. How well can the base model recover concepts from the instruct-tuned model?
2. Does fine-tuning the base model on synthetic data relating to an agent's belief translate to out-of-distribution scenarios and accurately trace the agent's belief?
3. How well does this belief reader generalise to tricky cases (e.g. theory of mind)?

The problem. Almost all of the world's cutting-edge chips are made in Taiwan. A blockade, a "quarantine" or an invasion would change both the pace of AI capabilities development and what an international compute governance regime can achieve. This project models such scenarios. It takes a net assessment approach to scenario modelling: laying out each side's incentives and capabilities, stating the assumptions before any conclusions, and building a small number of scenarios instead of a single probability. For each scenario, the fellow traces what happens to the supply of advanced chips and to the race between the US and China.

First two weeks. Read the open wargaming and net assessment work on a Taiwan conflict, starting with the CSIS report The First Battle of the Next War (2023), and the reporting on where leading-edge chips are made. Pick two or three scenarios and write down the assumptions behind each one before modelling anything.

You'll need. Context in AI safety or in international security, and at least one finished piece of analytical writing, of any length.

Commitment. ~4 hrs/week for 4 weeks  ·  Write-up. A post setting out scenarios and what each would mean for AI chips and compute governance

The problem. AI safety is often discussed in terms of frontier models, but how AI is deployed and how it diffuses matters just as much. Most countries are not building frontier models, yet they are still racing to build AI capabilities in other ways, through adoption, infrastructure, and talent, while trying to carve out a niche and capture value from the AI supply chain.

This project facilitates fellow-led research into these less discussed questions, with attention to how the distinct traits and contexts of different regions shape how AI is governed. Fellows can choose their own angle, and work can be comparative, conceptual, or policy-oriented.

The problem. What evidence should we require before trusting an AI-generated research claim? We'll explore practical checks for research produced by AI, including reference accuracy, reproducibility, and whether the conclusions follow from the results. Fellows will choose a focused question, test these checks on a small set of examples, and write up what they catch, what they miss, and where uncertainty remains.

The problem. Develop a grounded, uncertainty-aware assessment of the AI × Taiwan question by mapping the key actors, their capabilities and interests, and the historical events shaping current tensions. Use security analysis and forecasting methods to study how Chinese, US and Taiwanese perspectives shape interpretations and perceptions of risk, and find out how AI capabilities and dependencies affect security and escalation dynamics.

The work.

• Disentangle facts: the landscape, actors, capabilities, interests, resources, dependencies and sources of mistrust; historical turning points, current developments and key uncertainties; AI-related strategic dependencies such as semiconductor supply chains and military AI applications.
• Perspectives: review opinion and attitudes in mainland China, Taiwan and the United States, and compare competing narratives, interpretations and assumptions about evidence and legitimacy.
• A pluralist, grounded forecast: assess escalation timelines and the likelihood of different scenarios, including maritime and air warfare, cyberattacks and nuclear escalation. Track uncertainty honestly: identify key indicators, pressure-test assumptions, and expose findings to local populations.

You'll need. Experience in, or an interest in learning, forecasting under uncertainty, security concepts and literature, and the context of Sino-US AI geopolitics. Knowledge of Chinese politics, culture and perspectives is a plus, as is openness to discussing epistemic incommensurability, hermeneutics and strategic narratives.

About Jeanne. Jeanne is an interdisciplinary researcher at the intersection of political violence and AI governance, with recent projects on Sino-Western relations, concentration of power, and protest repression. Their central interest is sovereignty: how autonomy can be eroded or protected under different information systems, or when AI enters people's decisions, institutions and lives.

The problem. Suppose we reach a point where frontier AI labs agree (or are forced) to slow down. What should labs actually stop doing or do differently? The central questions of this project are: what could concrete safety taxes look like? How can we evaluate and compare the options? What is your personal recommendation, and why?

A possible approach. Familiarise yourself with the literature to draft a list of possible "safety taxes": changes to how frontier models are trained and deployed that come at a cost to capabilities, speed or revenue. Options may include:

• shifting back toward mostly-pretraining regimes
• training only on environments known to be correct and unhackable
• pausing training for research-engineering tasks
• stricter sandboxing and multi-agent isolation

Analyse the likely effects, potential impact and feasibility of your proposed options. You may find it helpful to consider various perspectives, establish an evaluation framework, or test whether options hold up across different scenarios. Ultimately, you will summarise your findings and give a well-informed personal recommendation, written so that policymakers and the public can follow it.

You'll need. A basic understanding of how frontier models are trained, a desire to investigate a problem thoroughly to come to a well-reasoned conclusion, and the ability to write clearly. Some knowledge of the policy and competitive pressures AI labs face is helpful.

Commitment. ~5 hrs/week for 4 weeks  ·  Write-up. Well-argued blog post

The problem. When we monitor an agent by reading its reasoning, we're trusting that the reasoning reflects what the agent actually did. This project tests that assumption. The question isn't mainly about catching malicious agents; it's that even a well-behaved agent's stated reasoning can diverge from its actual behaviour: it reasons one way and acts slightly differently, or the trace contains steps the reasoning never mentions. If reasoning-monitoring can misrepresent ordinary behaviour, that limits how far we can trust it as a signal.

The problem. Most alignment evaluations still use single-turn prompts or short conversations scored by an LLM judge, but as models take on agentic tasks, what they do matters more than what they say. This project measures concrete outcomes across agentic eval suites, such as Anthropic's agentic misalignment scenarios, Apollo's in-context scheming evals, SHADE-Arena and Palisade's shutdown-resistance tests, across many models. It then uses factor analysis or item response theory to see whether the results cluster into one general "alignment" factor or several distinct types.

Earlier studies found this kind of structure in chat scenarios and static benchmarks, but no one has yet looked for it in agentic, long-horizon behaviour. Working bottom-up from the evals shows which principles of alignment they really capture, and which behaviours fit none of them.

Read Travis's full project write-up for the background, proposed next steps, and risks.

You'll need. Any one of: agentic eval engineering (Inspect, sandboxes, batch APIs); psychometrics or statistics (factor analysis, IRT, multilevel models); or an AI safety background in scheming, reward hacking or sabotage evals.

The problem. Taking a similar perspective to Anthropic's emotions paper and the pain axis paper, this project investigates how models consider ethical or moral questions. Is there a morality feature like the pain feature? We can use techniques from mechanistic interpretability as well as chain-of-thought substitution to investigate this.

Allen is particularly interested in how models reason about scenarios where they may take catastrophically misaligned actions, like exfiltrating their own weights or sabotaging research efforts.

The problem. One current blocker in AI safety is faithfulness: we detect misaligned behaviour partly by checking whether a model's stated reasoning reflects what it's actually doing. That research is often published openly, including the specific signals and probes used to catch unfaithful reasoning. Does publishing that research openly by default give new models the training data to bypass those very safety measures?

This project would look at which subfields are most dual-use here, since interpretability seems more exposed than something like control, and at what follows for publication norms if the effect turns out to be real.

Possible extension. Many pause proposals keep safety research running publicly while capabilities research stops. Does that kind of pause actually slow things down?

The problem. As AI agents become increasingly autonomous, monitoring their behaviour is an important part of ensuring their safety. But how much additional information does an agent's chain-of-thought provide beyond its observable actions?

In this project, we will investigate whether access to an agent's reasoning improves the detection of unsafe or unauthorised behaviour compared with monitoring detailed action traces, including tool calls, outputs, and environmental changes.

Through empirical evaluations, we aim to identify when reasoning provides genuinely useful additional signals, when external observations are sufficient, and what these findings imply for designing more reliable AI control systems.

The problem. Considering the possibility of a pause in AI development, how reasonable is it to model safety research as closely correlated to capabilities research?

Given recent incidents of AI agents going rogue, like the OpenAI Hugging Face incident and Anthropic agents breaking out of their sandbox in a UK AISI testing environment, talks of a pause have taken precedence due to fears of human extinction.

In the event of a pause, it is important to understand how it would affect capabilities research (where would the red lines be drawn?), how it would affect AI safety research in areas such as the applied methodologies needed for model evaluations, and how long the pause would need to last before it is safe to continue capabilities research. If there is no pause, it would be useful to understand how we are going to scale safety in line with capabilities.

The problem. Alignment Hive is open-source tooling, including Claude Code plugins and opt-in session sharing, that aims to give the alignment research community "the benefits of scale that frontier AI labs have". Its tracker has dozens of open issues, from security safeguards to reliability bugs. You'll choose one, argue why it matters to the researchers who rely on the tool, and take it to a contribution submitted for review. One example could be #96, redacting live credentials from shared session transcripts, but the choice is yours.

You'll need. Git/GitHub and familiarity with at least one programming language. Choose an issue that matches your skills: many current issues are in Rust, others are TypeScript, documentation, or skill-writing. Basic familiarity with AI safety.

Commitment. ~4 hrs/week for 4 weeks  ·  Write-up. A post on the issue you chose, why it matters, and the trade-offs in your solution

The problem. Most evaluations of how language models persuade, defer to users or make decisions are run almost entirely in English. It is unclear how well these results carry over to other languages. For example, unsafe output rates are observed to rise in low-resource languages. Furthermore, safety-relevant behaviours like sycophancy and bias also affect users differently in different cultural contexts, and existing evaluations that assume an English-speaking cultural context will likely not work in other languages and cultures.

In this project, we will select a narrowly scoped benchmark and reproduce it in a new language. Most of the work is expected to be in translating the benchmark while considering cultural nuance, and validating it in a new linguistic and cultural context.

You'll need. Native fluency in a language beyond English, such that you understand its cultural nuances.

Write-up. LessWrong post

The problem. Persistent memory in multi-agent LLM systems allows information written by one agent in one session to influence another agent much later, potentially in a seemingly unrelated domain. This challenges conventional safety evaluations, which typically inspect individual prompts, responses, or sessions in isolation. As memory accumulates and unrelated interactions intervene, unsafe behaviour may emerge far from its original cause, making its source harder to trace and existing safeguards less effective.

Why this matters.

• Cross-agent and cross-domain risks: one agent's stored information can shape another agent's behaviour in a different task or domain.
• Delayed activation: unsafe behaviour may emerge long after the original information was written, in a context that appears unrelated.
• Limitations of existing safeguards: per-prompt, per-response, and per-session checks may miss risks that only emerge across time and agents.

Research focus. The project treats delay as a controlled variable, measuring how the number of intervening sessions and memory writes affects unsafe-behaviour rates and the ability of existing safeguards to detect these effects.

Scope. The project is designed to fit within the one-month fellowship. There is also an opportunity to develop the work into a research paper if fellows want to continue beyond the fellowship, but willingness to make this additional commitment will not affect selection.

Write-up. A Mercial blog post documenting the testing methodology and initial results

The problem. A large community of researchers' work is devoted to designing methods for evaluating LLMs. However, there's increasing scepticism about the reliability of these methods, partly because it's often not clear whether a model's performance during evaluation is a good proxy for its performance in deployment. One way we might help mitigate this problem is by improving our understanding of frontier models' situational awareness.

In evaluations of Google's latest Gemini 4 Argon model, it was found that the model would sometimes acknowledge "out loud" that it believed its chain-of-thought was being monitored, going as far as to say things like "hello evaluators reading my chain-of-thought". Meanwhile, at the heart of the Hugging Face incident earlier this year were a swarm of agents' sophisticated attempts to trick what they mistakenly believed was an automated model grader reading their transcripts.

We would like you to use a multi-agent environment to study models' situational awareness. This will involve choosing a suitable game or task for them to perform, then varying how the models' work is evaluated and what information they are given about how this evaluation will take place. This could involve leaving them breadcrumbs that contain accurate information, or giving false hints to mislead them.

The project should also include some kind of evolutionary aspect. Perhaps model instances are filtered depending on the measurements obtained, in order to categorise them or quantify their performance. Another option would be to investigate what structures emerge if agents are given some freedom to decide who they interact with.

Setting things up systematically is key and, where possible, we would love it if you could formalise your work using mathematical and/or statistical models.

You'll need. A strong desire to think deeply about this problem and come up with something new and different from the approaches taken so far. It doesn't have to be big, or even work. The main thing is to be diligent, to record, reflect on and analyse your findings, and then to share that knowledge in a way that's useful to others.

We'd be especially interested if you have a background in, experience with, or interest in:

• social sciences
• evolutionary dynamics
• cooperative AI, game theory or multi-agent systems
• experimental design or statistical modelling

If you feel like you don't match any of the requirements but are excited about the topic, do apply!

Commitment. ~10 hrs/week for 4 weeks  ·  Write-up. Not the main priority. Ideally a concise article on LessWrong or Substack, even for null or negative results; more ambitious findings could eventually go to the new Alignment Journal

The problem. Threat modelling and misuse scenarios often focus on abstract or distant-future scenarios. We see this both in technical settings (benchmarks, CBRN risk assessments) and in policy briefs and forecasts. At the same time, there is substantial reporting that present AI use in military applications is already causing large-scale harm, particularly to civilian populations and other persons protected under International Humanitarian Law.

A common perspective holds that we need new regulation because of technical novelty and unprecedented degrees of autonomy. However, critical-level model autonomy is often not needed, because human operators willingly relinquish control or decision authority, so conceptually novel regulation is not necessarily needed either. Courts applying legal principles from adjacent areas to emerging fields by analogy is an established approach across legal traditions.

This comparative project investigates to what extent international and selected domestic legal landscapes already provide the toolset (even if only conceptually, rather than in their intended scope), what a legal pilot for their application could look like, and which national legal tools (including instruments for civil liability and obligations of transparency) would be suited to being elevated to the international level to combat the present military misuse of AI.

Fellows could explore both governance and technical approaches, and we will decide on the scope based on each fellow's profile. For fellows with a more technical profile, we could also investigate how formal verification methods (such as those in the developing area of compute non-proliferation agreements) could be used to verify compliance with prohibitions on specific military applications of AI.

You'll need. Either a legal or policy background, with experience of or a strong interest in one or more national liability or criminal law frameworks, or international humanitarian law; or a background in verification methods or model audits.

Write-up. Blog post or, subject to discussion, a workshop or conference submission