Apply to the Research Fellowship

The problem. Write a report on what has changed in the AI landscape recently. This could include a risk report on GPT-6 Astra and an outline of what we can infer about the capabilities of OpenAI's new "internal model". Mentees could also cover the debate about LLMs' mathematical capabilities, including the many recent developments in this space (e.g. Navier-Stokes), as well as the Hugging Face incident and the subsequent response (e.g. the METR investigation and Amodei's "We Must Pace the Frontier"). As a main focus of the report, it would be valuable to investigate how close we are to recursive self-improvement and the resulting implications of this.

You'll need. Good understanding of the overall AI landscape, and a desire to thoroughly investigate a problem, considering multiple different angles, to come to a well-reasoned conclusion. You should have excellent writing ability.

Commitment. ~5 hrs/week for 4 weeks  ·  Write-up. Well-argued blog post

The problem. When we use a deception probe against models in reinforcement learning, they learn to trick the probe. As probes are essentially looking for some characteristic in the residual activation streams, the model may be learning a simple linear transformation, like a rotation, to alter its geometry to trick the probe, whilst maintaining structure. We test this hypothesis using an open-source codebase from Neel Nanda. I have made some more in-depth notes here.

First two weeks. Read the relevant papers, replicate their results, and train deception probes. Plan the best ways to find evidence for or against the hypothesis, and run some low-cost experiments.

You'll need. You should be comfortable with first-year university level maths, and thinking abstractly. Strong programming skills are not required, but you should be able to competently use Python. You should be able to learn quickly, and be committed to the project!

Commitment. ~6 hrs/week for 4 weeks  ·  Write-up. A post making the case for our assumption, and relevant evidence we find in our investigation. We will also talk about how much this might generalise to larger models.

The problem. Open weight models are about 4 months behind closed weight models. This project would look at the likelihood of a Rogue Agent Catastrophe with open weight models at the capability level of Mythos or Astra, and investigate how likely it is that action will be taken to protect humanity against this.

First two weeks. Spend time reading and gaining a deep knowledge of factors affecting likelihoods of different harm pathways. Critique current literature.

You'll need. Excellent writing abilities. Previous writing is a plus (e.g. you have a substack). You should think through arguments in depth and look for weaknesses in assumptions.

Commitment. ~4 hrs/week for 4 weeks  ·  Write-up. Well-argued post

The problem. Recently, an argument has been circulating on alignment forums that alignment is net negative, as it causes us to overstate how aligned models are, while underestimating catastrophic risk, and in fact it is better to let a smaller catastrophe happen now to wake up the public. The case is also made that certain alignment techniques, like RLVR, make models less aligned. It would be good to do a deep investigation on this, looking at the strengths and weaknesses of this position.

First two weeks. Familiarise yourself with the literature surrounding this, such as this LessWrong post.

You'll need. Strong writing and communication skills.

Commitment. ~4 hrs/week for 4 weeks  ·  Write-up. Blog post