Research Engineer, AI Cyber Evaluations

The team

Our client is a well-funded frontier AI security lab in San Francisco. They build the benchmarks, evaluation environments and training data that frontier AI labs use to measure and improve their models on cyber tasks, under contract to several of those labs.

Demand has outgrown the team, so they are making multiple hires over the coming months. The team is small, very sharp and highly technical. Engineers get full agency to propose and run their own projects, with no micromanagement. Deep cybersecurity expertise is not required: curiosity and the drive to go deep matter more.

What you will do

• Design and build cyber evaluation environments and benchmarks for frontier models: from CTF-style exploitation challenges to realistic scenarios in specific systems such as cloud infrastructure, malware analysis, incident response or industrial control
• Generate high-quality training data and RL environments that help labs improve their models' defensive cyber capabilities
• Build and improve the agent frameworks and scaffolding that let models perform at their best on long, multi-step tasks
• Dig deep into an unfamiliar environment, understand exactly how it works, and turn that understanding into rigorous, well-specified tasks
• Own the pipeline end to end: from APIs that serve frontier labs to analysis tools that turn 10,000+ agent trajectories into clear conclusions
• Work with expert red teamers and security specialists to ground tasks in real adversarial practice
• Publish results from time to time and present findings directly to lab partners

What we're looking for

• Hands-on experience building benchmarks, evals, RL environments or training data: at an evaluations or AI data company, or on benchmarking work at a frontier lab
• Exceptionally sharp, with strong analytical and problem-solving skills and a track record that shows it
• Deeply curious: you use AI agents heavily, but you understand exactly what they are doing, can explain it, and know how to steer them
• Strong production programming skills (Python), and the speed to go from prototype to reliable infrastructure
• Comfortable going down the stack in a domain you have never touched before
• Security experience is a plus, not a requirement
• Able to work onsite in San Francisco five days a week

Total compensation $250k to $500k+ including equity, depending on level. The process is two to three short conversations with the founders, then an in-person work trial.