It used to be that when you showed someone a new benchmark, you’d start off by telling them three things:

1) What the task is.
2) What solve % the top model gets right now.
3) What solve % a human would get.

Today, especially in many coding-related tasks, the era of the human baseline is over. We’re not going to be asking each other what a human gets on new benchmarks in these categories because it’ll be clear to everyone that a human, using a reasonable amount of time, would get 0% those coding benchmarks.

I think that ProgramBench was the first attempt at this new paradigm. It tasks agents with recreating an entire terminal program based just on its docs and interactions with the program itself, but with no ability to read or decompile any of its code. ProgramBench tasks include legendary programs like SQLite, FFmpeg, and the PHP interpreter.
These programs have been developed by large teams of engineers over decades. This benchmark isn’t just challenging, it’s clearly targeting a super-human ability. People initially thought it was too challenging (we got the same feedback for SWE-bench); but the rate of progress seems to imply that ProgramBench might’ve been not challenging enough.

One of the most important properties of a benchmark is how challenging it is. Benchmarks are used to direct model development: model builders iterate on training data, model architectures and optimization algorithms while checking the effects of changing those on benchmark results. Benchmarks set the north star for AI development. And, at least in coding, in many subdomains that north star should now be much beyond the abilities of any human.

SWE-bench was about solving bugs that would take a human engineer a few hours up to a few days to solve. SWE-bench has gotten pretty saturated at the frontier. We now have slightly tougher SWE-bench-like benchmarks, including DeepSWE and the TerminalBench 4 that have tasks that might take a human a week or two to solve. But those will also be saturated in the next few months. When a paradigm (SWE-bench in this case) gets saturated, that entire class of benchmarks gets saturated. It’s not enough to make a benchmark that is within the same paradigm and is 2x or 3x harder, that’ll just buy you a few more months. When a paradigm gets saturated, it’s time to move on to the next one.

And so if SWE-bench represents bugs that take a single human a few hours to days to solve, the next paradigm isn’t around tasks that would take an engineer a few weeks to solve; the next paradigm is around tasks that would take a team of engineers years or decades to solve.

Do I think that it’s a mistake to make a benchmark that represents tasks that humans can complete? No, absolutely not. Even at the frontier, even if we limit ourselves just to coding, models still lack human-level capabilities in a lot of subdomains. The vision capabilities in frontier multimodal models sometimes can’t count objects correctly and in general seem very weak (the newly relaunched SWE-bench Multimodal targets some of these capabilities; some top models solve just 40-50% of it right now).
Programming in less popular domains or using niche libraries can still be improved, and hallucination and context management are still an issue sometimes. AI seems to be automating many aspects of coding but every software engineer I know is working harder than ever. There’s still a lot of human-level stuff, in the SWE domain, that AI just can’t do. So there will be many more human-level benchmarks in the future. But I don’t think that many of the north stars for the AI community in the future will be human-level benchmarks.

Will future benchmarks be totally non-human, i.e. fully synthetic? No. SWE-bench was so obviously a good benchmark because it tasks agents with solving real bugs, from real, popular Python repositories on GitHub, that real humans solved at some point. Realness is what makes benchmarks challenging, it’s what makes them understandable both to model developers and model consumers, and it’s what makes models trained on these benchmarks useful in the real world. Being grounded in the real-world will continue to be super important in benchmarks. As benchmark developers, one of our most important tasks is to look at the real world and understand how we could arrange it in a way that would make for a good benchmark. This will continue! The best benchmarks will continue to be grounded, and will be built on top of real human data or environments or requirements.

Is it possible to build super-human benchmarks on top of real-world (human) data? Yes. The challenge is figuring out how to do it. ProgramBench takes real programs, some of which large teams of humans have built over decades, and asks agents to rebuild them in a single session. SWE-sweep takes a repo with real bugs, that were eventually all solved by many different contributors, and asks an agent to solve them all in one go. We’ll keep on finding these types of creative ways to take a human created artifact and turn it into a super-human testing/training ground.

And I think it will be super important to continue sourcing benchmarks from nature. As in, real stuff that real people have had to do to solve a real problem. If you ask a person to write a benchmark task for you, it’s probably not going to be very realistic, it’s probably not going to be very challenging, and it’s definitely not going to be at the super-human complexity level that the future paradigm requires. It’s because you get that person’s attention for a week or two of work; even if you have unlimited money you literally don’t have enough time to get a team of humans to work for a few decades to create something as complex as FFmpeg or SQLite.
Yes, there will still be a bunch of these purpose-built benchmarks as we continue to sweep away all the blemishes in sub-domains where frontier models are worse than humans; but in coding, I don’t think that these purpose-built benchmarks will not be the major north stars that direct the community.

Realness leads to scalability! A scalable benchmark is one like SWE-bench, ProgramBench or SWE-sweep, where new tasks can be automatically crawled from the internet with little to no human supervision, annotation or filtering. As mentioned previously, benchmarks set a north star for model developers to work towards; once a new north star is found we start working towards improving capabilities in that direction. And the best way to improve capabilities is to find training environments that are similar to the ones in the benchmark. If your benchmark was scalably sourced by crawling the web, your training data can be scalably sourced by crawling the web! And that makes improving your model in the direction of that benchmark much easier.

Imagine two competing model developers, one trying to hillclimb towards SWE-bench and one trying to hill climb towards TerminalBench. SWE-bench was sourced by crawling the web, so the first developer can, in a few days, crawl millions of SWE-bench-like issues from GitHub and with little manual labour have a substantial amount of working SWE-bench-like environments for use in training. TerminalBench tasks are ideated, constructed, audited by large teams of people over many weeks or months. So the second developer, that hillclimbs on TerminalBench, can’t just crawl training instances from the web. There’s not really any instances to be found on the web. This developer has to source, hire, train and supervise many teams of people to construct training instances, and not only is the cost high but the amount of time it takes is enormous and the potential for low-quality or bad instances is substantial.

In summary, scalable benchmarks come from scalable data sources which can be used for scalable training data generation which leads to improved model capabilities in a short amount of time, using little to no manual labor. This is true for human-level and superhuman-level tasks.

To conclude: in many subdomains of programming, a lot of our conversations going forward are going to shift from the past discourse of replicating human abilities, to now trying to build the future by training agents to do things humans will never be able to do. The jagged capability frontier makes this much harder for all of us to grasp. We’re going to have amazing capabilities in some narrow subdomains along with really bad performance in other subdomains, just within software engineering. Some of us will work on pushing performance on certain tasks from superhuman to supersuperhuman, while others will work on pushing performance on other tasks just to human-level. I wanted to write this blogpost to help me and others process this new beginning, this science-fiction role of us having to push forward a machine which can do some things much better than any of us can.

To push the AI frontier and make AI more useful for humans, we need to understand and then formalize the gap between ‘what agents can do’ and ‘what humans need’ via benchmark building. This is becoming harder than ever, but there’s definitely still a gap there so there’s work for us to do.

This is an amazing, stressful, confusing and inspiring time but not an unprecedented time. Horses can carry more weight than you or I, for longer distances, electric grids and radio networks transmit energy or information faster than any human could, and vaccines protect us in a way no human ever could. We’ll figure this out too.

My new coding benchmark checklist:

1. Is the task easy to explain? It should be explainable to a highschooler who has completed 1 programming class, in a short simple sentence. One of the goals of a benchmark is to explain to the wider community where models are in terms of capabilities; if your benchmark itself is too hard to explain it’ll be hard to use it to explain models. Beyond just the task, everything in the benchmark should be straightforward to explain, almost just obvious. Including the scoring/grading/metric, what the starting environment for each task is and what the directive given to the model is. The more complicated your benchmark is the harder it is to get it to go viral. Of course, the challenge here is to build a benchmark that is easy to explain to humans but hard for models to solve.

2. Minimize the benchmark-reality gap. How far away are the tasks in your benchmark from tasks that humans do in their day-to-day life or job? Would an agent that could do this task autonomously or semi-autonomously be useful to a large group of people? If you’re too niche that could lead to a harder time convincing people that your benchmark is useful. But if you’re too general the tasks will probably end up being too easy. Finding that balance is hard.

3. Is your benchmark currently hard for frontier models? Top accuracy should be <5%. There’s also a danger of making your benchmark too hard, but 99% of the time the mistake that people make is that they make their benchmark too easy.

4. Are the tasks robustly verifiable by a computer program? Don’t use LM-as-a-judge, human raters, or any type of subjective scoring. Benchmarks that can be run quickly and without human intervention enable model builders to iterate quickly.

5. Do you have enough tasks? This is a tricky one to define, but you should aim for at least 100 samples. This is extremely case-dependent but you should also make sure not to have too many benchmark samples, because then the benchmark will be too slow to run. As our benchmarks become more complex, the number of samples in them goes down. MMLU had more than 10k multiple choice questions, SWE-bench Verified had 500 programming tasks, ProgramBench had 200 tasks, SWE-sweep has 100 repos (which have 4k bugs within them), and the actual toughest current benchmark, the Millennium Prize Problems, only has 5 open tasks :)

6. Is test/train data creation for your benchmark scalable? Good benchmarks make model developers want to iterate on those abilities, by building pre-training and post-training environments and data. If your benchmark data creation pipeline is manual, that will make hill climbing your benchmark harder. One of the benefits of SWE-bench, ProgramBench and SWE-sweep is that the data pipelines for creating benchmark tasks are fully or almost-fully autonomous, meaning that creating training data for these benchmarks is also fully or almost-fully autonomous, enabling model builders to quickly hill climb these abilities.

7. The environment for running the benchmark is cheap and widely reproducible. If your benchmark requires 9 hours of CPU time that’s great. If it requires 1000 hours of raw hardware access to an AMD EPYC 9005 CPU, that makes reproducibility much harder. Of course if you limit yourself to just building benchmarks that can be ran for 50 cents per run, you’re not gonna get very far, but make sure your benchmarking environment is reproducible.