Decoding the Artificial Analysis Coding Agent Index v1.1

The headline is easy to remember: GPT-5.6 Sol (max) in OpenAI’s Codex harness scored 80 on the Artificial Analysis Coding Agent Index v1.1. But a single number is also easy to misuse. Before you pick a coding agent based on an 80, a 77, or a 65, it helps to know what the index actually averages, what changed in v1.1, and what the three benchmark components are really testing.

This post walks through the v1.1 composition, the scoring mechanics, the July 2026 leaderboard snapshot, and the practical way to read the index without treating it like a magic ranking.

Why v1.1 matters

Version 1.1, published in mid-2026, made one structural swap in the public benchmark suite:

• Added: DeepSWE — long-horizon software engineering tasks.
• Removed: SWE-Bench-Pro-Hard-AA.

That swap is more significant than it looks. SWE-Bench tasks are mined from real, merged pull requests. The fixes, discussion, and test files are publicly visible, which means a model can memorize patterns instead of reasoning through them. DeepSWE writes its 113 tasks from scratch across 91 active open-source repositories and never contributes them upstream, so the reference solutions stay outside the training data. It also grades with hand-written program verifiers rather than inherited test suites, which means a correct but differently-shaped implementation still passes.

The other two components, Terminal-Bench v2 and SWE-Atlas-QnA, were already in the index. v1.1 kept them and rebalanced the suite toward original, verifier-checked implementation work.

The three components, in plain language

The index does not collapse all coding into one task type. It deliberately mixes three different skills:

*Terminal-Bench v2 originally has 89 tasks; Artificial Analysis excludes five for environment compatibility.

In short:

• DeepSWE asks: can the agent ship a working patch?
• Terminal-Bench v2 asks: can it live in a shell and finish a real workflow?
• SWE-Atlas-QnA asks: can it understand and explain a codebase deeply?

Each component uses binary outcomes. An attempt can finish cleanly and still score zero if the verifier or judge is not satisfied.

How the index is built

The headline score is not a percentage of code correctness in the abstract. It is a task-normalized average of pass@1 scores.

For each of the 321 evaluated tasks, the harness gets three attempts. For each task, the three attempts are averaged. Then those task-level scores are averaged so every task has equal weight. The overall index is a simple average of the three component scores:

That simplicity is intentional. It forces the index to reward breadth. A model that is brilliant at terminal hacking but terrible at repository Q&A cannot hide behind a single strong component.

What the July 2026 snapshot looked like

When GPT-5.6 reached general availability in early July, Artificial Analysis published the first independent v1.1 results. The model-level headline was clear:

Source: Artificial Analysis GPT-5.6 benchmark article and LLM-Stats v1.1 leaderboard, July 2026.

Sol’s lead was narrow — about 2.8 points over Fable 5 — and came with a meaningful efficiency advantage. According to Artificial Analysis, Sol (max) used less than half the output tokens and less than half the time of Fable 5 while costing roughly 40% less. The mid-tier Terra also edged Fable 5, and Luna landed within a few points at a much lower price.

But the headline is only a starting point. The chart below shows what the same composite score looks like when you split it into the three benchmark components for two top configurations. Same average, different strengths.

Behind the headline: the per-component split

Artificial Analysis publishes the composite headline first, but the real story is in the bars. The breakdown below compares two high-scoring configurations — Claude Code with Opus 5 (xhigh) and Codex with GPT-5.6 Sol (max) — on each of the three components plus the final index. Both configurations land near 67 on the current harness comparison (v1.3 scoring), yet their component profiles are almost mirror images.

Source: Artificial Analysis Claude Code vs Codex comparison (same three components, current v1.3 scoring).

How to read this chart:

• Terminal-Bench v2 is the easy high bar for both. Both agents are strong in shell-driven workflows, so this component tends to compress near the top.
• DeepSWE separates the patch-shippers. Codex Sol max scores higher here (69% vs 60%), which matches its reputation for producing working diffs.
• SWE-Atlas-QnA is the swing vote. Claude Code Opus 5 (xhigh) wins this component (55% vs 43%), meaning it is better at explaining and reasoning about a codebase than at patching it.
• The index average hides the trade-off. Both configurations round to the same headline, but one is stronger at implementation, the other at repository understanding. Pick the tool that matches the work you actually do.

What the index adds — and where it still needs interpretation

The value of the v1.1 index is that it forces a multi-dimensional comparison. Instead of asking which model is best at coding? it asks three narrower questions and then averages them. That protects against two common failures:

1. Single-benchmark overfitting. A model tuned for SWE-Bench style patch tasks may not reason well in a terminal or explain architecture clearly. The three-component average exposes that.
2. Contamination from public commits. By swapping SWE-Bench-Pro-Hard-AA for DeepSWE, v1.1 shifts the suite toward original tasks with hand-written verifiers. The score is more likely to reflect problem-solving than recall.

The index also pairs performance with cost, token usage, and execution time. That is the part that changes a budget. A 78-point agent that costs $8 per task and a 74-point agent that costs $0.31 per task are not in the same league for high-volume automation.

Still, the index has boundaries:

• It does not measure UI work, design taste, or large-scale refactoring taste.
• It does not capture team workflow: code review etiquette, documentation style, or incident response.
• It does not test security, safety, or governance under production constraints.
• Scores are tied to a specific suite version. Comparing a v1.1 number to a future v1.3 number without checking the version history is comparing different exams.

Cost, speed, and the Pareto frontier

A higher index score is not automatically the right choice. The same benchmark suite can cost anywhere from $0.31 to $11.71 per task depending on the model and harness. The scatter plot below shows selected top configurations from the current Artificial Analysis comparison.

Source: Artificial Analysis Claude Code vs Codex comparison (current v1.3 scoring).

How to read this chart:

• Lower-right is the efficiency frontier. Points like Codex GPT-5.6 Luna max ($0.31, index 59) and Codex GPT-5.6 Terra max ($2.21, index 62) give a surprising amount of capability per dollar.
• Diminishing returns live on the right. Sol max and Opus 5 xhigh both score 67, but they cost roughly $7–8 per task. The last few index points are expensive.
• Reasoning effort slides a model across the chart. Sol climbs from roughly $1.72 (low) to $7.08 (max) while gaining about 13 index points. Whether those points are worth the spend depends on your use case.
• Use this as a filter, not a contract. Your tasks, context length, retry policy, and cache hit rate will move these numbers in production.

How to read the index like an engineer

If you are choosing or benchmarking a coding agent, treat the headline index as a filter, not a verdict.

1. Look at the bars and the scatter plot, not just the headline number. A single composite cannot tell you whether an agent is better at shipping patches, navigating terminals, or explaining architecture.

2. Look at the per-component breakdown. If your workflow is mostly edit this repo and open a PR, DeepSWE matters most. If you live in CI/CD and containers, Terminal-Bench v2 is closer to your reality. If you need onboarding and architecture explanations, SWE-Atlas-QnA is the signal.

3. Hold the harness constant. The same model scores differently in Claude Code, Codex, Cursor CLI, or Grok Build. The index measures the system of model + harness + reasoning effort + routing.

4. Pair score with cost per task and wall time. A cheaper, slower agent may be fine for overnight batch work; an expensive, fast agent may be worth it for interactive development.

5. Evaluate on your own tasks last. Benchmarks tell you which candidates to audition. Your actual codebase, your tests, and your acceptance criteria decide who gets the role.

6. Watch version drift. v1.3 has already refined SWE-Atlas-QnA scoring to align with Scale AI’s Task Resolve Rate methodology. Always compare scores from the same version.

The Artificial Analysis Coding Agent Index v1.1 made the leaderboard more honest by adding original implementation tasks and keeping terminal and Q&A work in the same frame. The next time you see an 80, a 77, or a 65, you will know what it is really averaging — and what it is not.