Max Narditdata & ai systems
← Articles
4 mindev

Gemini 4 Neon on SWE-sweep: 39.0% in the Antigravity CLI, 12.8% in mini-SWE-agent, and Google ran both rivals' agents itself

In the same minimal harness Neon is a point ahead of GPT-6.2 Sol; in each vendor's own agent, thirty. Most of that swing is one cell, Sol in Codex, which Google ran itself and which scores below Sol in the minimal harness.

Contents

Google’s Gemini 4 Neon announcement gives two SWE-sweep scores for one model: 12.8% with the minimal mini-SWE-agent harness and 39.0% in Google Antigravity. Google calls the second state of the art and credits the jump to what Antigravity unlocks.

Take both at face value. The 39.0% is still a score for a model plus an agent, set against other models in their own agents, and Google ran all three agents itself. I have not run Neon or SWE-sweep; the cells below are Google’s, and the ratios are my arithmetic.

What is SWE-sweep, and why are its scores so low?

SWE-sweep is a bug-hunting benchmark from Meta FAIR and university co-authors, published October 1. An agent gets a repository with many real bugs present at once and no hints about where they are, and has to find and fix as many as it can. It covers 4,068 bugs in 100 repositories and 22 languages. In the paper’s runs an agent had up to 1,000 steps and 12 hours per repository. Hidden tests from the original fixes grade each repair, and breaking the existing test suite zeroes the repository.

Finding is the hard part, so scores are small. The official leaderboard, updated November 2, runs every model in mini-SWE-agent and tops out at 8.9%. Claude Opus 5.5 at xhigh sits at 3.6%. Neither Neon nor GPT-6.2 Sol has a row there.

How much does the harness change Gemini 4 Neon’s lead on SWE-sweep?

Against Claude Opus 5.5, little. Against GPT-6.2 Sol, almost all of it. Here are Google’s six SWE-sweep cells, which its methodology reports as “subtask-weighted reward”:

SWE-sweep (Google’s table)Gemini 4 NeonClaude Opus 5.5GPT-6.2 Sol
Mini-SWE harness12.8%3.6%11.6%
Provider harness39.0% (Antigravity CLI 1.5.9)10.1% (Claude Code)8.3% (Codex)
Provider ÷ mini-SWE3.0x2.8x0.7x

Neon is 3.6 times Opus in the minimal harness and 3.9 times in the vendors’ agents. Against Sol, a 1.2-point lead becomes 30.7 points, because Sol in Codex scores below Sol in mini-SWE-agent. The thirty-point gap over GPT rests mostly on that one cell.

Who ran Claude Code and Codex for Google’s SWE-sweep comparison?

Google did. The evaluation methodology says competitor figures normally come from providers’ own reports, then makes an exception here:

SWE-sweep (provider harness) results are self-computed. For provider harness setting we use the Antigravity-CLI version 1.5.9 harness through official integration for Gemini 4 Neon, Claude Code for Opus 5.5, and Codex for Sol 6.2.

It says the mini-SWE row is self-computed too, with a “mini-swe-agent vision harness”. Opus 5.5’s 3.6% there matches the public board’s xhigh row; the PDF does not say whether that cell was rerun or carried over. Google’s default is to report rivals at their highest reasoning setting, but the SWE-sweep note names neither the effort level nor the Claude Code and Codex versions it used, and gives no cost or token count for the 39.0% run.

The note does offer one explanation for Neon’s number:

The public Antigravity-CLI harness includes a default tool that allows for sub-agents that fan out across the codebase, which is particularly suited to bug finding tasks like SWE-sweep.

That is a fair disclosure, and it is the only explanation Google gives. It cannot be the whole one: Claude Code gets nearly the same multiplier without that tool.

Did Claude Code and Codex help in the SWE-sweep paper’s own test?

Codex hurt, Claude Code barely moved. The paper ran its ablation on older models, over a set of 20 tasks and the 10 repositories with the most bugs:

Sol 5.6 (xhigh) scores 1.6% with the native Codex harness, significantly lower than with mini-SWE-agent (2.9%). Opus 5 (xhigh) scores 1.1% with the native Claude Code harness, only slightly higher than with mini-SWE-agent (1.0%).

The direction matches Google’s table in both cases. The size does not: Claude Code added 0.1 points there and 6.5 points in Google’s runs. Different models, different tasks, a different operator, so the two do not contradict each other. They do show that a harness effect is not a constant you can divide out.

Last month the asymmetry ran the other way. In the Argon table, Google’s DeepSWE score was its own mini-SWE-agent run next to rivals’ system-card figures.

Which other Gemini 4 Neon coding scores are harness-bound?

Terminal-Bench 4.1 is. Google lists Neon at 65.2% cheating-adjusted accuracy, sourced from Artificial Analysis running the Antigravity CLI 1.5.9. Opus 5.5’s 65.8% comes from the public leaderboard, which ranks model and agent pairs, here Claude Code. The top pair on that board is GitHub Copilot’s own agent at 71.2%.

The Artificial Analysis Coding Agent Index, where Google reports Neon first at 58.6, works the same way: its methodology scores agent variants, not bare models. A harness-bound score answers “which product”, and Google reports it in a table of models.

How should you read Gemini 4 Neon’s coding scores next to Claude Opus 5.5 and GPT-6.2 Sol?

Find the harness row first, then check who ran each cell. A rival’s agent run by the vendor competing with it is one interested party’s run until someone repeats it.

If you will work in the Antigravity CLI anyway, 39.0% is the number closest to your setup, provided your work looks like SWE-sweep: broad, unprompted bug hunting across a large repository. At $3 per million input tokens and $15 per million output, Google does not say what that run cost. If you are choosing a model for your own agent loop, read the mini-SWE row, where Neon leads Sol by about a point and Opus by a wide margin. The same habit applied to Carbon’s cost-per-task claim: ask which setup a board measured before you compare it with anything.

01 / Discussion

No comment section here — all discussions happen on X.

Max Nardit@mnardit

02 / More articles