Max Narditdata & ai systems
← Articles
9 minresearch

Gemini 4 Hydrogen in Google Search: the November 12 model swap your AI visibility report has to log

With six runs a day per surface, a before/after test catches a twelve-point move less than half the time, and even a large panel misses a five-point one a quarter of the time. Unless a power check shows your panel could see the move, whether to cut the series at the Hydrogen launch is a call the announcement makes, not your data. An event log, a split script with a rollout band, and the paragraph the report owes its readers.

Contents

On November 12, Google put Gemini 4 Hydrogen into Search. Robby Stein’s announcement calls it “the biggest model upgrade to Search yet”, and it lands in the middle of a reporting month. If you send a monthly AI visibility number for Google’s AI answers, your November most likely straddles a change of the model behind them, and a month-over-month line does not say so.

In the measurement framework I wrote that a time series should carry every known vendor release as an annotation. This is the practical version for one release. My argument is narrow: unless the panel is large and the move you care about is big, a before/after test around the 12th will usually fail to show whether Hydrogen moved your numbers, so the decision to cut the series has to come from the announcement, not from your data. The script below tells you which case you are in. I have not measured Hydrogen’s effect on anything. No Google surface was queried for this piece, and every number below comes from a simulation whose code is included.

That it is live for everyone, that query fan-out was upgraded, and that answers get more visual. It did not say which AI surface runs the model, and it gave no numbers. The sentence that matters for a time series is this one:

Hydrogen is now available globally in Google Search for all users, delivering more helpful, visual, and intuitive responses with improved model capabilities:

The two claims that bear directly on measurement:

Hydrogen’s advanced reasoning supercharges our query fan-out technique in Search, connecting you to a broader range of helpful links than before.
 
You’ll see more images, structured responses, comparison tables, and custom tools built for you.

Read it for what is missing. Neither post names AI Overviews. The only surface named in the Search post is AI Mode, and only in a sentence about language coverage (“across all supported AI Mode languages”). The DeepMind model announcement says Hydrogen “is rolling out globally”, while the Search post says “now available”. Neither post says how to tell, from an answer, which model wrote it, and Google has not said that AI Mode or AI Overviews now default to Hydrogen.

The model post touches two more surfaces a visibility panel might track. In the Gemini app, free users get Hydrogen “via auto model selection when it’s the appropriate model for the prompt”, so a panel collected through the app on auto selection is now a mix of models; log the account type, the model setting and, wherever the app reports it, the model per response. On the API, Hydrogen is a new model id, gemini-4-hydrogen. A collector that requests a fixed older version was not touched by this release, though a fixed id has never guaranteed fixed behavior.

Why does the Gemini 4 Hydrogen swap break a month-over-month AI visibility comparison?

Because a visibility rate describes the panel and the surface together, and on the 12th Google says the surface changed. A number before the 12th and a number after it may not be two readings of the same thing, and Google’s posts give no way to tell whether they are.

The fan-out claim names the mechanism. In a retrieval-based answer, the queries the system issues decide which pages get fetched, and the fetched pages decide what can be cited. “A broader range of helpful links” describes a change in exactly that step. So citations per answer, distinct domains per answer and the share of answers citing your URLs can all move with nothing on your site changing. The direction is not obvious: wider retrieval can lift your citation rate while your share of the citations shrinks. My guess is that the format change matters for mentions too, since a comparison table tends to name several products where a paragraph might name one, but that is a hypothesis to check on your own panel, not something Google said.

A November figure computed across the 12th averages two regimes, weighted by how many of your runs happened to fall on each side. Shift the collection calendar and the November number shifts with it.

How do you log the Gemini 4 Hydrogen swap in an AI visibility time series?

As data, next to the panel, before anyone looks at November. A free-text note in the report is not enough: the analysis code has to be able to read it.

date,surface,event,source,confidence
2026-11-12,google_aio,Gemini 4 Hydrogen in Search,https://blog.google/products-and-platforms/products/search/gemini-4-hydrogen-search-upgrade/ (read 2026-11-16),announced for Search; surface not named
2026-11-12,google_ai_mode,Gemini 4 Hydrogen in Search,https://blog.google/products-and-platforms/products/search/gemini-4-hydrogen-search-upgrade/ (read 2026-11-16),announced for Search; surface not named

The confidence column says how much of the row is Google’s statement and how much is my assignment. Google announced a Search-wide change on a date. Attaching it to AI Overviews and AI Mode is my assumption, because those are the Search surfaces that generate answers. Write it down that way, and if later evidence says otherwise you have one row to revise. Add rows for the Gemini app if you track it on auto selection, and none for an API series pinned to an older model.

Record the read date with the source, because vendor posts get edited: the model announcement already shows “Last updated November 13”. And because “rolling out” usually means the change did not reach every user at the same hour, keep the event as a single date and handle the uncertainty in the analysis with a band of excluded days around it, which the script below does.

How do you split a November AI visibility series at the Gemini 4 Hydrogen date?

Cut each affected surface at the event, keep the rest of the panel continuous, and put an interval on every segment. The script reads the event log above and a run record with one row per collection attempt. Only completed answers enter the mention rate; a timeout or a query where no AI answer appeared is counted as an attempt, not as “no mention”, and the attempt count is printed so a change in how often the AI answer shows up stays visible. The optional third argument is the rollout band in days.

python
# split_series.py: cut each surface's mention-rate series at logged events
import csv, math, sys
from collections import defaultdict
from datetime import date, timedelta
 
def wilson(k, n, z=1.96):
    p, d = k / n, 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return c - h, c + h
 
def z_two(k1, n1, k2, n2):
    p = (k1 + k2) / (n1 + n2) if n1 and n2 else 0
    if p in (0, 1):
        return None                   # empty segment, or no variation at all
    return (k2 / n2 - k1 / n1) / math.sqrt(p * (1 - p) * (1 / n1 + 1 / n2))
 
def main(events_file, runs_file, band_days=0):
    band = timedelta(days=band_days)
    cuts = defaultdict(list)
    for e in csv.DictReader(open(events_file)):
        cuts[e["surface"]].append(date.fromisoformat(e["date"]))
    cells = defaultdict(lambda: [0, 0, 0])  # mentions, completed, attempts
    for r in csv.DictReader(open(runs_file)):
        day, cs = date.fromisoformat(r["date"]), sorted(cuts[r["surface"]])
        if any(abs(day - c) < band for c in cs):
            continue                  # rollout band: neither before nor after
        cell = cells[(r["surface"], sum(day >= c for c in cs))]
        cell[2] += 1
        if r["status"] == "completed":  # a timeout or "no AI answer shown" is not "no mention"
            cell[0] += int(r["mentioned"])
            cell[1] += 1
    for (s, seg), (k, n, tries) in sorted(cells.items()):
        if n == 0:
            print(f"{s:<15} seg {seg}  no completed runs of {tries}")
            continue
        lo, hi = wilson(k, n)
        line = f"{s:<15} seg {seg}  {k:>3}/{n:<3} {k/n:6.1%}  [{lo:.1%}, {hi:.1%}]  of {tries} tries"
        if seg > 0:
            k0, n0, _ = cells.get((s, seg - 1), (0, 0, 0))
            z = z_two(k0, n0, k, n)
            line += "  z n/a" if z is None else f"  z {z:+.2f}"
        print(line)
 
if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2], int(sys.argv[3]) if len(sys.argv) > 3 else 0)

It treats every run as an independent trial, which real panels are not: runs of the same prompt on the same day share a lot. That makes the intervals too narrow. For production, keep prompt ids, weight prompts equally and compare the same prompts on both sides, as the framework describes. To see what a split can and cannot show, the simple version will do.

The test data is a simulated November: six runs a day on three surfaces, where only google_aio truly changes on the 12th (a 20 percent mention rate before, 32 after) and the other two stay flat. chatgpt_search has no event row, so it stays one segment and serves as the control.

python
# toy_runs.py: simulated November; only google_aio really changes on the 12th
import csv, random
from datetime import date
random.seed(7)
rates = {"google_aio": (.20, .32), "google_ai_mode": (.25, .25), "chatgpt_search": (.15, .15)}
w = csv.writer(open("runs.csv", "w", newline=""))
w.writerow(["date", "surface", "status", "mentioned"])
for d in range(1, 31):
    day = date(2026, 11, d)
    for s, (before, after) in rates.items():
        p = before if day < date(2026, 11, 12) else after
        for _ in range(6):
            w.writerow([day, s, "completed", int(random.random() < p)])
shell
python3 toy_runs.py && python3 split_series.py events.csv runs.csv
chatgpt_search  seg 0   34/180  18.9%  [13.8%, 25.2%]  of 180 tries
google_ai_mode  seg 0   25/66   37.9%  [27.1%, 49.9%]  of 66 tries
google_ai_mode  seg 1   26/114  22.8%  [16.1%, 31.3%]  of 114 tries  z -2.16
google_aio      seg 0   13/66   19.7%  [11.9%, 30.8%]  of 66 tries
google_aio      seg 1   42/114  36.8%  [28.6%, 46.0%]  of 114 tries  z +2.41

This particular draw (seed 7) happens to show both cases in one table. AI Mode’s true rate was 25 percent on both sides, yet the split shows a drop from 38 to 23 percent with a z beyond 2. The real rise on AI Overviews looks no more convincing than the false drop on AI Mode. A false alarm like that is the ordinary 1-in-20 case for a test at the 5 percent level, not a flaw in the split. The row that matters more is AI Overviews, and you only see why once you repeat the draw:

python
# power.py: how often a 66/114-run split flags a change, over 1,000 simulated Novembers
import random
from split_series import z_two
 
def share_flagged(before, after, n0=66, n1=114, draws=1000):
    hits = 0
    for seed in range(draws):
        rng = random.Random(seed)
        k0 = sum(rng.random() < before for _ in range(n0))
        k1 = sum(rng.random() < after for _ in range(n1))
        z = z_two(k0, n0, k1, n1)
        hits += z is not None and abs(z) > 1.96
    return hits / draws
 
for before, after in [(.25, .25), (.20, .32), (.20, .36), (.20, .40)]:
    print(f"{before:.0%} -> {after:.0%}: flagged in {share_flagged(before, after):.1%}")
25% -> 25%: flagged in 4.4%
20% -> 32%: flagged in 42.0%
20% -> 36%: flagged in 62.4%
20% -> 40%: flagged in 81.9%

With 66 runs before the 12th and 114 after, a twelve-point rise is flagged in fewer than half of the simulated Novembers. The rate has to roughly double, from 20 to 40 percent, before the split catches it four times out of five. Smaller real effects mostly come back as “no significant change”, which is not the same as no change. Those runs are also independent by construction, so a real panel does worse.

So run it with your own numbers before deciding: set n0 and n1 to your completed runs on each side of the 12th, before to your current rate, and after to that rate plus the smallest move you would report. The answer depends heavily on size. At 300 and 520 runs, the twelve-point rise above is flagged 97 percent of the time, but a five-point move (20 to 25 percent) only 36 percent. At about 730 and 1,270, roughly what the framework’s example of 6,000 answers a period on three surfaces leaves per surface, the five-point move is still missed about a quarter of the time. If the move you care about is flagged less than about four times in five, your data cannot settle whether the series broke, and the announcement has to.

Running with a band of 2 (python3 split_series.py events.csv runs.csv 2) drops November 11 to 13 from both sides. Compare the estimates and their intervals, not only whether a z crosses 2: a band also removes runs, so a result that flips with it may just have lost power.

Should you re-baseline Google’s AI surfaces after Gemini 4 Hydrogen if your data shows no jump?

Yes, unless the power check above says your panel could have seen the moves you care about. State it up front as a reporting policy: treat the 12th as a break for Google’s answer surfaces whether or not your data shows a jump. A test that misses the relevant move more often than it catches it cannot be the thing that decides whether the series is continuous. Without a cut, a vendor change can pass as the effect of your own work.

The cut has costs. The pre-swap segment is short, so October is compared with eleven days of November instead of thirty, and any comparison across the 12th loses power. The rule for which vendor news earns a cut has to be written down before the next one arrives; mine would be an announced model or retrieval change on a surface you track. Everything else (ChatGPT, Perplexity, API series pinned to an older model) stays continuous, because nothing in this announcement touches them.

Then look at the claims Google made, on your own panel, because they are cheap to check once you store full answers: citations per answer, distinct domains per answer, the share of answers with a table, and how often an AI answer appears at all. A large shift in those on one surface around the 12th is a hint that the swap reached it, though it cannot prove the model was the cause. A flat mention rate at these sizes tells you little about your position, in either direction.

Search Console needs the same treatment. Its generative AI performance report counts AI Overviews and AI Mode impressions together, and that series crosses the 12th as well. Annotate it with the same event row.

What should a November AI visibility report say about Gemini 4 Hydrogen?

What changed, what you did about it, and what the numbers can no longer show, in the body of the report rather than a footnote. A paragraph along these lines does it:

On November 12 Google moved Search to a new model (Gemini 4 Hydrogen) and said the new model would connect users to “a broader range of helpful links”. Google did not say which AI surfaces changed. We treat the date as a break for the Google series. November 1 to 11 is compared with October; November 12 onward starts a new baseline. The December comparison will set a full month against a partial one that includes Black Friday, so we will compare the same prompts and report the gap with its interval. ChatGPT and Perplexity are unaffected and reported as usual.

An arrow on a slide is simpler, and in January, when someone asks why Google visibility jumped in November, the arrow will have nothing to say.

What can a Gemini 4 Hydrogen annotation not tell you?

Which part of the release moved your number. Google describes a new model, a reworked fan-out step and new answer formats in one release, and the index underneath kept changing as usual. The annotation records that the surface changed on a date; it cannot separate the parts, and a cut at the announcement date is only as sharp as the rollout was.

It also covers only what was announced. Consumer surfaces change models and retrieval without a blog post, and a model is a dependency that won’t hold still: the event log catches the releases someone wrote down, and for the rest you are left reading the series itself, where a sudden move across many prompts at once is one of the few signals and gradual drift may show nothing. When I measured Beetroot’s visibility across 359 answers, the same product scored anywhere from 0 to 100 percent depending on the surface and the prompt, all on one day. A model swap behind one of those surfaces adds time to the same problem.

01 / Discussion

No comment section here — all discussions happen on X.

Max Nardit@mnardit

02 / More articles