I ran a simple check using the ten highest-scoring models on the current Artificial Analysis Intelligence Index v4.1.1.

AA combines nine benchmarks. Most have publicly accessible test data, except for AA-Omniscience: only a representative 600-question sample is public; the remaining 5,400 questions are held private. Omniscience measures factual accuracy and whether a model avoids answering when it does not know.

For each model, I collected:

  1. Its overall AA score.
  2. Its AA-Omniscience Index.
  3. Its AA score after completely removing Omniscience and renormalizing the remaining 88%. Omniscience accounts for 12% of the original AA score, leaving 88% for the other eight benchmarks.

Using Omniscience accuracy (A) and Omniscience hallucination rate (H), both in percentage points:

\[\text{AA without Omniscience} = \frac{\text{AA}-0.08A-0.04(100-H)}{0.88} \tag{1}\]

Next, I examined whether each model’s Omniscience score was unusually low relative to its adjusted AA score.

I treated AA without Omniscience as (x) and the Omniscience Index as (y). Using the ten models, I fitted an ordinary least-squares regression line, meaning the line that minimizes the total squared difference between predicted and actual Omniscience scores.

The regression slope and intercept are:

\[b = \frac{\sum_i(x_i-\bar{x})(y_i-\bar{y})} {\sum_i(x_i-\bar{x})^2} = 2.7906 \tag{2}\] \[a=\bar{y}-b\bar{x}=-146.05 \tag{3}\]

Therefore:

\[\text{Predicted Omniscience} = -146.05 + 2.7906 \times \text{AA without Omniscience} \tag{4}\]

Finally:

\[\text{Anomaly} = \text{predicted Omniscience} - \text{actual Omniscience} \tag{5}\]

A positive anomaly (bad) means that the model’s Omniscience score is lower than expected from its performance on the rest of AA. A larger positive value indicates a larger downward deviation.

AA rank (↓) Model Vendor AA score (↑) AA without Omniscience (↑) Omniscience Index (↑) Anomaly (↓)
1 Claude Opus 5 — max Anthropic 63.05 64.34 37.07 −3.58
2 Claude Opus 5 — xhigh Anthropic 62.52 63.80 35.38 −3.39
3 Claude Fable 5 — max, fallback Anthropic 62.07 62.94 43.30 −13.71
4 Claude Opus 5 — high Anthropic 61.48 62.74 33.72 −4.69
5 GPT-5.6 Sol — max OpenAI 60.93 63.48 21.97 +9.12
6 Grok 4.6 — high xAI 60.92 61.86 30.48 −3.91
7 Kimi K3 — max Moonshot AI 59.70 61.39 19.70 +5.56
8 GLM-5.3 — max Z.ai 59.51 61.35 14.30 +10.85
9 Qwen3.8 2.4T A95B Alibaba 57.70 59.97 4.32 +16.98
10 Muse Spark 1.2 — xhigh Meta 56.76 57.34 27.20 −13.24

The largest anomalies are Qwen3.8, GLM-5.3, GPT-5.6 Sol and Kimi K3. These models perform strongly on the rest of the AA suite but considerably worse than expected on the mostly private AA-Omniscience question set. Qwen3.8 2.4T A95B performed exceptionally worse on AA-Omniscience than its overall benchmark performance would predict, receiving an Omniscience Index of just 4.32 and the largest anomaly, +16.98.

This is not proof of benchmark contamination. Differences in capabilities, refusal behavior and hallucination calibration can also explain the gaps. The analysis also includes only ten models, so the regression is sensitive to this particular sample.

Still, models with large positive anomalies may be a bit suspicious and deserve closer investigation, especially when much of their apparent strength comes from benchmarks with public test sets.