From solving word problems to proof juries: five years of AI mathematics

Mathematics has long been a malevolent mirror for language models. The model could write a persuasive essay on number theory, and then immediately mess up division, mistake a necessary condition for a sufficient one, or contradict himself in the last line. His tongue sounded smarter than his calculation.

But in just five years, almost everything has changed: tasks, metrics, models, and the idea of what "AI has solved a mathematical problem" actually means. In 2021, we measured whether one model hits the short answer on the first try. In 2026, we're looking at systems that create a population of proofs, critique each other, fix them, run them through Python or Lean, and only then select a winner in a tournament.

This text is a technical history of that transformation. Not a ranking of brands, but a map of principles: which systems have won in each period, why they have won, and which seemingly smart ideas have repeatedly failed. In the end, I will translate it into a math mode design for our HyperFusion and experiments on LUMI.

Main takeaway: the biggest jumps didn't just come from the bigger model. They were brought about by new ways of using the calculation: detailed procedure, diversity of candidates, voting, tools, verification, targeted correction and finally the orchestration of the entire proof process.

First, one inconvenience: the results are not directly comparable

The number "90% in math" can mean four very different things:

• the model generated the correct number once;
• the correct answer appeared at least once among hundreds of trials;
• most of the many solutions agreed on the correct result;
• the formal proof passed the kernel validator without gaps.

This includes model size, access to tools, number of samples, context length, time limit, test hardware, data public, and sometimes human formalization of the assignment. Therefore, I will distinguish at least the following metrics for each result:

This distinction is more important than the order in the table. If a system with 2000 samples beats a model at pass@1, it does not automatically mean a better base model. It means that it had a more efficient combination of generator, budget and selector.

2021: MATH and GSM8K reveal that fluency is not reasoning

MATH: competitive problems like a cold shower

The MATH benchmark yielded 12,500 competitive high school problems in algebra, geometry, number theory, probability, and other areas. Each had not only an answer, but also a step-by-step solution. The big transformers of the time only reached units of a percent; the authors reported a range of roughly 2.9 to 6.9%.

That was the key moment. On common NLP benchmarks, scaling worked almost like a one-size-fits-all recipe. MATH has shown that a larger language model can better mimic the form of mathematical text without reliably maintaining a long chain of dependencies. The error in the first guess turned into an elegantly planted nonsense in the sixth step.

In addition, the benchmark later had a major cultural impact: it established a common goal against which training on mathematical data, prompting, voting, and tools could be measured. At the same time, however, it gradually became saturated. Once a test becomes a training compass for an entire field, it ceases to be a neutral landscape.

GSM8K: simple arithmetic, heavy language structure

In the same year, GSM8K: 8,500 word problems at the elementary school level, typically with two to eight steps, was created. It wasn't a difficult theory. It was about translating the story into the correct sequence of operations.

It was at GSM8K that one of the most enduring principles of the entire history appeared: it is easier to generate than to choose reliably. OpenAI had the model produce many candidate solutions and trained a verifier to rank them. Improving the verifier could have had a similar effect to dramatically increasing the generator.

But even then a future problem became apparent: with the growing number of candidates, solutions appear that are faulty but can fool the verifier. More samples only help until the selector recognizes quality faster than the generator produces sophisticated errors.

2022: model learned to "show work"

Chain of Thought: intermediate results like working memory

The work Chain-of-Thought Prompting showed that a few examples with a detailed procedure can prompt a significantly better multi-step solution for a large model. The PaLM 540B with eight samples achieved the then top result at GSM8K.

Why did it work? Not because the textual procedure is automatically evidence. Generated tokens created an external desktop. The model did not have to hold all intermediate results in one hidden calculation; he could condition on them in the next steps.

But CoT has developed a dangerous habit: mistaking a convincing-sounding procedure for a faithful record of internal reasoning. A language chain can be a rationalization. The correct number can be reached by the wrong path or an error can occur only in the last transcription. CoT is a powerful candidate generator, not a certificate.

BREAK 1: EIGHT EXAMPLES CHANGED THE METHOD OF CALCULATION

Chain of thought: the model received several solved examples and started writing the intermediate results to its own output. The text became external working memory. It was a move from a one-off tip to an auditable trajectory, not yet proof.

Self-consistency: one path is a sample, not a verdict

Self-consistency replaced greedy decoding with a simple idea: sample multiple different processes and marginalize them over the final result. On GSM8K, it improved the original work by 17.9 percentage points, on SVAMP by 11 and on AQuA by 12.2 points.

The algorithm is almost trivial:

solutions = sample(model, problem, n=64, temperature>0)
answers   = normalize_final_answers(solutions)
winner    = most_frequent(answers)
return best_explanation_among(solutions_with(winner))

Its power lies in the assumption that the correct answer has many independent paths, while the errors are scattered. But if all samples share the same false stereotype, majority voting will only amplify the error. Text diversity is not the same as error diversity.

BREAKTHROUGH 2: +17.9 PERCENTAGE POINTS AT GSM8K

Self-consistency: sample different procedures, extract the final answer from each, convert equivalent notations to the same form, and choose the most frequent result. The same principle added 11 points on SVAMP and 12.2 points on AQuA.

Minerva: data specialization plus test-time compute

Minerva followed up on PaLM and continued training on tens of billions of technical content tokens, including math websites and arXiv. Minerva 540B scored 50.3% on MATH with majority voting. In 2021, such a number would seem absurd.

A combination of three things won:

1. strong base model;
2. domain-rich continuation pretraining;
3. tens to hundreds of independent solutions during inference.

This is the first clear historical pattern: training determines the distribution of available strategies, inference decides how many of them we search.

2022 to 2024: The calculator is back in the room

A language model is good at designing a procedure, but wasting its capacity for exact multiplication is akin to not allowing a mathematician to use paper. The next generation of systems therefore separated planning from execution.

PAL: the model writes the program, the runtime calculates

Program-Aided Language Models had LLM generate the program and entrusted the final result to Python. PAL with Codex on GSM8K outperformed the much larger PaLM with chain of thought by 15 percentage points.

The division of labor is elegant:

• the model recognizes entities and relationships;
• creates a symbolic or program procedure;
• the deterministic runtime will perform the arithmetic;
• the system returns the result as well as an auditable trace of the calculation.

This does not make all errors disappear. The model can translate the input incorrectly or write a program that correctly calculates the wrong quantity. However, the tool removes the entire class of numeric crosstalks and enables automatic tests.

BREAKTHROUGH 3: +15 PERCENTAGE POINTS ON GSM8K

PAL: the model no longer counted each number token by token. He translated the relations into a program, Python did the arithmetic, and the result was returned in the answer. The division of roles beat PaLM by an order of magnitude with a pure chain of thought.

ToRA: alternating text and tools

ToRA doesn't just use the tool at the end. The model alternates natural language with calls to Python or a symbolic solver, observes the result, and continues. ToRA-7B scored 44.6% on MATH and the ToRA-Code-34B variant exceeded 50%.

Architecturally, this is more important than the number itself. The solution changed from a monologue to a closed loop:

plán → kód → výsledek nástroje → revize plánu → další kód → odpověď

This loop is the forerunner of today's agents. The difference is only in the number of roles and whether the correction is performed by the same model, another model, or a formal system.

DeepSeekMath: data, GRPO and a small model that is no longer small in capabilities

DeepSeekMath 7B continued pretraining on 120 billion math tokens and implemented Group Relative Policy Optimization, later known as GRPO. Without tools and voting, it achieved 51.7% on MATH, with a self-consistency of 64 samples of 60.9%.

While Minerva showed the power of a giant specialized model, DeepSeekMath showed that a high-quality mathematical corpus, targeted post-training and test-time sampling can take a 7B model into the realm of hundreds of billions of parameters until recently.

BREAKTHROUGH 4: +9.2 PERCENTAGE POINTS THROUGH 64 ATTEMPTS

DeepSeekMath: the same 7B checkpoint had 51.7% on pass@1 and 60.9% with self-consistency over 64 samples. The difference shows exactly how much ability was already in the generator, but one attempt could not reliably extract it.

Benchmark defends: robustness, multimodality and really hard questions

As MATH and GSM8K became too well known, tests were created to attack their weaknesses.

OlympiadBench: text is not enough

OlympiadBench contains 8,476 bilingual multimodal math and physics problems. The diagram is no longer a decoration; carries part of the information. At launch, GPT-4V scored 17.97% overall. The benchmark separated the ability to read the image, formalize the situation and perform the calculation.

GSM-Symbolic: change the numbers and see what remains

GSM-Symbolic generates symbolic variants of the same templates. So the model cannot benefit only from the specific wording of a known question. The authors showed significant variance between numerical instances and large dips after adding irrelevant information.

This is a caveat for any leaderboard: a high average score does not necessarily mean an invariant algorithm. A model may have learned local heuristics that work on a typical data surface but break down under small transformation.

FrontierMath: returning a question that the model probably didn't see

FrontierMath was created as a collection of new, expert-created problems from higher university to research level. At launch in 2024, the leading models were around two percent at most. Only a smooth imitation of the Olympic style did not help here.

FrontierMath also requires governance scrutiny. Epoch AI later explained that some questions had been commissioned for OpenAI and the company had access to all but the holdout. This does not invalidate the benchmark, but it changes how the score should be interpreted. Data provenance and access are part of the metric.

2024: AIMO 1 and the victory of the open 7B system

The first AI Mathematical Olympiad Progress Prize attracted more than a thousand teams. Project Numina won with a system built on DeepSeekMath-Base 7B. In the public NuminaMath solution, the advantage did not come from one model call, but from a carefully designed data and inference pipeline.

Why the contestants didn't use the frontier API

The rules were deliberately unlike a normal chatbot benchmark. Evaluation used a hidden set of 50 AMC 12 and AIME-like problems, each with an integer answer. Only public open-weight models released before the cutoff date were allowed. A submission ran as an offline Kaggle notebook with no internet or external APIs, on one P100 or two T4 GPUs, for at most nine hours. This explains the 7B checkpoint: the competition measured the entire local system under a fixed compute budget, not who could call the most expensive closed model. The winning solution's technical report documents the constraints and recipe.

Numina collected approximately one million mathematical problems, cleaned and annotated the solutions, divided the training into classical chain of thought and tool-integrated reasoning, and used SC-TIR for inference. The public description lists approximately 860,000 CoT examples and 70,000 TIR examples. The final DeepSeekMath-Base 7B was fully fine-tuned first to CoT, then to TIR, and ran as an 8-bit AutoGPTQ variant.

Replicable winning recipe: 48 candidates times 4 cycles

For each problem, the system generated N = 48 candidates. When a candidate opened a Python block, generation paused, the code ran, and stdout or the traceback was appended to the context. This could repeat for at most M = 4 rounds. Incomplete trajectories were discarded. Final integers were extracted from the remaining outputs, and the most frequent normalized answer won. Larger N or M no longer helped within the nine-hour limit.

for candidate in 48_parallel_samples:
    repeat at most 4 times:
        generate until Python block or final answer
        if Python block: execute and append stdout or traceback

answers = normalize(extract_integer(completed_candidates))
return most_frequent(answers)

Normalization is not an LLM judge. It is a deterministic function that extracts the required integer, removes a wrapper such as \boxed{...}, unifies equivalent notations, and rejects an invalid format. The frequencies are then counted. Candidates do not vote according to the length or persuasiveness of the reasoning. Each valid result adds one vote to its number.

BREAKTHROUGH 5: 48 CANDIDATES × 4 PYTHON CYCLES

Numina SC-TIR: open 7B model, 8-bit quantization, 48 trajectories, at most four returns from Python, and plain majority of normalized numbers. The result was 29 correct out of 50.

The original schematic of the winning Numina team. Source: project-numina/aimo-progress-prize, Apache License 2.0.

2025: AIMO 2 and the moment when the simple majority was no longer enough

The second Progress Prize was won by NVIDIA NemoSkills with a score of 34/50. The system is described by the paper OpenMathReasoning and the public repository NeMo Skills.

The rules were again local and offline. The private test had 50 problems; the notebook received four L4 GPUs and five hours. The winning checkpoint started from Qwen2.5-14B-Base. It trained for eight epochs on 2.2 million DeepSeek-R1 CoT solutions, followed by a lightweight TIR fine-tune on 15,000 examples for 400 steps at learning rate 1e-5. The authors then linearly merged checkpoints in the ratio 0.3 × CoT + 0.7 × TIR.

Winning inference used FP8, ReDrafter speculative decoding, and an almost-greedy temperature of 0. The model could produce up to 16 candidates, but the system stopped once the first four or five completed answers agreed. Each problem received a base budget of 350 seconds; unused time moved into a reserve, and a hard problem could receive up to 560 seconds. Python was allowed at most six times per trajectory, with a two-second timeout and only the first 200 output characters returned to the model.

BREAKTHROUGH 6: 34 OF 50 UNDER THE FIVE-HOUR LIMIT

AIMO 2 winning run: 14B checkpoint composed at 0.3 CoT and 0.7 TIR, FP8, ReDrafter, up to 16 near-greedy candidates and an early exit when matching four to five answers. The key was not another giant model, but working with time and quickly finding a stable majority.

Important correction: GenSelect was not on a winning run

Indeed, OpenMathReasoning has developed GenSelect, which is given a problem and several complete solutions and generatively selects the most promising candidate. However, the winning AIMO 2 submission did not use it due to the time limit. A previous version of this article combined the two parts of the paper, which was inaccurate.

In the GenSelect research protocol, the system started with 64 generated solutions. It repeatedly selected from subsets of 16 candidates, ran that selection 64 times, and then majority-voted over the normalized answers of the selected solutions. It became unstable beyond 32 generations because not all trajectories fit in one prompt. This is a valuable research direction, but it is not the recipe that won AIMO 2.

majority:  argmax_a  sum_i 1[normalize(answer_i) = a]
GenSelect: repeat 64 times select_one(problem, subset_of_16_solutions)
           then vote over normalized answers of selected solutions

A later joint AIMO evaluation showed how strongly results grow with budget. A combination of public systems reached 47/50 with a very large number of attempts. That is impressive, but it is also pass@very-high-k. Compute cost and the selection mechanism belong in the headline alongside the score.

Rules of three AIMO competitions in one table

Therefore, the largest available frontier model does not automatically win in AIMO. The rules remove the Internet and proprietary APIs, limit hardware and time, and require a locally controllable artifact. The competition thus measures the model, quantization, serving, working with instruments, time management and selection as a whole.

Parallel branch: from the answer to the proof accepted by the kernel

Classic benchmarks often check the final number. But mathematics is not the only answer. In the proof, it is important that each step follows from the previous statements.

miniF2F and PutnamBench

miniF2F yielded 488 formal assertions in systems such as Lean, Metamath, and Isabelle. PutnamBench expanded the ambition to 1,692 formalizations of 640 undergraduate competition theorems.

Formal evidence changes the referee. A small trusted kernel replaces the scoring model. The proof either fits typologically and logically, or it doesn't. This removes a large part of the aesthetic bias, but comes with another price: converting the natural input into a formal language, searching for library lemmas, and a huge space of possible tactics.

AlphaProof: silver on IMO, but with manual formalization

In 2024, AlphaProof and AlphaGeometry 2 together scored 28 out of 42 points at the International Mathematical Olympiad, the silver medal level. They solved four out of six tasks.

AlphaProof used Lean and reinforcement learning to search for formal proofs. However, the assignments had to be manually converted into a formal language, and some jobs took several days to be processed by the system. So the result was not "a chatbot solved IMO in a normal conversation", but a triumph of a combination of formalization, search and verification.

AlphaGeometry 2, on the other hand, benefited from a narrow domain and a symbolic geometry engine. That's another history lesson: a specialized solver with hard invariances can beat a more general system in a narrow class.

DeepSeek-Prover and Goedel-Prover: compiler as teacher

DeepSeek-Prover-V2 worked with recursive decomposition into subgoals and reinforcement learning. The authors reported 88.9% on miniF2F and 49 solved problems out of 658 in PutnamBench, but with the 671B model and with a specific sampling budget.

Goedel-Prover-V2 showed how quickly efficiency shifted. His 8B variant reports 84.6% on miniF2F at pass@32; 32B with self-correction 90.4%. The fix uses Lean compiler feedback: the model doesn't get a vague "try again", but a specific location and type of error.

BREAKTHROUGH 7: 90.4% WITH COMPILER IN CONTROL LOOP

Goedel-Prover-V2 32B: will design a formal proof, Lean will return an accurate diagnosis, and the model will correct the localized problem. Kernel doesn't rate style or confidence. The proof either passes or it doesn't.

The fundamental principle is universal:

navrhni důkaz → spusť ověřovač → přečti diagnostiku → oprav lokální chybu → znovu ověř

This is much more reliable than asking to "check your answer" because the criticism comes from a different mechanism than generation.

2025: natural language gold at IMO

Google DeepMind announced that advanced Gemini Deep Think has reached IMO 2025 35 points out of 42, the gold medal level. He solved five out of six problems, the solutions were officially evaluated and were created in natural language within the competition limit of 4.5 hours.

Compared to AlphaProof 2024, the qualitative difference is huge: no manual input formalization and no multi-day search. Deep Think uses parallel reasoning. Again, it is not about one perfectly straight track, but about searching multiple paths and combining them.

However, it is fair to say that it is a closed system. We do not know all the details of the training, the number of internal candidates or the exact selector. The result is official, the reproducibility of the architecture is not.

2026: AIMO 3 and the end of the illusion that only a model competes

AIMO 3 brought 110 new problems from National Olympiad to IMO level, from algebra, combinatorics, geometry and number theory. The answer had five digits. Competitors worked in an offline laptop with the H100 and could build on open scales such as gpt-oss-120b or Qwen3-Next.

The announcement of winners lists Exalted Joseph, varianceofx, SKobayak, TAMU-TACO and yemao ye among the top honorees. It highlighted two hallmarks of the new era: a very strong common base model with high variance and shared notebooks that quickly spread the best harnesses between teams. The leading results have clustered. The difference has moved from "I have a secret model" to "I can consistently extract good candidates from the model and not let them drop."

For the first place, there is not one compact recipe with a safely quotable number of candidates in the official summary. I don't want to imagine him. But the second-place public system is exactly replicable varianceofx: local gpt-oss-120b via vLLM, stateful Jupyter Python sandbox, up to eight parallel trials, and a combination of simple majority with entropy-weighted scoring. This is more useful than the vague claim that "the 120B model won".

BREAKTHROUGH 8: UP TO 8 TRIALS AND ENTROPICALLY WEIGHTED VOICE

AIMO 3 2nd Place Public Recipe: gpt-oss-120b generates up to eight solutions with access to stateful Python. The agreement of the answers is the main signal, but the certainty of the tokens adjusts their weight. So a candidate is not stronger just because he is tall or stylistically sleek.

AIMO 3 thus confirmed that it is necessary to publish the entire experimental contract:

• checkpoint and quantization;
• sampling parameters;
• number and length of attempts;
• response normalization;
• recovery in case of incomplete reasoning output;
• timeouts and hardware usage;
• selection algorithm.

Without it, the score is just the result of an invisible mix of model and infrastructure.

Proof Pilot: 25 calls of one 32B model as a proof organization

After AIMO 3, the Proof Pilot aimed at human readable proofs was launched. Yi-Chia Chen's winning open solution is extremely instructive precisely because it does not rely on a giant proprietary model. The public repository and deploy bundle describe a model derived from OLMo 3.1 32B Think, with a DeepSeek-V4 tokenizer, distillation from DeepSeek-V4-Flash, and a quantized service.

But the most important thing is the inference. One checkpoint switches roles with a prompt:

1. six provers will produce independent proofs;
2. each proposal receives two verification reviews, i.e. twelve critics;
3. three refiner passes refine the most promising candidates;
4. four selectors vote on the winner.

The 6 / 2 / 3 / 4 configuration means a total of 25 model calls per task. In addition, the published harness has a watchdog, working with a very long context, saving an unfinished <think> block, and fallbacks. These are not the operational details below the line. In the reasoning model, they are part of the algorithm.

BREAKTHROUGH 9: 25 CALLS DIVIDED INTO FOUR PROFESSIONS

Proof Pilot: 6 independent proofs, 12 critics, 3 targeted fixes, and 4 selection votes. It is not only the number of samples that is important, but the order of the roles and the information they share with each other.

The model bundle reports an IMO-ProofBench v2 average score of 4.48/7with the Claude cross-check and 3.808/7with the DeepSeek grader; teacher's DeepSeek-V4-Flash is listed at 4.83/7. The difference between the two graders is itself a warning: in natural evidence, the judge is not just a measuring device, but another model with its own error.

Technically, the difference between a teacher and a student is also interesting. The model card lists around 18.74GB for the GPTQ variant versus around 65GB in BF16. So the capability was not only transmitted through hard answers, but through multi-phase distillation and soft targets. But with compression, the authors mention tendencies towards repetition and loops. It is precisely this type of degradation that a multi-agent system can multiply if it does not have a watchdog.

What really won in history

The table is not one leaderboard. Each row has a different dataset and budget. However, it shows a consistent trend: systems that turn redundant computation into controlled diversity and quality selection win.

Which, on the contrary, repeatedly did not work

1. "Let the model think longer" without checking

Longer reasoning can bring correction, but also rumination, looping and self-reinforcement. A particularly quantized model can produce thousands of tokens of the same error over and over again. A budget without verification is not a guarantee of quality.

2. More samples without oracle gap measurement

If oracle@64 = 90 % but the selector reaches 65%, the problem is not in the generator. Additional samples can overwhelm the selector even more. First of all, it is necessary to analyze why the correct solution does not win.

3. Majority among models with the same error

Three checkpoints from the same family, trained on similar data and prompted by the same prompt are not three independent voices. They are three measurements of the same bias. For fusion, sometimes a weaker but different solver is more valuable than another copy of the strongest one.

4. The generative judge as the only source of truth

An LLM judge easily mistakes length, confidence or familiar style for correctness. May prefer same family response. Where there is Python, SymPy, unit test or Lean, the hard signal must take precedence over the aesthetic judgment.

5. A synthesis that will "enhance" the correct foundation to unrecognizability

In internal HyperFusion tests on DRACO, we saw an extreme failure of the old generative fusion: a GPT-5.5 response rated at 88.7% dropped to 13.9% after aggressive compression, leaving 295 characters out of about 27,000 characters. The synthesizer didn't make a tiny mistake; destroyed evidentiary content.

The newer v3augment strategy first selects the strongest basis, preserves it, and only purposefully adds non-conflicting knowledge. It averaged 84.9% across the set, while the best single raw response was 81.6%. For mathematics, a stricter rule follows from this: choose and verify; don't overwrite unless you have a localized bug.

6. Benchmark without data provenance

The result without information about the date, dataset version, access to the test and method of evaluation ages badly. For mathematical benchmarks, contamination is particularly dangerous: a familiar task can look like a generalization even though it is a reconstruction of a memorized template.

How would I build HyperFusion Math from it

The current HyperFusion uses a panel of models, an anonymized judge, and a synthesizer. For general searches, select-and-augment is reasonable. However, for the exact math, I would add a separate mode with confidence rank:

1. deterministic test: calculation, substitution, numerical control, invariance;
2. formal test: if the task is formalizable and the library is available;
3. independent criticism: searching for errors in premise and completeness;
4. weighted match: only as an additional signal;
5. style and clarity: the last filter, not the first.

For example, a candidate's score may take the form:

score(c) = α · V_exec(c)
         + β · V_formal(c)
         + γ · log P_selector(c)
         + δ · Σ_i w_i · 1[answer_i = answer_c]
         - λ · risk(c)

Where:

• V_exec is the result of a code, substitution, or test;
• V_formal is acceptance by a formal verifier;
• P_selector is the probability of the generative selector;
• w_i is the solver weight estimated on the training part;
• risk penalizes a parse fail, an incomplete reasoning, an unsubstantiated step, or a contradiction between the proof and the answer.

Coefficients may not be created on the final test. It is necessary to divide the dataset by task families, not randomly by rows, otherwise almost the same templates will get into training and test.

The experiment I would run next on LUMI

If we have the answers of several local models stored, we do not need to spend additional inferences first. We can measure offline which combinations even have potential.

Phase A: no judge, only from existing answers

For each model and save mode:

• normalized final answer;
• correctness;
• parse fail and truncate;
• length, time, tokens and memory;
• model family and mode identifier think/nothink.

Then for all pairs, triples and quadruplets, calculate:

1. individual pass@1;
2. majorities/weightedvote;
3. oracle@k: whether at least one member was right;
4. disagreement rate;
5. pairwise correlation of errors;
6. performance on categories and difficulties;
7. price for another percentage point.

The combination with the highest sum of individual scores is not the most interesting. It is a combination with a high oracle ceiling and low error correlation.

Phase B: selector training without data leakage

I would suggest the division as follows:

If the dataset is small, I would use nested cross-validation by job types. For the 60 items of GPQA-Diamond type, the uncertainty interval is wide; a difference of a few points can be a single question. For mathematical benchmarks, a significantly larger holdout is ideal.

Phase C: expensive judge only where it has informational value

I wouldn't always call Judge. I would only run it if:

• surviving candidates give different answers;
• hard tests did not decide;
• oracle analysis shows that the correct solution often exists, but simple voting cannot find it.

This creates a cascade: easy questions are solved by cheap consensus, computational Python tasks, formal Lean tasks, and only real conflicts get an expensive jury.

The specific first panel

For the first experiment, I would keep our proposed trio panel Gemma-4-31B think + Qwen-27B nothink + GLM-5.2 Q4 nothink because it combines different modes and families. But I wouldn't automatically name Gemma a judge just because she has the best individual score. First, I would measure on historical responses:

• how many times Gemma corrects the dispute correctly;
• how many times he prefers his own style;
• what is the difference between its selection accuracy and simple majority;
• whether a smaller specialized verifier will not bring the same result cheaper.

The strongest solver and the best judge are two different roles. The history of GSM8K, GenSelect and Proof Pilot is surprisingly consistent in this.

What to watch instead of a single number

For each experiment I would publish at least this vector:

pass@1
oracle@k
maj@k
selector@k
formal_or_exec_pass
mean_tokens
p95_latency
peak_memory_gb
parse_fail_rate
cost_per_correct

It can be used to diagnose where the system loses:

• low oracle@k → lack capable generator or diversity;
• high oracle@k, low selector@k→ judge fails;
• high selector@k, low formal_or_exec_pass→ judge appreciates plausibility;
• high score, high parse_fail_rate → pipeline is laboratory strong, operationally fragile;
• small profit for many times higher tokens → test-time compute already has a diminishing return.

What about research from 2026?

Research is rapidly moving from simple best-of-N to evidence populations. MaxProof, for example, describes one model in four roles: generator, verifier, refiner, and ranker. It uses population search and tournament selection. The authors of the preprint report 35/42at IMO 2025 and 36/42at USAMO 2026. It is important to add that this is the author's result of the preprint, not the official competition verdict.

DeepMind for the system Aletheia describes the generation, verification and revision cycle and performance growth with test-time compute. The trend is the same: the model is changing from a responder to a process.

At the same time, it turns out that generative verifiers are not neutral. The work on scaling generative verifiers draws attention to prompt sensitivity and the difference between choosing the correct answer and evaluating the quality of the evidence. Reinforcement learning can improve procedural metrics without improving the accuracy of the final answer selection.

Therefore, in my opinion, the next decisive step will be less effective than "an even bigger reasoning model": calibrated, heterogeneous and auditable verification.

Conclusion: mathematics was not won by one model, but by a new way of organizing the calculation

From 2021 to 2026, one big story repeats itself.

MATH and GSM8K showed that language fluency is not enough. Chain of thought created a job trail. Self-consistency turned a single trace into a population. PAL and ToRA have left the exact calculation to tools. Numina has shown the power of an open model, data and voting. The winning NeMo Skills combined TIR, checkpoint merge, fast decoding and early stop; its GenSelect research arm showed smarter selection for future systems. Lean turned the compiler into a referee. AlphaProof combined formalization with RL search. Gemini Deep Think has struck gold in natural language. Proof Pilot showed that a single 32B checkpoint can be an organization of four professions.

For HyperFusion, it doesn't follow that we have to blindly copy 25 calls per question. This leads to something more practical:

1. cultivate real error diversity, not just more text;
2. measure the oracle ceiling before we buy a more expensive judge;
3. preferring hard validation to LLM flair;
4. correct a localized error, not rewrite the correct basis;
5. separate generator, verifier, refiner and selector even when they are physically played by one model;
6. state the inference budget as prominently as the score;
7. protect the test from leakage when learning weights and routing.

The best mathematical system of the future may not be the model that "knows the most". It will be the system that knows best when it doesn't know which of the others just knows more and how to verify it before sending the answer on.

---

Primary sources and reproduction materials

• Hendrycks et al.: MATH
• Cobbe et al.: GSM8K and trained verifiers
• Wei et al.: Chain-of-Thought Prompting
• Wang et al.: Self-Consistency
• Lewkowycz et al.: Minerva
• Gao et al.: PAL
• Gou et al.: ToRA
• Shao et al.: DeepSeekMath
• He et al.: OlympiadBench
• Mirzadeh et al.: GSM-Symbolic
• Epoch AI: FrontierMath
• Numina: AIMO 1 winning solution and technical analysis 48 × 4 SC-TIR
• NVIDIA: AIMO 2 paper and NeMo Skills
• AIMO Prize: rules and launch of AIMO 3 and announcement of winners
• varianceofx: breeding recipe 2nd place AIMO 3
• DeepMind: AlphaProof and AlphaGeometry 2
• DeepMind: Gemini Deep Think at IMO 2025
• Zheng et al.: miniF2F
• Tsoukalas et al.: PutnamBench
• Ren et al.: DeepSeek-Prover-V2
• Goedel-LM: Goedel-Prover-V2
• Yi-Chia Chen: Proof Pilot code and deploy bundle
• Jiang et al.: LLM-Blender
• Wang et al.: Mixture-of-Agents
• DeepSeek-AI: DeepSeek-R1
• Chen et al.: MaxProof

Note on figures: each result in the text belongs to a specific protocol of the authors. Pass@1, self-consistency, pass@k, official Olympic points and LLM-judge scores are not interchangeable. For preprints, I present the results as the authors' claims, not as an independently confirmed record.