How AI Learned to Program: From HumanEval to an ICPC Gold Medal

Programming has one major advantage over mathematical reasoning: we can run the answer. A program either compiles or it does not. On a specific input, it returns a specific output. We can measure time and memory and compare the result with a hidden checker. In our recent article on AI's journey from GSM8K to proof agents, we followed how mathematical systems learned to generate multiple approaches, vote and verify proofs. This article continues the same story in an environment where the result can not only be judged, but actually executed.

This leads to the idea that competitive programming is easier for AI than math. It is actually a tougher and cleaner lab. The model must derive a hidden algorithm from a long input, prove its correctness, translate it into a flawless program, and survive inputs it has never seen. The only omitted boundary condition means zero.

In five years, therefore, not only the performance of the models has changed, but the entire way in which we let AI be programmed. In 2021, we asked if one model would complete the short function on the first try. By 2025, multiple agents were already designing algorithms in parallel, running the compiler and tests, sharing bugs they found, and iterating to an ICPC gold medal-level solution.

This text is a technical history of this transformation. It's not just about which model got how many points, but mainly about which algorithm of inference, selection and verification produced this result. It is these principles that are transferable to HyperFusion and to experiments at LUMI.

Main conclusion: The biggest leap was not made by the model that wrote one more perfect program. It was brought about by systems that created a population of different algorithms, cheaply weeded out obvious bugs, looked for counterexamples, fixed only proven bugs, and intelligently decided which programs were even worth shipping.

From whispering to vibe coding

The term vibe coding was introduced by Andrej Karpathy on February 2, 2025 for a way of working where a person describes the intent in natural language, accepts large changes created by AI, and gradually guides the system without necessarily reading every line. Within months, the technical slang had become a cultural term, and Collins selected it as the Word of the Year 2025.

EXPLANATORY NOTE: WHAT IS VIBE CODING

In its pure form, it's not just autocomplete or occasional chatbot advice. The programmer shifts attention from syntax to intent, application behavior, and other instructions for the model. The code can be created in large blocks and the author does not need to understand it in detail. This is great for a prototype, but dangerous anywhere that lacks tests, security review, and someone who understands the resulting system.

It's important not to confuse the rapid growth of AI tools with pure vibe coding. Stack Overflow Developer Survey 2025 reports that 84% of respondents use or plan to use AI tools, up from 76% a year earlier, and 51% of professional developers use them daily. At the same time, however, 72% of respondents stated that vibe coding is not part of their professional work. AI assistance has become widespread, not automatically the willingness to stop checking the code.

A more sober correction was added by the METR experiment from the first half of 2025. Sixteen experienced open-source developers solved 246 tasks in their own repositories and their work took an average of 19% longer with the AI ​​tools of the time. It's a narrow snapshot in time, not a universal verdict on AI programming. But he reminds that the rapid expansion of the tool does not in itself prove higher productivity.

Competitive programming is therefore an interesting counterpoint to vibe coding. We can generate code very freely, but in the end it has to pass a precise checker. It is not the speed of writing itself that is of research value, but the ability to turn a vague stream of suggestions into a verified program.

First the rules: Codeforces, IOI and ICPC are not the same discipline

A number like "85th percentile in programming" sounds more definite than it actually is. Codeforces, International Olympiad in Informatics and ICPC test related but different skills.

Also, metrics measure different things:

Therefore, in the article I will not draw one falsely smooth curve of "general programming intelligence". For each point in the timeline, I list the platform, number of attempts, tools, and how to convert to human ranking.

Human Equivalent Timeline

The first systems had no credible human percentile yet. DeepCoder in 2017 sped up the enumerative search for short programs by about an order of magnitude, but worked in a limited DSL and only solved problems of difficulty similar to the simplest competition entries. Codex achieved 28.8% pass@1 on HumanEval in 2021, but HumanEval does not have a leaderboard of human competitors. These milestones therefore belong on an axis, not on a percentile scale.

As of 2022, we already have several direct, though still not entirely interchangeable, comparisons with humans. I put only the frontier system of the given time and its publicly documented result in the graph:

• AlphaCode, Codeforces 2022: outperformed approximately 45.7% of competitors, performing around the median.
• AlphaCode 2, Codeforces 2023: passed an average of 85% of participants. However, in the top two competitions, he outperformed more than 99.5% of participants. It is this variance that shows why the average itself hides an important part of the story.
• o1, Codeforces 2024: general o1 reached the 89th percentile. A variant further trained for IOI reached Elo 1807 and the 93rd percentile.
• o1-IOI, IOI 2024: Under the actual limit of 50 submissions per task, he scored 213 points and in the 49th percentile of elite high school Olympians.
• GPT-5, LiveOIBench 2025: scored 81.76 on 403 tasks from the 2023 to 2025 Olympics. percentile. The authors point out that the best people in this set tend to be above the 90th percentile.
• Gemini 2.5 Deep Think, ICPC 2025: ten problems solved would mean second place out of 139 teams, better than about 98.6% of the leaderboard. This is a hypothetical recalculation as Gemini was not an officially ranked varsity team.

I only connect the line between results from the same Codeforces family. IOI, ICPC and LiveOIBench remain as separate points because their populations and rules are different. The vertical line in AlphaCode 2 is not a confidence interval, but the difference between the average and the two best competitions. Thus, the graph shows two truths at the same time: frontier systems reached almost the 99th percentile in some competitions, while in another elite discipline the same year they remained around the median.

2021: HumanEval showed that ability is a distribution, not a single trial

The work Evaluating Large Language Models Trained on Code introduced Codex and with it HumanEval, 164 handwritten programming tasks. The model got a function signature, a docstring and several unit tests hidden from generation.

Codex solved 28.8% of tasks on the first try. But when the authors generated 100 programs from the same model, at least one correct solution appeared for 70.2% of the problems.

This is one of the most important numerical differences in all of history:

pass@1   = 28,8 %
pass@100 = 70,2 %
oracle gap = 41,4 procentního bodu

So the model often was able to generate the correct solution, but one attempt failed to extract it reliably. This gave rise to two distinct research questions:

1. how to improve candidate distribution;
2. how to recognize the right program among a hundred similarly convincing proposals without knowledge of hidden tests.

BREAKTHROUGH 1: 28.8% CHANGED INTO 70.2%

Repeated sampling: ask the same model the same task many times. The correct program appeared much more often than pass@1 indicated. But a selection problem was born: the oracle knows which candidate will pass the hidden tests, the real system does not.

APPS: the programmer's job was no longer just adding a function

The APPS benchmark in the same year yielded 10,000 problems ranging from simple tasks to competitive algorithmic challenges. The input was in natural language and correctness was determined by test cases. The GPT-Neo of the time managed about 20% of the individual tests on initial problems, not 20% of fully solved heavy tasks.

APPS introduced an important distinction that is sometimes lost to this day: the percentage of test cases passed is not the same as the percentage of fully resolved issues. A program that passes nine tests out of ten may have a practically useful basis. But it still gets a zero in a binary checker contest.

2022: AlphaCode changed code generation to population search

AlphaCode was the first system to reach about the median participant level in the new Codeforces competitions. The evaluation used ten competitions more recent than the training data. It was not a short function addition, but a long assignment requiring combinatorics, graphs, dynamic programming and precise complexity analysis.

Architecture was more important than a single checkpoint:

zadání
  -> miliony různorodých programů
  -> kompilace a veřejné testy
  -> behaviorální clustery
  -> seřazení kandidátů
  -> nejvýše 10 submitů

AlphaCode pretrained transformers on public code from GitHub and retrained them on competition tasks and solutions. During inference, it generated an order of magnitude more programs than previous work. He then filtered, clustered and reduced them to ten trials.

In the simulation, he finished in the top 54.3% of the leaderboard on average. This is often misspelled as the 54th percentile. This correctly means that approximately 45.7% of participants had a worse outcome.

BREAKTHROUGH 2: MILLIONS OF PROGRAMS, TEN SUBMITS

Population search: the model was not a contestant who wrote one solution. He was a population generator. A separate layer of compilation, testing, deduplication and selection decided which members of the population would get the rare opportunity to be sent.

CodeContests: the dataset also contained erroneous human solutions

DeepMind released the CodeContests dataset and execution tools. The data includes tasks from Codeforces, AtCoder, CodeChef and other sources, test inputs and outputs, as well as correct and incorrect human solutions.

Bad code is surprisingly valuable for training. It makes it possible to teach not only "what the correct solution looks like", but also "which small changes turn the received program into an almost correct program". For the future critic and repair model, this boundary is more useful than another thousand flawless templates.

AlphaCode's secret weapon: clustering by behavior, not text

Two programs can look completely different and yet implement the same algorithm. Conversely, two nearly identical implementations can differ by a single sign and have different behavior at the range boundary.

Therefore, AlphaCode did not derive diversity from textual similarity alone. He generated additional inputs, ran candidate programs on them, and for each program created a signature of its outputs:

testy = generate_inputs(problem)

signature(program) = [
  run(program, test_1),
  run(program, test_2),
  ...,
  run(program, test_m)
]

clusters = group_by_similar_signature(programs)

If ten thousand programs return the same thing on all diagnostic inputs, it makes no sense to send ten cosmetic variants of them. It is better to select one representative and devote the remaining submission budget to other behavioral hypotheses.

This is the programming equivalent of voting on a normalized math answer, but with a much richer signal. We don't just vote on the final number. We compare features according to their behavior on a full set of targeted questions.

Why the largest cluster is not automatically true

Behavioral majority works if the correct algorithm produces a wide region of similar implementations and the errors are fragmented. But it can fail when models share the same popular fallacy. A typical example is a solution that passes the demo but ignores overflows, duplicate values, or the asymptotic worst case.

Therefore cluster size is not proof. It is a prior. Hard tests, adversarial entries and a quality scorer must decide whether the largest group is the wise crowd or the crowd with the same bug.

2023: AlphaCode 2, one million candidates narrowed down to ten

AlphaCode Tech Report 2 offers an extremely specific recipe. The basis was a family of models derived from the Gemini Pro, not a single generator. The models went through two rounds of fine-tuning on the updated CodeContests dataset and a higher quality supplementary set.

The updated corpus had approximately 15,000 problems and 30 million human programs. A family of policy models was created with different hyperparameters. So diversity wasn't just left to random temperature, it was built into the generator family.

For each of the 77 problems from the twelve new Codeforces competitions, the system did the following:

1. generated up to 1,000,000 C++ programs;
2. randomly changed temperature and metadata in prompt;
3. evenly divided the budget between several policy models;
4. launched candidates in public examples;
5. removed non-compiling and obviously bad programs;
6. created behavioral clusters from approximately 50,000 survivors;
7. left the ten largest clusters;
8. selected the best representative of each cluster with a trained Gemini scorer;
9. sent no more than ten programs.

Public tests and compilations eliminated about 95% of candidates on average. This is an important economic principle: the dear learned judge should not be wasted on code that cannot even pass the first example.

The result was 43% of problems solved within ten attempts against 25% of the original AlphaCode. Estimated rank moved from performing better than about 46% of participants to performing better than 85% of participants.

Even more impressive is the efficiency. About 100 samples of AlphaCode 2 were enough to reach the performance of the original AlphaCode with a million samples. The authors therefore report a more than ten thousand-fold improvement in sample efficiency at this comparative level.

BREAK 3: 25% CHANGED TO 43% AND THE MEDIAN TO THE 85TH PERCENTILE

Diverse policies + execution + clustering + scorer: a stronger model did not eliminate the need for search. He changed its economy. A hundred new samples could be worth a million old ones, but the final result still depended on a hard filter and selection.

How to generate a million candidates: sampling is not the same as tree search

The intuitive explanation is often that the system was searching the solution tree. But history is more subtle.

DeepCoder in 2017 actually used classical search. The neural network estimated from the examples which features of the restricted language would be needed in the program, and depth-first search, enumerative solver or SMT solver followed those probabilities. Thus, the model reduced and ordered the explicit combinatorial space of short programs.

AlphaCode and AlphaCode 2, on the other hand, did not generate a million candidates using one giant Monte Carlo tree search. The generative transformer repeatedly autoregressively sampled the entire program token by token. Individual rollouts could be run independently and massively parallel. The diversity was created by different policy models, languages, prompts, metadata and randomly chosen temperature. Only finished programs went through compilation, public tests, behavioral clustering and scorer.

nezávislý rollout 1  -> celý program -> testy
nezávislý rollout 2  -> celý program -> testy
...
nezávislý rollout N  -> celý program -> testy

nikoliv nutně:
kořen -> token -> token -> společný explicitní strom všech možností

Why did he go to make a million in the first place? Competition programs are relatively short, sampling can be easily batched on accelerators, and rollouts do not have to wait for each other. A million candidates does not mean a million allowed submissions. AlphaCode 2 reduced a million raw programs to about 50,000 survivors after the first filters, then ten behavioral clusters, and finally ten actual submissions.

For o1-IOI, we know the external interface, not the complete proprietary mechanism. OpenAI says the system sampled many candidates, selecting 50 using public and generated tests and a learned scorer, and allowed 10,000 submissions per job in a released experiment. But the public report does not claim that MCTS was running inside. We know that o1 improves with longer test-time compute and that it can correct its own progress. The exact shape of the internal branching has not been published.

In addition, today's agent systems do not create diversity just by independent sampling. A typical loop is plan -> implement -> compile -> failing test -> targeted fix. A new candidate will arise as a branch triggered by a specific proof of error. Beam search, DFS or MCTS can be built over such a loop, but they are not an automatic explanation of every reasoning model.

The most important trend is economic: stronger policies dramatically reduce the number of blind samples needed. AlphaCode 2 achieved with about 100 samples the performance that the original AlphaCode needed a million to achieve. Therefore, it makes more sense for HyperFusion to create dozens of deliberately different plans, measure their behavior, and branch only where a test reveals a conflict, than to mindlessly produce a million cosmetic variants.

The compiler is not only a referee, he is also a teacher

Programming offers cues that math text often doesn't:

• the parser will show a syntax error;
• the compiler returns the problem type and line;
• the runtime detects an exception or overflow;
• profiler will show time or memory complexity;
• checker returns the specific input on which the solution failed;
• differential testing will show where candidates diverge.

Not all signals are equally valuable. A compilation error often directly points to a local fix. "Wrong answer on hidden test 37" says almost nothing. Therefore, the best architectures do not perceive execution feedback as a single message, but as a gradual refinement of the diagnosis.

Test generation as an active experiment

The most useful test is not random input. It is the input at which two plausible solutions do not agree. It has a high informational value because it splits the hypotheses.

Let's imagine three candidates:

• And it uses a greedy algorithm;
• B dynamic programming;
• C same greedy procedure but fixes duplicates.

A random test can return the same results for everyone. The active generator is supposed to search for the small case with duplication, the case with zero, the maximum range, and the construction on which greedy violates the global optimum. Only then does testing become a scientific experiment, not a lottery.

2024: o1-IOI accurately measured the selector value

OpenAI evaluated a specialized model derived from o1 at the International Olympiad in Informatics 2024. According to the official description, the system had ten hours for six problems and a maximum of 50 submissions per problem.

He generated many candidates for each problem. He selected the fifty submitted solutions according to three signals:

1. performance on public IOI tests;
2. performance on tests generated by the model;
3. learned scoring functions.

The result was 213 points and the 49th percentile of competitors. At the same time, OpenAI estimated that a random selection of candidates would only score 156 points on average.

stejný generátor + náhodný výběr       ≈ 156 bodů
stejný generátor + test-time selection = 213 bodů
hodnota selektoru                       ≈ +57 bodů

This is the purest ablation of cognitive architecture value available. The correct programs were already in the population. Almost 60 points was not created by additional pretraining, but by the ability to invest the limited 50 submissions in better candidates.

When the limit was artificially relaxed to 10,000 submissions per problem, the system scored 362.14 points, above the gold medal threshold, even without test-time selection. This result is valuable as an oracle experiment, but it is not a fair competitive result. It shows the upper limit of the generator at almost unlimited attempts, not the ability to work under the IOI rules.

BREAKTHROUGH 4: +57 POINTS WITHOUT CHANGING THE GENERATOR

Public tests + generated tests + learned scorer: the selector decided which of the many solutions would consume the limited submission budget. The random selection left much of the model's capability unused.

Codeforces rating: same family, different view

OpenAI also simulated Codeforces competitions with ten submissions allowed. GPT-4o achieved an Elo of 808, around the 11th percentile. o1-preview achieved Elo 1258 and the 62nd percentile, o1 Elo 1673 and the 89th percentile, and specialized o1-IOI Elo 1807, the 93rd percentile.

These numbers are not to be confused with the 49th percentile on the IOI. IOI selects the world's most talented high school Olympians and uses a different type of tasks and scoring. It is the difference between Codeforces' 93rd percentile and IOI's 49th percentile that reminds us that the "human percentile" always needs a denominator.

2024: the benchmark began to resist contamination

HumanEval was essential, but it only has 164 jobs and quickly became part of the public ecosystem. Once the solutions, tests, and derived variants appear in the training data, high scores can measure memory as well as generalization.

LiveCodeBench therefore continuously collects new tasks from LeetCode, AtCoder and Codeforces. The first release contained 400 tasks published from May 2023 to May 2024. The task release date allows the models to be tested on problems that arose after their probable data cut-off.

In addition, LiveCodeBench does not only evaluate program generation. Contains scenarios for:

• self-repair of faulty code;
• predicting program output;
• work with execution feedback;
• classical solution generation.

This is methodologically important. A model that can write a solution may not be able to find a specific error. A model that correctly fixes a known bug does not need to be able to invent an algorithm from scratch. A single coding score unnecessarily discounts these abilities.

2025: Gemini Deep Think brought a multi-agent team to the ICPC

The ICPC final is a different type of pressure. The team has five hours, a single submission received means a point and each failed attempt adds a time penalty. It is not enough to have correct@1000. You need to quickly find the right algorithm, implement it, and know when to send it.

Advanced version of Gemini 2.5 Deep Think solved ICPC World Finals 2025 remotely under the supervision of the organizers. She started ten minutes later than the humans, had the same five-hour time limit, and solved 10 of the 12 tasks. She managed eight tasks in the first 45 minutes, two more within three hours. With a combined penalty time of 677 minutes, they would finish second out of 139 teams when compared to the official table.

DeepMind described several key components:

• multi-step reasoning;
• parallel thinking;
• several Gemini agents with their own designs;
• terminal access;
• running code and tests;
• iteration according to the results of all trials;
• reinforcement learning on very difficult programming tasks.

Final Gemini programs for ICPC 2025 have also been released. However, the complete architecture, sample counts and all internal prompts cannot be reproduced. This is an important competitive milestone, not an open NuminaMath quality recipe.

BREAKTHROUGH 5: 10 OF 12 TASKS IN FIVE HOURS

Parallel agents + terminals + iteration: agents didn't just write several independent programs. They saw the results of the execution, corrected themselves, and used the previous trials as a common experimental memory.

Problem C: no university team found a solution

Gemini solved Problem C in about half an hour, which was not accepted by any human team. The task required optimal distribution of liquid through networks of channels and reservoirs with an infinite number of possible settings.

According to the published description, Gemini implemented priority values for stacks. For fixed priorities, the best channel settings could be found by dynamic programming. Then, using the minimax argument, he transformed the original task into finding the priorities that produce the tightest constraints. He used nested ternary search in convex space.

This example is important because it's not just a flawless implementation of a familiar template. Success required multiple representations of the same problem:

fyzická síť
  -> priority zásobníků
  -> dynamické programování pro pevné priority
  -> minimaxová reformulace
  -> numerické hledání optimálních priorit

It is the ability to change the representation that separates a strong implementer from algorithmic discovery.

2025 and 2026: sober dates against too quick victory

News of medal systems can give the impression that competitive programming is settled. More open benchmarks show a more complex picture.

LiveCodeBench Pro: zero percent on hard without tools

LiveCodeBench Pro was also compiled by researchers and medalists of the International Algorithmic Olympiads. They continuously updated tasks from Codeforces, ICPC, and IOI, annotated them according to algorithmic categories, and analyzed the faulty model solutions line by line.

Without external tools, the best rated model achieved 53% pass@1 on moderate tasks and 0% on hard. The authors found that the models are more powerful in implementation-intensive problems than in fine-grained algorithmic reasoning and complex case analysis. They often produced confident but incorrect rationales.

This does not contradict the Gemini result. It shows the difference between stand-alone model without tools and expensive closed multi-agent architecture with terminals, execution and iteration.

LiveOIBench: offline olympics with human benchmarking

LiveOIBench, the third version of which was released on July 8, 2026, contains 403 expertly selected tasks from 72 competitions in 14 Informatics Olympiads from 2023 to 2025. Each task has an average of about 60 official test cases and a detailed subtask rubric. The entire assessment can be run offline.

Out of 34 evaluated models, GPT-5 scored 81.76. percentile, still below the best human competitors. Open-weight gpt-oss-120b scored around the 60th percentile. Moreover, analysis of reasoning traces indicated that more robust models prefer accurate input analysis over excessive chaotic searching.

BREAKTHROUGH 6: BENCHMARK NO LONGER MEASURES JUST ACCEPTED / WRONG ANSWER

Fresh problems + subtasks + human percentiles: a new generation of benchmarks tracks task date, partial points, test coverage and direct comparison with elite humans. A high pass@k on a small static set is no longer enough.

What really won in history

1. Diversity of algorithms, not just words

Changing the temperature will often create a different style of the same algorithm. Real diversity means a different data structure, invariant, asymptotics, or decomposition of the problem. AlphaCode 2 supported it with several fine-tuned policies and changed metadata.

2. Hard filters before the expensive judge

Compilation, public tests, limits, sanitizers and static checking are cheaper and more reliable than LLM judge. AlphaCode 2 thus eliminated approximately 95% of the candidates even before the final selection.

3. Behavioral deduplication

Textually different code can be functionally identical. Clustering according to outputs on diagnostic inputs protects the submission budget from ten copies of the same error.

4. Actively generated counterexamples

The best test differentiates between two strong candidates. The test generator should maximize the information gain, not the number of random inputs.

5. Learned scorer as the last layer, not the only truth

The scorer is useful where the execution has not decided. It should not overwrite the fact that the program crashed, violated a limit, or failed a known test.

6. Targeted repair

When we know the failing input, we fix a particular invariant or line. Free synthesis of two programs can destroy parts that were already correct.

7. Test-time compute with a budget

A hundred candidates without a selector is an oracle experiment. A real architecture must optimize the expected number of points per GPU-second, the compilation and the submission allowed.

8. Department of professions

The planner, implementer, test generator, critic, and selector have different goals. One model can alternate roles, but the protocol must prevent the author from only confirming his own solution.

Which, on the contrary, repeatedly did not work

1. One hundred almost identical implementations

They increase pass@k less than the sample count suggests. Correlated error survives both voting and reranking.

2. Optimization for public demos only

The sample test explains the format, not the full specification. The model easily learns the patch for the three examples and misses the general algorithm.

3. "Code looks right" as the main metric

A language judge often prefers clean titles, comments, and compelling explanations. Hidden checker doesn't care about style.

4. Random submission

The o1-IOI experiment measured a nearly 60-point difference between random and intelligent selection of the same population.

5. Self-reflection without new evidence

A “check yourself” prompt often creates a longer rationalization. A fix should get a new test, traceback, counterexample, or conflict with another candidate.

6. Mixing oracle@k with deployable performance

Ten thousand submissions above the gold limit does not mean a gold medal under the limit of fifty attempts. Oracle is a generator potential diagnostic.

7. Benchmark without temporal provenance

For a public problem, we need the publication date, the model cut-off date, the availability of the solution, and the hash of the specific version of the dataset.

How would I build HyperFusion Code from it

History shows an architecture that is not simple by majority vote. It is a controlled experimental cycle.

The original opening illustration takes on a more precise role here: it depicts the fusion of multiple independent streams of thought, not programming itself.

Phase 1: problem footprint

The router first estimates:

• algorithmic domain;
• expected complexity;
• the need for a mathematical proof;
• risky borderline cases;
• whether C++, Python, or another language is more appropriate;
• how much inference budget the job deserves.

The router is not supposed to solve it right away. He is to create an experimental plan.

Phase 2: blind algorithmic design competition

Three to five different local models will produce only algorithm design, invariants, and complexity estimation. They cannot see the suggestions of others. This prevents premature convergence on the first confident idea.

Proposals are grouped according to the actual principle:

• greedy;
• dynamic programming;
• graph transformation;
• flow or matching;
• mathematical construction;
• search with pruning;
• randomized or numerical procedure.

A maximum of two representatives of each family advance to the next stage.

Phase 3: more implementation of strong plans

Each surviving plan is implemented by at least two models. This separates the algorithm error from the write error. If two independent implementations of the same plan give different results, we get a very valuable diagnostic conflict.

Phase 4: test forge

A stand-alone agent creates:

• minimal cases;
• null and empty input if allowed;
• duplicates and identical values;
• maximum size;
• values at data type boundaries;
• cases violating the greedy intuition;
• inputs on which candidates differ;
• metamorphic transformation with a known relation of outputs.

If we know a slow reference algorithm for smalln, we use differential testing. For small cases, we exhaustively compare fast candidates with brute force.

Stage 5: targeted repair

The candidate will only receive:

• original plan;
• custom code;
• specific failing input;
• expected and actual output;
• possibly diagnostics of the sanitizer or the limit.

The task is not "write a better solution", but "find the minimal cause of this failure and fix it without changing the other invariants".

Stage 6: tournament selector

The ranking of candidates combines hard and soft signals:

score =
    1000 * known_tests_passed
  - 1000 * crash_or_timeout
  +  120 * independent_implementations_agree
  +   80 * adversarial_tests_passed
  +   40 * predicted_correctness
  +   20 * algorithmic_diversity_bonus
  -   10 * unexplained_complexity_risk

Specific scales must be locked to the validation set. The test set is only used for a one-time report.

Stage 7: auditable submit

The system stores:

• all algorithmic designs;
• source model and seed;
• programs and their hashes;
• compiler and flags;
• all generated tests;
• execution results;
• corrections and their reasons;
• selector's final score;
• the actual result of the hidden checker.

Without this trace, it is impossible to determine whether the improvement was brought about by the model, the test generator, the repair loop, or just a greater number of attempts.

From design to data: I actually ran the experiment on LUMI

The proposal above did not remain on paper. On Finland's LUMI supercomputer I built a local panel of open-weight models, a safe execution sandbox and an experiment over recent LiveCodeBench problems. I wanted to know whether hard feedback from a running program could break the selector wall I had previously encountered on GPQA science questions.

I used a stratified sample of 40 tasks released from March 2025 onward: 13 easy, 14 medium and 13 hard, with 26 AtCoder and 14 LeetCode problems. Each problem had about 43 public and hidden tests on average. gpt-oss-120b, Qwen2.5-72B, GLM-4.7-flash and Mistral-Small-3.2-24B ran locally on LUMI; Kimi K2.7-code and Kimi K3 via API served only as a reference ceiling.

First surprise: a hard signal must discriminate

Merely running the code was not enough. Candidates that passed every public test returned the same correct outputs there, leaving the selector blind. On roughly one problem in five, such a public passer still failed a hidden test. In this controlled pilot I measured the value of a discriminating layer using a clean split between selection tests and an independent scoring set; in a production system that role must be filled by generated counterexamples and differential testing.

The most important LUMI result: the most valuable artifact may not be another program. It may be the single test that separates the correct solution from code that only looked correct on every public example.

Second surprise: the panel did not raise the ceiling

The combined oracle stayed at 35/40, and one strongest coder already solved those same 35 tasks alone. Unlike GPQA, the leading models' errors were barely complementary. The practical recipe was therefore not “as many model families as possible” but one strong model × more samples × discriminating test selection. It raised the floor from about 30 to 35 tasks, not the ceiling beyond 35.

Five hardest problems, four orchestration strategies

I then attacked the remaining five problems using only local gpt-oss-120b, without upgrading to a more expensive model:

Together these runs cracked 2 of the 5 tasks that no baseline panel model had solved. One fell only to proof-pilot after the other three modes scored zero. That is an important brake on the phrase “the model cannot do it”: sometimes the capability exists, but the default inference procedure cannot extract it. The model did not call tools itself (tool_calls = 0); the harness reliably ran tests and returned concrete failures.

One methodological trap was just as instructive. With a 12,000-token limit, Kimi K3 solved only 17/40 because 35% of responses ended mid-reasoning before any code appeared. Raising the budget to 32,000 tokens lifted it to 31/40. Without that control run, I would have published a harness error as a model weakness.

Inference speed as a new dimension of AI applications

The LUMI experiment asked how to obtain more correct code from a limited compute budget. In Fast LLM Inference with Cerebras, I tried the opposite extreme: what changes when waiting for a large model almost disappears?

The course uses open-weight gpt-oss-120b on Cerebras wafer-scale infrastructure. The current Cerebras model catalog lists approximately 3,000 output tokens per second for the endpoint. That is a provider figure, not my universal guarantee. In my own runs I measured:

A similar billiards prompt took about 39 seconds in one GPT-4.1 mini run. This is not a clean hardware benchmark: the models, infrastructure and network paths differ. It is still a vivid demonstration of how the working experience changes when tens of seconds collapse toward one.

Two seconds to a first playable prototype

The first Tetris output was genuinely playable: falling tetrominoes, movement, rotation, faster drop, cleared rows and scoring. This does not mean “a finished product in two seconds.” It means two seconds to a working base that can immediately be run, tested and improved.

The chess game was much rougher, which made it more interesting as an experiment. In about two seconds the model produced a board, piece controls, basic move validation and a simple computer reply. It is neither rules-complete nor a replacement for a proven chess engine. Yet going from an empty prompt to an interactive game in the time it normally takes to open an editor still feels remarkable.

Why this is more than a faster chat

At ordinary latency, we often generate one proposal and wait. At hundreds or thousands of tokens per second, one interaction can contain several candidates, tests, repairs and a final selection. Speed enables a different architecture, not merely faster text animation.

For HyperAdvisor and HyperFusion, three modes follow naturally:

Best-of-N, continuous tests and linting, an architect paired with a fast executor, parallel specialists and more natural voice assistants no longer need to punish users with long silence. The most useful metric is therefore not tokens/s alone, but:

How quickly, and at what cost, does the system produce a verified correct result?

Cerebras does not automatically replace the strongest proprietary models, and fast code is not automatically good code. The combination with the LUMI lesson is powerful, however: speed supplies more attempts, while execution and discriminating tests supply a reason to trust the selected one.

Parallel branch: competitive programming is not ordinary software engineering

HumanEval, Codeforces, IOI, and ICPC have a clear specification, clean input, expected output, and relatively isolated program. The real repository is a different world. The input can be incomplete, the tests out of date, and a proper fix can require understanding hundreds of files.

Therefore, I would not mix SWE-bench and agents fixing GitHub issues in the main timeline. It deserves a separate article. The principles are related, but the role of the tests is different: in competition the tests define the truth, in real development they may themselves be part of the problem.

What to take away

The history of competitive AI programming is not a simple story of ever-larger models. It is a history of ever better organization of attempts:

1. HumanEval revealed a huge difference between pass@1 and pass@100.
2. AlphaCode turned generation into a population search.
3. AlphaCode 2 combined various policy models, execution, behavioral clustering and learned scorer.
4. o1-IOI measured that selection of the same population was worth approximately 57 points.
5. LiveCodeBench added fresh data and more capabilities than just generating.
6. Gemini at ICPC used parallel agents, terminals, and iterative repair.
7. LiveCodeBench Pro and LiveOIBench showed that elite algorithmic reasoning is still not solved.
8. My LUMI experiment recovered 80–83% of the available oracle gap with discriminating tests, while public tests alone were insufficient.
9. Four orchestration strategies for the same local model cracked two of five problems the baseline panel could not solve.
10. Cerebras exposed a second axis: inference can be fast enough to fit generation, checking and repair into one interaction.

Final Principle: The smartest programming system is not the one that thinks the longest in one monologue. It's a system that can generate independent hypotheses, design experiments, collect hard evidence, fix specific errors, and manage a limited submission budget.

Programming can be the ideal laboratory for HyperFusion. Here we have something that is often missing from open humanities questions: cheap feedback from reality. When we use it well, we don't need one model to be perfect. We need the whole system to be able to learn from every failed run.

Primary sources and reproduction materials

• DeepCoder: Learning to Write Programs
• Evaluating Large Language Models Trained on Code, Codex and HumanEval
• Measuring Coding Challenge Competence With APPS
• Competition-level code generation with AlphaCode
• DeepMind: Competitive programming with AlphaCode
• CodeContests dataset and execution tools
• AlphaCode 2 Technical Report
• OpenAI: Learning to reason with LLMs, section on IOI and Codeforces
• LiveCodeBench
• LiveCodeBench repository and evaluation harness
• LiveCodeBench Pro
• LiveOIBench
• Gemini 2.5 Deep Think at ICPC World Finals 2025
• Published Gemini programs from ICPC 2025
• Original post by Andrej Karpathy on vibe coding
• Collins Word of the Year 2025: vibe coding
• Stack Overflow Developer Survey 2025: AI and vibe coding
• METER: measuring the productivity of experienced open-source developers with AI
• Fast LLM Inference with Cerebras, DeepLearning.AI course
• Cerebras Inference model catalog and listed gpt-oss-120b speed
• Cerebras Cloud