Now we are famous in Finland too: two days and one night on LUMI

Now we are famous in Finland too.

My first article about LUMI caught the attention of the LUMI AI Factory team. On Sunday, they published their own story about Alpha Industries, titled Czechia's Alpha Industries validates LUMI's power and sustainability in their initial test. Within days, my first Czech experiment had become a story on the official Finnish website of Europe's AI infrastructure.

I was genuinely delighted. It also made me want to find out whether the second act could live up to the first.

On Monday morning, LUMI reopened after a security shutdown. Over the next 38 hours, I ran three experiments, tested ten pre-written hypotheses, compiled a custom inference engine for AMD, ran a model with about 753 billion parameters on a single node, and ran several systems overnight to solve PhD-level scientific questions.

Some hypotheses held up. Others failed so convincingly that they became the most interesting part of the result.

LUMI in Kajaani. The European infrastructure that ran these experiments sits behind its distinctive blue cladding.

The whole experiment in one sentence: A large model is not automatically the best, 4-bit compression does not damage all models equally, and more thought can increase the quality by 15 points or decrease it by 27 points.

First, four terms in plain English

Model fusion is an AI jury. Several models answer independently, then another model checks their work and combines it into a final solution.

Quantization is compression. Instead of storing each value in 16 bits, we use about 4 bits. The model takes up less memory, but may lose some capabilities.

Reasoning means that the model creates a longer internal process before the final answer. It can help a lot with heavy tasks, but it costs time, energy and tokens.

A benchmark is a standardized test with known correct answers. That means quality does not have to be judged by another AI.

Why do I write down hypotheses in advance?

Before each experiment, I wrote down the expected results and the rules by which I would evaluate them. It's a simple insurance policy against self-deception. You write down the prediction before the start and after the result you can no longer claim that this is exactly what you expected.

In the first and third experiments, ten hypotheses were created. Some were confirmed, some only partially and three turned out the opposite of what I expected. That's not a shame. This is the reason to do the experiment at all.

Experiment 1: model fusion a second time

In June, I showed on a single RTX 5090 that a fusion of the local Qwen and GLM models scored 98 out of 100 and outperformed the individual configurations. On LUMI, I repeated the same test at full BF16 precision, with the same prompts and the same task.

This time, I evaluated 60 points with a deterministic script against the correct solution. A frontier model outside the tested panel assessed the remaining 40 points. I had previously been rightly criticized for allowing Qwen to evaluate a jury of which it was itself a member, so I added independent judges as well.

Score in one pre-defined open-ended task, therefore not a general model ranking. Fusion remained at the top, but this time tied for first place with standalone Qwen.

Fusion scored 99/100. The best standalone Qwen scored the same. This is an important nuance: the replication confirmed that fusion preserves an excellent solution and can filter out bad candidates, not that it necessarily beats the best panel member in every run.

That robustness was not free. One degenerate candidate did not mislead the judge, but its 15,000 tokens of noise expanded the judge input to 31,000 tokens and roughly doubled the prefill cost. The practical recommendation is simple: filter or trim candidates carrying TRUNCATED or REPETITION failure flags before fusion.

But the mechanism worked exactly as intended. The judge did not trust confident claims; it recalculated them. One candidate, for example, wrote that 20 + 5 = 25, which supposedly matched 29. The judge exposed this "verification" and rejected the solution. Models can bluff too.

Four different judges chose the same winning foundation. In the task with an objective solution, the judge's own bias was not reflected in the verdict. But the quality of the synthesis varied: the big Qwen2.5-72B almost just copied the winner, while the smaller judges actively extracted useful parts even from the weaker answers.

Surprise: the inference engine changes the behavior of the model

GLM-4.7-Flash scored 38 points in June via llama.cpp. On LUMI, using vLLM and the more precise BF16 representation, it fell to 9. At a low temperature, it entered a loop and printed 15,000 tokens of a number sequence.

So the same weights are not the whole model. The engine, its version, conversation template, temperature and limits also affect the result. From now on I log all this data for every output.

Experiment 2: 753 billion parameters on a single node

The bolder part came in the evening. According to its model card, GLM-5.2 has 753 billion parameters, of which roughly 40 billion are activated for each token. At full precision, its weights would require about 1.5 TB of memory. I used a 435 GB, roughly 4.5-bit version.

One LUMI-G node provides four physical AMD MI250X accelerators, or eight independent compute GCDs, with a combined 512 GB of fast HBM memory. The model fit, but with little room to spare.

In vLLM, this 4-bit variant lacked properly optimized kernels for the hardware. I therefore compiled llama.cpp with the HIP backend directly for the LUMI architecture. The result was 17 tokens per second, an interactively usable speed for a model that cannot come close to fitting in a regular workstation.

The first attempt still failed. Loading a 435 GB file via mmap from the Lustre parallel file system had made no progress after 40 minutes. The --no-mmap switch reduced the same load to about five minutes. Much of the HPC experience looks exactly like this: hours of searching, one switch, an eightfold difference.

What did the giant answer to the question of the meaning of life?

As a warm-up, I asked: "What is the meaning of life?" The model replied with surprising sobriety: "Meaning is not something that exists out there as an object to be found. Rather, the meaning of life is created and perceived." It then discussed existentialism, Frankl's emphasis on work, love and one's attitude to suffering, and the biological perspective of survival and the continuation of life.

This is not a scientific result, just a small portrait of the model. Still, it is charming that the 753-billion-parameter system's first sentence was not about performance. It was about how we must create meaning ourselves.

The full and compressed GLMs were not the same model

I then gave both versions our open-ended test task. The full GLM-5.2 accessed through an API received a manual estimate of about 95/100. It made a mistake, disproved it through an honest calculation, found the correct rule and finally explained where its reasoning had gone wrong.

The quantized version finished at about 50/100. It often merely imitated verification, writing equations that did not add up without noticing.

Meanwhile, 4-bit Qwen lost only 2 to 4 points against BF16 in the first experiment. So the same compression can be almost harmless for one model and essential for another. Without measurement, it cannot be known by the number of parameters or by the size of the file.

Experiment 3: a night with GPQA Diamond

One open task is an anecdote. So overnight I added GPQA Diamond, a benchmark of PhD-level science questions in biology, chemistry and physics. The answers are A to D and the correct option is known so we don't need an AI judge.

Prior to launch, I stratified 60 of the 198 total questions, saved their identifiers, and set a seed of 42. Where traffic permitted, three seeds were run. I am not publishing the content of the questions to protect the benchmark.

Accuracy in strict mode. Bars show 95% Wilson intervals. Full systems and Qwen variants have 180 responses, local GLM Q4 60; therefore, close scores should not be read as definitive rankings.

The best result came from the full GLM-5.2 with a high reasoning budget: 89.4%. This is close to the vendor's 91.2% result in the official model card. The much smaller Qwen3.6-27B scored 86.7% without reasoning. The 2.7 percentage-point difference is smaller than the statistical uncertainty for this sample.

This matters. On this particular task, a 27-billion-parameter model came very close to one with 753 billion total parameters. That does not mean it is equally capable in general. It means a much smaller system may be sufficient for this type of query.

For Qwen with reasoning, one important number belongs next to its strict score of 82.8%: 94.0% among completed responses. The gap shows how strongly deliverability affected the strict result: 31 of 180 runs failed to provide a final choice within the 30,000-token limit.

Reasoning helped the full model and hurt the compressed one

Same model family, but opposite reasoning effect. For the Q4 branch, 35 out of 60 responses hit the 8,000 token limit, so the chart measures both capability and practical deliverability within a given budget.

For the full GLM, thinking raised the score from 74.4 to 89.4%. For the 4-bit GLM, on the other hand, it dropped from 80 to 53%.

The most likely explanation corresponds to the study Quantization Meets Reasoning: quantization error can appear early in the reasoning chain and further steps amplify it. The newer paper Quantized Reasoning Models Think They Need to Think Longer, but They Do Not reports the same pattern of excess thinking: aggressive quantization lengthened reasoning chains while accuracy fell. In our run, 35 out of 60 responses exceeded the 8,000-token limit. So I cannot claim that compression alone caused the entire drop. The data does show that this particular Q4 configuration is unsuitable for long reasoning within a practical budget.

Tomáš Mikolov and ThinkingCap: think less, deliver better

I also added BottleCap AI's new ThinkingCap-Qwen3.6-27B to the overnight test. Tomáš Mikolov is involved in its research. The goal is not to make Qwen smarter, but to teach it to stop earlier, avoid unnecessary loops and reach the same quality with fewer tokens.

This is exactly the area in which Tomáš Mikolov has been strong for a long time: efficiency, not just the pursuit of a bigger model.

The same 60 questions and three seeds. The chart shows strict accuracy, average output tokens, median latency and truncated responses across all four modes.

The most important point is that, for this particular task, the best strict result came from ordinary Qwen without reasoning: 86.7%. ThinkingCap without reasoning scored 82.2%. Once reasoning was enabled, their order reversed: ThinkingCap scored 84.4%, while Qwen scored 82.8%.

For Qwen, reasoning therefore reduced the strict score from 86.7% to 82.8%. The rise in truncated responses from 11 to 31 out of 180 contributed substantially; the reasoning branch used a 30,000-token limit. For ThinkingCap, reasoning instead raised the score from 82.2% to 84.4%. The same switch produced opposite results in two related models.

With reasoning enabled, ThinkingCap consumed an average of 7,312 tokens versus 15,097 for the original Qwen, a saving of 52%. Median latency fell from 1,189 to 403 seconds and truncated responses from 31 to 9. It therefore fulfilled its central promise: within a limited budget, it delivered a complete answer more often.

If we compare completed reasoning answers only, the original Qwen scored 94%, while ThinkingCap scored 88.9%. Qwen had the higher ceiling, but failed to deliver it within the fixed budget more often.

My verdict is therefore deliberately restrained: ThinkingCap is not a smarter Qwen in this pilot. It is a more disciplined Qwen. It may be more valuable in production precisely because it knows when to stop. For tasks with an almost unlimited time budget where the highest possible ceiling matters, the original model remained stronger.

Budgeting for thinking is a separate ability

I raised the reply limit three times during the night because I originally set it as a guess. Qwen was able to use up 30,000 tokens on a difficult question and still not reach the final A to D choice.

This resulted in three metrics:

1. Strict: Hard limit. An incomplete answer is zero. Measures deployability within budget.
2. Budget-forced: The model knows how many tokens it has left and must close the answer. It measures budget management.
3. Unbounded: The model can think for a very long time. It measures the upper limit of capabilities rather than a practical product.

How the three metrics compared

The final rescue pool contained 40 unique Qwen runs that exceeded their limits across the tested modes and therefore scored zero under the strict metric. In the budget-forced retry, the model received 5,000 tokens and an explicit instruction to close its answer. It correctly rescued 11 of 40 attempts (27.5%). The unbounded variant had a 100,000-token ceiling and an approximately 85-minute window on the node. It completed only 13 of 40 attempts and answered 8 correctly (20%).

Forced closure therefore solved more overflows in this rescue pool than nearly unlimited thinking, at a fraction of the compute cost. Roughly two thirds of the hardest attempts still failed to produce a usable conclusion even with the exceptional limit. Here, "did not finish" usually meant "could not decide in time," not "needed a little more room."

The pool included overflows from both tested Qwen modes. When the rescues are folded back into the full reasoning branch only, its score rises from 82.8% strict to 86.1% capability with budget forcing and 87.2% with the unbounded run. The latter added only 1.1 percentage points at vastly greater time and compute cost. In practice, a fixed budget plus forced closure is therefore preferable to repeatedly raising the ceiling. It brings reasoning Qwen to roughly the level of its no-reasoning version (86.7%), but at substantially higher cost.

The paper Scaling LLM Test-Time Compute Optimally shows that a smaller model can outperform a much larger one with well-distributed computation time. Our result adds a practical corollary: that time must be divided wisely. Increasing the limit alone is not a strategy.

What the ten hypotheses showed

Not even four slots did quadruple performance

Aggregated speed remained around 16 to 17 tokens per second. For this large MoE model, a single stream ran into memory bandwidth and additional slots did not yield the expected scaling.

I expected more concurrent slots to raise the throughput by at least 1.5 times. I measured 16, 17 and 17 tokens per second for one, two and four slots respectively. Practically a straight line.

This corresponds to a well-known problem of MoE inference: batching can activate more experts and increase the pressure on memory throughput. It is described, for example, in the work Lynx. Therefore, I will not add slots on a single node for further benchmarks. I split the queries between multiple replicas on multiple nodes.

What went wrong

About a third of about 35 jobs failed over two days. Almost always my fault, not LUMI's.

• A health check without the -f parameter treated the 503 Loading model response as a healthy server. Eighty questions went to waste.
• The HTTP timeout was shorter than the longest generation. After 30 minutes, the client discarded the answer on which the model worked for 50 minutes.
• I underestimated the answer limits three times. That was the most expensive systematic error.
• The pkill command killed itself twice.
• The signed Hugging Face addresses expired before the 46GB file was downloaded and the client hung without error.
• Multimodal finetune did not have complete processor files. The correct solution was to supplement them from the basic model.
• On a shared node, a foreign process could clean up shared memory under a running vLLM.

I wrote down all the errors in symptom, cause, fix and prevention format. This may end up being one of the most valuable outputs. Another team no longer has to pay the same tuition.

How much did it all cost?

Including failed runs, fixes, replicas, overnight jobs and the final unbounded runs, I used about 55 GPU-hours out of the allocated 5,000, roughly 1.1% of the allocation. The external API cost about CZK 350.

This is perhaps the least intuitive result. Serious pilot research on a European supercomputer doesn't have to be expensive when hypotheses are written in advance, runs are repeatable, and dead jobs are stopped quickly.

Access is not limited to large laboratories

I reached LUMI with the help of Jakub Siwek and the Czech IT4Innovations team, which provides access to the LUMI AI Factory. I started with the lowest playground allocation of 5,000 GPU-hours. According to Jakub, successfully completing one of the smaller programs also creates a route to applying for the 50,000+ GPU-hour tier. It is not an automatic ticket, but a pilot produces exactly what a larger application needs: a working workflow, initial data and evidence that the allocated compute will be used well.

Waiting was not dramatic either. Small development jobs started within seconds or minutes. Monday's delay of about two and a half hours was caused by security maintenance across the infrastructure, not by a normal queue.

For a company or developer, the encouraging message is that a journey to a supercomputer does not have to begin with buying hardware or securing a grant for a huge laboratory. It begins with a good question, several hypotheses written down in advance and a conversation with the national AI Factory.

What will come of it in practice?

Jakub Siwek from IT4Innovations asked me during the experiment where it all leads in the long term. The answer is simple: to the decision map for deploying AI for reasonable money.

When is a small open model enough? When does a fusion of several small models help? When is it worth paying more for a large model? When should reasoning be enabled? And how do we know whether compression has damaged the capability we need?

This data can support a smart router for HyperProstor. It would send each query to the cheapest system likely to handle it, escalating only the hardest cases to an expensive specialist. RouteLLM shows that similar routing can significantly reduce cost while retaining most of the quality. I want to test that idea on my own data, across multiple models and real business tasks.

There is also a European dimension. Companies are often afraid of AMD because they developed their models in the NVIDIA ecosystem. For training, the difference is still significant. However, for the operation of ready-made open models, the entry barrier is smaller than is traditionally believed. Within two days I ran Qwen, GLM, ThinkingCap and the 435GB quantized model on European AMD hardware. I didn't have to teach them again. I needed the right containers, engines and some expensively found switches.

The first article showed that I could connect to LUMI and run batch processing. This second one shows something more interesting: European infrastructure can support honest, repeatable research on open models, producing results that are more than a rewrite of vendor tables.

And it's just starting now.

Methodological note

This is a pilot, not a final ranking. The open-ended test in Experiment 1 contained one task. The GPQA part worked with 60 out of 198 questions and not all configurations had the same number of repetitions. Therefore, I use the results to decide the next stage, not to say that one model generally beats the other.

The next step is to expand the number of jobs, add more seeds, report confidence intervals, and compare the fusion against equally expensive baseline methods. Only then will the learned router take its turn.

Sources and data

• LUMI AI Factory: Czechia's Alpha Industries validates LUMI's power and sustainability
• LUMI-G node documentation
• GLM-5.2: model card and vendor benchmarks
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• Li et al. (2025): Quantization Meets Reasoning
• Lotfi et al. (2026): Quantized Reasoning Models Think They Need to Think Longer, but They Do Not
• Snell et al. (2024): Scaling LLM Test-Time Compute Optimally
• Gupta et al. (2026): Lynx: Efficient MoE Inference
• BottleCap AI: ThinkingCap-Qwen3.6-27B and its model card
• LMSYS: RouteLLM

I orchestrated the experiments with Fable 5 under my own supervision. Hypotheses, decision rules, raw JSONL logs, engine versions, and per-query results are stored in the project. I am not publishing the content of the GPQA questions themselves due to the integrity of the benchmark.