OpenAI Puts Low-Resource Code to a Byte-for-Byte Test

# OpenAI Puts Low-Resource Code to a Byte-for-Byte Test

*Lukas Hoffmann, Europe & Frontier Correspondent, Berlin — October 20, 2026*

On October 19, OpenAI published a detailed technical report↗ titled "Advancing coding in low-resource programming languages," introducing two new evaluations designed to measure something the AI coding benchmark ecosystem has largely ignored: how well frontier models handle languages that are underrepresented in public training corpora. The results are striking, the methodology is unusually transparent in places, and the caveats — buried but important — deserve as much attention as the headline numbers.

The timing is not accidental. Legacy software modernisation has become one of the most commercially compelling use cases for large language models, and the competitive pressure to demonstrate credible progress is intense. IBM data cited in the report states that COBOL alone underpins more than 40% of online banking systems and 95% of ATM transactions globally. The market for migrating, maintaining, and extending that code is enormous. OpenAI is signalling, carefully, that its newest models are beginning to close the gap between what AI can do in Python and what it can do in COBOL.

Two Evaluations, Two Different Problems

The publication describes two distinct benchmarks, each targeting a different failure mode.

MultiLangBench asks a model to implement a SQLite-compatible database engine from scratch across 20 programming languages, including C, C++, COBOL, Fortran, Java, JavaScript, Lean, OCaml, Python, Rust, Visual Basic, and Zig. The task is deliberately non-trivial: a working database engine requires correct handling of data types, query parsing, and storage semantics — all of which vary substantially across languages. The evaluation uses a pass-rate metric, checking whether the generated code compiles and passes a suite of functional tests.

The COBOL migration evaluation is narrower but arguably more commercially relevant. It tests whether a model can translate legacy COBOL programs into modern target languages — C++, Java, Kotlin, Python, or Rust — while preserving byte-for-byte parity in observable output. That last requirement is the hard part. COBOL's arithmetic semantics differ from those of every modern language in ways that matter enormously for financial applications: the language truncates rather than rounds in certain decimal operations, and a migration that silently changes that behaviour could corrupt financial records. The benchmark is designed to catch exactly those failures.

GPT-6 Astra vs. the Field

The performance gap between GPT-6 Astra and its predecessors is the most immediately legible finding in the report. On MultiLangBench, GPT-6 Astra achieved an average pass rate of 87.4% across all 20 languages, compared to 26.8% for GPT-5.6 Sol. That is not a marginal improvement; it is a qualitative shift. Astra scored above 83% in 19 of the 20 languages, suggesting that the performance floor has risen substantially, not just the ceiling.

The COBOL migration results are even more dramatic. GPT-6.1 Sol — a mid-tier model, not the flagship — achieved 86.6% accuracy at low reasoning intensity and 97.9% at higher reasoning intensity on the migration task. The baseline for comparison is GPT-5.6 Sol, which managed 5.4%. Independent analysis from CellCog↗ confirms these figures align with what the raw evaluation environment produces, and Interesting Engineering's coverage↗ notes that the jump from 5.4% to 86.6% represents the kind of threshold crossing that makes enterprise procurement conversations materially different.

The cost dimension is also worth noting. GPT-6 Luna, the lower-cost variant, achieved a 40.3% pass rate on MultiLangBench at approximately 78% lower cost per sample than GPT-5.6 Sol. For organisations running large-scale code analysis pipelines, that cost-performance trade-off is directly relevant to build-versus-buy decisions.

What the Benchmark Actually Measures — and What It Does Not

OpenAI's transparency on methodology is partial but meaningful. The MultiLangBench evaluation environment, test suite, and results have been released on GitHub↗, allowing independent replication. That is a genuine contribution to the evaluation ecosystem, and it stands in contrast to the opacity that has characterised many previous benchmark releases from frontier labs.

The COBOL migration evaluation is less open. The specific program samples and hidden tests have not been fully disclosed, which limits independent verification. OpenAI acknowledges this asymmetry in the report, framing it as a necessary precaution against benchmark contamination — a reasonable concern, but one that also means the 97.9% accuracy figure cannot be fully audited by external researchers.

More importantly, the report is explicit about what the benchmarks do not measure:

• Production readiness: OpenAI states directly that these results do not indicate that AI-assisted COBOL migration is ready for unsupervised deployment. Real-world migrations involve business rules, external dependencies, and edge cases that no benchmark can fully enumerate.
• Semantic correctness beyond observable output: Byte-for-byte parity in output is necessary but not sufficient. A migration could pass the benchmark while introducing subtle logic errors that only manifest under specific input conditions not covered by the test suite.
• Human expert review: The report frames AI assistance as a tool for augmenting, not replacing, the domain expertise of engineers who understand both the legacy system and the target environment.

"Effective modernisation of legacy enterprise systems remains complex, requiring rigorous human expert review and testing against business rules and external dependencies that automated benchmarks cannot fully capture." — OpenAI, October 19, 2026

That caveat is not boilerplate. It reflects a genuine epistemic limitation that any organisation considering AI-assisted migration should take seriously.

The Low-Resource Language Problem in Context

The framing of "low-resource" languages in the context of AI coding is worth unpacking. The term refers not to the computational resources required to run the language, but to the scarcity of publicly available code in that language — which directly affects how much training signal a model receives. Python, JavaScript, and TypeScript are high-resource: GitHub, Stack Overflow, and the broader web are saturated with examples. COBOL, Fortran, and Visual Basic are low-resource: most of the code that exists in those languages lives in private enterprise repositories, mainframe systems, and internal documentation that never reaches the public internet.

This creates a systematic bias in model capabilities that has been well-documented in the research literature but rarely addressed directly in commercial benchmark releases. OpenAI's decision to publish evaluations specifically targeting this gap is notable, even if the results are self-reported and the methodology is partially opaque.

The Broader Competitive Landscape

OpenAI's benchmark release does not exist in isolation. The past two weeks have seen a cluster of technically substantive announcements from across the Western AI lab ecosystem that collectively define the competitive context.

Mistral AI has been particularly active. On October 6, the Paris-based lab launched Mistral Large 4↗, a natively multimodal model with 1 trillion total parameters and 49 billion active parameters, trained on 3,800 NVIDIA Grace Blackwell GPUs at Mistral's European datacentres. The model is positioned as a sovereign alternative to closed-weight frontier systems, with particular strength in cybersecurity and legal/financial tasks. On October 15, Mistral followed with Timestral↗, a family of time-series foundation models ranging from 5M to 209M parameters, with the 209M variant claiming first place on the GIFT-Eval and TIME benchmarks. Both releases reflect Mistral's consistent strategy: open weights, European infrastructure, and domain-specific performance claims that are verifiable against public benchmarks.

Google DeepMind has been expanding its research access programme. Reuters reported on October 14↗ that DeepMind and Google Cloud committed to providing up to 200 European researchers with access to frontier models and $3 million in compute credits through the EU RAISE initiative. That is a relatively modest number in absolute terms, but it is strategically significant: it positions Google as a partner to European academic institutions at a moment when the EU AI Act is reshaping the regulatory environment for frontier model deployment.

On the regulatory front, Reuters reported on October 19↗ that the US government is moving to ban DeepSeek from federal agency IT systems, citing national security concerns. The ban does not directly affect Western labs, but it reinforces the bifurcation of the global AI market along geopolitical lines — a dynamic that benefits European and American providers in government and regulated-industry procurement contexts.

Why Legacy Code Matters Now

The commercial logic behind OpenAI's benchmark release is straightforward. The global installed base of COBOL code is estimated at 800 billion lines, with an annual maintenance cost that runs into the tens of billions of dollars. The engineers who wrote that code are retiring faster than new COBOL developers are being trained. The problem is not theoretical; it is an active operational risk for banks, insurers, and government agencies that depend on systems built in the 1970s and 1980s.

AI-assisted modernisation has been discussed as a solution for several years, but the gap between benchmark performance and production reliability has been wide enough to prevent serious enterprise adoption. The question OpenAI's publication implicitly raises is whether that gap is now narrow enough to justify pilot deployments — not full automation, but AI-assisted workflows where a model generates candidate migrations that human engineers then review and validate.

The answer, based on the published results, is: possibly, for well-defined migration tasks with robust test coverage. The 97.9% accuracy figure for GPT-6.1 Sol at high reasoning intensity is impressive, but it applies to a controlled benchmark environment. The relevant question for an enterprise is not whether the model can pass OpenAI's tests, but whether it can handle the specific quirks of their specific COBOL codebase — which may include undocumented business logic, non-standard compiler extensions, and decades of accumulated technical debt.

What Developers and Architects Should Watch

For practitioners evaluating AI-assisted modernisation tools, several specific considerations emerge from the OpenAI publication:

• Reasoning intensity matters: The gap between 86.6% and 97.9% accuracy on the COBOL migration task corresponds to different levels of compute applied at inference time. Higher reasoning intensity means higher cost and latency. Organisations need to model that trade-off against the cost of human review for errors the model misses at lower intensity.
• Language-specific performance varies: MultiLangBench results show that even GPT-6 Astra's performance is not uniform across all 20 languages. The report does not publish per-language breakdowns in full, but the aggregate data suggests that some languages remain significantly harder than others. Organisations with COBOL, Fortran, or Visual Basic codebases should not assume that aggregate benchmark scores translate directly to their specific language.
• Test coverage is the binding constraint: Byte-for-byte parity is only verifiable if you have comprehensive tests that exercise the relevant output paths. Organisations with weak test coverage for their legacy systems will find that AI-assisted migration amplifies, rather than reduces, their risk exposure.
• Open-source alternatives are maturing: Mistral Large 4's open-weight release, scheduled for later this month, will provide a sovereign alternative for organisations that cannot or will not route sensitive code through OpenAI's API. The performance gap between open and closed models on coding tasks has been narrowing steadily.

"The model must preserve specific legacy quirks, such as how COBOL handles decimal truncation versus rounding, which is critical for financial accuracy." — OpenAI technical report, October 19, 2026

Open Questions

The publication raises as many questions as it answers. OpenAI has not disclosed which specific COBOL programs are included in the migration evaluation, making it impossible to assess whether the benchmark is representative of real-world legacy codebases or optimised for tasks where current models perform well. The MultiLangBench GitHub release is more transparent, but the hidden test suite for the COBOL evaluation remains a black box.

There is also the question of what happens at the edges. A 97.9% accuracy rate sounds impressive until you consider that a 2.1% error rate on a financial system processing millions of transactions per day could mean thousands of incorrect calculations. The benchmark measures correctness on a fixed set of test cases; it cannot measure the tail risk of errors on inputs the test suite does not cover.

Finally, the competitive response from Anthropic, Google DeepMind, and Mistral on this specific benchmark will be worth watching. None of the other major labs have published comparable evaluations for low-resource language coding, which means OpenAI is currently setting the terms of the conversation. Whether competitors accept those terms or propose alternative evaluation frameworks will shape how the industry thinks about AI-assisted legacy modernisation over the next twelve months.

The benchmark is a genuine contribution. The caveats are real. The commercial opportunity is large. The gap between those three facts is where the interesting work will happen.

---

*Lukas Hoffmann covers frontier AI research and European AI policy for Neuron. He is based in Berlin.*