On October 19, OpenAI published new evaluations for coding in languages it describes as low-resource, including COBOL, Fortran, and Visual Basic. The results invite a practical question beyond whether a model can write code: can a team preserve what a working system does while changing how it is built? OpenAI reports that GPT-6.1 Sol reached 86.6 percent accuracy at low reasoning effort and 97.9 percent at higher effort on its COBOL migration evaluation. Those are company-reported benchmark results, not independently reproduced evidence or a production migration.

“Low-resource” describes the amount of code and documentation available for model training. It does not mean that a language has few users, little value, or no future. A system written in a less-represented language may still carry work people depend on today. Its age says little by itself about whether it should stay, change, or go.

A benchmark is a narrow light

The new report separates two tasks. One asks a model to build a SQLite-compatible database engine in 20 programming languages. The other asks it to translate existing COBOL programs into languages such as Java, Python, or Rust while preserving observable behavior. The distinction matters: creating a new example and changing a system with accumulated rules are different kinds of work.

For the first task, OpenAI has published the MultiLangBench repository. Its README describes a model implementing SQLite 3.40.1, with generated programs checked against database-output tests. The repository contains 24,000 converted tests but recommends a 272-test subset for ordinary evaluation; the full set is possible for higher-scoring samples. The October 19 article does not say which configuration produced its figures. That leaves an important limit around each score.

The separate COBOL migration result uses hidden tests and requires exact parity on observable outputs, according to the report. That is a meaningful benchmark condition: code that runs but changes a result does not pass. It still says nothing by itself about whether the hidden cases cover a particular organization’s rules, connected systems, or exception paths. The report itself says production use requires expert review and testing against business rules and dependencies.

A March CangjieBench preprint tested 248 tasks in Cangjie, another low-resource general-purpose language. The authors report that code-to-code translation often underperformed text-to-code generation. The paper is a preprint, covers Cangjie rather than COBOL, and cannot validate OpenAI’s results. It does underline why “the model can code” is too broad when the task is translation.

The code is also institutional memory

Programs keep decisions alive after the meeting that made them has disappeared. Some rules are visible in documentation; others appear only in a condition, a rounding operation, or the order in which records are handled. A maintainer has to understand both the language and the reason a behavior may matter.

OpenAI’s example makes the point concrete without making it universal. Its report notes that COBOL fixed-point arithmetic can discard excess decimal digits unless rounding is explicitly requested. A rewrite that rounds those values may execute successfully and still produce a different financial result. The detail is not a quaint feature of old code. It is a decision that has consequences when the code runs.

Current UK Government guidance on managing legacy technology names missing documentation, lost staff, unclear ownership, and skills shortages as migration barriers. It recommends phased change, documentation, and compatibility checks across systems. This is government guidance, not research on every team, but it shows how software knowledge can live in people, data, and dependencies as well as code.

That is why age is an incomplete diagnosis. The UK’s updated definition of legacy IT centers the label on persistent, unacceptable burden or risk, not on whether a technology is old or unfashionable. A system can be decades old and still meet a need. Another can be newer and already difficult to support safely. The decision should begin with evidence about the system and the work around it.

Migration asks what must remain true

Behavior-preserving migration is not a promise to keep every oddity forever. It is a way to distinguish intended behavior from accidental behavior before changing either. An output that looks strange may be a bug, a policy choice, a workaround for another system, or a rule nobody has recorded. The code alone may not tell the team which one.

A careful migration can begin with an inventory of observable decisions: what inputs arrive, what outputs leave, what records change, what exceptions interrupt the path, and which other systems rely on each result. For each decision, write down the evidence, the person or role that can explain its purpose, and any unanswered question. This is an editorial method, not a procedure tested by the cited benchmarks.

Then freeze representative cases from the current system. Include ordinary transactions as well as boundaries: missing records, decimal precision, duplicate entries, dates at a period change, and rejected inputs. Have knowledgeable reviewers mark which outputs are required, which are allowed to change, and which are not yet understood. Label a deliberate redesign as a redesign; do not let it hide inside a translation.

Run the old and new versions against the same cases. Compare the observable results, inspect every meaningful difference, and trace it to an approved rule or a defect being fixed. Test the connections to other systems, not only the new program in isolation. If the team cannot explain a difference, it has found a question to resolve, not a reason to declare equivalence.

Make the knowledge travel with the code

The work does not end when the test suite turns green. A useful migration record preserves the source version, test cases, expected behavior, approved exceptions, unresolved assumptions, and the reason for each accepted difference. It names who can make a decision when the system behaves in a way the test set did not anticipate. The next maintainer should be able to distinguish a documented rule from a guess that happened to pass once.

Use a phased cutover where the system allows it. Keep a route to compare results and restore the previous behavior while the changed version is still being observed. The UK guidance recommends iterative migration and ongoing documentation; applying those ideas to a particular system requires its own safety and service plan. A benchmark score cannot supply that plan.

The most valuable outcome may not be a replacement language. It may be a clearer account of what the program does, why it does it, and who can answer when the facts run out. Old code remains present tense because someone is maintaining it, depending on it, and carrying its consequences. A responsible change makes that knowledge easier to share before it asks the code to speak in a new form.

Sources and limitations