First Try Is the Differentiator: How LLMs Translate Python into Rust

Leo Walker · October 2026

TL;DR

Frontier labs now use their own models to port entire codebases to Rust. Coming from data science, I wanted to measure how well these models translate numerical Python, the NumPy-based code common in scientific work, into Rust.

I gave 17 models 39 numerical Python tasks to translate: 32 from NPBench, a NumPy benchmark, and 7 from Code Metal's radarlib. Every model got the same tools, reasoning effort set to medium, and up to four submissions per run. I scheduled three runs per model and task; one was lost to provider timeouts, leaving 1,988 scored runs. I graded each final answer on test inputs the model never saw.

Seven models reached 100% after repair. First-submission accuracy showed larger differences between models.
Cheaper per token is not cheaper per task. The open-weight models wrote about 2.3 times as many tokens and needed more repairs, so on average they cost more per task than the frontier models.
Best value: GPT-6 Luna and Claude Haiku 5.5, with GPT-6.1 Sol the most efficient of the top-tier models. DeepSeek V4.1 Flash is the strongest open-weight model.

All the code, tasks, and results data are open source at github.com/leoalord/numpy-to-rust-eval.

1 Introduction

The central question is whether the translated code does what the original did.

In numerical code built on matrices and arrays, compiling is only part of the job. The harder failures are silent: the program runs and returns the wrong numbers. A scaling term is dropped, two indexes are swapped, or a calculation runs at lower floating-point precision than the original. Nothing crashes, so nothing signals the error.

New models arrive every few weeks, usually with benchmarks from other domains, so I built an evaluation closer to my own background. I chose NPBench for its breadth, from linear algebra to physics simulations. Code Metal's work on verified code translation led me to add radar-processing tasks from their radarlib.

I compared models using the same tools and submission budget, measuring both accuracy and cost. Passing tests is evidence, not proof, so "correct" here means the Rust matches the Python outputs within the grading tolerances on inputs the model never saw.

2 Related work

TransCoder judged a translation by whether it passes tests, not by how the code looks, an idea that underpins evaluations like this one.

Recent projects apply similar checks to whole codebases. The Bun team ported Bun, about 535,000 lines of Zig, to Rust in 11 days with Claude Code. It passed Bun's test suite and still introduced 19 regressions, since fixed. Google reports Gemini 4 Argon agents porting libraries such as re2 and libgav1, and the Fuchsia Zircon kernel, to Rust. Both rely on the same two signals I use: compiler errors to drive repair, and outputs compared against the original.

Code Metal frames the goal as behavioral equivalence: the new code must behave like the old, checked on a spectrum from side-by-side testing to formal proof. This study sits at the testing end of that spectrum.

Their type-mapping work argues that the types at a function's boundary should come from a fixed schema rather than from the model. My task contracts fix the Rust function signatures in advance. Their workflows-versus-agents paper motivated this project and shaped the design of my harness.

3 Setup

3.1 The workflow

Each run is a single model working on a single task, driven by a four-node LangGraph graph.

Solid edges are the repair loop; dashed edges end a run early. Held-out results come only from the last node, so they never reach the model.

The model writes or edits a Rust file, which is then compiled and tested. Each submitted version is one submission, and each run allows up to four submissions. After a failure, the model sees the compiler errors or where its values were wrong, but never the values it should have produced. The run stops when every development test passes or after the fourth submission, and that last submission is graded on the held-out cases. Like the expected values, held-out results are never shown to the model.

3.2 Preparing a submission

A model call is one request to the LLM. A submission can involve several model calls, but its first successful code edit sends the file straight to compile and test. The model has three tools:

Tool	What it does	Submits the file?
`write_code`	Replaces the whole Rust file	Yes, if it succeeds
`replace_code`	Replaces one block of the file	Yes, if it succeeds
`declare_dependencies`	Sets which Rust libraries (crates) to install	No

A model must declare any dependencies before submitting, so they are installed when the code compiles. The prompt tells the models to answer with tools, and a reply without a tool call submits the file as it stands. If an edit fails, the model gets the error and can try again, up to ten model calls per submission. That cap rarely mattered: only 34 of the 2,640 submissions went forward without a successful edit. Preparing a submission took 1.16 model calls on average.

Models had no search tool in this round. They see the Python source, the Rust function signature, the starter crates, and their latest feedback, and never the grading code, a reference solution, or the expected values.

3.3 Tasks and models

NPBench (32 tasks) covers linear algebra, stencils (grid updates where each cell reads its neighbors), dynamic programming, FFTs, neural-net layers, and physics simulations.
radarlib (7 tasks) adds six synthetic-aperture radar stages plus the full Range-Doppler Algorithm, which chains two of them (range compression and RCMC) with azimuth compression.

Each task fixes its Rust signature up front, and the held-out cases include at least one input bigger than anything the model was tested on during the run.

All 17 models ran at reasoning effort medium, and each ran every task three times to reduce the influence of a single lucky or unlucky run. Seven are frontier models from OpenAI, Anthropic, and Google, called through their own APIs; the other ten are open-weight models I ran on Fireworks AI. One run was lost to provider timeouts, leaving 1,988 scored runs. The whole batch cost about $69.

4 Results

First-submission accuracy was 82.7%, and final accuracy after repair was 95.7%. The radar tasks widened the gap mainly among models outside the top six; section 4.4 breaks down that split. Each model keeps its maker's color in every chart.

Colors throughout show who served each model:

Open-weight models on Fireworks, labeled by maker:

4.1 First try separates the models

After repair, seven models score 100% and fourteen score 97.4% or higher, so final scores cluster near the ceiling. First-submission accuracy shows larger differences:

Pass rate by model

Filled dot is first try, colored by provider, with a bar for its 95% interval. The ring is the final held-out rate.

Intervals resample tasks, with trials kept together. A perfect score gives a zero-width interval, which says nothing about tasks outside this set.

Each model had 117 scored runs (116 for Inkling).

GPT-6.1 Sol and Claude Opus 5.5 got every task right on the first try. The smaller Claude Haiku 5.5 (98.3%) and GPT-6 Luna (91.5%) offer strong accuracy for their cost.

Haiku outscoring Sonnet was unexpected. Sonnet reported zero reasoning tokens in most runs, though this batch cannot show whether that explains the gap. Gemini 4 Argon's 50% on the neural-net tasks was also unexpected.

Among open-weight models, DeepSeek V4.1 Flash was both efficient and accurate, scoring slightly higher than GPT-6 Luna on the first try. NVIDIA's Nemotron Lightning 3.5 trailed the field at 30.8%.

4.2 Higher spending did not consistently improve accuracy

Cost per run varies about 70-fold, from $0.0013 for GPT-6 Luna to $0.093 for Gemini 3.8 Flash, and it barely tracks accuracy.

Mean cost per run vs first-try rate

Cost axis is logarithmic. Prices are the 2026-10-19 roster; a model is on the frontier when no cheaper model has a higher first-try rate.

Several models used many reasoning tokens (the "thinking" a model does before it answers) without a matching gain in accuracy, and that thinking adds both time and cost. Open-weight models are often cheaper per token, but they wrote about 2.3 times as many tokens and needed more submissions (1.51 on average against 1.06), so they cost more per run ($0.038 against $0.030). Kimi K3, Ember-1, and Inkling each cost more than Claude Opus 5.5 and scored 12 to 40 percentage points lower on first-submission accuracy.

GPT-6 Luna and Claude Haiku 5.5 offer the best value in this batch. Among models with 100% first-submission accuracy, GPT-6.1 Sol is the cheaper option, though it costs seven times as much as Haiku for 1.7 percentage points more. Gemini 3.8 Flash used so many tokens that it cost more than three times as much as Gemini 4 Argon, and it scored lower. DeepSeek V4.1 Flash nearly matches Argon (92.3% against 93.2%) at 40% of the cost.

4.3 "Medium" effort means different things

Every model was set to medium reasoning effort, but the setting translates very differently across providers. GPT-6.1 Sol's median was 123 reasoning tokens per run. Claude Opus 5.5, GPT-6 Luna, and Gemini 4 Argon were also in the hundreds; Claude Haiku 5.5's median was 1,974. Gemini 3.8 Flash's median was 21,565. Median run times ranged from 13 to 159 seconds.

Models that used more reasoning tokens or took longer were not consistently more accurate in this batch. Across models, longer runs tended to score lower. Within repeated runs of the same model and task, first submissions that failed used fewer reasoning tokens up front on average than those that passed. Neither comparison establishes that changing the reasoning budget would change accuracy. The two most accurate models were also reasonably fast, with median runs of 19 seconds for Claude Opus 5.5 and 26 seconds for GPT-6.1 Sol.

Median reported reasoning tokens per run

Log scale. OpenAI, Anthropic and Google report how many reasoning tokens they used but don't return the reasoning text; only the Fireworks-hosted models return readable reasoning. Claude Sonnet 5.5's median is 0: it thought in 38 of 117 runs. Inkling is left off; see Notes.

Median time per run vs first-try rate

Log scale. Time per run is wall-clock: every model call plus Docker compiles and tests, on one shared machine. Provider infrastructure speed is part of it, so treat it as what a user would wait, not a pure measure of thinking.

4.4 Difficulty sits in a few tasks

Across all models and trials, 26 of the 39 tasks have a first-submission pass rate of at least 80%. The hardest are resnet (33.3%) and scattering_self_energies (46.0%); four of the next seven are radar tasks (62.7–72.5%).

First-try passes out of three trials, every model and task

0 of 31 of 32 of 33 of 3held-out failed in a trialradar task

Rows sorted by first-try rate, columns hardest on the left. Inkling's scattering_self_energies cell has two trials because one was lost to gateway timeouts.

family average

Task first-try:95% and up80–95%60–80%below 60%

The six models with the highest overall first-submission rates—GPT-6.1 Sol, Claude Opus 5.5, Claude Haiku 5.5, Claude Sonnet 5.5, Gemini 4 Argon, and DeepSeek V4.1 Flash—average slightly higher scores on radar than on NPBench. The other eleven average about 20 percentage points lower on radar. The full radar pipeline, radar_rda, is harder than any of its stages on its own.

4.5 Repair fixes three in four failures

Of the 375 first submissions that failed during the batch, repair fixed 289: 203 on the second submission, 61 on the third, and 25 on the fourth. The second submission accounted for 70% of successful repairs. Of the runs that used all four submissions, 23% passed.

1. Pass rate after each submission

Share of runs that have passed held-out by each submission. Each grey line is one model; hover it to see which.

2. What went wrong on the first submission

An undeclared crate is code that uses a crate (usually num-complex or rustfft) without declaring it, so it fails to compile even when the code itself is right. GPT-6.1 Sol and Claude Opus 5.5 never failed one. The label shows how many of each model's failures repair turned into passes.

3. Which failures repair can fix

Share of first failures of each type that end up passing held-out

Weaker models mostly failed to compile and recovered less often; mid-tier models compiled but produced wrong values. No run that passed its visible tests went on to fail the held-out ones.

5 Discussion

First-submission accuracy was the more useful way to distinguish models in this task set, since final scores clustered near 100%. For an unattended translation tool, I would weigh that alongside repair success and cost. The frontier models' lower average cost per run came with fewer tokens and fewer submissions.

The numerical failures are worth a closer look. In 13 of resnet's 15 final failures, the model stored a padded buffer at single precision (f32) where NumPy uses double (f64). The tests caught the resulting errors. Fixing the input and output types alone does not prevent that mistake: the signature stays the same while the precision changes inside the function.

The harness itself needs improvement. The prompt described starter crates as "available" without making clear that they still had to be declared. Next time it should say "Starter crates are not included automatically; declare any you use." The one-edit-per-submission rule can also discard useful work: one GLM 5.3 run lost two correct follow-up edits and then broke the code again on its next submission.

Limitations and future work

Tests are evidence, not proof. The held-out cases only check the values they reach; Bun's port passed its full test suite and still introduced 19 regressions.
Correctness only. I do not measure how fast the translated Rust runs.
Three trials rank tiers, not neighbors. None of the top six differ from each other by a statistically significant margin.
One prompt, one effort setting, and no documentation search.
Prices and timing. Costs use the prices recorded in the October 19, 2026 model roster. All runs shared one machine, so run times include waiting and may differ on other hardware or under a different workload.
Next: the remaining 22 NPBench tasks, and a Rust documentation search tool (planned on Exa) to compare against this closed-book round.

Notes

Starter-crate correction. The prompt listed six starter crates as "available" without saying they still had to be declared, and 49 first submissions failed to compile for that reason alone. I replayed each one with the code unchanged and only the crate declared; 31 passed, and the first-try figures count them. Three models change: Muse Spark Open (69.2% as recorded), Claude Sonnet 5.5 (93.2%), and Inkling (57.8%).

Reasoning tokens for Claude Sonnet 5.5 and Inkling. Sonnet's median of 0 is real: with adaptive thinking at medium it chose to think in only 38 of its 117 runs (41 of 130 model calls), against 116 of 117 calls for Opus and all 120 for Haiku. LangSmith's records for every Sonnet run agree. Inkling did reason, and Fireworks returns its count, but in a field LangChain does not map, so neither my harness nor LangSmith recorded it; it is left off the reasoning chart. Its raw responses put the median at about 5,200 reasoning tokens per run. Costs and pass rates are unaffected, and the other nine Fireworks models' counts were recorded correctly.

How candidates are built and graded. Every candidate builds and runs in Docker with no network, a read-only filesystem, and memory, CPU, and time limits; only crate downloads touch the network. Outputs must match NumPy's shapes and types exactly, integers must be equal, and floats must agree within a relative tolerance of 1e-5 for single precision and 1e-9 for double. After a failure the model sees the error, or up to five mismatched positions, but never the expected values. A run that hits a provider error, $5 of spend, or repeated infrastructure failures ends early and is not counted against the model.

Code. github.com/leoalord/numpy-to-rust-eval. The data behind every figure is in its results/ folder.