NVIDIA's skill set has an agent rewrite Spark functions, and a test matching the old version comes first
NVIDIA released a skill set for agents to rewrite Spark UDFs for GPUs and reports a 22x speedup. Three checks to accept a rewrite only when its test matches the original.
Your data team has a slow function it wrote itself in Apache Spark (a platform for processing large datasets), and someone suggests handing it to a coding agent (an AI assistant that writes code) to rewrite for speed. On October 13, 2026, NVIDIA published a guide to doing exactly that and reported a large result: for one function, the operator time (the time spent in the step that calls the function) fell from 274.55 hours to 12.78 hours. What is worth copying into your own work is the order of evidence in the workflow, not the speedup: a rewrite is accepted only once a test has matched it against the original.
Where NVIDIA says the problem lies
A function you write yourself in Spark is called a UDF (user-defined function). NVIDIA writes that a UDF contains arbitrary code that the system can neither analyze nor run in parallel. When you use cuDF, the plugin that lets Spark run on a GPU (graphics processor), the data has to be converted from columns to rows, sent back to the CPU (central processor) to run the UDF one row at a time, and converted back. That slows down an entire query that was otherwise running on the GPU.
The post names three ways to rewrite, from easiest to hardest:
| Approach | Complexity, per NVIDIA | Best for |
|---|---|---|
| Rewrite as a SQL expression (a database query language) | Low | Simple relational logic, small functions |
| Rewrite with cuDF | Medium | Most cases, such as string parsing and nested types |
| Rewrite in CUDA (a GPU programming language) | High | Complex logic or maximum performance |
The example in the post is a function that converts degrees Celsius to Fahrenheit. The original returns null (empty) when the input is empty, and the formula is celsius * 9.0 / 5.0 + 32.0. The SQL version leaves only that formula inside the query.
The part worth learning: the agent works only inside a loop with evidence
NVIDIA writes that much of this work suits a coding agent “with proper guardrails”: the agent searches unfamiliar APIs (programming interfaces), writes a replacement and iterates quickly, while executable tests and performance measurements decide whether each change is accepted. Its skill set, cuDF skills for Apache Spark, splits the job into four feedback loops: testing, conversion, benchmarking and optimization. Each loop defines the work to perform and the evidence required before moving to the next. The skills come with a project template that includes a test runner, a benchmark harness and routines that check the output against the original.
On results, NVIDIA describes a large consumer technology company, unnamed, that converted more than 30 UDFs in production data pipelines. For one UDF that affected more than 3,000 jobs (tasks that run on a cluster), the operator time for that function fell from 274.55 to 12.78 hours on a 600-billion-row workload, about 22 times faster. NVIDIA says the rewrites preserved the behavior of the original CPU UDFs. These are figures published by NVIDIA, and the post gives no data from that workload for an outsider to rerun.
What you can do this week
This article has not run NVIDIA’s skill set, so the three steps below are the article’s own suggestion, based on the order NVIDIA describes, and have not been tried on any system.
- Pick a small, easy-to-read UDF and save the original’s output on a sample dataset. The sample needs empty rows, because NVIDIA’s original handles that case separately.
- Before the agent rewrites anything, write a test that compares the output row by row between the original and the new version. The input is the sample set; the expected output is zero differing rows. The agent must run this test after every edit.
- Measure time only after the test matches. If the test reports a difference on any row, send the rewrite back to the agent with the differing rows, and do not accept it because it is faster.
If the test fails on the empty rows, the new version has changed behavior, and you should keep the old UDF until the differing rows are explained.
Sources
- NVIDIA Technical Blog, “How to Rewrite Apache Spark UDFs for GPU Acceleration with AI Agents” (October 13, 2026) — describes the UDF problem, the three rewrite approaches, the agent’s four skill loops and one customer company’s result; this is the vendor’s own account and has not been independently verified.