Journal / 10 September 2026 / mage-001
A faster call can hide a slower kernel
The first profiles changed the question. Before choosing a language, I need to understand which part of the work became faster.
I want to push further into writing kernels. The first versions give me something concrete to work with: five mathematical operations, implementations in Rust and Triton, a PyTorch reference, and profiles from the same 4090.
Some of the rewrites improve the measured time. Others make it worse. The more interesting result is that even the answer to “which is faster?” changes when I change the boundary of the measurement.
Where does the work begin?
Take bias followed by GELU, a nonlinear transformation used in neural networks. The mathematical operation is the same across the three implementations. In our CUDA-event measurements, the Rust call spans about 13 microseconds and the Triton call about 24. In a separate GPU profile, the Triton kernel executes in about 8 microseconds and the Rust kernel in about 11.
The interactive Mage graphs let you switch between these views and the number of kernel launches.
An event span can include a gap while the host submits work. A kernel duration describes execution on the GPU. The application experiences a larger system: allocation, submission, computation, synchronization, and whatever comes next. A shorter span around one call does not by itself establish that its kernel does less work or that a service will respond faster.
These measurements come from separate runs with different launch rhythms, so subtracting them would not isolate Python overhead. That uncertainty is useful. It tells me which experiment to build next.
The equation still leaves choices open
The same pattern appears elsewhere. The first custom matrix multiplications are slower than the library-backed PyTorch call. Both custom triangle contractions use one kernel launch, but PyTorch’s three-launch expression still has the shortest total GPU execution time in that capture.
Reducing the number of launches is one possible improvement. It does not settle the choices inside the kernel: which values are reused, how threads share partial answers, how memory is arranged, or which instructions execute the arithmetic.
This is the part of revisiting linear algebra that interests me. An equation names a relationship. Implementing it forces me to think about the arrangement of information that makes the relationship affordable to compute.
There is another relationship worth keeping in view. If a kernel accounts for a fraction $f$ of a request’s time and becomes $s$ times faster, the simplified overall speedup is $1/(1-f+f/s)$. If that fraction is 20% and the kernel becomes twice as fast, total latency falls by only 10%, assuming everything else stays the same. The larger system limits what an isolated improvement can deliver.
What I want to try next
The next investigation starts with the launch path. I want to compare the same fused operation through warmed calls, batches of launches, and graph replay where the pinned tools support it. A compiled PyTorch baseline belongs in that comparison too. The aim is to find out whether the apparent native advantage survives a more closely matched execution pattern.
Then there are more demanding questions: how reductions share work, how a graph with uneven numbers of neighbors changes scheduling, and when a different tensor layout exposes useful reuse. Lower precision and tensor-core experiments are another direction, with their own accuracy contracts. I want each change to test an explanation, and I want to keep the regressions alongside the improvements.
For the production question, the current results are a starting point. They favor retaining the library-backed dense operations and investigating Triton for custom PyTorch work. Rust also raises questions about native integration and control of execution. A representative model or service will have to decide whether those advantages matter in context.
AI makes it easier for me to reach these experiments. What I want to develop alongside that speed is the ability to say what an experiment means. That is how this notebook can become useful beyond the code it produces.
The next research steps are in Mage. The kernels, full measurements, and reproduction instructions stay there; this journal follows the questions and what changes as I work through them.
Measured values
Five FP32 operations, three implementations, one RTX 4090 under WSL2. The figures below are the retained measurements behind this entry; the method and reproduction steps are in the measurement record.
Kernel time and time around the call for the same five operations, in one figure:
Top row: time inside the kernels. Bottom row: time around the call. Each column has its own scale, so implementations compare within a column. Values are in the two tables below.
GPU kernel time, summed per operation. Separate Nsight Systems capture, 100 iterations per measurement; gaps between launches are excluded. The WSL timestamp fallback has reduced precision.
Values (µs)
| Operation | PyTorch | Triton | Rust |
|---|---|---|---|
| Matrix multiplication | 56.0 | 83.8 | 344.0 |
| Bias + GELU | 15.9 | 7.9 | 11.2 |
| LayerNorm | 11.4 | 8.2 | 18.6 |
| Triangle contraction | 28.5 | 93.9 | 81.2 |
| Neighbor aggregation | 78.3 | 7.9 | 10.3 |
Mean of 300 warmed CUDA-event spans, collected in three rounds with rotating implementation order. Whiskers show the range of the three round means, not a confidence interval. A span can include gaps while the host submits work.
Values (µs)
| Operation | PyTorch | Triton | Rust |
|---|---|---|---|
| Matrix multiplication | 50.9 | 88.8 | 326.7 |
| Bias + GELU | 27.7 | 24.3 | 13.0 |
| LayerNorm | 17.8 | 22.6 | 19.5 |
| Triangle contraction | 55.6 | 91.9 | 78.5 |
| Neighbor aggregation | 89.7 | 21.8 | 13.2 |
Captured launches divided by 100 iterations. Both custom implementations use one kernel per operation. Fewer launches explain part of the result; they do not determine kernel duration.
Values
| Operation | PyTorch | Triton | Rust |
|---|---|---|---|
| Matrix multiplication | 1 | 1 | 1 |
| Bias + GELU | 2 | 1 | 1 |
| LayerNorm | 1 | 1 | 1 |
| Triangle contraction | 3 | 1 | 1 |
| Neighbor aggregation | 4 | 1 | 1 |
These are forward-only learning kernels at five fixed shapes, on one WSL workstation with unlocked clocks. Compilation, transfers, and service startup are excluded. Each row has its own scale, so implementations are comparable within a row and not across operations. The two views come from separate runs with different launch rhythms, so subtracting one from the other does not isolate Python overhead.