Open-VC Attention: an open-source, error-tunable variant of VC-Attention

On B200, FP8 doubles tensor-core throughput, but the special-function unit (MUFU) that computes exponentials did not get faster (NVIDIA). With one exponential per attention score, a kernel that sends them all through MUFU tops out near 2.53 PFLOP/s, about half the FP8 peak. VC-Attention (Li et al., 2026) removes that bottleneck with ExpCast: an FP8 (E4M3) byte is almost an affine function of the logarithm of the value it encodes, so the FP8 code of a softmax probability can be computed straight from the score with a single fused multiply-add, with no exponential and no FP32-to-FP8 conversion. The paper pairs ExpCast with V-Smooth, a value-clustering scheme that reduces V’s quantization error. Open-VC keeps ExpCast and builds everything around it differently.

ExpCast is the paper’s contribution; making it fast is ours. Once the exponential is nearly free, the cost moves to everything around it: scale handling, rescales of the running output, data movement and input preparation. This section covers what Open-VC does about each, and how it handles V’s quantization error without V-Smooth.

Scheduling around ExpCast

One instruction per score. Q and K are quantized to FP8 with one scale per 128-token block, and V with one scale per head. The kernel never materializes dequantized Q or K. Instead, the softmax scale, the Q and K dequantization scales, log₂e, ExpCast’s factor of 8 and the running maximum all fold into the one fused multiply-add that produces each ExpCast code, and byte permutes pack four codes into a single register. Because K’s scale changes from tile to tile, raw score maxima from different tiles are not comparable, so the kernel tracks the running maximum on the true score:

A consistent denominator. The softmax row sum adds up the same decoded FP8 probabilities that feed the PV product. The output is therefore a weighted average of value rows under exactly the weights the kernel used, so ExpCast’s approximation shows up only as a mild, bounded reweighting of keys, never as a mismatch between numerator and denominator.

Mid-window key traversal. ExpCast has no headroom above the running maximum. BF16 FlashAttention-4 can skip a rescale when the maximum grows only slightly, but under ExpCast a score just 0.074 nats above a stale maximum would be clipped. Open-VC therefore applies every increase, and each one costs a rescale of the running output on the CUDA cores, which are already the bottleneck. The cheapest fix is to find the maximum early. Video tokens are flattened in space-time order, and keys near the query’s own position usually score highest, so the kernel starts a few blocks past the query’s diagonal block, sweeps down to block 0, then visits the tail. Every block is still visited exactly once; only the order changes.

A warp-specialized pipeline. Separate warps load K and V tiles, issue the tensor-core matrix multiplies, and run ExpCast and the softmax bookkeeping. QK^T, score conversion, the denominator, PV and the output correction form one pipeline, coordinated by four hand-offs: scores ready, probabilities ready, PV issued and output rescaled. Each CTA holds two 128-row query tiles that alternate, so one tile’s softmax runs while the other’s matrix multiplies execute. On B200, each K tile’s scale is loaded early enough to overlap the wait for that tile’s scores.

Packed V. The PV product reduces over keys, so Open-VC stores V transposed and padded, as [H, 128, S_padded], with the key dimension contiguous. That matches the operand layout of the FP8 PV path, so the tensor cores read V with contiguous loads.

Fused input preparation. Activations change at every denoising step, so the BF16 inputs have to be quantized on every call. On B200 this takes three stream-ordered kernels: one quantizes Q and K per block, one reduces each head’s V maximum, and one converts V straight into the packed layout together with its scale. The result is byte-identical to the general preparation path and can be captured in a CUDA graph. At 73,397 tokens × 56 heads it adds 0.86 ms (1.5%) to the attention call, and since preparation grows linearly with sequence length while attention grows quadratically, that share shrinks for longer videos.

Why we left out V-Smooth

V-Smooth clusters value tokens with online k-means, permutes K and V so each 128-row block holds similar tokens, and subtracts the block mean before quantizing V. It is a sound idea for low-bit formats, but at FP8 it does not pay for itself.

The reason is how E4M3 error behaves. Error grows with magnitude, so subtracting a mean lowers V’s quantization error only by the share of energy the mean carried. The paper’s own energy shares predict its measured error ratios almost exactly (0.957 vs 0.961, and 0.696 vs 0.707).

The cost, meanwhile, lands right on the bottleneck. Restoring the means adds per-tile work on the CUDA cores, and a step that regroups with k-means takes 4.7 ms longer at 56 heads, or 7% of VC’s attention time. Permuting keys by value cluster also scrambles the positional order that the mid-window traversal depends on.

At 4 bits, things look different. With NVFP4, the same relative gain applies to a much larger base error, which is where VC-Attention reports V-Smooth’s speedups.

Why V residual repair

Dropping V-Smooth still leaves V’s quantization error, which is most of the total: VC-Attention attributes 82% of attention-output error to V on Wan2.2. With one scale per head, every value token j leaves a residual:

That residual is not spread evenly. A small number of tokens round badly and carry most of the error. So rather than reshaping the whole sequence to shrink every residual a little, as V-Smooth does, we fix the worst few exactly.

For each head, we rank tokens by residual energy Σ_c r_j,c² and keep the top ρ·S. Each selected residual becomes an extra FP8 value row, paired with a copy of its original key. The kernel treats these rows like any other key, so it adds the missing P_:,jr_j term without any change to its inner loop. Repair rows are visited last and the denominator is restored, so they only affect the numerator. Nothing is clustered or permuted, so the mid-window order stays intact, and the budget ρ sets how much extra time you spend for lower error.

The first 0.5% of tokens deliver most of the gain at a small extra cost. Beyond that, cost grows linearly while the gain flattens, so 0.5% is the operating point we use.

On B200, with activations taken from a MiniMax-H3 video model, Open-VC sustains 2.70-2.76 PFLOP/s in the attention kernel. That is about 60% of the FP8 dense peak, and above the 2.53 PFLOP/s ceiling we derived earlier for any kernel that sends its exponentials through MUFU. It is about 2× faster than BF16 FlashAttention-4, still about 2× once FP8 input preparation is included, and about 1.2× faster than our implementation of the VC-Attention method. Output error against BF16 is about 3% relative L2, and slightly lower with V repair turned on.

We timed three implementations of the same operator in one harness on one B200: BF16, the unmodified upstream FlashAttention-4 4.0.0b33 release; VC, our efficient implementation of the published VC-Attention method (ExpCast plus V-Smooth value grouping) on the same kernel family; and Open-VC. The inputs are BF16 Q, K and V from a real MiniMax-H3 denoising step, shape [73,397, 56, 128]. Seven heads is the per-GPU shape when this model runs with 8-way Ulysses sequence parallelism. Every call is CUDA-graph captured, and we report medians of 12 rounds in randomized order.

The speedup holds from 7 to 56 heads, so the sharded shape used in multi-GPU inference benefits as much as the full model. Both FP8 kernels use the same ExpCast probabilities, so the 1.21-1.23× between VC and Open-VC comes entirely from Open-VC’s scheduling and data layout.

Accuracy is close between the two FP8 kernels: VC’s value smoothing is 3-4% better in relative terms, and Open-VC’s 0.5% V repair closes about half of that gap at a fraction of the cost.

Kernel metrics only tell part of the story, so we also generated full videos with MiniMax-H3, swapping only the attention kernel. Each panel uses the same prompt and seed.

As with any change in numerics, the FP8 runs are not pixel-identical to BF16. Small differences early in denoising can change a background element or a camera path, while subject, layout and style stay with the prompt.

Open-VC Attention targets forward inference on long, non-causal video sequences: head dimension 128, multi-head attention, and at least 32,768 tokens for the packed fast path. Other calls fall back to a general path with the same semantics. Fused preparation and V repair currently run on B200, and B300 has its own schedule.

Open-VC is derived from the FlashAttention-4 CuTe DSL kernel and implements ExpCast-FP8 from VC-Attention (Li et al., 2026). Open-VC is an implementation inspired by the published paper.

Kernels like this one are part of the inference stack behind the video models we serve. Talk to us if you want this stack for a private deployment in your VPC or dedicated cloud.

Curious how your models perform on MachGen? We’ll run a low-touch PoC in 2-4 weeks, handling the optimization end-to-end while your team stays focused. You’ll get a live endpoint to evaluate the results on your own workloads.