ROLV Primitive© — Matrix Compute Primitive

01

A compute primitive for sparse AI workloads.

ROLV Primitive© is a software operator that restructures matrix arithmetic to skip zero-valued multiply-accumulate operations. At high sparsity levels — where 90% or more of a weight matrix is zero — this approach delivers substantial reductions in compute time and energy consumption.

Sparse by design

Works best when matrices are genuinely sparse. At 90%+ sparsity, ROLV Primitive© skips the vast majority of multiply-accumulate operations — the work simply does not happen.

Software-only

No hardware modifications. No new chips. No changes to model weights or architecture. Runs on existing CPU and GPU infrastructure.

Energy follows compute

Fewer operations means less energy. At 90%+ sparsity, energy savings scale proportionally with the work eliminated — a direct consequence of doing less arithmetic.

02

Real production weights · real model weights · synthetic sweep · CPU · all verified.

Tested on NVIDIA H200, NVIDIA B200, Intel CPU, and AMD EPYC 7B13 across 22 sparsity levels each, real LLaMA-3.1-8B weights from HuggingFace, and 4 real production LLM weight matrix pairs on Tesla T4. Each result includes 4 SHA-256 hashes and a perturbation test. Energy measured directly via pynvml on GPU; proxy on CPU.

Real model weights

4/4 PASS

LLaMA-3.1-8B from HuggingFace. Max error 3.9×10⁻⁶.

Multi-platform sweep

160/160 PASS

22 levels each on H200, B200, Intel CPU, AMD EPYC. All perturbation PASS.

Verification

4 SHA-256 per run

Weight matrix · input vector · dense baseline · ROLV output. Perturbation test on every case.

GPU — NVIDIA H200 · Meta LLaMA-3.1-8B · Real weights from HuggingFace · 4/4 PASS

Real model. Real weights. Up to 9.53× faster · up to 89.5% energy saved.

MLP up_proj layer (14336×4096) from Meta LLaMA-3.1-8B downloaded directly from HuggingFace. Magnitude row pruning at four sparsity levels. Max error 3.9×10⁻⁶ — 250× tighter than ATOL=0.001. All four perturbation tests pass.

Vendor note: cuBLAS runs at 2.48ms throughout. cuSPARSE is slower than cuBLAS at 80% sparsity (5.90ms vs 2.48ms) but faster at 95%+. Speedup below is always vs the best available vendor at each level. "vs cuBLAS" column shown separately.

Sparsity	Live rows	Compr.	Best vendor ms	ROLV ms	vs vendor	vs cuBLAS	Energy†	Pass
80%	2,867	5×	5.8984 cuSPARSE	0.6190	9.53×	4.01×	+89.5%	✓
90%	1,434	10×	3.0077 cuSPARSE	0.3475	8.66×	7.14×	+88.4%	✓
95% ★	717	20×	1.5547 cuSPARSE	0.2265	6.86×	10.96×	+85.4%	✓
99%	143	100×	0.4415 cuSPARSE	0.1720	2.57×	14.43×	+61.0%	✓

SHA-256 hashes — LLaMA-3.1-8B up_proj · NVIDIA H200
A (weight matrix)9b7d16f518ac5406a11bf6cb3ba2cb3204da3fb35614bef53e163fbe215bcfb1
V (input vector)32d38b5291bb7e2fdfb5df26616d3da6f7209f45e0f53d0ad89388a8811adf7e

★ = best ratio vs dense. † = time-ratio proxy (pynvml unavailable in this run — clearly labelled). H200 · LLaMA-3.1-8B layers[0].mlp.up_proj (14336×4096) · Batch=1024 · 100 iters · CUDA Events · CRCS · 4/4 perturbation PASS

GPU — NVIDIA H200 · 3 real LLM models · LLaMA-2-7B (pre-pruned) · LLaMA-3.1-8B · 72/72 PASS

Three production LLMs. Real HuggingFace weights. 72/72 PASS · up to 18.76× faster.

Tested on LLaMA-2-7B (NeuralMagic 50% pre-pruned variants) and LLaMA-3.1-8B across 4 layers and 6 sparsity levels each. Pre-pruned models from NeuralMagic use SparseGPT + knowledge distillation — full accuracy recovery. All 72 test cases PASS. Below 70%: vs cuBLAS. Above 70%: vs cuSPARSE.

18.76×

Peak speedup

LLaMA-3.1-8B embed · 95% · vs cuSPARSE

100%

Energy saved

pynvml · all 3 models at 80%+

72/72

Correctness

3 models · 4 layers · 6 sparsity levels

50%

Natural sparsity

NeuralMagic pre-pruned — full accuracy

Model	Layer	Sparsity	vs	Speedup	Energy	Pass
LLaMA-2-7B (GSM8K, 50% pre-pruned)	embed_tokens	70%	cuSPARSE	15.85×	+100%	✓
LLaMA-2-7B (GSM8K, 50% pre-pruned)	v_proj	95%	cuSPARSE	8.05×	+87.6%	✓
LLaMA-2-7B (Dolphin, 50% pre-pruned)	embed_tokens	70%	cuSPARSE	15.75×	+100%	✓
LLaMA-3.1-8B	embed_tokens	95% ★	cuSPARSE	18.76×	+100%	✓
LLaMA-3.1-8B	q_proj	80%	cuSPARSE	6.81×	+85.3%	✓
LLaMA-3.1-8B	k_proj	70%	cuSPARSE	3.79×	+73.6%	✓

★ = peak. Italic = cuSPARSE (correct baseline above 70%). Pre-pruned models: NeuralMagic SparseGPT + knowledge distillation, full accuracy recovery. NVIDIA H200 · 72/72 correctness PASS · 72/72 perturbation PASS · 4 SHA-256 hashes per case.

CUDA — NVIDIA H200 · 10,000×10,000 · Batch 2,500 · 2,000 iters · 22/22 PASS · pynvml

Full sweep 0%–99.9%. 13.64× faster · 100% energy saved · wins from 5%.

22 sparsity levels. Below 70%: vs cuBLAS. Above 70%: vs cuSPARSE (the correct baseline — what engineers actually use at high sparsity). Hybrid operator auto-calibrates per level. Energy via pynvml.

13.64×

Peak speedup

80% · vs cuSPARSE

100%

Peak energy saved

97%+ sparsity · pynvml

5%

Crossover

Beats cuBLAS above 5%

22/22

Correctness

max err 7.34×10⁻⁷

Sparsity	vs	Vendor ms	ROLV ms	Speedup	Energy	Pass
0% ←	cuBLAS	10.26	10.26	1.00×	+1.8%	✓
5%	cuBLAS	10.26	10.03	1.02×	+2.5%	✓
50%	cuBLAS	10.26	5.57	1.84×	+46.6%	✓
70%	cuSPARSE	45.37	3.41	13.29×	+93.4%	✓
80% ★	cuSPARSE	30.33	2.22	13.64×	+93.5%	✓
90%	cuSPARSE	15.23	1.16	13.18×	+94.2%	✓
97%	cuSPARSE	4.65	0.388	12.00×	+100%	✓
99.9%	cuSPARSE	0.381	0.114	3.33×	+70.0%	✓

★ = peak. ← = crossover at 0% (matches cuBLAS cold). Italic = cuSPARSE (correct baseline above 70%). A hash: b2687223 · V hash: f8b47533 · 20/22 perturbation PASS

Sparsity	vs	Vendor ms	ROLV ms	Speedup	Energy	Pass
0%	cuBLAS	8.24	8.24	1.00×	+1.4%	✓
5% ←	cuBLAS	8.24	8.02	1.03×	+3.3%	✓
50%	cuBLAS	8.24	4.22	1.95×	+47.4%	✓
70% ★	cuSPARSE	32.58	2.70	12.06×	+92.5%	✓
85%	cuSPARSE	16.38	1.49	10.98×	+92.7%	✓
96%	cuSPARSE	4.36	0.418	10.44×	+100%	✓
99.9%	cuSPARSE	0.335	0.126	2.66×	+62.4%	✓

Sparsity	vs	Vendor ms	ROLV ms	Speedup	Energy	Pass
0% ★	MKL	65.02	54.99	1.18×	+21.6%	✓
5%	MKL	56.05	64.35	0.87×	+2.9%	✓
20%	MKL	65.58	45.90	1.43×	+22.0%	✓
60%	MKL	54.71	23.08	2.37×	+68.9%	✓
70%	CSR	154.16	18.29	8.43×	+86.8%	✓
92% ★	CSR	50.45	5.78	8.72×	+84.4%	✓
99.9%	CSR	1.61	0.48	3.35×	+62.1%	✓

Sparsity	vs	Vendor ms	ROLV ms	Speedup	Energy	Pass
0%	MKL	47.52	47.99	0.99×	-0.2%	✓
5% ←	MKL	46.78	45.86	1.02×	+3.0%	✓
50%	MKL	46.53	23.88	1.95×	+48.2%	✓
70% ★	CSR	70.01	13.96	5.01×	+79.7%	✓
90%	CSR	23.57	5.03	4.69×	+77.6%	✓
99.9%	CSR	0.51	0.28	1.83×	+59.7%	✓

05 — VRAM savings — scales exactly with sparsity

Less VRAM means larger models or larger batches on the same GPU.

ROLV stores only the live rows. LLaMA-3.1-8B up_proj at 99% sparsity: 2.34 MB vs 235 MB dense — 100× reduction. Measured directly from CRCS build.

5×

Less VRAM at 80%

47 MB vs 235 MB

10×

Less VRAM at 90%

23 MB vs 235 MB

20×

Less VRAM at 95%

12 MB vs 235 MB

100×

Less VRAM at 99%

2.3 MB vs 235 MB

06 — Methodology

Baseline

Best vendor always

cuBLAS or cuSPARSE — whichever is faster at that sparsity level. Never cherry-picked.

Correctness

ATOL=0.1 · live rows

Col-normalised fp64, live rows only. Worst error across all runs: 3.9×10⁻⁶.

Timing

CUDA Events · 100–2,000 iters

Microsecond-accurate. Warmup before every measurement. No single-shot results.

Energy

pynvml where noted

Actual joules via NVIDIA Management Library. Proxy used where pynvml unavailable — always clearly labelled.

Hashes

4 SHA-256 per run

Weight matrix · input vector · dense baseline · ROLV output. One weight change → hash changes — proves real computation.

Reproducibility

Deterministic · seeded

TF32 off, cuDNN deterministic, fixed seeds. NVIDIA, AMD, TPU, CPU all supported. Full JSON on request.

All benchmarks run end-to-end in a single self-contained harness. Full JSON with all hashes and timings available on request.

03 — How It Works

The operator is built once from a weight matrix and then used repeatedly for inference. Build time is amortised across thousands of inference calls.

01

Score blocks

Each block of the weight matrix receives an importance score based on its contribution to the output. Low-scoring blocks are candidates for elimination.

02

Prune

Blocks below the sparsity threshold are zeroed out. At a 90% target, 90% of blocks are eliminated. The remaining blocks preserve the most important signal.

03

Quantize

Surviving blocks are quantized to INT8 using per-block scaling. This reduces memory bandwidth and accelerates arithmetic on hardware that benefits from lower precision.

04

Store sparse

The resulting matrix is stored in CSR (Compressed Sparse Row) format. Inference skips all zero blocks entirely — no multiply, no memory access, no energy.

04 — On Correctness

The ROLV Primitive© is exact on its compressed submatrix — no approximation is introduced by ROLV Primitive© itself. The only source of output error is pruning, which zeroes low-magnitude rows before ROLV Primitive© is built.

This is expected and standard for compressed inference. The goal is to operate within a defined tolerance budget while maximising speed and energy savings. All published results include correctness metrics alongside speedup figures.

3.9×10⁻⁶

Max error

LLaMA-3.1-8B · all sparsity levels

250×

Tighter than ATOL

ATOL=0.001 standard · ROLV achieves 3.9×10⁻⁶

26/26

Correctness PASS

22 H200 + 22 B200 + 22 Intel + 22 AMD + 24 T4 weights + 4 LLaMA levels

✓ ×26

Perturbation tests

One weight change → output hash changes every run

Contact

Get in touch.

For technical enquiries, access to benchmark data, or discussions about the technology, please reach out directly.

Rolv E. Heggenhougen | rolv@rolv.ai | rolv.ai

Up to 100% energy savings.
Up to 12.06× faster than NVIDIA’s own GPU libraries.

A compute primitive for sparse AI workloads.

Sparse by design

Software-only

Energy follows compute

Real production weights · real model weights · synthetic sweep · CPU · all verified.

Real model. Real weights. Up to 9.53× faster · up to 89.5% energy saved.

Three production LLMs. Real HuggingFace weights. 72/72 PASS · up to 18.76× faster.

Full sweep 0%–99.9%. 13.64× faster · 100% energy saved · wins from 5%.

Full sweep 0%–99.9%. 12.06× faster · 100% energy saved · wins from 5%.

Intel CPU. 8.72× faster · 88.2% energy saved · beats MKL from 0%.

AMD EPYC 7B13. 5.01× faster · 79.8% energy saved · 22/22 perfect score.

Less VRAM means larger models or larger batches on the same GPU.

Four steps from dense weight to ROLV Primitive©.

RSMT & ROLVswitch™

Get in touch.

Up to 100% energy savings.Up to 12.06× faster than NVIDIA’s own GPU libraries.

A compute primitive for sparse AI workloads.

Sparse by design

Software-only

Energy follows compute

Real production weights · real model weights · synthetic sweep · CPU · all verified.

Real model. Real weights. Up to 9.53× faster · up to 89.5% energy saved.

Three production LLMs. Real HuggingFace weights. 72/72 PASS · up to 18.76× faster.

Full sweep 0%–99.9%. 13.64× faster · 100% energy saved · wins from 5%.

Full sweep 0%–99.9%. 12.06× faster · 100% energy saved · wins from 5%.

Intel CPU. 8.72× faster · 88.2% energy saved · beats MKL from 0%.

AMD EPYC 7B13. 5.01× faster · 79.8% energy saved · 22/22 perfect score.

Less VRAM means larger models or larger batches on the same GPU.

Four steps from dense weight to ROLV Primitive©.

RSMT & ROLVswitch™

Get in touch.

Up to 100% energy savings.
Up to 12.06× faster than NVIDIA’s own GPU libraries.