Benchmarks

LLaDA-8B-Instruct, Q4_K_M. 13th Gen Intel(R) Core(TM) i5-13420H, 12 threads, 33.3 GB RAM, performance power profile. Each prompt: 128 generated tokens, 32 denoising steps; median of 3 runs x 3 prompts.

Versions: mojo-dllm bdd268b, Mojo 1.1.0 (8189361e), llama.cpp 836d571, diffuse-cpp 1d2bd6a. Evidence: bench/results/2026-10-03-cpu-diffuse.json, bench/results/2026-10-03-cpu-llada.json.

† Not the same work: the inter-step cache reuses stale K/V and entropy_exit can stop early. Shown because it is diffuse-cpp's recommended mode.

LLaDA-8B-Instruct on the GPU, Q4_K_M. NVIDIA GeForce RTX 3050 6GB Laptop GPU (6144 MiB, driver 595.84), host 13th Gen Intel(R) Core(TM) i5-13420H. Each prompt: 128 generated tokens, 32 denoising steps; median of 3 runs x 3 prompts. Peak RSS is host memory.

Versions: mojo-dllm 78014b4, Mojo 1.1.0 (8189361e), llama.cpp 836d571. Evidence: bench/results/2026-10-12-gpu-llada.json.

Dream-v0-Instruct-7B, Q4_K_M. 13th Gen Intel(R) Core(TM) i5-13420H, 12 threads, 33.3 GB RAM, performance power profile. Each prompt: 128 generated tokens, 32 denoising steps; median of 3 runs x 3 prompts.

Versions: mojo-dllm bdd268b, Mojo 1.1.0 (8189361e), llama.cpp 836d571. Evidence: bench/results/2026-10-03-cpu-dream.json.

Dream-v0-Instruct-7B on the GPU, Q4_K_M. NVIDIA GeForce RTX 3050 6GB Laptop GPU (6144 MiB, driver 595.84), host 13th Gen Intel(R) Core(TM) i5-13420H. Each prompt: 128 generated tokens, 32 denoising steps; median of 3 runs x 3 prompts. Peak RSS is host memory.

Versions: mojo-dllm 036ef79, Mojo 1.1.0 (8189361e), llama.cpp 836d571. Evidence: bench/results/2026-10-12-gpu-dream.json.

LLaDA-8B-Instruct, llama.cpp thread sweep and forward pass alone: 13th Gen Intel(R) Core(TM) i5-13420H; median of 2 runs, one prompt, canvas 148 tokens.

The ms / step column runs llama-diffusion-cli. The forward pass alone is mojo-dllm's measured forward pass against llama.cpp's llama-bench at the same token count. llama-bench applies a causal mask and computes logits for one position; mojo-dllm's figure includes logits for up to 32 positions and copying the canvas in. CPU busy before a run: at most 6.5%. Evidence: bench/results/2026-10-09-fair-cpu-llada.json.

LLaDA-8B-Instruct on the GPU, forward pass alone: NVIDIA GeForce RTX 3050 6GB Laptop GPU; median of 3 runs, one prompt, canvas 148 tokens.

The forward pass alone is mojo-dllm's measured forward pass against llama.cpp's llama-bench at the same token count. llama-bench applies a causal mask and computes logits for one position; mojo-dllm's figure includes logits for up to 32 positions and copying the canvas in. CPU busy before a run: at most 8.9%. Evidence: bench/results/2026-10-12-fair-gpu-llada.json.

What is compared

For LLaDA, all three runtimes load LLaDA-8B-Instruct Q4_K_M and denoise a canvas of L = prompt + 128 tokens in 32 steps. For Dream, mojo-dllm and llama.cpp (diffuse-cpp's Dream support needs its own converted file, not benchmarked here) do the same with Dream-v0-Instruct-7B Q4_K_M; both run Dream's timestep schedule over the whole canvas. A step is one full forward pass over the canvas in each of them, so the step counts and shapes match.

diffuse-cpp cannot read llama.cpp's GGUF files, so it loads its own conversion of the same model (diffuse-cpp/LLaDA-8B-Instruct-GGUF, llada-8b-q4km.gguf, 5.5 GB): F16 embeddings and a slightly different Q4_K / Q6_K mix. It has no tokenizer and receives the token ids mojo-dllm produces. It prints no timings either, so the harness timestamps the two lines it writes around its generate call.

llama.cpp runs its timestep schedule because its block mode needs the prompt length to be a multiple of the block size. The schedules commit tokens in a different order, which changes the text but not the cost of a step.

† diffuse-cpp's recommended mode reuses K/V between steps and stops early when the canvas has settled. It does less work, so it is not a like-for-like row.

On the GPU

bench/config-gpu.json and bench/config-dream-gpu.json compare mojo-dllm --device gpu with the same llama-diffusion-cli built with CUDA and run with -ngl 99; the harness refuses a CUDA run unless llama.cpp reports every layer on the GPU. Canvas, steps and prompts match the CPU runs. These configs leave the pytest watchdog off ("kill_pytest": false), so each result records how busy the CPU was before every run. Peak RSS is host memory; neither runtime reports GPU memory. The GPU tables above come from these configs.

The test GPU is a laptop part held to 30 W, whose clocks fall under sustained load (bench/checks/2026-10-12-gpu-power-under-load.txt) and vary from day to day: the same llama.cpp build ran LLaDA steps much slower on 2026-10-12 than on 2026-10-09 (compare bench/results/archive/2026-10-09-gpu-llada.json with the current file). Both runtimes in one result file run interleaved, prompt by prompt, so compare them within a file, not across dates.

bench/config-fair-cpu.json and bench/config-fair-gpu.json address two questions about fairness: whether llama.cpp does better below 12 threads on a hybrid CPU (a sweep over -t 4, 6, 8, 12), and how the forward passes compare without either runtime's sampler (mojo-dllm's measured forward pass against llama.cpp's llama-bench at the same token count).

Method

bench/run_bench.py does the measuring:

• three prompts, three repetitions, runtimes interleaved within each repetition;
• every run starts only when the CPU has been under 10% busy for two seconds;
• the power profile is performance for the duration and restored after;
• a watchdog kills any pytest process (other work on the machine started test suites mid-run), and the kills are logged in the result file;
• generation time excludes model loading, which is reported as startup;
• peak RSS comes from /usr/bin/time.

The table above is generated from bench/results/*.json by scripts/bench_table.py, and CI fails if the two disagree.

Steps and quality

A diffusion model commits tokens / steps tokens per step. The table above uses 4 tokens per step, which is fast and shows in the text as repeated or dropped words. More steps cost proportionally more time:

LLaDA-8B-Instruct.Q4_K_M.gguf, 128 generated tokens.

Evidence: bench/quality/2026-10-03-LLaDA-8B-Instruct.Q4_K_M.json (mojo-dllm bdd268b).

"Repeated words" counts immediate repetitions such as "the the", a rough proxy for parallel-decoding damage. bench/quality.py produces this table.

Accuracy of --threshold and --dual-cache

Both options change what a step computes, so their answers can differ from the plain sampler's. A paired GSM8K run checks how often they still get the answer right (bench/eval_gsm8k.py, see Development):

LLaDA-8B-Instruct.Q4_K_M.gguf, GSM8K, 200 problems, flags --max-tokens 256 --block-length 32 --steps 256 --device gpu --json, mojo-dllm 3b2dac4.

Discordant counts problems only the baseline solved / only the arm solved. Time per problem is the mean over the first 50 problems of each arm, a round that ran with nothing else on the machine.

Evidence: bench/eval/2026-10-29-baseline-vs-dualcache.json, bench/eval/2026-10-29-baseline-vs-threshold09.json, bench/eval/2026-10-29-baseline-vs-dualcache-first50.json, bench/eval/2026-10-29-baseline-vs-threshold09-first50.json.

Reading the numbers

• ms / step is the cost of one denoising pass over the canvas.
• tokens / s is 128 divided by generation time. A diffusion model's tokens/s depends on the step count you choose, so compare ms / step first.
• The ablation row shows what computing logits only for candidate positions saves.
• On the GPU, read ms / step and peak RSS together. llama.cpp is faster per step; mojo-dllm holds far less in host memory, because it uploads the weights once and releases the file's pages behind it.
• The thread sweep and the forward-pass-alone tables test the CPU comparison. If llama.cpp ran faster below 12 threads, or mojo-dllm's lead came only from skipping logits, those tables would show it. The sweep stops at 12 because the test CPU runs 12 threads.
• A note under a table counts the runs that started on a busy CPU and gives the medians over the quiet runs alone.