Quick start

You need Linux on x86-64 with AVX2, glibc 2.34 or newer, about 6 GB of free memory, and uv. uv installs the pinned Mojo toolchain (Mojo 1.1.0, max-core 26.6.0) into the project's .venv; nothing goes system-wide.

1. Set up

curl -LsSf https://astral.sh/uv/install.sh | sh    # skip if you have uv
git clone https://github.com/vladimirrott/mojo-dllm
cd mojo-dllm
./mojo-dllm setup

The uv installer puts uv in ~/.local/bin. If your shell does not find it afterwards, open a new terminal or run source ~/.local/bin/env.

setup runs four steps and prints each one:

1. Toolchain. It checks the platform, glibc, AVX2 and free memory, stops on anything that cannot work, and runs uv sync --frozen.
2. Build. It compiles build/mojo-dllm, or reports that the binary is newer than the sources.
3. Model. It asks before downloading LLaDA-8B-Instruct Q4_K_M (4.9 GB) into the Hugging Face cache, and checks the file's sha256 against the one the hub publishes. --yes answers the question for you; --no-model skips the step; --model picks another model.
4. Check. It generates 16 tokens to prove the binary, the model and the device work together.

Without a terminal to ask on (a script, CI), setup does not download anything unless you pass --yes. It then exits 3 and prints the command that finishes the job.

./mojo-dllm doctor runs the same checks without changing anything, and adds the GPU, the binary's age and the model cache. Every line that is not ok names the command that fixes it.

2. Generate

./mojo-dllm run "Explain speculative decoding in two sentences."
./mojo-dllm run "Write a haiku about rain." --max-tokens 64 --steps 32 --verbose

The prompt is the first word that is not a flag; quote it. run rebuilds the binary if a source file changed, finds the model, picks the device and hands the rest of the flags to build/mojo-dllm run. Command line lists them all.

--max-tokens sets the canvas size, --steps the number of denoising passes. Fewer steps run faster and commit more tokens per step, which costs quality. Each step runs the full model over prompt and canvas, so expect seconds per step on a laptop CPU. --verbose prints load time, per-step time and peak memory.

The prompt goes through the model's chat template. Pass --no-chat to feed raw text instead.

The device

--device auto is the default. It picks the first NVIDIA GPU with compute capability 8.0 or newer that nvidia-smi lists, and the CPU when there is none. Status lines go to stderr, so stdout holds only the model's text:

device: gpu (NVIDIA GeForce RTX 3050 6GB Laptop GPU, compute 8.6, 585 MiB free)

auto does not compare the card's free memory with the model's size. In that line another program held most of the card, so the run that followed would have said how much memory it needs and exited. Pass --device cpu to run anyway.

3. Other models

-m (or --model) takes one of three forms:

Without :QUANT, a repo resolves to its Q4_K_M file, then Q8_0, then the first .gguf it lists, the order llama.cpp uses. A quant that matches nothing lists the files the repo has. Split (sharded) GGUF files are not supported.

mojo-dllm runs LLaDA and Dream GGUF files made by llama.cpp's converter. Another architecture downloads and then stops with an error naming the architecture.

./mojo-dllm pull MODEL downloads without generating. Models live in the Hugging Face cache (~/.cache/huggingface/hub, or $HF_HUB_CACHE), shared with every other tool that uses it. A gated or private repo needs HF_TOKEN.

The sha256 check runs on download. A file already in the cache is checked against the hub's sha256 the first time it is used, and the launcher remembers the result in ~/.cache/mojo-dllm/verified.json (under $XDG_CACHE_HOME when that is set). A file that fails the check is deleted, and the next run downloads it again. With --offline the check cannot run, so an unchecked file is used after a warning.

4. Offline

./mojo-dllm pull                      # on a connected machine
./mojo-dllm run "Hello" --offline     # later, with no network

--offline (or HF_HUB_OFFLINE=1) uses the cache only and fails with the pull command to run when the model is missing.

5. Look inside a model

./mojo-dllm pull                             # prints the model file's path
./mojo-dllm inspect PATH.gguf | head -40

You get the metadata, a count of tensor types, and every tensor with its shape. The binary's own commands (inspect, tokenize, logits) go to build/mojo-dllm unchanged, after the same rebuild check. Any other word stops with the closest command name.

Without the launcher

The launcher is a convenience. The binary works on its own:

uv sync --frozen
scripts/build.sh
build/mojo-dllm run --model PATH.gguf --prompt "Hello" --device cpu

build/mojo-dllm defaults to --device cpu and takes only file paths.