SovereignAI

Blog · Guide · August 25, 2026

SovereignAI on NVIDIA: DGX Spark, RTX desktops, RTX workstations

SovereignAI itself needs almost nothing — one Node process, no GPU. The models you run under it are where the hardware matters. This guide picks the install path for each NVIDIA machine, gives Ollama the GPU properly, and tells you which models genuinely fit at 8, 24, 48, 96 and 128 GB — including the parts that do not work yet.

By the SovereignAI team at Unify Dynamics · about twelve minutes · every command below was checked against its vendor's documentation on the date above

The one thing to understand first

SovereignAI is the memory, knowledge, and exit layer. It does not run models; it talks to an engine that does — Ollama, FreeToken, or any OpenAI-compatible server — over HTTP on your own machine. So "setting up SovereignAI on NVIDIA hardware" is two jobs: install an engine that uses your GPU, then install SovereignAI and point it at that engine. The engine is the only part that cares whether you have a Spark, a 4090, or a workstation card. The container SovereignAI ships in does not need --gpus at all.

Which install path fits which machine

Step 1 — give Ollama the GPU

On the host (recommended). Install NVIDIA's driver first and confirm nvidia-smi lists the card. Then Ollama's one-line installer, which detects amd64 or arm64 for you:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen3:8b            # a first model; more sizing below
ollama ps                       # after a chat: the PROCESSOR column should say 100% GPU

On a DGX Spark this is exactly what NVIDIA's own Ollama playbook does; if you prefer the manual route, Ollama publishes an ollama-linux-arm64.tar.zst tarball. Ollama installs as a systemd service on Linux and listens on 127.0.0.1:11434, which is all SovereignAI needs.

In Docker (if you would rather). Install the NVIDIA Container Toolkit, register it with Docker, and pass the GPU in:

sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
docker exec -it ollama ollama pull qwen3:8b

If you use our docker-compose.yml with the --profile ollama service, note that it does not reserve a GPU by default — a containerized Ollama without the reservation runs on the CPU. The reservation ships as an overlay file, docker-compose.gpu.yml, which you add on the command line:

docker compose -f docker-compose.yml -f docker-compose.gpu.yml \
  --profile ollama up -d
docker compose exec ollama nvidia-smi   # confirm the container sees it

It is a separate file rather than the default because Compose has no conditional form for a device reservation: on a host without the NVIDIA Container Toolkit it fails the whole up with "could not select device driver" instead of falling back to the CPU. The alternative, and the one the trial command uses, is to run Ollama on the host and point SovereignAI at it — a host Ollama already has the GPU and needs no overlay.

Step 2 — run SovereignAI and point it at the engine

x86_64 (RTX desktop, RTX workstation): the open trial is the real product in a container, already wired to a host Ollama:

docker run -d --name sovereign -p 127.0.0.1:4321:4321 -v sovereign:/state \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -e SOVEREIGN_TOKEN=pick-a-long-secret ghcr.io/mlmrx/sovereignai:latest

Open http://localhost:4321/#token=pick-a-long-secret with your secret. The wizard probes Ollama, lists the models you pulled, and lets you pick one as the default. On Windows, Docker Desktop supports host.docker.internal natively and Ollama's Windows installer uses the GPU through the same driver — the command is identical.

DGX Spark (aarch64): the same trial command works — the GHCR image is multi-arch and Docker pulls the arm64 layer on the Spark. Prefer no container? The Releases page has a Linux arm64 single binary, and source under Node 22 (which exists for Linux arm64) is the third option:

# Node 22.5+ for arm64 (nodejs.org, or your preferred version manager), then:
cd SovereignAI
SOVEREIGN_TOKEN=pick-a-long-secret node --no-warnings bin/sovereign.js start
# the local Ollama at 127.0.0.1:11434 is the default endpoint - nothing to configure

The source is on GitHub, and the arm64 image and binary ship with every release from v0.6.0 on.

Whichever path: run sovereign doctor (or node bin/sovereign.js doctor) once. It checks the home directory, the config, the database, every enabled provider, and the model you set as default — and it tells you if it sees a FreeToken engine running that you have not enabled.

Step 3 — pick models that actually fit

Two rules of thumb the app applies for you on the starter shelf: a dense model needs about 0.6 GB per billion parameters at four-bit, and the shelf assumes it may use about 60% of your RAM (and, since last week, your GPU's memory). A model that fits in VRAM runs fast; one that spills to system RAM still runs on Ollama, slower. Here is what that means per machine, using Ollama's published sizes:

Inside the app, Model Studio → the starter shelf shows each entry with a fit badge for the machine it is on, its license, and a one-click "use as default" — for the sparse tier, both a RAM badge and a GPU badge. Set SOVEREIGN_HARDWARE_PROBE=off if you would rather it did not run nvidia-smi.

Step 4 — FreeToken, where it applies

FreeToken keeps a sparse model's experts in host RAM and streams the active set to the GPU, which is how a 4–8 GB card runs the 20B–120B mixture-of-experts class. It requires Linux x86_64, an NVIDIA driver r580 or newer (CUDA 13), the CUDA 13 toolkit with nvcc on the path, and Python 3.10+. On a big workstation card you may not need it; on a gaming card it is the whole difference. The FreeToken guide covers install, serving, enabling it in SovereignAI, and the Docker wiring.

Step 5 — reach it from other machines (optional)

A Spark or a workstation often lives under a desk while you work on a laptop. Start SovereignAI with --lan: it binds your network interfaces, mints a bearer token, and prints a #token= URL to open from the laptop; the token rides in the URL fragment, so it never appears in server logs. Use a tailnet or a trusted LAN — plain HTTP does not encrypt anything. Ollama itself can stay on loopback; SovereignAI is the only thing that needs to be reachable.

What does not work yet, plainly

FreeToken does not run on the Spark (x86_64 only) or on Apple Silicon. Our Compose file's optional Ollama service is CPU-only unless you add the docker-compose.gpu.yml overlay shown above, which needs the NVIDIA Container Toolkit on the host. Rented-GPU provisioning (byoc gpu) has been tested against provider APIs, not on a live bill. And SovereignAI's database is not encrypted at rest — on a shared workstation, use full-disk encryption and per-user accounts.

One engine on the GPU, one process beside it.

That is the whole architecture. The ledger lists what each layer does and does not guarantee.

Run it now

Sources