From Two Gaming GPUs to a 124 tok/s Private LLM on VergeOS

0:00

/0:30

Here's what we're building, up front: two consumer RTX 5060 Ti cards passed through to an Ubuntu VM on VergeOS, serving a 35-billion-parameter model at ~124 tokens/second through the llama.cpp WebUI. No cloud, no per-token bill, no data leaving the building. Two gaming cards and enterprise infrastructure, doing real inference.

There are two halves. First, get the GPUs into the VM (passthrough, the fiddly part). Second, build and run the model (the fun part). I'll keep the passthrough half to the steps, because it's mostly about knowing which obvious-looking choice is the trap. Every step below includes a trap to watch out for; hit them in order and you won't burn a reboot like I did.

Part 1: Pass the GPUs through to a VM

0. Prerequisites

VT-d / IOMMU enabled in the BIOS. VergeOS cannot enable this itself. On an HP Z440: F10 → Security → System Security → enable Virtualization Technology and Virtualization Technology for Directed I/O.
A second display GPU on the node, or run it headless. Never pass through the host's console GPU, a boot device, or a storage controller. Doing so can make the node unbootable.

1. See the cards (not where you'd first look)

Your instinct is vrg gpu list. It returns [], and that's correct: the whole vrg gpu subsystem is for NVIDIA vGPU, which doesn't exist on consumer GeForce cards. Your cards live in a lower-level table, node_pci_devices, reachable through the API:

HOST=https://your-vergeos-host
curl -ks -H "Authorization: Bearer $TOKEN" "$HOST/api/v4/node_pci_devices?fields=all" \
 | jq -r '.[] | select(.node==5) | select(.vendor|test("NVIDIA";"i"))
          | "slot=\(.slot) hex=\(.vendor_device_hex) driver=\"\(.driver)\" iommu=\(.iommu_group)"'

You'll see four devices, not two. Each GPU carries a companion HDMI-audio function in the same IOMMU group:

slot=04:00.0 hex=10de:2d04 driver="" iommu=51   ← GPU 1
slot=04:00.1 hex=10de:22eb driver="" iommu=51   ← GPU 1 audio
slot=06:00.0 hex=10de:2d04 driver="" iommu=52   ← GPU 2
slot=06:00.1 hex=10de:22eb driver="" iommu=52   ← GPU 2 audio

2. Confirm VT-d actually took

Trap: the iommu_group field reads empty when VT-d is off, but it's also just unpopulated in plenty of normal cases. Empty is not proof. Trust the boolean nodes.<key>.iommu instead.

curl -ks -H "Authorization: Bearer $TOKEN" "$HOST/api/v4/nodes/5?fields=all" | jq '{iommu, restart_reason}'

iommu: false with "A driver reload is required but IOMMU is not enabled" means go back to the BIOS. Once VT-d is genuinely on, the iommu_group numbers populate; that's how you confirm.

3. Create the resource group: type `pci`, not `host-gpu`

The expensive trap. vrg's help calls host-gpu "Full-GPU passthrough pool for AI workloads," which sounds exactly right and is wrong: host-gpu is the vGPU host-driver path that keeps the card bound to the host nvidia driver. You want pci.

vrg resource-group create --name gpu-5060ti --type pci --device-class gpu \
  --description "2x RTX 5060 Ti one-to-one PCI passthrough"

The tell that you got it right comes later: the device driver field must read vfio-pci (host released the card), never nvidia (host still owns it). The returned $key is a UUID, not an integer.

4. Add a resource rule (API-only)

Devices join the group by a filter expression, not by hand-picked keys, and vrg has no command for this, so it's the raw API:

curl -ks -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"resource_group\":\"$UUID\",\"filter\":\"vendor_device_hex eq '10de:2d04'\"}" \
  "$HOST/api/v4/resource_rules"

Trap: a bare string ("10de:2d04") is accepted by the POST and then matches nothing. You need the expression form, field eq 'value'.

Both cards are the same model, so they share vendor_device_hex, so one rule pools both. You'll know it matched when nodes.<key>.reload_drivers_required flips to true.

5. Reboot the node to release the cards

"Reload drivers" means reboot. Before you do, make sure nothing important lives on the node:

vrg -o json vm list | jq -r '.[] | select(.node_name=="node5") | .name'   # should be empty
vrg node restart node5 -y

Trap: vrg node restart routes through maintenance mode, and a storage cluster allows only one node in maintenance at a time. If a phantom node is stuck in maintenance, this fails with a generic "too many 500 error responses." Fall back to rebooting from the GUI.

6. Verify the handoff (the decisive check)

curl -ks -H "Authorization: Bearer $TOKEN" "$HOST/api/v4/node_pci_devices?fields=all" \
 | jq -r '.[] | select(.node==5) | select(.vendor|test("NVIDIA";"i")) | "\(.slot) \(.driver)"'

04:00.0 vfio-pci
04:00.1 vfio-pci
06:00.0 vfio-pci
06:00.1 vfio-pci

vfio-pci across the board = success. If any still say nvidia, stop: the resource group is the wrong type, and another reboot won't fix it. (The audio functions get auto-reserved with max_instances: 0, so asking for count=2 later correctly grabs just the two GPUs.)

7. Create the VM and attach the GPUs

Deploy the VM with a recipe. Answer every question or the deploy silently skips stages, and mind the inconsistent units: RAM in megabytes, drive size in raw bytes, cluster by integer key, NIC by name:

vrg recipe deploy "Ubuntu Server 24.04 (Noble Numbat)" --name gpu-node \
  --set YB_CLUSTER=1 --set SELECT_OS_TIER=1 \
  --set YB_CPU_CORES=8 --set YB_RAM=65536 \
  --set YB_DRIVE_OS_SIZE=214748364800 \
  --set YB_NIC_ETH0=External --set YB_IP_ADDR_TYPE=dhcp \
  --set USER=adminuser --set SSH_KEY="$(cat ~/.ssh/id_ed25519.pub)"

Attaching the GPUs is API-only, and it attaches to the machine (machines.$key), not the VM, because VergeOS keeps those as separate key spaces:

MKEY=$(curl -ks -H "Authorization: Bearer $TOKEN" "$HOST/api/v4/vms" \
       | jq -r '.[] | select(.name=="gpu-node") | .machine')

curl -ks -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"machine\":$MKEY,\"name\":\"gpu-5060ti\",\"type\":\"node_pci_devices\",\"resource_group\":\"$UUID\",\"count\":2}" \
  "$HOST/api/v4/machine_devices"

type must be the literal node_pci_devices. Then pin the VM to the node (passthrough is physically wedded to the hardware) and start it:

curl -ks -X PUT -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"preferred_node":"5"}' "$HOST/api/v4/vms/<vm_key>"
vrg vm start gpu-node

8. Install the guest driver: the `-open` module

The host needs no NVIDIA driver at all; vfio hands the raw card to the guest. Inside the VM:

ssh adminuser@<vm-ip>
lspci -nn | grep -i nvidia          # 2 VGA + 2 audio
sudo apt-get install -y nvidia-driver-595-server-open
sudo reboot

Trap: Blackwell (RTX 50-series) will not load the closed modules. You must use the -open variant. nvidia-driver-595 without -open fails.

nvidia-smi
GPU 0: NVIDIA GeForce RTX 5060 Ti | 16311MiB
GPU 1: NVIDIA GeForce RTX 5060 Ti | 16311MiB
Driver 595.71.05  |  CUDA 13.2

Two cards, 32 GB of VRAM, inside a VM. Now the fun part.

Part 2: Serve a 35B model on the llama.cpp WebUI

The result, up front

I benchmarked the same model across three engines on this exact hardware:

Engine	Model	tok/s
llama.cpp `llama-server`	Ornith-35B Q4_K_M	124.6 🏆
Ollama	Ornith-35B Q4_K_M	88.5
Ollama	qwen3.6:27b (dense)	22.1

Two things worth internalizing from that table:

MoE beats dense, decisively. Ornith-35B is a Mixture-of-Experts model: all 35B parameters sit in VRAM, but only a few billion activate per token. The dense 27B model fires every parameter for every token. On cards that are VRAM-rich but compute-modest (exactly what a 5060 Ti is), sparse activation is a huge win. Ornith is larger and 4× faster, at lower power.

llama.cpp beats Ollama by ~41% on the identical model. Ollama wraps llama.cpp but adds scheduling overhead and conservative defaults. If you want the throughput, go straight to llama-server.

Build llama.cpp with CUDA

There's no prebuilt Linux CUDA binary, so you build from source. You need the CUDA toolkit (the driver doesn't include a compiler), and Ubuntu's stock one is too old for Blackwell:

sudo apt-get install -y git build-essential cmake ninja-build libcurl4-openssl-dev

# CUDA 13 from NVIDIA's repo (stock nvidia-cuda-toolkit is 12.0, too old)
cd /tmp
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb && sudo apt-get update
sudo apt-get install -y cuda-toolkit-13-0

Build with the right architecture: 120 is sm_120 = Blackwell. Get it wrong and it won't use the GPUs:

git clone --depth 1 https://github.com/ggml-org/llama.cpp ~/llama.cpp
cd ~/llama.cpp
export CUDA_HOME=/usr/local/cuda PATH=/usr/local/cuda/bin:$PATH
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 \
      -DLLAMA_CURL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j8 --target llama-server

Pick a quant and launch

Against 32 GB of total VRAM, Q4_K_M (~21 GB) is the sweet spot: it needs both cards and leaves ~10 GB for KV cache. Anything above ~15.5 GB is forced to split across both GPUs, which is also your proof you're using both.

MODEL=/opt/models/ornith-1.0-35b-Q4_K_M.gguf
./build/bin/llama-server \
  -m "$MODEL" --alias Ornith-1.0-35B \
  -ngl 99 \
  -c 262144 \
  --jinja \
  --parallel 2 --kv-unified \
  --host 0.0.0.0 --port 8033

The load-bearing flags: -ngl 99 offloads all layers to GPU; -c 262144 sets the context to the model's full trained 256k (more on why that matters in a second); --jinja enables the chat template. Open the WebUI at http://<vm-ip>:8033.

The one gotcha that will make you think it's broken

Ornith is a reasoning model (it "thinks" before answering), and that interacts badly with two defaults.

First: give it a generous token budget. With a short max_tokens, the model spends its entire allowance thinking and the actual answer comes back empty. Measured, for "say hello in one short sentence": at 150 tokens you get content: ""; at 1500 you get ~880 tokens of reasoning and then "Hello, and welcome!" It really did use ~880 thinking tokens to produce three words. Budget accordingly.

Second, and this is the real one: the WebUI replays the model's entire chain-of-thought back to the server on every turn, out of the box. The context compounds turn over turn until the server hard-errors mid-conversation and the UI goes blank:

srv send_error: request (70989 tokens) exceeds the available context size

The fix is a single checkbox, no server change:

WebUI → Settings → Developer → tick "Exclude reasoning from context."

That setting ships unchecked. Every ~3000-character thinking block gets resent each turn until even a 256k window fills. Raising -c only buys time; the checkbox is the actual fix. (And don't bother with --context-shift: this model's hybrid recurrent architecture doesn't support KV shifting, so llama.cpp disables it regardless.)

Confirm both cards are working

nvidia-smi --query-gpu=index,memory.used,utilization.gpu,power.draw --format=csv

Both cards should load to ~11 GB each and climb together when you send a prompt. If only one moves, the model fit on a single GPU and you're leaving performance on the table. A quick throughput check:

curl -s http://127.0.0.1:8033/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Write a 400-word essay on GPU virtualization."}],"max_tokens":600}' \
 | python3 -c "import sys,json; print('%.1f tok/s' % json.load(sys.stdin)['timings']['predicted_per_second'])"

Expect ~125 tok/s. (If a plain curl on the root URL returns a 415 gzip is not supported error, that's not a fault: the UI is gzip-compressed and browsers handle it; add --compressed.)

Make it survive a reboot, and avoid the silent CPU trap

A nohup launch dies on reboot, so wrap it in a systemd unit. But there's a nasty race worth knowing about: both Ollama and llama.cpp probe for GPUs exactly once at startup and cache the result. If the service wins the race against the NVIDIA kernel modules, it finds nothing and runs CPU-only forever, with no error, just 5–7× slower. The only symptom in Ollama is ollama ps showing 100% CPU.

The fix is an ExecStartPre guard that blocks until the driver answers:

# /etc/systemd/system/llama-server.service  (excerpt)
ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do /usr/bin/nvidia-smi -L >/dev/null 2>&1 && exit 0; sleep 2; done; exit 1'
ExecStart=/home/adminuser/llama.cpp/build/bin/llama-server -m /opt/models/ornith-1.0-35b-Q4_K_M.gguf \
  --alias Ornith-1.0-35B -ngl 99 -c 262144 --jinja --parallel 2 --kv-unified --host 0.0.0.0 --port 8033
Restart=on-failure

Pick one engine: two auto-starting services racing for the same VRAM is a coin-flip on every boot, so sudo systemctl disable --now ollama. After systemctl enable --now llama-server and a reboot, expect a ~40-second gap between the service going active and the model appearing in VRAM (that's the GPU-wait guard plus streaming 20 GB off disk). That's normal.

One more: if you pkill -f llama-server over SSH, the pattern matches your own shell and kills your session. Kill by port instead: sudo fuser -k 8033/tcp.

Secure it before you leave it running

The WebUI has no authentication, and binding to 0.0.0.0 puts an open GPU endpoint on your network. As a persistent service, that's a standing exposure: anyone who can route to the box can consume your GPUs. Close it off from the start, pick one:

API key: add --api-key <secret> to the launch; the WebUI will prompt for it.
Localhost + SSH tunnel: --host 127.0.0.1, then ssh -L 8033:127.0.0.1:8033 adminuser@<vm-ip>.
Firewall port 8033 to trusted sources.

What this adds up to

The endpoint is OpenAI-compatible, which is the quiet payoff of the whole exercise: anything that speaks that API can now use a model running entirely on your own hardware: a chat UI, a coding agent, an automation script, whatever. The model, the GPUs, and the data all stay in one building you control.

And the hardware story is the part I keep coming back to. The received wisdom is that private AI means a five-figure datacenter GPU you wait months for. It doesn't. Two consumer gaming cards, an operating system that doesn't care what silicon is underneath it, and a model architecture that plays to VRAM over raw compute, and you're serving a 35B model at 124 tokens a second. The passthrough traps are real, but they're a one-time tax. What you're left with is genuinely useful, genuinely fast, and genuinely yours.

From Two Gaming GPUs to a 124 tok/s Private LLM on VergeOS

Part 1: Pass the GPUs through to a VM

0. Prerequisites

1. See the cards (not where you'd first look)

2. Confirm VT-d actually took

3. Create the resource group: type `pci`, not `host-gpu`

4. Add a resource rule (API-only)

5. Reboot the node to release the cards

6. Verify the handoff (the decisive check)

7. Create the VM and attach the GPUs

8. Install the guest driver: the `-open` module

Part 2: Serve a 35B model on the llama.cpp WebUI

The result, up front

Build llama.cpp with CUDA

Pick a quant and launch

The one gotcha that will make you think it's broken

Confirm both cards are working

Make it survive a reboot, and avoid the silent CPU trap

Secure it before you leave it running

What this adds up to

Read more

What Is a... Site-to-Site Sync?

vrg Meets Bubble Tea: I Built a VergeOS TUI

Self-Hosting a Private LLM API on VergeOS with Lemonade Server

What is a... great VergeOS demo box?

Part 1: Pass the GPUs through to a VM

0. Prerequisites

1. See the cards (not where you'd first look)

2. Confirm VT-d actually took

3. Create the resource group: type pci, not host-gpu

4. Add a resource rule (API-only)

5. Reboot the node to release the cards

6. Verify the handoff (the decisive check)

7. Create the VM and attach the GPUs

8. Install the guest driver: the -open module

Part 2: Serve a 35B model on the llama.cpp WebUI

The result, up front

Build llama.cpp with CUDA

Pick a quant and launch

The one gotcha that will make you think it's broken

Confirm both cards are working

Make it survive a reboot, and avoid the silent CPU trap

Secure it before you leave it running

What this adds up to

Read more

What Is a... Site-to-Site Sync?

vrg Meets Bubble Tea: I Built a VergeOS TUI

Self-Hosting a Private LLM API on VergeOS with Lemonade Server

What is a... great VergeOS demo box?

3. Create the resource group: type `pci`, not `host-gpu`

8. Install the guest driver: the `-open` module