Self-Hosting a Private LLM API on VergeOS with Lemonade Server

David Vincent

David Vincent

— 6 min read
Share

Quick preface, so nobody mistakes this for a gap-filler: VergeOS already has AI capabilities built into the platform, its Private AI feature set. So this isn't "VergeOS can't do AI, here's a workaround." It's the opposite. The platform handles AI natively; I just got curious about Lemonade Server specifically and wanted to get hands-on with it, to learn how a standalone, OpenAI-compatible inference server behaves when you run it on a VergeOS VM. Pure exploration, and a good excuse to learn something new.

Every time you call a cloud AI API, your prompts (and increasingly your code and your documents) leave your building and land on someone else's servers. For a lot of homelabbers and plenty of enterprises, that's the whole problem. The fix isn't "don't use AI," it's "run it where you control it."

So I stood up a private, OpenAI-compatible LLM inference server on VergeOS: a right-sized Debian VM, Lemonade Server for the serving layer, and a model chosen to actually run at usable speed on plain CPU. The endpoint speaks the OpenAI API, so any tool or SDK that already talks to OpenAI can point at it instead. The difference is that nothing leaves your network.

This is a field report: the exact build, and the places it tried to trip me.

Why VergeOS for an AI box

You could do this on a bare metal server, but VergeOS gives you three things that matter for an AI workload:

  • Right-size it on demand. I gave this VM 8 cores and 32 GB of RAM with a one-line deploy. If it turns out I want 16 cores next week, that's a setting, not a shopping trip.
  • Thin-provisioned NVMe. LLM weights are big (the model below is ~17 GB). VergeOS's vSAN thin-provisions and dedupes, so I allocated 100 GB without committing 100 GB of physical flash up front.
  • It lives where everything else lives. The inference server sits on the same platform as the rest of my infrastructure, on an isolated network, reachable by exactly the things I choose. No separate appliance, no separate management plane.

Step 1: Deploy the Debian VM

VergeOS ships an official Debian recipe, so the base VM is a single deploy. I sized it for CPU inference (cores and RAM are what matter) and gave it a static IP so it's predictably reachable, plus my SSH key so I can drive the rest headless:

vrg recipe deploy "Debian 13 (Trixie)" \
  --name lemonade-server \
  --set HOSTNAME=lemonade-server \
  --set YB_CLUSTER=1 \
  --set YB_CPU_CORES=8 \
  --set YB_RAM=32768 \
  --set SELECT_OS_TIER=1 \
  --set YB_DRIVE_OS_SIZE=107374182400 \
  --set YB_IP_ADDR_TYPE=static \
  --set YB_NIC_ETH0=External \
  --set YB_NIC_ETH0_IP_ADDR=192.168.1.214 \
  --set YB_NIC_ETH0_CIDR=/24 \
  --set YB_NIC_ETH0_GW=192.168.1.1 \
  --set YB_NIC_ETH0_NS=192.168.1.1 \
  --set USER=debian --set PASSWORD='...' \
  --set "SSH_KEY=$(cat ~/.ssh/id_ed25519.pub)"

A few things the recipe will not do for you, learned the hard way:

  • It doesn't power the VM on. recipe deploy creates the VM and its drive, then leaves it stopped. Follow with vrg vm start lemonade-server.
  • Wait for the OS drive to finish importing. Right after deploy the drive sits in media: import while the cloud image streams in. Start it too early and you get "Cannot power on a VM while drives are importing." Wait until the drive settles.
  • SELECT_OS_TIER has no default, so set it. Skip it and the OS-drive step fails silently and you get an unbootable VM with no error. (1 here, the NVMe tier.)
  • Cloud-init holds the apt lock on first boot. Before you install anything, ssh debian@<ip> 'cloud-init status --wait' so you're not fighting unattended-upgrades for the dpkg lock.

A minute later I had Debian 13, 8 cores, 31 GiB RAM, a 100 GB disk, and key-based SSH.

Step 2: Install Lemonade Server

Lemonade ships a Debian package. The install is three commands:

wget https://github.com/lemonade-sdk/lemonade/releases/latest/download/lemonade-server_10.8.1-debian13_amd64.deb
sudo apt install -y ./lemonade-server_10.8.1-debian13_amd64.deb
sudo systemctl enable --now lemond
sudo systemctl --no-pager status lemond

Note the service unit is lemond (the daemon), not lemonade-server. After that, status lemond should show active (running), and Lemonade is serving its API on port 13305.

Step 3: Expose the API on the network

Here's the first thing that'll quietly stop you: out of the box, Lemonade binds to 127.0.0.1:13305, localhost only. Great for security, useless if you want to call it from another machine. Don't hack the vendor's unit file directly; drop in a systemd override:

sudo mkdir -p /etc/systemd/system/lemond.service.d
sudo tee /etc/systemd/system/lemond.service.d/override.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/bin/lemond --host 0.0.0.0
EOF
sudo systemctl daemon-reload && sudo systemctl restart lemond

The empty ExecStart= clears the original; the second line replaces it. Now it binds 0.0.0.0:13305 and survives package updates. Confirm with ss -tlnp | grep 13305.

⚠️ Only bind to 0.0.0.0 on a trusted, isolated network. This is fine in a homelab that sits behind a firewall on an isolated segment, which is exactly my situation here. It is not something to do on an untrusted or internet-facing network. On 0.0.0.0 with no authentication, anyone who can reach port 13305 can run your model and read your prompts. If this needs to be reachable beyond a segment you fully trust, keep it bound to localhost (reach it over an SSH tunnel or reverse proxy), and at minimum require an API key (see the caveats below).

Step 4: Pull a model that actually fits CPU

This is where it gets interesting, because CPU inference makes model choice the whole game. My first instinct was the big general model Lemonade suggested, a 27-billion-parameter dense model. It loaded and answered at about 2.4 tokens per second. A 300-token reply took two minutes. Unusable for anything interactive. Worse, it was a reasoning model: it spent its entire token budget "thinking" and returned empty content unless I gave it huge limits, which made the speed problem exponentially worse.

The fix is a model architected for this. Lemonade has a built-in registry (lemonade list), and the standout for CPU was a Mixture-of-Experts model:

lemonade pull Qwen3-Coder-30B-A3B-Instruct-GGUF

The trick is in the name: 30B total parameters, but only ~3B active per token (that's the "A3B"). On CPU, per-token compute tracks the active parameter count, so this 30B model runs at roughly small-model speed while keeping big-model quality. It's also an Instruct (non-reasoning) model, so answers land in content directly, and it advertises tool-calling. The result: about 18 tokens/second warm. That's 7.5× the dense 27B, and genuinely usable.

Two finishing touches so it's always ready and right-sized:

# keep it loaded across idle, give it real context, and remember the settings
lemonade load Qwen3-Coder-30B-A3B-Instruct-GGUF --ctx-size 32768 --pinned --save-options
  • --pinned keeps the ~17 GB model resident so callers never pay a cold-load wait.
  • --ctx-size 32768 matters: the model loaded at a tiny 4 K context by default, which truncates anything beyond a short prompt. 32 K is a sane size that still fits comfortably (16 GB used, 14 GB free).
  • --save-options persists those across reboots.

The payoff: an OpenAI-compatible endpoint

That's the whole point. Lemonade speaks the OpenAI API, so any app, SDK, or tool you already use with OpenAI can target it instead:

  • Base URL: http://192.168.1.214:13305/api/v1
  • Model: Qwen3-Coder-30B-A3B-Instruct-GGUF
  • API key: any non-empty string (none required unless you set one, see below)

A plain request looks exactly like OpenAI:

curl http://192.168.1.214:13305/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3-Coder-30B-A3B-Instruct-GGUF",
       "messages":[{"role":"user","content":"In one line, what does git rebase do?"}],
       "max_tokens":80}'

Point your OpenAI client's base URL at that and you're running entirely on your own metal.

The honest caveats

  • It's CPU-only, and that's a real ceiling. ~18 tok/s is fine for completions and light interactive use; it is not GPU-snappy. The natural upgrade path is GPU passthrough: VergeOS can hand a physical GPU to a VM, which would change the speed conversation entirely. That's the next experiment.
  • Reasoning vs. instruct models behave very differently. If you pick a "thinking" model and get blank content, that's why: it's reasoning until it runs out of tokens. For an API you call programmatically, prefer Instruct variants.
  • The endpoint is open by default. Once it's on 0.0.0.0, anyone who can reach the box can use it. Lemonade reads LEMONADE_API_KEY from /etc/lemonade/conf.d/*.conf. Set one and require it on every route if this reaches beyond a trusted segment.
  • Model load is not free. A 17 GB model takes ~30 seconds to load; pinning hides that, at the cost of keeping the RAM resident.

Why this matters beyond the lab

Strip away the homelab framing and this is a data-sovereignty story. The same setup that lets me keep my prompts off the cloud is the one an enterprise uses to keep regulated data, source code, and customer information inside its own walls, running on infrastructure it already owns and manages, instead of paying per token to ship that data somewhere else.

And notice it's the same platform doing it. The VergeOS cluster running your VMs, your storage, and your networks can also run your private inference: right-sized, isolated, and co-located with the applications that consume it. The homelab convenience and the enterprise capability are, once again, the same feature at different scales.

Cloud AI asks you to send your data out to get an answer back. Self-hosting flips it: the model comes to your data. On VergeOS, that's an afternoon and a one-line deploy.

Read more