Kubernetes on VergeOS, from zero to a reachable LoadBalancer

David Vincent

David Vincent

— 7 min read
Share

The result first

$ curl -s --max-time 5 http://192.168.1.202/ | grep -o '<title>.*</title>'
<title>VergeOS + Kubernetes: integration success</title>

That hit a custom nginx pod on a single-node k3s cluster, through a LoadBalancer Service whose IP was allocated by vergeos-cloud-controller-manager from a pool I declared in Helm values, made reachable from the LAN by a hand-added VergeOS firewall rule the CCM doesn't generate. End-to-end working time, not counting the recipe deploy: about ten minutes of helm install and a couple of curl -X POST calls.

The cluster has to live on the same vnet as the VergeOS host API. Both the CSI driver and the CCM call it constantly, and no other placement keeps both happy. With that decided, the install is short — every command you actually run, plus the firewall rules you need to add yourself.

The install

Prerequisites

  • vrg configured against the host API. Verify with vrg --query name vm list | head — should list host VMs, not error.
  • An IP range on External you're willing to dedicate to LoadBalancer services. I used .200–.204.
  • Two free static IPs on External for the cluster. I used .50 for the k3s server and .51 for the CSI pool VM.

1. Deploy the k3s node and the CSI pool VM

External has DHCP disabled, so static config goes through --set:

vrg recipe deploy k3s-server \
  --set name=k3s-server-01 --set vnet=External \
  --set ip=192.168.1.50 --set gateway=192.168.1.1

The CSI pool VM is a thin Linux VM that holds the unattached block drives the CSI driver hotplugs onto cluster nodes. Any small Linux template works:

vrg vm create -t "Debian 12 (Bookworm)" --name csi-pool-01 \
  --vnet External --ip 192.168.1.51 --gateway 192.168.1.1 \
  --cpu 1 --ram 1024

After both come up, capture the pool VM's vms.$key — you'll need it for the CSI Helm values:

POOL_VM_ID=$(vrg -o json vm get csi-pool-01 | jq -r '."$key"')
echo $POOL_VM_ID   # in my homelab: 39

This is vms.$key, not machines.$key. They are different integers in different tables. Wrong one gives VM with ID X not found. (Background: see VergeOS raw API patterns.)

2. Get a kubeconfig

scp it off the k3s node and rewrite the server URL:

ssh [email protected] sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed 's|127.0.0.1|192.168.1.50|' > ~/.kube/config
kubectl get nodes
# k3s-server-01   Ready   control-plane,etcd   1m   v1.35.4+k3s1

3. Install the CSI driver

Add the verge-io chart repo and install. The chart annotations (catalog.cattle.io/*) reveal these are also published as Rancher catalog charts, so a Rancher-managed VergeOS cluster gets the same install via UI click — the subject of the next post in this series.

helm repo add verge-io https://verge-io.github.io/helm-charts
helm repo update

helm install vergeos-csi verge-io/vergeos-csi \
  --version 0.2.0 -n kube-system \
  --set vergeos.host=https://192.168.1.111 \
  --set vergeos.apiKey=$VERGE_TOKEN \
  --set vergeos.verifySSL=false \
  --set block.enabled=true \
  --set block.poolVmId=$POOL_VM_ID

vergeos.host is the host-API URL; the CSI driver makes HTTPS calls to it for every PVC. block.poolVmId is the vms.$key from step 1. NAS support is off here — see fine print.

kubectl get pods -n kube-system -l app.kubernetes.io/name=vergeos-csi
# vergeos-csi-block-controller-...   4/4 Running
# vergeos-csi-node-...               3/3 Running

4. Install the CCM with an IP pool

Same repo, separate chart. The networkID is the VergeOS vnet $key you want LB IPs allocated on — for an External-resident cluster, that's the External vnet:

NETWORK_ID=$(vrg -o json --query '[?name==`External`]."$key" | [0]' network list)
# in my homelab: 3

helm install vergeos-ccm verge-io/vergeos-cloud-controller-manager \
  --version 0.2.0 -n kube-system \
  --set vergeos.host=https://192.168.1.111 \
  --set vergeos.apiKey=$VERGE_TOKEN \
  --set vergeos.verifySSL=false \
  --set loadBalancer.networkID=$NETWORK_ID \
  --set-json 'loadBalancer.ipPool=["192.168.1.200","192.168.1.201","192.168.1.202","192.168.1.203","192.168.1.204"]'

The pool size caps how many LoadBalancer Services you can have running concurrently. Pick accordingly.

5. Test the LB path

kubectl create deployment hello --image=nginx --replicas=1
kubectl expose deployment hello --type=LoadBalancer --port=80 --name=hello
LB_IP=$(kubectl get svc hello -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
NODE_PORT=$(kubectl get svc hello -o jsonpath='{.spec.ports[0].nodePort}')
echo "LB=$LB_IP NodePort=$NODE_PORT"

Within seconds the CCM will:

  • POST a vnet_addresses row binding the chosen IP on External
  • POST a vnet_rules row of the form direction=incoming, action=translate, dst=<LB_IP>, target=<node>:<NodePort>
  • VergeOS auto-applies the firewall within ~1 minute

Verify from k3s-server-01 itself:

ssh [email protected] curl -s -o /dev/null -w '%{http_code}\n' http://$LB_IP/
# 200

If that returns 200, the integration's k8s-side is fully wired. The next step is the part nobody documents.

6. Add the SNAT companion rules

The CCM creates only the DNAT half of the LB rule. From any client on External, traffic to LB_IP:80 gets DNATed to node:NodePort, but k3s-server-01 lives on the same broadcast domain as the LAN client, so its reply ARPs the client directly with src=node-IP instead of src=LB-IP. The client drops the unrecognized reply.

The fix is an outgoing translate rule per (LB IP, source IP) pair that re-sources the post-DNAT packet to the vnet's own IP, forcing the reply path back through conntrack:

HOST=https://192.168.1.111
GATEWAY_ADDR_KEY=$(curl -ks -H "Authorization: Bearer $VERGE_TOKEN" \
  "$HOST/api/v4/vnet_addresses?fields=all" \
  | jq -r '.[] | select(.vnet==3 and .ip=="192.168.1.111") | ."$key"')
# in my homelab: 6

for SRC in 192.168.1.1 192.168.1.110; do
  curl -ks -X POST -H "Authorization: Bearer $VERGE_TOKEN" -H "Content-Type: application/json" \
    -d "{
      \"vnet\": $NETWORK_ID,
      \"name\": \"k8s-LB-snat ($LB_IP from $SRC)\",
      \"enabled\": true, \"protocol\": \"tcp\",
      \"direction\": \"outgoing\", \"action\": \"translate\",
      \"source_ip\": \"$SRC\",
      \"destination_ip\": \"192.168.1.50\",
      \"destination_ports\": \"$NODE_PORT\",
      \"target_ip\": \"address:$GATEWAY_ADDR_KEY\"
    }" "$HOST/api/v4/vnet_rules"
done
vrg network apply-rules External

address:$GATEWAY_ADDR_KEY references the vnet_addresses row for .111 itself. After this rule lands, curl http://$LB_IP/ from your Mac and your router both succeed. Tailscale via a subnet router works too — the subnet router SNATs tailnet traffic to its own LAN IP before it reaches the vnet, so it matches.

Why the sequence is what it is

Both the CSI and the CCM need host-API HTTPS, and the cluster lives where they can reach it. That single constraint forces the cluster onto External. Four other placements were tried before this one stuck — tenant-internal, host-vnet, Tailscale-as-bridge — and each broke in a different way (NAT-no-hairpin, blocked inter-vnet TCP, asymmetric routing). Once you accept External, the install is short.

block.poolVmId takes vms.$key, not machines.$key. VergeOS has two VM key spaces — vms (what vrg vm shows) and machines (what NICs and drives are keyed by). The Helm chart reads from vms. Using the wrong integer fails fast and noisily.

The CCM's networkID is the vnet that hosts the LB IPs, not the cluster's vnet. They're often the same (because the cluster is on External and you want LB IPs on External), but the chart treats them independently. If you ever migrate the cluster to a different vnet, only networkID would need to change.

The SNAT companion rules are the difference between "internal-only LB" and "actually reachable LB." Without them, every off-vnet client times out. With them, every covered source IP works. The CCM should arguably emit these alongside the DNAT rule; until it does, an agent loop or a controller of your own can walk kubectl get svc -A against /api/v4/vnet_rules and reconcile the missing pairs.

The fine print

NAS volumes need a UI click. The CSI NAS driver wants a k8s-nas VergeOS NAS service backed by the Services recipe. The recipe lives in the catalog but vrg recipe deploy services returns Changes are not allowed on non-local repositories. Activate it in the UI, then re-install the CSI chart with --set nas.enabled=true.

The node controller can't find VMs by hostname. The CCM logs VM not found for node "k3s-server-01" on every sync — it correlates Nodes to VergeOS VMs by hostname and the lookup misses. Storage and LoadBalancer both work fine without it; you just don't get VergeOS-specific Node annotations. Workaround is to set spec.providerID on each Node to the matching VM key.

The IP pool is a hard limit on concurrent LBs. A 5-IP pool means at most five Services of type LoadBalancer. The CCM doesn't queue or recycle gracefully — sixth Service stays in <pending> until you delete one. Size the pool for the workloads you actually expect.

Companion SNAT rules don't auto-clean. kubectl delete svc <name> removes the CCM-created DNAT rule and the address binding, but the SNAT pairs you POSTed are invisible to the CCM. Either delete them in the same script that deletes the Service, or run a periodic reconcile that drops orphans.

Don't put production workloads here. External is the same broadcast domain as the VergeOS UI, the tenant UI, and the LAN router. A misbehaving pod is one network hop from your control plane. For a homelab it's fine; for anything multi-tenant or internet-facing, the right answer is a dedicated API endpoint reachable from a separate workload vnet.

What this all adds up to

The install is ~10 minutes once you know the rules. Two VMs, two Helm releases, an IP pool, and a couple of POST calls per Service. Most of the rough edges show up the first time you try to talk to an LB IP from outside the vnet — the SNAT companion section above is the work that gap turns into.

The SNAT companion is the only piece that wants automation. Everything else either runs once at install time or is handled by the CCM. The SNAT pairs scale with (LB count × source IP count) and need to be reconciled on Service create/delete. A small controller — or a periodic agent loop — closes that gap cleanly. Until the upstream CCM emits them itself, that automation is the integration's missing eighth step.

Verify against the API, not the UI. Every step here can be checked with vrg, kubectl, or a curl against /api/v4. The UI is fine for spot-checking; the API is what an agent loop reads. Build for the latter and the former takes care of itself.

The next post picks up the meta thread — what the agent loop got right and wrong while wiring this integration together, and the breakthrough that took the off-vnet reachability problem from "still investigating" to a one-rule fix.

Read more