kubehz Platform

lok8s can optionally connect a cluster to kubehz, a managed-Kubernetes platform for Hetzner. lok8s supports two ways to use it:

Self-hosted clusters. You run the whole cluster on your own Hetzner account. The integration gives you dashboard visibility (node status, control-plane health, certificate expiry).
Hosted control planes. kubehz runs the control plane. You run only the workers, on your own Hetzner account. This needs a kubehz account and an API token in KUBEHZ_TOKEN. The quickest start is lo init cluster <domain> --driver kubehz-hosted, which writes a hosted cluster spec (see lo init).

Sign-up, prices and terms are on kubehz.cloud. Sign-up can need an invite code.

No lock-in

The integration is strictly opt-in. lok8s works fully without kubehz: the default is no platform contact at all, every kubehz call during provisioning is non-fatal (a failed registration warns and continues), and deregistering removes everything. Nothing in the provisioning or deployment path depends on the platform being reachable.

`spec.kubehz`

Platform integration is configured per cluster in cluster.lok8s.yaml:

yaml

spec:
  kubehz:
    hosting: self                        # self | hosted | shared
    access: registered                   # none | registered | managed
    agent: cronjob                       # cronjob | operator — which agent beats
    apiUrl: https://api.kubehz.cloud     # required when hosted/shared, or access != none
    connectHcloudToken: false            # opt-in: hand kubehz your HCLOUD_TOKEN (see below)

Two independent axes:

Axis	Values	Meaning
`hosting`	`self` (default)	You provision and own the control plane: the normal lok8s flow.
	`hosted`	kubehz runs the control plane in its infrastructure; you run only workers. Requires `apiUrl` and a kubehz account: `lo provision` creates the cluster with your `KUBEHZ_TOKEN`. With `kind: Lo` it additionally requires `spec.runner`.
	`shared`	kubehz runs a control plane shared by many customers and you get namespaces on it, with nodes you register yourself. Requires `apiUrl` and `kind: Kubehz`. See Spaces.
`access`	`none` (default)	No platform contact whatsoever.
	`registered`	The in-cluster agent registers the cluster and sends authenticated heartbeats: read-only dashboard visibility.
	`managed`	Everything `registered` does, plus kubehz's management features (healing policies, capacity watches, desired-state management) driven from the dashboard. Subscription-gated (Supporter+): the platform enforces the tier once you claim the cluster. Acting is pull-based: the in-cluster agent fetches desired state and applies it locally with the cluster's own credentials; the platform never pushes into your cluster, and per-feature execution switches let you keep acting off. Acting also needs `agent: operator`: the default CronJob agent reports, it does not act.

Plus two more keys:

Key	Default	Meaning
`agent`	`cronjob`	Which in-cluster agent sends the heartbeats: the bash CronJob or the Go live agent. Exactly one of them beats. See Choosing an agent.
`connectHcloudToken`	`false`	When `true` and a `KUBEHZ_TOKEN` is set, `lo provision` also stores your `HCLOUD_TOKEN` with the platform (encrypted at rest, used only to manage your clusters) so the dashboard can drive provisioning (worker pools, SSH keys). Without it `lo` uses the token only locally and never sends it. A read-only token gets stored for account-exact pricing but leaves provisioning locked.

apiUrl is required when hosting is hosted or shared, or when access != none, and it must be HTTPS: the agent's bearer token travels on this URL, so lo refuses plain-HTTP endpoints outright.

The hosted control-plane shape

A hosted control plane has no plan and no tier. Its shape describes it:

Value	Range	Default	Meaning
`spec.controlPlane.replicas`	1 to 3	1	apiserver replicas. `lo provision` sends this count to the platform.
state limit	1 GB to the datastore maximum	2 GB	How much cluster state the control plane holds (etcd: up to 6 GB, kine: up to 32 GB).
auto-grow step and cap	1 GB steps, up to the datastore maximum	1 GB, the datastore maximum	The platform raises the limit by one step at 80% use, until the cap.

The presets in the dashboard (dev: one apiserver, ha: three) are defaults only. The platform bills every started hour of what runs. Each unit is capped at 730 hours a month, so the monthly price on the price page is the most a unit costs. The state limit, the step and the cap live in the dashboard.

The platform admits a hosted cluster by the shape it asks for. When the pool cannot hold one more control plane of that shape, lo provision stops with the api's 503 AT_CAPACITY. The message names what is full and counts it the way that gate counts: slots in use and free for a shape or for the free pool, and nodes present against nodes required when a cluster asks for dedicated hardware. Then it gives the ways out: retry later (with a suggested wait), pick a smaller shape with fewer apiserver replicas or drop the option, watch the live capacity, run a self-hosted cluster meanwhile, or contact support. GET <apiUrl>/api/capacity lists the live capacity per shape without a token.

Upgrades and maintenance windows

Two more optional blocks under spec.kubehz tell the platform how far it may upgrade your cluster without being asked, and when that work is allowed to run:

yaml

spec:
  kubehz:
    upgrades:
      channel: patch            # none | patch | minor — absent = patch
      defer: window             # window | immediate — absent = window
    maintenanceWindow:
      enabled: true
      daysOfWeek: [Sat]
      startTime: "02:00"        # HH:MM, in the window's timezone
      durationMinutes: 240
      timezone: Europe/Berlin
      exclusions:               # absolute freezes — a date, or a from/to range
        - "2026-12-20/2027-01-06"

channel is consent, not a schedule: patch (the default) allows patch releases; minor allows minor releases too. none disables automatic upgrades entirely. Not recommended: you forgo automatic security patches, and it does not exempt the cluster from the end-of-support floor. The platform still upgrades clusters that run releases too far behind, regardless of channel.

Changed default. An absent channel previously meant "no automatic upgrades"; it now means patch. A spec without a channel receives automatic patch upgrades once upgrade execution ships. Set channel: none explicitly if you want the old behavior.

defer decides when an allowed upgrade starts: window (the default) waits for the next maintenance window; immediate starts as soon as the upgrade is available.
maintenanceWindow bounds platform-driven work generally, not just upgrades. exclusions are absolute freezes: nothing platform-driven runs inside one, except forced security/EOL upgrades.

How an upgrade lands depends on who owns the machines:

Nodes you own (self-hosted clusters, and machines you registered yourself) get upgraded in place: the upgrade swaps the kubelet underneath the running workloads, and the swap itself does not restart them. On the managed tier, the live agent will walk the nodes one at a time once upgrade execution ships.
Hosted worker pools get replaced, surge-first: new nodes join on the target version before old ones drain and leave. Per-pool tuning (surge size and the like) rides the pool definition on the platform side, not this file.

lo checks the shape: the two enums, and that every exclusion is a YYYY-MM-DD date or from/to range. Full semantic validation happens where each block is consumed: for hosted clusters and pools, the platform API validates its own payloads; for self-hosted clusters the block is the declared input for the managed-tier agent's upgrade walk. The agent will read it locally once upgrade execution ships, so your cluster's upgrade behavior will never depend on the platform being reachable. Until then the block is declarative only. lo does not catch calendar validity beyond the shape (a February 30th) today.

Spaces

hosting: shared is the third shape: kubehz runs a control plane shared by many customers, you get namespaces on it, and you register your own machines as the nodes your workloads run on. There is no control plane to provision, so the driver is thin: kind: Kubehz.

yaml

# clusters/acme.example.org/cluster.lok8s.yaml
apiVersion: cluster.lok8s.dev/v1beta1
kind: Kubehz
metadata:
  name: acme
spec:
  cluster:
    domain: acme.example.org
  kubehz:
    hosting: shared
    apiUrl: https://api.kubehz.cloud
    space:
      slug: acme          # optional — defaults to the domain's first label
      name: Acme Prod     # optional — defaults to the slug
      nodes:              # optional — one join ticket minted per name
        - worker-1
        - worker-2
      limits:             # optional — omit a number to take the platform's
        nodes: 2          # the node ceiling
        namespaces: 1     # the namespace ceiling
        objectCapKiB: 256 # KiB per Secret/ConfigMap

A space is three numbers, and they live under space.limits. nodes is how many machines the space takes, namespaces is how many namespaces it holds, and objectCapKiB is the size cap for one Secret or ConfigMap in those namespaces. There is no plan to pick.

The ceiling on each number belongs to your account, not to lo. It differs per account, and extra resources are bought, so the platform is the only place that knows it. lo checks that a value is a whole number of 1 or more and sends it; the platform answers with your account's real ceiling if you asked for more. A number you leave out is not sent at all, and the platform applies its own default. Your current ceiling is in the kubehz dashboard.

space.limits.nodes is a count; space.nodes is the list of machine names to mint a ticket for. They are different fields, and a list under space.limits.nodes is refused with both names.

lo provision creates the space (or adopts it, if it already exists) and mints a single-use join ticket for every machine listed under space.nodes. lo destroy removes the space.

bash

lo provision                  # create/adopt the space + mint join tickets
lo kubehz join worker-3       # mint another ticket, any time
lo kubehz status              # space phase, the three numbers, the nodes
lo kubehz deregister          # remove the space

A few things worth knowing before you reach for it:

KUBEHZ_TOKEN is required, not optional as it is elsewhere: a space belongs to your account from the moment it exists, so there is no anonymous path and no claim step.
The join script comes with the ticket. The platform ships the recipe the docs describe (containerd, kubelet, the cluster CA verified against the ticket, bootstrap config) and lo writes it to a fresh private directory, $TMPDIR/kubehz-join-<random>/kubehz-join-<node>.sh (0700 over 0600), and prints the scp/ssh line — it runs nothing itself. The file carries the ticket: read it, copy it to the machine, run it there as root, delete it (and its directory) once the node has joined. The terminal does not repeat the ticket; lo kubehz join <node> --print-token shows it when you need it in the open. Older platforms without the script print the ticket and a pointer to the node-join guide instead.
The three numbers apply when the space is created. A later lo provision adopts the space and changes nothing on the platform. An edit under space.limits therefore gets a note in the output instead of a silent no-op. Change the numbers of a live space in the dashboard.
A join ticket is minted once, is bound to one node name, and expires quickly. Nothing stores the plaintext outside the script: a lost ticket gets re-minted, never recovered.
The kubeconfig holds no credential. The control plane is platform-operated and is not handed out. lo provision writes .kubeconfig/<domain>.yaml, the kubelogin file the api serves for the space: kubectl --context kubehz-<slug> signs you in with your kubehz account (OIDC) on first use. lo kubeconfig prints that path.
spec.bootstrap does not apply. The platform baseline (CNI, ingress, certificates, DNS) is already running on the shared plane, and a space cannot install cluster-scoped things anyway.
Machines are yours. lok8s does not provision them and kubehz does not bill for them. Deregistering a node withdraws its credentials; shutting the machine down is still your call.
The shard is the upgrade unit. The platform upgrades the shared control plane as a whole, on its own schedule, and retires old versions on that schedule too. Here upgrades.channel is a preference, not consent: an eager channel moves your space early in a shard's upgrade window, a conservative one later, but the platform migrates every space by the deadline. There is no opting out.

Limits, and whose problem each one is

A space runs inside three independent budgets, and the error codes keep them apart deliberately:

Your space's own three numbers: limits.nodes, limits.namespaces and limits.objectCapKiB. Exceeding one answers 403 QUOTA_EXCEEDED and names the limit. Fix: remove something, or raise the limit in the dashboard. The numbers in the spec apply when the space is created, so raising one there does nothing to a space that already exists.
What your account allows: the ceiling on each of the three numbers, which differs per account and rises with what the account has bought. Ask for more than it allows and the create answers 403 SPACE_LIMITS_ABOVE_FREE; ask for more than a shared plane can hold for any account and it answers 400 SPACE_LIMITS_ABOVE_SHARED, which means a cluster that large needs a control plane of its own, so use hosting: hosted. A request that still carries a plan answers 400 SPACE_PLAN_RETIRED: remove spec.kubehz.space.plan and use spec.kubehz.space.limits. Each refusal names the account's real ceiling, and lo prints that message together with the values it sent. The current ceiling is in the kubehz dashboard.
The platform's capacity: how much a shared control plane can carry. When the platform is the limit you get 409 SHARD_AT_CAPACITY (a node join the current plane genuinely cannot take) or 409 NO_SHARD_AVAILABLE (a space create with nowhere to land). Both say so explicitly: "this is a platform capacity limit, not an account limit". A larger account does not change them, and nothing you delete will either.

The platform manages its own headroom (it adds shared planes as they fill), so the 409s are expected to be rare and transient. Retry later, and if one persists, that is a support ticket, not a configuration hunt.

The platform docs cover joining a node (requirements, the recommended host firewall, the bare-metal lane): kubehz docs → Spaces.

Still no lock-in

A space is the one shape where the control plane is not yours, so it is worth being explicit: everything else in lok8s keeps working without kubehz, and moving to a cluster of your own (hosting: hosted or self) is a supported path rather than an export-and-rebuild.

Bootstrap addons on hosted clusters

Your spec's bootstrap section applies to a hosted cluster too: after lo provision has the control plane running and the kubeconfig downloaded, lok8s applies the same bootstrap DAG a self-hosted cluster gets: your gateway, sso-gate, cert-manager, whatever you declared. A few hosted-specific rules:

cilium and ccm are platform-owned and always skipped (matched on the addon directory, so a name:-renamed entry skips too): the hosting platform manages the CNI as a system application and runs the cloud integration on its side (kubehz, for one, ships its hosted CNI with WireGuard encryption on by default: a platform property, not something this skip configures). Declaring them is harmless; bootstrapping your own copy would fight the platform for the datapath, so lo refuses on your behalf.
You need working kubectl access: the default hosted kubeconfig is OIDC (kubelogin + a browser sign-in). Headless/CI runs without it get a clear notice and a skipped bootstrap. Install the kubelogin plugin and re-run lo provision or lo bootstrap interactively.
Workers first: a control plane with no Ready workers can't run addon workloads, and wait:-gated entries would hang, so lo skips the bootstrap with a notice and you re-run once workers have joined. The apply is idempotent.

Nodes you bring (static pools)

On a hosting: hosted cluster, kubehz normally provisions the workers for you on your own Hetzner account. A static pool is the other option: a worker pool whose machines are yours (bare metal, a second cloud, a virtual machine that already runs). Nothing provisions them and nothing rotates them. You install a kubelet, run one join command, and the node appears in your cluster next to the pools kubehz manages. One cluster can hold both kinds.

Declare the pool with kind: static in the dashboard, then work from the machine itself:

bash

lo kubehz node join --pool metal      # mint a ticket and join THIS machine
lo kubehz node status                 # the nodes you brought, and the slot count
lo kubehz node remove --name box-1    # take a node out and free its slot

Cluster-level commands are unchanged

lo kubehz join mints a join ticket for a Space (hosting: shared), and lo kubehz deregister removes a whole cluster. The node group is the node-level surface, so a verb never touches more than its name says.

`lo kubehz node join`

Run it as root on the machine you are adding. It mints a join ticket from the platform, then runs the kubeadm join line the platform composes. The platform composes that line because the join address and the CA fingerprints are properties of a control plane you do not run.

Flag	Default	Meaning
`--pool`	the one pool this cluster's nodes share	The `kind: static` pool to join. Name it for the first node of a pool, or when the cluster holds more than one.
`--name`	this machine's short hostname, lowercased	The node name. A DNS label: lowercase letters, digits and dashes.
`--cluster-id`	the active domain's cluster	The platform cluster id (`cl-…`). Set it to reach a cluster your local specs do not describe.
`--node-ip`	(none)	The address other nodes reach this machine on. Add it when the machine's default-route address is not the one the cluster can dial.
`--kubelet-version`	read from this machine	The kubelet version to declare. The platform refuses a version more than two minors behind the control plane.
`--print-only`	off	Mint the ticket, print the command, and run nothing.

The command refuses before it mints anything when kubeadm is absent, or when the shell is not root. Use --print-only to mint on one machine and join on another.

Re-run with sudo -E, not plain sudo

kubeadm join must run as root, but the join reads KUBEHZ_TOKEN and the PATH_* lo environment first. Plain sudo starts a clean environment and drops both, so the mint would fail to authenticate. Use sudo -E lo kubehz node join … to carry your environment through, or mint with --print-only as your own user and run the printed kubeadm join line under root separately.

Requirements on the machine: a container runtime, plus kubelet, kubeadm and kubectl from a supported Kubernetes release. The machine must reach the join endpoint on TCP 6443 and the node tunnel on TCP 8088. A machine behind NAT or CGNAT needs --node-ip with an address the cluster can reach.

The ticket is a credential. It is bound to one node name and to ten minutes, it works once, and its plaintext exists only in that one response: a lost ticket gets minted again, never recovered. A second mint for the same name revokes the first.

`lo kubehz node status`

Lists the nodes you brought, with the slot count against the cluster's static-node limit:

Cluster: cl-1234abcd (acme.example.org)
Nodes:   2/20

  NAME                     POOL             STATUS     JOINED
  box-1                    metal            Ready      2026-08-30T10:00:00Z
  box-2                    metal            Joining    -

This does not list nodes in kubehz-provisioned pools: those appear as pool counts on the cluster itself. A warning appears while the control plane has published no join address: the platform refuses a mint until it does.

`lo kubehz node remove`

Removes one node from the cluster and frees its slot at once. The platform cordons the node, then deletes it from your cluster.

Removal takes the membership, never the hardware. The removal does not evict pods on the machine first, because the machine is yours and asking for it back is not a reason for the platform to move your workloads. Drain the node yourself when you want a clean handover. Run kubeadm reset on the machine before you join it anywhere again.

A dark node still holds its slot

A node that stops reporting keeps its slot until you remove it. The platform notifies you, and removes an unacknowledged dark node after 30 days. Use lo kubehz node remove when you already know the machine is gone.

When a join is refused

Every refusal names what to do next. The four you are most likely to meet:

Refusal	What it means
Static pools not enabled	The feature rolls out account by account. Ask kubehz support to enable it.
The kubelet is too old	A node must run a kubelet no more than two minor versions below the control plane. Upgrade the kubelet.
No free slot	The cluster is at its static-node limit. Remove a node you no longer use, or ask support to raise the limit.
No join address yet	The control plane publishes its join address and CA fingerprints after the cluster declares its first `kind: static` pool. Check `lo kubehz node status`, then try again.

Registration

When access is registered or managed, the cluster registers itself from inside the cluster: the heartbeat agent (below) owns this. On its first run the agent generates two 256-bit secrets locally and announces only their SHA-256 hashes to POST /api/clusters/agent-register:

Secret	Key in `kubehz-agent`	Role
agent-token `A`	`agent-token`	The live credential every heartbeat authenticates with (`Authorization: Bearer A`). It never leaves the cluster and is never printed by any command.
claim-code `C`	`claim-code`	A one-time proof you paste into the dashboard to claim the cluster.

The platform stores only sha256(A) and sha256(C): it can never push a credential into your cluster, and a database leak yields no usable secret. The agent generates the secrets once and never rotates them, so re-applying it is a no-op. Registration is provider-agnostic: no SSH key, no hcloud token, nothing Hetzner-specific.

Legacy: lo kubehz register

A pre-agent announce still exists and runs automatically at the end of lo provision (or standalone via lo kubehz register): it POSTs the cluster domain plus an SSH-key MD5 fingerprint to /api/clusters/register, making the cluster visible before the agent is deployed. It is vestigial for self-hosted clusters (the agent's self-registration and the claim code below are the real path) and stays only as a best-effort announce and as the anchor for the platform's hosted-control-plane auto-claim. It proves nothing and needs no token; if the API is unreachable it warns and provisioning continues.

When the platform is busy: 503 or 429

The platform can refuse a registration for a short time. Then it answers 503 or 429 with a Retry-After header and a code, for example REGISTRATION_UNAVAILABLE or RATE_LIMITED. This applies to all three register modes: the direct claim with KUBEHZ_TOKEN, the claim key with HCLOUD_TOKEN, and the fingerprint announce.

lo waits for the time in Retry-After and tries again. One register call tries at most 3 times, and one wait is at most 60 seconds. So one call waits at most 2 minutes.
Ctrl-C stops the wait at once.
If the platform still refuses after the third try, lo stops with an error. The error names the code and the command to run when the platform is available again. lo does not try the next register mode, because the platform gives the same answer to all of them.
The command in the error depends on where the registration ran:
- After lo kubehz register, run lo kubehz register again.
- After lo provision, run lo provision --bootstrap. The error stops the provision after the infrastructure is up and before the bootstrap, and the provision exits with status 1. The cluster runs, but it has no spec.bootstrap addons and no platform registration. lo provision --bootstrap registers the cluster and then applies the bootstrap. It does not provision the infrastructure again.
lo does not try a 503 without Retry-After again, because the answer gives no time to wait. The registration then continues the same way as when the API does not answer. The warning names the status, for example (HTTP 503).
The 2-minute limit applies to each register call. If a call gets two refusals with Retry-After and then a different refusal, lo tries the next mode, and that mode can wait again. In that rare case, the waits of the three modes add up to at most 6 minutes.

Claim before you deploy: the bind secret

The pre-agent announce also hands back a one-time bind secret. lo kubehz register stores it at clusters/<domain>/.kubehz-bind (mode 0600). lo kubehz deploy stages it into the cluster, and the agent presents it on its first agent-register. The platform then adopts the announced row (the row that lo kubehz register created), wherever it sits by then. The row can still be pending, or already claimed in the dashboard. Either way the agent binds there instead of making a second row, so a cluster you claim before the agent deploys still shows its own heartbeats.

The secret proves only that the operator who deploys is the operator who announced. It rotates on every re-announce and is spent on the first adoption. It is a credential: keep .kubehz-bind out of version control.

A re-run of lo kubehz register sends the stored secret, so the re-run keeps the same cluster record and stores the new secret it gets back. lo uses the file only when it is readable and holds exactly 64 lowercase hex characters, and it never puts the value on a command line.

The platform does not adopt the announced row by its domain alone. An agent that registers without the secret gets a separate pending row, and the announced row gets no heartbeats. So lo kubehz deploy stages the secret before it applies the agent:

The deploy applies the namespace kubehz-system, then the Secret kubehz-agent-bind with a server-side apply. The kubeconfig needs patch on Secrets in kubehz-system, and create when the Secret is new. It does not need delete.
If the deploy cannot stage the secret, it stops before it changes an agent.
If .kubehz-bind is missing (for example, lo kubehz register ran on another machine) or holds no valid secret, the deploy warns and continues without the secret.

To deploy without the secret on purpose, remove .kubehz-bind. The agent then registers a separate pending row. Claim that row with the code from lo kubehz claim-code. The announced row gets no heartbeats.

Claiming

Claiming attaches a registered cluster to your account. It is provider-agnostic and needs no hcloud token: you prove ownership when you read a one-time code out of the cluster (which only its operator can do) and paste it into the dashboard while signed in.

Deploy the heartbeat agent (below). On its first run it registers the cluster and generates the claim code.
Read the claim code with your kubeconfig pointed at the cluster:
bash
```
lo kubehz claim-code
```
It prints the code (khzc_…) plus a paste hint. The equivalent raw command is:
bash
```
kubectl -n kubehz-system get secret kubehz-agent \
  -o jsonpath='{.data.claim-code}' | base64 -d
```
Paste it into the dashboard Claim page while signed in. The platform attributes the cluster to your account and consumes the code (single-use).

The claim code is the only secret that transits your browser, and only once. The agent-token (the credential that authenticates heartbeats) never leaves the cluster, and no command prints it, lo kubehz claim-code included.

Choosing an agent

Dashboard visibility comes from an in-cluster agent. lok8s ships two, and spec.kubehz.agent picks which one sends the heartbeats:

	`cronjob` (default)	`operator`
Workload	CronJob `kubehz-heartbeat`	Deployment `kubehz-live-agent`
What it is	A shell script in a stock `alpine/k8s` image	The Go agent from the public kubehz-agent repo
Cadence	Every 5 minutes	Within seconds of a change, and at least every minute
Reports	Node status, control-plane component health, certificate expiry, the 24 h assessment	Node status with capacity and instance type, control-plane component health, certificate expiry, pod counts by phase, warning events, machine-controller failures, your addon inventory
Can act	No	Yes: worker scaling, self-healing and worker upgrades, when the platform authorizes them
Costs	~50 m CPU for a few seconds every 5 minutes	~25 m CPU and 64 Mi resident, always on

Deploy either one with:

bash

lo use my-cluster.example.com   # point the kubeconfig at the cluster
lo kubehz deploy                # applies the agent your spec names
lo kubehz deploy --dry-run      # print the manifests instead

Change spec.kubehz.agent and re-run lo kubehz deploy to switch. The command applies the agent you named and stops the other one, differently in each direction. Switching to operator keeps the CronJob and silences it: it still bootstraps and enrolls the identity Secret, and it reads the KUBEHZ_HEARTBEAT_OWNER marker before every beat. Switching to cronjobdeletes the live agent's Deployment, because the Go agent reads no marker and beats whenever it runs.

Exactly one agent may beat

The platform keeps the latest heartbeat and nothing older. The two agents report different fields, so if both beat, each one erases what the other reported: your live view would empty itself every five minutes.

You do not have to police this as long as you switch with lo kubehz deploy. spec.kubehz.agent holds one value, so a spec cannot ask for both; the command applies, silences and deletes in an order that leaves no gap in either direction, and it stops rather than continues if any step fails; and the cluster itself carries the decision as KUBEHZ_HEARTBEAT_OWNER in the kubehz-agent-config ConfigMap, which the CronJob reads before every beat.

That marker is a one-way interlock, and it is worth knowing which way. It can silence the CronJob lok8s ships: that manifest reads the marker before every beat, so applying it by hand while the live agent owns the heartbeat starts no second producer. A CronJob you write yourself reads nothing and beats. It cannot silence the live agent either: the Go agent has no equivalent switch and beats whenever it runs, so what stops it is removing its Deployment, which is exactly what lo kubehz deploy does before it re-arms the CronJob. Apply the live agent's Deployment by hand while the marker says cronjob and you will have two producers. Let the command do the switching.

What the live agent gives you

With agent: operator, and once you claim the cluster:

Live view. Node capacity and instance type, pod counts by phase, recent warning events, and machine-controller provisioning failures, pushed within seconds of a change. Pod names and namespaces stay in your cluster unless you opt in.
Addon updates in kubectl. The agent reports your lok8s ClusterInventory and writes the platform's answer back onto it, so kubectl get clusterinventory cluster -o yaml shows which addons have a newer version. No dashboard needed.
Worker scaling, self-healing and worker upgrades, when all of these hold: your access is managed, your tenant tier allows acting (Supporter+), and the platform has the feature switched on. The agent asks the platform what it should do and obeys the answer. Nothing local turns acting on (there is no flag for it), and without access: managed the platform grants no acting permission at all.

Acting always runs with your cluster's own credentials. kubehz holds no credential for your cluster and cannot connect into it; the agent pulls, and kubectl delete namespace kubehz-system ends it.

What you give up for now

While the live agent owns the heartbeat, the platform does not receive control-plane component health, certificate expiry, or the 24 h assessment: the live agent does not collect them yet, so lo kubehz assess will go stale. Set agent: cronjob and re-run lo kubehz deploy to get them back.

Permissions the live agent holds

The base install is read-only. It can get, list and watch nodes, pods and events, and read your ClusterInventory. It can list CertificateSigningRequests (for the certificate-expiry field) and get the /readyz and /version API paths (for component health). It can get one Secret by name: its own identity Secret. It has one write: patch on clusterinventories/status, which is where the addon-update answer lands. It cannot read pod logs, exec into a pod, or read any other Secret.

With access: managed it also gets, in kube-system only:

patch on machinedeployments.cluster.k8s.io, two fields only: the replica count (scaling) and the kubelet version (worker upgrades). No create, no delete.
get/list/watch and delete on machines.cluster.k8s.io. The delete verb exists for one reason: self-healing removes an unhealthy worker Machine so your MachineSet rebuilds it. It is the sharpest permission the agent holds. Remove that one verb and self-healing fails closed with a reported error, while everything else keeps working.
delete on pods, cluster-wide. This one exists only to finish a deletion the cluster cannot finish itself: when a healed node is truly dead, the pods stuck Terminating on it can never confirm their eviction, so the machine sits in deletion and the server keeps billing you. The agent force-deletes exactly those pods, once per machine, and only while the node is still unreachable.

Every one of these is documented at the rule in rbac-managed.yaml, and lok8s ships that file byte-identical to the agent's own repo: the permissions you grant are the ones the public source documents. That link points at the exact release tag lok8s vendored, which is what makes "byte-identical" checkable: the SHA-256 of both vendored files is recorded in .lok8s/libs/kubehz/manifests/live-agent/UPSTREAM.sha256, and sha256sum --check UPSTREAM.sha256 from that directory verifies it.

The image is ghcr.io/kernpilot/kubehz-agent, pinned by digest and signed. To check what you are about to run:

bash

cosign verify \
  --certificate-identity-regexp '^https://github\.com/kernpilot/kubehz-agent/\.github/workflows/release\.yaml@' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/kernpilot/kubehz-agent@sha256:<the digest in the manifest>

The heartbeat agent

The default agent is a CronJob kubehz-heartbeat in the kubehz-system namespace that runs every five minutes and POSTs a status snapshot to /api/clusters/<domain>/heartbeat. It also owns the cluster's identity, which is why it stays deployed even when the live agent takes over the heartbeat: it creates the identity Secret and enrolls it, and the live agent only reads that Secret. Both agents tolerate the control-plane taint, so a cluster with no worker yet, or one that is control-plane-only by design, runs them on the control plane. A first lo kubehz deploy with agent: operator runs the CronJob's bootstrap once as a one-off Job and waits for it. The live agent then finds its identity Secret at once, not at the CronJob's next tick. Properties worth knowing:

Self-bootstrapping identity. On each run the agent ensures one Secret, kubehz-agent in kubehz-system, holding a self-generated agent-token A (khz_agt_…) and claim-code C (khzc_…). The agent creates it once and never rotates it; the platform only ever receives sha256(A)/sha256(C).
Authenticated. Every heartbeat carries Authorization: Bearer A. Once a cluster has enrolled a token, the platform rejects anonymous heartbeats for it: the identity is proven from the first beat.
Outbound-only. The agent pushes; the platform never connects into your cluster and holds no credentials for it (only verifier hashes).
Least-privilege RBAC. The kubehz-agent ServiceAccount can read cluster state: get/list on nodes, namespaces, componentstatuses and CSRs, list pods in kube-system only (control-plane health), and get on the /readyz
- /version API paths. On top of that it holds get+create (never update/patch/delete) on Secrets in its own kubehz-system namespace, to bootstrap its identity Secret once. It reads no Secrets elsewhere and can rotate nothing. The assessment adds read-only probe grants (storageclasses, persistentvolumes, services list, webhook configuration list, get on the one CAPI CRD, kube-system daemonset list + the kubeadm-config ConfigMap) and get/patch scoped to the single kubehz-agent-config ConfigMap for its send-gate marker.
Domain-keyed. The payload's clusterId is the cluster domain, the same identity it registers under.
Hardened. Stock alpine/k8s image (pinned by digest), non-root, read-only root filesystem, all capabilities dropped.

The snapshot contains the Kubernetes version, node list (name/readiness/ role/instance type), certificate expiry, and control-plane component health. The agent reads all nodes with one API call, so the number of calls does not grow with the cluster. It sends at most 500 nodes, the limit of the platform's heartbeat API. A larger cluster sends its first 500 nodes, and the agent writes the total to its log:

json

{
  "clusterId": "my-cluster.example.com",
  "timestamp": "2026-07-04T12:00:00Z",
  "kubernetes": { "version": "v1.31.8" },
  "nodes": [{ "name": "cp-1", "status": "Ready", "roles": "control-plane", "instanceType": "cx32" }],
  "components": [
    { "name": "apiserver", "status": "Healthy" },
    { "name": "etcd", "status": "Healthy" },
    { "name": "scheduler", "status": "Healthy" },
    { "name": "controller-manager", "status": "Healthy" }
  ],
  "certificates": { "expiresAt": "2027-06-15T00:00:00Z" }
}

Each node's instanceType mirrors the well-known node.kubernetes.io/instance-type label (set by the cloud provider, e.g. cx32 on Hetzner). This is what lets the dashboard show a price overview for self-hosted clusters. Nodes without the label (bare metal, kind, no CCM) report an empty string; like every other probe, it never breaks the heartbeat.

Component health is honestly derived (apiserver and etcd from the apiserver's /readyz?verbose checks, scheduler and controller-manager from the kube-system static pods) and fail-soft: a probe that errors omits its component instead of guessing, a control plane without static pods (managed/external) simply reports fewer components, and the heartbeat itself never fails because a probe did.

Deploy it with lo kubehz deploy, which reads spec.kubehz.agent, substitutes your apiUrl and cluster domain, and applies the result with your current kubeconfig:

bash

lo use my-cluster.example.com
lo kubehz deploy

On its first run the agent creates its identity Secret, self-registers, and starts beating. Then read the claim code with lo kubehz claim-code and paste it into the dashboard to claim the cluster.

The manifests ship with lok8s as kustomize directories: .lok8s/libs/kubehz/manifests/agent/ (CronJob) and .lok8s/libs/kubehz/manifests/live-agent/ (live agent, base and managed). So you can read exactly what deploy applies, or render it first with lo kubehz deploy --dry-run.

Status and deregistration

bash

lo kubehz status        # local spec + live platform status for the domain
lo kubehz deregister    # remove the cluster from the platform

status prints the configured hosting/access/apiUrl, then reads your registry row on the platform: the cluster's live status, its id, and the last heartbeat. deregister deletes the platform registration: it resolves the cluster id from your registry and sends DELETE /api/clusters/<id> (and runs automatically during lo destroy when access != none). Both look the cluster up through the tenant registry, so both need KUBEHZ_TOKEN set to a clusters:write token of the owning tenant. Without it, status reports the HTTP refusal and deregister refuses without deleting anything.

`KUBEHZ_TOKEN`: claim at provision time

KUBEHZ_TOKEN is a clusters:write API token you mint in the dashboard (Access → API Tokens). When you set it during lo provision, the platform claims the cluster directly to your tenant at registration: no claim code to paste, no fingerprint step. It shows up in your dashboard already owned. Pair it with spec.kubehz.connectHcloudToken: true to also wire dashboard-driven provisioning in the same run. Without KUBEHZ_TOKEN, lo provision registers a self-hosted cluster anonymously and you claim the cluster afterwards. A hosted cluster always needs the token: the platform creates it for your tenant.

Deregistering deletes the platform cluster record and, when HCLOUD_TOKEN is set, the kubehz-claim-<domain> SSH key from your Hetzner Cloud account. The in-cluster agents survive. To remove them as well:

bash

kubectl delete namespace kubehz-system
kubectl delete clusterrole,clusterrolebinding -l app.kubernetes.io/part-of=kubehz
kubectl -n kube-system delete role,rolebinding -l app.kubernetes.io/part-of=kubehz

All three are needed. Both agents run in kubehz-system, but neither keeps all of its RBAC there. The cluster-scoped roles go in the second command. Both agents also hold a namespaced Role/RoleBinding in kube-system: the CronJob's pod-list and assessment reads (kubehz-agent), and, at access: managed, the MachineDeployment and Machine grants (kubehz-live-agent-machinedeployments). Those survive the first two commands.

After the three, nothing labelled app.kubernetes.io/part-of=kubehz is left:

bash

kubectl get clusterrole,clusterrolebinding -l app.kubernetes.io/part-of=kubehz
kubectl get role,rolebinding -A -l app.kubernetes.io/part-of=kubehz

Both should print No resources found. If you moved the worker pools out of kube-system with KUBEHZ_MD_NAMESPACE, the managed Role moved with them. The -A query above finds it wherever it is.

Agent keys

An agent key is a machine identity for an AI agent or a pipeline. A tenant owner or admin creates it in the dashboard or with POST /api/agent-keys. The answer shows the client id and the client secret once, the token endpoint, and the scope to request.

lo kubehz token turns the key into an access token. kubectl runs it as an exec credential plugin:

yaml

users:
- name: agent
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1
      command: lo
      args: [kubehz, token, --token-url, "https://id.kubehz.cloud/oauth/v2/token", --scope, "<tokenScope>"]
      interactiveMode: Never

Set KUBEHZ_AGENT_CLIENT_ID and KUBEHZ_AGENT_CLIENT_SECRET in the agent's environment, or keep the secret in a file and pass --secret-file. The file can be a pipe, for example --secret-file <(pass show kubehz/agent). The secret is never a flag, because every user on the machine can read a command line.

The same token works against the platform api:

bash

export KUBEHZ_TOKEN=$(lo kubehz token --format token)
lo kubehz status

The command caches the access token in ${XDG_CACHE_HOME:-~/.cache}/lok8s/kubehz-token/ (mode 0600). It asks the token endpoint again five minutes before the token expires. --no-cache always asks and writes nothing. When an owner revokes the key, the api refuses it at once. A token that was already issued stays valid at the apiservers until it expires (at most 12 hours).

A key that was created a moment ago can be refused for a short time, until the identity provider knows it. Thus the command tries again after 0.25, 0.5 and 1 second when the connection fails, the endpoint answers with a 5xx status, or it answers invalid_client. It prints only the last failure and never caches one. An attempt ends after 30 seconds and is not tried again, so a stalled endpoint holds kubectl for 30 seconds at most.

The output of this command is a bearer token. Thus the command is hidden from lo kubehz --help, no lo mcp server offers it as a tool, and lo chat denies it to the model.

Agent tools

lo kubehz space and lo kubehz cluster call the kubehz api with an agent key. They need no project and no cluster spec. lo mcp turns them into tools, so an AI agent that runs lo mcp on your machine can create a space for a task, extend its lease, and delete it when the task is done.

The environment

The commands read five values from the environment, never from a flag:

Variable	Value
`KUBEHZ_API_URL`	The kubehz api, for example `https://api.kubehz.cloud`. HTTPS only.
`KUBEHZ_AGENT_CLIENT_ID`	`clientId` of the key
`KUBEHZ_AGENT_CLIENT_SECRET`	`clientSecret` of the key
`KUBEHZ_AGENT_TOKEN_URL`	`tokenEndpoint` of the key
`KUBEHZ_AGENT_SCOPE`	`tokenScope` of the key

The answer that created the key holds the last four values. lo gets the access token the same way lo kubehz token does, and uses the same cache. Without an agent key, the commands send KUBEHZ_TOKEN (an API token from the dashboard). When both are set, lo uses the agent key.

No flag sets the api URL or the credential. A model that could set the URL could send the token to another host.

Commands

bash

lo kubehz space list                          # the spaces the key reaches
lo kubehz space get sp-1a2b3c4d               # one space: limits, nodes, endpoint, lease
lo kubehz space create --name "CI run" --slug ci-1234 --lease-hours 3
lo kubehz space lease sp-1a2b3c4d --hours 6   # the platform deletes the space 6 hours from now
lo kubehz space delete sp-1a2b3c4d
lo kubehz space kubeconfig sp-1a2b3c4d --file agent.yaml

lo kubehz cluster list                        # the clusters the key reaches
lo kubehz cluster get cl-1a2b3c4d
lo kubehz cluster lease cl-1a2b3c4d --hours 4 # hosted clusters only
lo kubehz cluster kubeconfig cl-1a2b3c4d --file agent.yaml

space create also takes --nodes, --namespaces and --object-cap-kib (the three numbers) and --region. A number you leave out is not sent, and the platform applies its default.

Every command takes -o text|json|yaml. text prints a table for a list and one line per field for one record. json and yaml print fixed fields (see Output formats). Use -o json in a script or an agent: -o yaml prints a string such as yes without quotes, and a YAML 1.1 reader takes it for a boolean (#246). A field that the api leaves out is null. Before lo prints a string from the api, it removes the control characters: C0, DEL, C1 (U+0080 to U+009F), the line and paragraph separators (U+2028, U+2029) and the bidi controls (U+202A to U+202E, U+2066 to U+2069).

kubeconfig writes the agent kubeconfig to the file you name (mode 0600) and prints the path. The file holds no secret: its exec stanza runs lo kubehz token, so kubectl uses the agent key from the environment.

bash

lo kubehz space kubeconfig sp-1a2b3c4d --file agent.yaml
KUBECONFIG=agent.yaml kubectl get pods

The command does not replace a file that exists, so --file ~/.kube/config cannot remove your other contexts by accident. A file that appears while lo downloads stays too. Add --force (or -f) to replace the file. The new file belongs to you and has mode 0600. A hard link to the old file keeps the old content.

When the path is a link, --force replaces the file that the link points to, and the link stays. lo does this only when you own the link and its target. Thus a link that another user puts in a shared directory such as /tmp cannot make lo replace one of your files. A directory, a link to one, and a link to nothing are refused.

A path in /proc or /dev is refused, also when a link in the path points there. Some of these paths change with the process that opens them: for lo, /proc/self/exe is the lo binary, and /dev/stdout is the file that its output goes to. lo checks the path before it calls the api, and again just before it writes.

What the api enforces

lo adds no authority. It checks the shape of an id (sp-…, cl-…) and of a number, then sends the request. The api decides:

The role of the key. A viewer key can read. A create, a lease or a delete answers 403 TOKEN_SCOPE_MISSING. Use a key with the role editor or admin. With KUBEHZ_TOKEN, the same code names the scope that the token does not hold.
The scope of the key. A key with the scope resources reaches only the clusters and spaces it names and the ones it creates. Another space answers 404, the same as a space that does not exist.
The spend cap. When the spend of this month on what the key created reaches its cap, a create or a lease extension answers 409 AGENT_KEY_SPEND_CAP. What already runs keeps running until its lease ends.
The lease. A space or a hosted cluster that an agent key creates gets a lease: 2 hours, or the hours that --lease-hours names (1 to 720). When the lease ends, the platform deletes the resource. lease --hours N sets the end to N hours from now. An agent key cannot remove a lease.

A refusal prints the HTTP status, the code and message of the api, and the next step.

As MCP tools

lo mcp puts each command in the tier its marker names:

Tool	Tier
`lo_kubehz_space_list`, `lo_kubehz_space_get`, `lo_kubehz_cluster_list`, `lo_kubehz_cluster_get`	default (read only)
`lo_kubehz_space_create`	`--allow-mutating`
`lo_kubehz_space_delete`, `lo_kubehz_space_lease`, `lo_kubehz_cluster_lease`	`--allow-destructive`
`lo kubehz space kubeconfig`, `lo kubehz cluster kubeconfig`	never a tool

lease is in the destructive tier because a shorter lease brings the delete closer. The two kubeconfig commands count as credential output, as lo kubeconfig does, so no tier offers them. The agent kubeconfig holds no secret, but run these two commands yourself, outside the MCP server.

Start the server with the agent key in its environment, not with your own credentials:

json

{
  "mcpServers": {
    "lok8s": {
      "type": "stdio",
      "command": "lo",
      "args": ["mcp", "start"],
      "env": {
        "LO_MCP_ALLOW": "mutating",
        "KUBEHZ_API_URL": "https://api.kubehz.cloud",
        "KUBEHZ_AGENT_CLIENT_ID": "<clientId>",
        "KUBEHZ_AGENT_CLIENT_SECRET": "<clientSecret>",
        "KUBEHZ_AGENT_TOKEN_URL": "<tokenEndpoint>",
        "KUBEHZ_AGENT_SCOPE": "<tokenScope>"
      }
    }
  }
}

This file holds the client secret: keep it out of version control. With mutating, the agent can create spaces. Each space ends when its lease ends: after two hours, or after the hours that the create names (720 at most). With destructive, the agent can also change leases and delete.

lo mcp refuses a call when a positional argument starts with a dash, and nothing runs. Thus an argument such as --force from the model cannot set a flag that the tool does not offer. lo mcp never offers lo chat: it passes its arguments on unread, and --lo <path> would run any file with the agent key in its environment.

lo chat cannot run these tools yet. It drives the bash variant of lo mcp (the argsh builtin), which names a command three levels deep by its last two words (lo_space_list) and then cannot run it (#245). The lo chat defaults deny lo_space_kubeconfig and lo_cluster_kubeconfig.

Assessment and handover

The heartbeat agent also collects a compact assessment of the cluster: Kubernetes version, datastore (etcd/kine/unknown), CNI, pod/service CIDRs, StorageClasses, a PV summary grouped by provisioner, LoadBalancer and webhook counts, node/control-plane counts. It is kubectl-only, read-only, fail-soft, and sent at most once per 24 h (or when the platform requests a refresh through the heartbeat response). The platform turns it into a handover feasibility verdict.

bash

lo kubehz assess    # pretty-print the stored assessment + feasibility

assess authenticates like lo kubehz status (KUBEHZ_TOKEN against spec.kubehz.apiUrl).

Receiving an ejected control plane

When you move a hosted control plane onto your own machine, the platform exports a bundle (cluster CA, service-account keys, front-proxy CA, the secrets-at-rest encryption key, an etcd snapshot location, and the endpoint DNS name). Two target-side commands consume it:

bash

# kubeadm path — run ON the fresh target node (writes /etc/kubernetes and
# /var/lib/etcd, runs kubeadm):
lo kubehz handover receive --bundle ./export-bundle [--snapshot ./snap.db] [--single-node]

# kubeone path — pre-seed only the PKI on node0, then provision as usual;
# kubeadm reuses existing CA files, preserving cluster identity:
lo kubehz handover preseed --bundle ./export-bundle --node <node0-ip>
lo provision

receive refuses a node that already carries Kubernetes state (use kubeadm reset first, or --force if you really mean it), restores the etcd snapshot (--snapshot for a pre-downloaded file; s3:// locations need an aws CLI on the node), seeds the exported PKI, runs kubeadm init with the pre-seeded state, and verifies the resulting cluster CA is byte-identical to the bundle's before telling you to cut DNS over. If that verify fails, do not cut over: the restored cluster did not keep its identity.

Troubleshooting

No heartbeats arriving? Check the CronJob is present, not suspended, and actually firing, then read the last run's logs:

bash

kubectl -n kubehz-system get cronjob kubehz-heartbeat   # SUSPEND + LAST SCHEDULE
kubectl -n kubehz-system logs -l app=kubehz-heartbeat

A 4xx from the API in those logs usually means the agent could not register or authenticate: a 401 is a missing or rejected agent-token, a 404 means the cluster is not registered yet. The agent self-registers on its first run and retries every tick, so a transient API outage clears itself.

No heartbeats, and you set agent: operator? Read the live agent instead:

bash

kubectl -n kubehz-system get deploy kubehz-live-agent
kubectl -n kubehz-system logs deploy/kubehz-live-agent

An ImagePullBackOff means the node cannot reach ghcr.io. Mirror the image and override the registry in the manifest, keeping the digest. A 401 means the agent-token was rejected: run lo kubehz re-enroll. If the Deployment is missing entirely, run lo kubehz deploy again.

To confirm the CronJob really has stood down:

bash

kubectl -n kubehz-system get configmap kubehz-agent-config \
  -o jsonpath='{.data.KUBEHZ_HEARTBEAT_OWNER}'   # must print: operator

If that prints cronjob while the live agent is running, both are beating. Run lo kubehz deploy to settle it.

Registration failed during provisioning? Provisioning no longer depends on it: a self-hosted cluster registers from inside, once the agent is deployed. Deploy the agent (above); it registers and starts beating on its own.

No claim code yet? lo kubehz claim-code reads it from the kubehz-agent Secret. If the Secret or its claim-code key is missing, the agent has not run yet. Confirm the CronJob fired at least once, then retry. Make sure your kubeconfig points at the cluster you mean to claim.

spec.kubehz.apiUrl is required / HTTPS errors? Set apiUrl whenever hosting is hosted or shared, or access != none, and use an https:// URL: lo rejects plain HTTP.

kubehz Platform ​

spec.kubehz ​

The hosted control-plane shape ​

Upgrades and maintenance windows ​

Spaces ​

Limits, and whose problem each one is ​

Bootstrap addons on hosted clusters ​

Nodes you bring (static pools) ​

lo kubehz node join ​

lo kubehz node status ​

lo kubehz node remove ​

When a join is refused ​

Registration ​

Claiming ​

Choosing an agent ​

What the live agent gives you ​

Permissions the live agent holds ​

The heartbeat agent ​

Status and deregistration ​

KUBEHZ_TOKEN: claim at provision time ​

Agent keys ​

Agent tools ​

The environment ​

Commands ​

What the api enforces ​

As MCP tools ​

Assessment and handover ​

Receiving an ejected control plane ​

Troubleshooting ​

kubehz Platform

`spec.kubehz`

The hosted control-plane shape

Upgrades and maintenance windows

Spaces

Limits, and whose problem each one is

Bootstrap addons on hosted clusters

Nodes you bring (static pools)

`lo kubehz node join`

`lo kubehz node status`

`lo kubehz node remove`

When a join is refused

Registration

Claiming

Choosing an agent

What the live agent gives you

Permissions the live agent holds

The heartbeat agent

Status and deregistration

`KUBEHZ_TOKEN`: claim at provision time

Agent keys

Agent tools

The environment

Commands

What the api enforces

As MCP tools

Assessment and handover

Receiving an ejected control plane

Troubleshooting