The fixes from an in-place rebuild of a production cluster (kubehz.in.net, 2026-10-23/24). The secrets store no longer makes up a value that is already encrypted, every command uses the store of its own domain, the Hetzner provider gets the rebuild and the load balancer right, lo destroy refuses a cluster name that another domain uses, and the kubehz heartbeat agent can read the cluster again. The changes are in Go and in bash. Read Behaviour changes before you upgrade.
Behaviour changes
lo destroy -rwith the Hetzner provider needs the cluster name typed (#299). It lists what it will delete and asks for thecluster_nameon a terminal. Without a terminal, pass--confirm-cluster <name>.--forcedoes not skip it. A refusal exits 3. A script that ranlo destroy -r --forcewithout a terminal now stops until it passes--confirm-cluster. lo refuses a descriptor withoutcluster_name.- A domain without its own secrets store is refused when other domains have one (#300).
lo build(not with--no-secrets),lo bootstrap, the KubeOne addon render,lo k8sand thelo secretscommands that read the store stop and name the store they would otherwise use. A project with only the flat.secrets/store works as before. An explicitPATH_SECRETSdoes not override the refusal. Create the empty directoryclusters/<domain>/secrets/for a domain that needs no secrets. - The secrets plugin refuses an entry whose
.encexists and whose plaintext does not (#298). It used to make up a new random value. Runlo secrets decrypt --domain <d>first. To rotate a value, remove both files, or useupdate: true. lo doctorreports a missing project.bin/argsh(#301). For a project whose domain names a provider, or that runs commands in bash, it is a failure (exit 1).- The
lo bootstrap --dry-runreport names each entry by the name--onlyaccepts (cilium:, not01-cilium:) (#301). The files in--render-dirkeep the number prefix. - Hetzner cloud-VM Nodes keep the
lok8s.dev/rolelabel (#297). Bare-metal Nodes still drop it.
Fixes
- Secrets (#298, #300):
lo secrets decryptcompares each plaintext it keeps with its.encin memory and warns by name when they differ ("N equal, M differ" on stderr). It never prints a value, a hash or a length. A failed decrypt prints only the file name.- The plugin checks every entry a Secret uses before it writes any, so a refusal leaves no half-written store.
- The KubeOne addon render uses the domain's store and the project's
.kustomizeplugin home (it used the flat store before). - Provider credentials are read from their
.encin memory when the plaintext is missing.
- Hetzner provider (#297, #296):
- A number in the descriptor passes as written (a volume
"size": 100became--size null). lo looks up only the reference keys. - A rebuild puts the descriptor's
sshPublicKeyinto the user-data and checks that it matchessshPrivateKeyfirst: a rebuilt VM that lok8s did not create could not be reached before. - A
cloudInit.paththat does not exist is an error. - An adopted load balancer is reconciled to the descriptor: a target with a different
use-private-ipis removed and added again (lo retries the add), and lo updates a changed service.
- A number in the descriptor passes as written (a volume
lo destroy(#299): it deletes only the resources it listed and you confirmed, and only those that still carry the cluster label. A failed list stops it.lo init cluster(#299) refuses a domain whose first label another domain uses as its name orcluster_name.- kubehz heartbeat agent (#301, B423): its in-cluster
kubectlcalls no longer pass--request-timeout, which made kubectl skip the in-cluster config. The marker read and the assessment probes work again, bounded bytimeout 5. A probe that fails or times out now holds the assessment back for that tick instead of sending 0. lo bootstrap --only(#301) also accepts the key as written inspec.bootstrap(./targets/flux).lo recover(#301): its children no longer read the consent from stdin, so a piped answer reaches the prompt.
Follow-ups
#302 (lo recover scope), #303 (agent RBAC), #304 (restore.d error text), #305.