Skip to content

The fixes from an in-place rebuild of a production cluster (kubehz.in.net, 2026-10-23/24). The secrets store no longer makes up a value that is already encrypted, every command uses the store of its own domain, the Hetzner provider gets the rebuild and the load balancer right, lo destroy refuses a cluster name that another domain uses, and the kubehz heartbeat agent can read the cluster again. The changes are in Go and in bash. Read Behaviour changes before you upgrade.

Behaviour changes ​

  • lo destroy -r with the Hetzner provider needs the cluster name typed (#299). It lists what it will delete and asks for the cluster_name on a terminal. Without a terminal, pass --confirm-cluster <name>. --force does not skip it. A refusal exits 3. A script that ran lo destroy -r --force without a terminal now stops until it passes --confirm-cluster. lo refuses a descriptor without cluster_name.
  • A domain without its own secrets store is refused when other domains have one (#300). lo build (not with --no-secrets), lo bootstrap, the KubeOne addon render, lo k8s and the lo secrets commands that read the store stop and name the store they would otherwise use. A project with only the flat .secrets/ store works as before. An explicit PATH_SECRETS does not override the refusal. Create the empty directory clusters/<domain>/secrets/ for a domain that needs no secrets.
  • The secrets plugin refuses an entry whose .enc exists and whose plaintext does not (#298). It used to make up a new random value. Run lo secrets decrypt --domain <d> first. To rotate a value, remove both files, or use update: true.
  • lo doctor reports a missing project .bin/argsh (#301). For a project whose domain names a provider, or that runs commands in bash, it is a failure (exit 1).
  • The lo bootstrap --dry-run report names each entry by the name --only accepts (cilium:, not 01-cilium:) (#301). The files in --render-dir keep the number prefix.
  • Hetzner cloud-VM Nodes keep the lok8s.dev/role label (#297). Bare-metal Nodes still drop it.

Fixes ​

  • Secrets (#298, #300):
    • lo secrets decrypt compares each plaintext it keeps with its .enc in memory and warns by name when they differ ("N equal, M differ" on stderr). It never prints a value, a hash or a length. A failed decrypt prints only the file name.
    • The plugin checks every entry a Secret uses before it writes any, so a refusal leaves no half-written store.
    • The KubeOne addon render uses the domain's store and the project's .kustomize plugin home (it used the flat store before).
    • Provider credentials are read from their .enc in memory when the plaintext is missing.
  • Hetzner provider (#297, #296):
    • A number in the descriptor passes as written (a volume "size": 100 became --size null). lo looks up only the reference keys.
    • A rebuild puts the descriptor's sshPublicKey into the user-data and checks that it matches sshPrivateKey first: a rebuilt VM that lok8s did not create could not be reached before.
    • A cloudInit.path that does not exist is an error.
    • An adopted load balancer is reconciled to the descriptor: a target with a different use-private-ip is removed and added again (lo retries the add), and lo updates a changed service.
  • lo destroy (#299): it deletes only the resources it listed and you confirmed, and only those that still carry the cluster label. A failed list stops it.
  • lo init cluster (#299) refuses a domain whose first label another domain uses as its name or cluster_name.
  • kubehz heartbeat agent (#301, B423): its in-cluster kubectl calls no longer pass --request-timeout, which made kubectl skip the in-cluster config. The marker read and the assessment probes work again, bounded by timeout 5. A probe that fails or times out now holds the assessment back for that tick instead of sending 0.
  • lo bootstrap --only (#301) also accepts the key as written in spec.bootstrap (./targets/flux).
  • lo recover (#301): its children no longer read the consent from stdin, so a piped answer reaches the prompt.

Follow-ups ​

#302 (lo recover scope), #303 (agent RBAC), #304 (restore.d error text), #305.

Released under the MIT License.