The rook-ceph addon narrows the RBAC of the Ceph CSI controller plugin and drops a cluster-wide COSI binding from its render (#267). The bootstrap gets a dry run and a per-entry filter: --dry-run, --render-dir, --no-diff and --only on lo bootstrap and lo provision --bootstrap (#268). The Lo (kind) driver gets two fixes: an unset spec.kubernetes.version now uses the default node image of kind (#269), and the default route of a kind node stays on the project network (#270). lo audit reads a dated Kubernetes support table (#258). Every change is in Go and in bash. Read Behaviour changes before you upgrade: a spec file must now hold one YAML document.

Behaviour changes

• One YAML document per spec file (#269). lo now refuses a cluster.lok8s.yaml or deploy.lok8s.yaml with more than one YAML document. This includes a trailing --- with nothing after it, empty documents before the spec, and text after a ... line. Before, the Go implementation read only the first document and the bash implementation read every document. So one file gave two different specs. The error names the lines to remove:

  [error] clusters/example.dev/cluster.lok8s.yaml holds more than one YAML document. Remove line 8 (a '---' line after the spec).

  • One --- line before the spec and one ... line after it stay valid.
  • These commands refuse the file before they change something: lo provision, lo up, lo bootstrap, lo destroy, lo down, lo recover, lo build, lo deploy (also for the cluster spec that spec.clusterRef names), lo addons and lo lint.
  • lo status shows the error in its Cluster section. lo audit reports the file as one unknown finding. lo use and lo doctor show the driver of the domain as ?.
• The bash implementation needs newer tools. This applies only to a command that lok8s.yaml routes to bash (spec.implementation). The Go binary reads YAML and checks charts itself, so it needs neither tool.
  • yq v4.47.1 or later (#269). The bash readers of spec.kubernetes.version (the Lo driver, lo lint and lo audit) use the yq flag --yaml-fix-merge-anchor-to-spec. An older yq stops the read with an error that says so.
  • helm v3.3 or a later 3.x on PATH for the chart dependency check of the bootstrap dry run (#268). Without it, a dry-run entry that renders a local chart with dependencies is inconclusive, and the run exits 2. A real bootstrap does not run this check.
• A bootstrap entry cannot set the variables of lo (#268). The env: of a spec.bootstrap entry may not set LOK8S_BOOTSTRAP_*, BOOTSTRAP_*, bootstrap_*, PATH_SECRETS, KUBECONFIG, KUSTOMIZE_PLUGIN_HOME or HELM_PLUGINS. Such an entry is an error on a real run and on a dry run, and lo names the variable. An entry that the run skips stays skipped. Both implementations give the env: of an entry only to the render (kustomize and envsubst), never to lo itself.

Security

• The CSI controller plugin ClusterRole is least privilege (#267). This is the controller-side follow-up to the node plugin fix of v0.9.2 (#264). The controller plugin Deployment (ceph-csi and the external-provisioner, -attacher, -resizer and -snapshotter sidecars) got the full upstream ClusterRole of the ceph-csi-drivers chart: 29 rules for RBD and NVMe-oF, 27 for CephFS, 16 for NFS. Four rules were wider than any caller:

  • secrets get/list/watch in all namespaces. The sidecars only get the secret that the StorageClass names. Nothing lists or watches.
  • configmaps get, serviceaccounts get and serviceaccounts/token create. Only a KMS uses them. serviceaccounts/token create alone mints a token for any ServiceAccount in the cluster.

  Before, a takeover of the controller plugin pod gave read access to every Secret in the cluster and a token for any ServiceAccount. The addon now removes the four rules from the RBD, CephFS and NVMe-oF roles, and the one cluster-wide secrets get from the NFS role. tokenreviews and subjectaccessreviews create stay. They are not escalation paths, and the snapshot-metadata sidecar needs them. Every PV, PVC, snapshot and attachment rule stays.

  • Secret access comes back namespaced. A new Role in the rook-ceph namespace grants get on two Rook secrets by name: rook-csi-rbd-provisioner and rook-csi-cephfs-provisioner. The StorageClass parameters provisioner and controller-expand and the VolumeSnapshotClass parameter snapshotter name these secrets. A RoleBinding gives the Role to the controller plugin ServiceAccount of each driver, under the pinned and the chart default name.
  • The removal uses the guarded JSON patches of #264. A test on the complete rule comes before each remove. A chart bump that reorders, merges or widens a rule fails the render.
  • Go tests compare the patched roles with a committed fixture of the chart. An online test renders the pinned chart again in CI and fails when the fixture is stale.
  • rook-ceph-csi-kms now also binds its rules to the RBD controller plugin ServiceAccount. A KMS runs on both sides: the controller creates the key, the node unlocks it. The warning of that addon now covers the controller pod too.
  • Verify on a cluster:

    $ SA=system:serviceaccount:rook-ceph:rook-ceph-rbd-csi-ceph-com-ctrlplugin-sa
    $ kubectl auth can-i list secrets --as="${SA}" --all-namespaces
    no
    $ kubectl auth can-i create serviceaccounts/token --as="${SA}"
    no
    $ kubectl auth can-i get secrets/rook-csi-rbd-provisioner -n rook-ceph --as="${SA}"
    yes

    Both no lines read yes on a cluster that bootstraps rook-ceph-csi-kms.

• The COSI ClusterRoleBinding leaves the render (#267). The rook-ceph chart ships ClusterRoleBinding/objectstorage-provisioner-role-binding. It gives the objectstorage-provisioner ServiceAccount secrets get, create, update and delete in every namespace. The addon deploys no CephCOSIDriver, and no pod runs as that ServiceAccount. The addon now drops the binding from its render ($patch: delete). The ClusterRole and the ServiceAccount stay.

  • A new cluster never gets the binding.
  • The bootstrap applies without prune. An existing cluster keeps the binding until you delete it once (see Upgrade).
  • A cluster that runs the COSI driver adds the binding again from its own target. The snippet is in the addons guide.

New

• Bootstrap dry run and --only (#268). A bootstrap applies every spec.bootstrap entry again with kubectl apply --server-side --force-conflicts. That overwrites live drift in the addons it touches, for example a live-patched config or a replica count that an autoscaler owns. New flags on lo bootstrap and lo provision --bootstrap show the change first and apply one entry at a time:

  lo provision --domain my-cluster.example.com --bootstrap --dry-run --render-dir /tmp/bootstrap
  lo provision --domain my-cluster.example.com --bootstrap --dry-run --no-diff
  lo provision --domain my-cluster.example.com --bootstrap --only cert-manager
  lo bootstrap --only cert-manager,envoy-gateway --dry-run

  • --dry-run renders every entry as a real run does and applies nothing. It uses the same project, value stack, envsubst, khelm and order, and the same HOME, credentials and CA. It renders each entry two times. Then it asks the API server for a read-only kubectl diff --server-side --force-conflicts of each entry.
  • The dry run prints one line per entry and a summary. The status is unchanged, differs changed=N new=M, inconclusive (…), rendered, skipped or error. restore.d gets its own line first.
  • Marks after the status: (non-deterministic: N) for objects that change on every render (for example the CA and Hubble TLS Secrets of Cilium), (non-repeatable) for new secret material that the store does not hold, and (partial: …) for a part that the dry run did not run or read.
  • Exit codes, the same for both commands and both implementations: 0 when nothing would change, 1 only when a completed diff found a change, 2 for all other failures. An inconclusive entry, a restore.d file that does not decrypt and a bad flag exit 2.
  • --render-dir <dir> writes NN-<entry>.yaml and NN-<entry>.diff for each entry. The directory holds rendered Secrets in plaintext. lo creates it with mode 0700. It changes an existing directory that the group or others can read to 0700. lo refuses a path with a symlink in it.
  • --no-diff renders only and never contacts the cluster.
  • --ignore-nondeterministic diffs an entry without the objects that change on every render. lo still lists them.
  • --only <entry>[,…] applies (or dry-runs) only the named entries, by entry name or addon name. It does not expand dependsOn. Only the edges among the named entries order the run. lo warns when an omitted entry is a prerequisite of a named one. An unknown or blank name is an error that lists the valid names and applies nothing.
  • --only leaves restore.d out. --with-restore adds it again.
  • A dry run skips the infrastructure gate and the steps after the bootstrap in lo provision (the cluster registration, the inventory, gitops). It ignores --force.

  What the dry run protects against. The dry run prevents accidental side effects of legitimate addon code and configuration. In a dry run, lo and its built-in generators never apply anything and never write new state:

  • the secrets plugin keeps new or changed values in a store of this run only,
  • the cert generator never writes CAROOT,
  • the Lo driver never writes .registries.json,
  • lo never ejects a framework addon.

  lo refuses executable addon code unless you pass --allow-generators. This includes an exec plugin other than the Secret and ChartRenderer of lok8s, a KRM exec or container function, a remote base, a helmCharts chart that is not in chartHome, a build of chart dependencies, a helm plugin and a bash: secrets generator. --allow-generators runs this code as trusted, with the side effects of a real run.

  What the dry run does not protect against. The dry run is not a security boundary against a malicious addon. Such an addon can still run code during a dry run, and the real bootstrap runs the same addon with full credentials. The helm cache can get new charts: khelm downloads a remote chart into the helm cache of your HOME, as a real run does. The full table is in Deploying to Clusters.

  Chart dependency check. khelm can build the dependencies of a local chart into the chart directory. A dry run does not render such a chart. It checks, as Helm does, that the chart directory holds every dependency. If not, the entry is inconclusive (chart dependencies not vendored). The Go implementation uses the Helm v3.21.2 library for the check. The bash implementation runs helm dependency list and helm lint, both read-only.

Fixes

• An unset spec.kubernetes.version uses the default node image of kind (#269). Before, a Lo spec without the version rendered image: "kindest/node:null" for every node. lo provision then failed at kind create with manifest for kindest/node:null not found. At that point it had already created the project docker network and the registry containers. Now the kind config names no image for an unset version, and kind uses the default node image of its release.
  • One reading rule serves the Lo driver, lo lint and lo audit, in both implementations.
  • Unset: the key is missing or null (~), or the value is "" or the string "null".
  • Valid: any OCI image tag, 1 to 128 letters, digits, _, . and -, with no . or - first. An optional @sha256:<64 hex> digest can follow. v1.31.4, latest and the tag of a kind build node-image image all work.
  • Refused: a value that is not a tag, for example kindest/node:v1.31.4. A shape that can hold a version but that the reader cannot read is an error, never unset. Examples are a merge cycle on the path, a version that is a map or a list, and a value with a control character. The error names the next action.
  • The reader resolves YAML aliases and merge keys as yq does. It reads only the path to the value, so an alias bomb in another part of the file costs nothing.
  • In the bash implementation, each version read has a time limit of 10 s. It also has a memory limit of 256 MiB (ulimit -d) where the operating system enforces it. The time limit uses no GNU timeout, so it holds on macOS too.
  • lo provision checks the version before it creates a VM, a docker network or a container. lo lint reports the same problems for a Lo cluster spec.
  • lo audit reads the version with the same rule. A Lo cluster without a version is a pass.
  • LOK8S_SPEC_KUBERNETES_VERSION is now empty for the string "null".
• The default route of a kind node stays on the project network (#270). With shared registries (spec.registries.shared.enabled), a kind node is on two docker networks: the project network and lok8s-registries. Docker takes the default route from the network with the highest gateway priority. Between equal priorities, it takes the network whose name sorts first. Before, both networks had priority 0. So a project network whose name sorts after lok8s-registries lost the default route. Traffic from the node to a service then left with the wrong source address. rp_filter dropped part of the replies, and the Cilium agent could not reach the API server at 10.96.0.1.
  • On Docker 28 or later (client and daemon), lo up connects a node to the registry network with docker network connect --gw-priority -1. The project network keeps priority 0 and the default route, also after a node restart.
  • On an older Docker, lo up connects without the flag. Then it moves the default route in the node:

    ip route replace default via <project gateway> dev <project interface>

    This move does not survive a node restart (see Known limitations).
  • lo up also checks each node of the cluster that is already on the registry network, also when shared.enabled is false or not set. It moves a wrong route and prints one [warn] line for each node.
  • To check a node, run docker exec <node> ip route show default. The gateway must be the gateway of the project network.
• lo audit reads a dated Kubernetes support table (#258). The k8s-version-support check read a static list of supported minors (1.34 1.35 1.36). A cluster on Kubernetes 1.37 got a "may be stale" warning, and 1.34 would keep its pass after its end of life. Now both implementations hold one row per minor with the date its upstream patch support ends:

  1.37:2027-10-28 1.36:2027-06-28 1.35:2027-02-28 1.34:2026-10-27 1.33:2026-06-28

  • A minor is supported while its date is in the future. So 1.34 changes from pass to fail on 2026-10-27 with no code change. A dev cluster gets a warn.
  • When every row is past its date, the check warns that the table is out of date.
  • A new minor still needs a new row in both tables. A test fails when the Go table and the bash table differ.
  • _AUDIT_TODAY (yyyy-mm-dd) pins the day of the check. The tests use it.
• lo never writes through a symlink in a staged addon (#268). An addon that stacks values renders from a copy. lo now writes the files it changes in the copy (chart.yaml, values.inline.yaml, values.merged.yaml) as new private files. The copy keeps its symlinks, and a file that a symlink points to stays unchanged. This applies to a real run too.

Changes

• The core lo binary now links the Helm v3.21.2 library for the chart dependency check of the dry run. The binary grows by about 0.4 MB.
• Dependencies: bubbletea v2.1.0, bubbles v2.2.1, lipgloss v2.0.6 and the MCP go-sdk v1.8.0. CI: actions/checkout v7.0.1, actions/setup-node v7.1.0, docker/login-action v4.6.0, docker/build-push-action v7.4.0, golangci-lint-action v9.3.0.

Upgrade

• Check your spec files first. Run lo lint with v0.10.0. It names each spec file with more than one YAML document and the lines to remove. Delete those lines. Until then, every command refuses that domain.
• The bash implementation. A project that routes commands to bash installs yq v4.47.1 or later. For the bootstrap dry run it also needs helm v3.3 or a later 3.x. A project with its own bash tree in .lok8s/ (lo assets eject bash) runs lo assets update bash to get the bash twin of each change.
• An ejected copy of the rook-ceph addons. lo ejects an addon into .lok8s/addons/<name>/ on first use and never overwrites it. A local copy wins over the embedded one. lo assets diff addons/rook-ceph shows the changed files. lo assets update addons/rook-ceph applies them when your copy is untouched. Do the same for addons/rook-ceph-csi-kms when the cluster bootstraps it.
• StorageClasses with their own secret. Before you apply #267, check each Ceph StorageClass. A StorageClass whose controller-side secret has another name, or lives outside the rook-ceph namespace, loses every controller call that needs the secret. These calls are provision, delete, attach, detach, expand and snapshot. The sidecar gets a 403 and a new PVC stays Pending. Mounted volumes keep working. Add a Role and a RoleBinding for that secret from your own target. The snippet is in the addons guide. The same applies to a VolumeGroupSnapshotClass or VolumeReplicationClass that names another secret.
• Apply #267 to an existing cluster: dry run first. The narrowed roles and the new Role arrive with the next bootstrap apply.
  1. Preview the bootstrap. Expect differs for rook-ceph and exit code 1:

     lo provision --domain <domain> --bootstrap --dry-run --render-dir /tmp/bootstrap

     Read NN-rook-ceph.diff. It shows the narrowed controller plugin ClusterRoles and the new Role and RoleBinding. The render directory holds Secrets in plaintext, so delete it when you are done.
  2. When other entries differ and you do not want to change them now, apply only the rook-ceph entries. Add rook-ceph-csi-kms to the list when the cluster bootstraps it:

     lo provision --domain <domain> --bootstrap --only rook-ceph

     Otherwise run lo provision --domain <domain> --bootstrap (or lo bootstrap --domain <domain>).
  3. Until the RoleBinding is in the cluster, a provisioner call can get a 403 for a few seconds. The sidecars try again. No pod restarts.
  4. A cluster that does not run the COSI driver deletes the old binding once, by hand. Then it checks the result:

     $ kubectl delete clusterrolebinding objectstorage-provisioner-role-binding
     $ kubectl auth can-i get secrets -n kube-system \
         --as=system:serviceaccount:rook-ceph:objectstorage-provisioner
     no

  5. Run the three kubectl auth can-i lines from Security. Then create a test PVC for each Ceph StorageClass. It must bind, attach, expand, take a snapshot and delete.
• Shared registries (#270). Run lo up once for each cluster whose nodes are on lok8s-registries. It moves a wrong default route. On Docker 28 or later, a node that an older lok8s attached gets only the temporary route move (see Known limitations).

Known limitations

• The route move of #270 is temporary in two cases. On a Docker older than 28, the move in the node does not survive a node restart, for example a host reboot. On Docker 28 or later, lo up does not change the priority of an endpoint that exists. A node that an older lok8s attached at priority 0 gets only the temporary move too (#273). After a node restart, run lo up again. For a permanent fix on Docker 28 or later, detach the node (docker network disconnect lok8s-registries <node>) and run lo up, or recreate the cluster.
• The route repair supports only the Docker engine. lok8s does not support a docker command from Podman for it.
• KRM functions in a dry run. Without --allow-generators, kustomize skips a KRM exec function with no error. Its objects are missing from the preview, and the entry is (partial: KRM function skipped). On macOS and BSD, the bash implementation cannot read the KRM hint and prints no partial mark (#276).
• The chart dependency check can differ between the implementations. A helm v3 release other than v3.21.2 can give a different dependency status. Then one implementation is inconclusive where the other renders.

Follow-up

• #265: feature-specific, namespaced grants for rook-ceph-csi-kms instead of the upstream rule set.
• #272: the rest of the unset-version work. Remove the unused --kubernetes flag, stop two readers that send the string "null", and move the version reader to a neutral package.
• #273: on Docker 28 or later, move a node that an older lok8s attached to gateway priority -1.
• #275: a bats provision test reaches the real docker binary.
• #276: small follow-ups from the review of the dry run: portable realpath in the bash KRM hint, clearer status labels and messages.
• #277: one helper for the bounded yq reads of the bash implementation, and cheaper merge-key reads.

Commits

• 4d52e09, da1912d, e46cca5, f36f5d7, ed75603, 7d8156e, e999a98, 76b936f, 703fb8e chore(deps), ci(deps): nine dependency bumps (#209 to #216, #235)
• 96f4da1 fix(audit): read a dated Kubernetes support table, in both implementations (#258)
• bfcbe50 fix(rook-ceph): least-privilege ClusterRole for the CSI controller plugin (#267)
• f86ccfe fix(lo): keep the default route of a kind node on the project network (#270)
• 2bd98b9 feat(bootstrap): --dry-run, --render-dir, --no-diff, --only (#268)
• 10b8d6a fix(lo): an unset spec.kubernetes.version uses kind's default node image (#269)