This release narrows the RBAC of two addons: system-upgrade-controller and redis-operator. lo provision now stops on a busy kubehz platform instead of a silent skip. The kubehz heartbeat works again on clusters with more than about 18 nodes. Read the behaviour changes and the upgrade steps before you update.

Behaviour changes

• The system-upgrade-controller addon no longer binds the controller to cluster-admin. The Rancher chart binds the ServiceAccount cattle-system/system-upgrade-controller to cluster-admin. The addon now deletes that ClusterRoleBinding from its render and binds four roles instead, each named with the prefix lok8s-: a ClusterRole and a Role (cattle-system) with the calls the controller makes, a Role in default for the events on nodes, and a ClusterRole for the cordon and drain steps of a Plan whose Job runs as the controller's ServiceAccount. The ServiceAccount can no longer change RBAC, impersonate, request tokens for other ServiceAccounts, read Secrets outside cattle-system, or write outside Nodes, pods (delete and evict), Jobs, Leases, events, the Plan status and CRDs. It can still read every Secret in cattle-system, start pods there through Jobs (privileged ones too, unless an admission policy stops them), change every Node, and evict or delete any pod. Root on a node stays in reach through such a Job. A Plan's upgrade container that runs as system-upgrade-controller and calls the Kubernetes API for more than cordon and drain now gets forbidden. Give that Plan a ServiceAccount of its own with the grants it needs, or bootstrap the new opt-in addon (below). The rules, what the ServiceAccount can still do, and which Plan feature needs which drain rule: system-upgrade-controller.

• The addon has no namespace: transformer any more. Its objects name their namespace themselves (the chart hardcodes cattle-system); the Role for the node events lives in default. A modified copy of the addon that relied on the transformer must set metadata.namespace on its own objects.

• The system-upgrade-controller addon pins its images by digest. The controller is rancher/system-upgrade-controller:v0.20.1@sha256:aaf4dbf6…, the kubectl image of the cordon and drain containers is rancher/kuberlr-kubectl:v8.1.1@sha256:74ab36f8…. The tags are the ones the chart pinned before, so the Deployment rolls once and runs the same code. A private mirror set through global.cattle.systemDefaultRegistry must keep the digests.

• The redis-operator addon no longer aggregates into admin, and the operator runs with narrowed RBAC. The chart's ClusterRole redis-operator grants create and delete on namespaces, Secrets, ConfigMaps, pods and pods/exec in every namespace, and its label rbac.authorization.k8s.io/aggregate-to-admin copies those rules into the built-in ClusterRole admin: every subject with a RoleBinding to admin could delete its own namespace. The addon now deletes the chart's ClusterRole and ClusterRoleBinding from its render and binds three roles, each named with the prefix lok8s-: a ClusterRole with the calls the operator makes for the Redis CRs (Secrets get only, no namespaces, no ConfigMaps), a ClusterRole with pods/exec create that only a RedisCluster needs, and a Role in redis-operator for leader election. No object of the addon carries an aggregate-to-* label. The operator can still create StatefulSets, read Secrets by name, patch pods and delete PVCs in every namespace. The rules, their sources and what the token can still do: redis-operator.

• pods/exec stays enabled by default in the redis-operator addon. The binding lok8s-redis-operator-exec keeps a RedisCluster working after the update, so the exec grant is not reduced until you remove it: the token can still run any command in any pod. Only a RedisCluster needs exec (creating the cluster, scaling, rebalancing); Redis, RedisReplication and RedisSentinel do not. A cluster without a RedisCluster can remove it: delete the binding document from the ejected rbac.yaml, apply, then run kubectl delete clusterrolebinding lok8s-redis-operator-exec (the bootstrap does not prune). The old ClusterRole/redis-operator grants exec too, until you delete it (see Upgrade). A RedisCluster added later needs the binding back first. The steps and the checks: pods/exec. A by-name switch is #316; a lo doctor warning is #315.

• A busy kubehz platform stops lo provision instead of a silent skip (#312). When the kubehz api answers a registration with 503 or 429 and Retry-After (for example REGISTRATION_UNAVAILABLE while its rate limit store is down), lo waits and tries again: at most 3 tries, at most 60 seconds for each wait. Until now lo dropped the answer, tried the other register modes, and continued without the registration. Now, if the platform still refuses after the third try, lo provision stops with status 1. What stops: the provision, after the infrastructure is up and before the bootstrap. What is left: a running cluster without its spec.bootstrap addons and without the platform registration. To recover, run lo provision --bootstrap when the platform is available again. It registers the cluster and then applies the bootstrap; it does not provision the infrastructure again. lo kubehz register stops the same way and names itself as the command to run again. A refusal that lo does not try again (a 503 without Retry-After, for example) keeps the old path, and its warning now names the status and the code. Both implementations behave the same. Details: When the platform is busy.

Fixes

• kubehz heartbeat agent: the node list is one call (#317). The heartbeat CronJob made three kubectl get node calls per node. At about 18 nodes a run passed the job's 120 s deadline, and the job ended before the heartbeat went out: the cluster sent no more heartbeats, and nothing reported an error. The agent now reads all nodes with one kubectl get --raw /api/v1/nodes call under timeout 30, so the number of calls does not grow with the cluster. The node cap was 20 and is now 500, the limit of the platform's heartbeat API. A larger cluster sends its first 500 nodes, and the agent writes the total to its log. The CronJob's limits change: CPU from 200m to 1000m (at 200m each kubectl start was throttled) and memory from 128Mi to 256Mi (the pod parses the node list). The requests stay at 50m and 64Mi. Details: the heartbeat agent.

New

• Opt-in addon system-upgrade-controller-cluster-admin (not recommended). It applies the chart's ClusterRoleBinding to cluster-admin again, unchanged:

  spec:
    bootstrap:
      - system-upgrade-controller
      - system-upgrade-controller-cluster-admin:
          dependsOn: [system-upgrade-controller]

  With it, the controller and every Plan Job that runs as its ServiceAccount can do anything in the cluster again. To opt out, remove the entry and also run kubectl delete clusterrolebinding system-upgrade-controller: the bootstrap applies without prune, so the binding stays otherwise.

• Opt-in addon redis-operator-chart-rbac (not recommended). It binds the rules of the chart's ClusterRole to the operator again, under the name lok8s-redis-operator-chart-rbac and without the label aggregate-to-admin:

  spec:
    bootstrap:
      - redis-operator
      - redis-operator-chart-rbac:
          dependsOn: [redis-operator]

  To opt out, remove the entry and also run kubectl delete clusterrolebinding lok8s-redis-operator-chart-rbac and kubectl delete clusterrole lok8s-redis-operator-chart-rbac.

Upgrade

• First, update the pins. Pin lo, the kustomize-secret plugin and the .lok8s env ref to v0.12.0, then run b update. Do this before the steps below: lo assets diff and lo assets update compare against the files inside the lo you run. From a release older than v0.11.1, follow the notes of each release in between first.
• Check what the .lok8s sync changed. With b's default replace strategy, a non-interactive b update of an env ref that syncs .lok8s/ writes the new addon files over .lok8s/addons/<name>/, also over a copy you changed. Run git diff .lok8s/addons/ before the bootstrap and put your own changes back. To keep local changes during the sync, use b update --strategy=merge (a three-way merge) instead.
• Both addons: update both copies, apply the bootstrap once, then do the deletes. The steps below are per addon, and each one says to apply the bootstrap. When you update both addons, one apply covers both.
• system-upgrade-controller: update the ejected copy. Without that sync, lo ejects an addon into .lok8s/addons/<name>/ on first use and never overwrites it, so a project that bootstrapped system-upgrade-controller before keeps the old version. lo assets diff addons/system-upgrade-controller shows the changed files. lo assets update addons/system-upgrade-controller applies them when your copy is untouched. A modified copy takes rbac.yaml, the patches: block, the kustomization without its namespace: line and the image pins by hand.
• Then delete the old binding once. The bootstrap applies without prune, so a cluster that ran the addon before keeps ClusterRoleBinding/system-upgrade-controller and with it cluster-admin. Apply the bootstrap first (it adds the new bindings), then run kubectl delete clusterrolebinding system-upgrade-controller. A cluster that opts back in skips this step: the opt-in addon owns that binding.
• Then prove it. Restart the controller (kubectl -n cattle-system rollout restart deployment/system-upgrade-controller), run one Plan that does what your Plans do (cordon, drain, Secrets), and look for forbidden in the controller log. Until now cluster-admin hid any call the new roles miss. The kubectl auth can-i checks are in the upgrade note.
• redis-operator: update the ejected copy.lo assets diff addons/redis-operator shows the changed files; lo assets update addons/redis-operator applies them when your copy is untouched. A copy without a .lo-origin marker needs --force once the diff shows only this release's changes. A modified copy takes rbac.yaml and the patches: block by hand.
• redis-operator: then delete the old ClusterRole and binding once. The bootstrap applies without prune, so a cluster that ran the addon before keeps ClusterRole/redis-operator (and with it the rules in admin) and ClusterRoleBinding/redis-operator. Apply the bootstrap first (it adds the new bindings), then run kubectl delete clusterrolebinding redis-operator and kubectl delete clusterrole redis-operator. admin loses the copied rules asynchronously: the delete queues the clusterrole-aggregation controller. Poll admin until no copied rule is left, and check the other ClusterRoles that aggregate into it (the guide has the commands). Restart the operator (kubectl -n redis-operator rollout restart deployment/redis-operator) and look for forbidden in its log. The checks, and how to drop the pods/exec binding on a cluster without a RedisCluster, are in the upgrade note.
• kubehz heartbeat: run lo kubehz deploy once on each cluster where lo kubehz deploy installed the agent with agent: cronjob (the default) (lo use <domain> first). It applies the new CronJob: the one-call node list and the new limits. Until then the cluster runs the old heartbeat, which can pass its deadline at about 18 nodes. With agent: operator the CronJob stops before the node list, so the old one cannot hit this deadline. A deploy there does no harm.