One security fix in the rook-ceph addon, in both trees (#264). The Ceph CSI node plugin loses five ClusterRole rules that only volume encryption with an external KMS uses. Before, root on any node could read every Secret in the cluster and mint a token for any ServiceAccount. A new opt-in addon, rook-ceph-csi-kms, gives the five rules back to the clusters that need them. No new command, no new field.

Security

• The CSI node plugin ClusterRole is least privilege. The node plugin DaemonSet tolerates every taint, so it runs on every node, control planes included. Its ClusterRole is what an attacker holds after a takeover of any one node. The upstream ceph-csi-drivers chart grants that role ten rules (RBD and NVMe-oF) or eight (CephFS). Five of them serve only volume encryption with an external KMS: secrets get/list/watch, configmaps get, serviceaccounts get, serviceaccounts/token create and tokenreviews create. Before, every node held them. secrets get across all namespaces reads every Secret, and serviceaccounts/token create alone mints a token for any ServiceAccount in the cluster. A plain, unencrypted RBD or CephFS mount uses none of the five: kubelet passes the Ceph credentials to the plugin inside the CSI request. The addon now removes the five rules from the RBD, CephFS and NVMe-oF node plugin ClusterRoles. The remaining rules (nodes get, persistentvolumes and volumeattachments get/list, persistentvolumeclaims get, events) stay. The NFS role carries nodes get only and is unchanged.
  • The removal is a guarded JSON patch in the addon's kustomization.yaml. A test on the complete rule precedes each remove. A chart bump that reorders, merges or widens a rule fails the render instead of removing the wrong rule.
  • Two Go tests guard the patch: one applies the real patches over a committed fixture of the chart's node plugin ClusterRoles at the pinned version, one re-renders the pinned chart in CI and fails when the fixture is stale.
  • RBAC takes effect at once. The plugin pods need no restart.
  • Verify on a cluster:

    $ SA=system:serviceaccount:rook-ceph:rook-ceph-rbd-csi-ceph-com-nodeplugin-sa
    $ kubectl auth can-i create serviceaccounts/token --as="${SA}"
    no
    $ kubectl auth can-i get secrets --as="${SA}" --all-namespaces
    no
    $ kubectl auth can-i list volumeattachments --as="${SA}"
    yes

Changes

• New opt-in addon rook-ceph-csi-kms. Three ceph-csi features read the removed objects through the Kubernetes API: encrypted volumes with a KMS (a StorageClass with encrypted: "true" and an encryptionKMSID), mounter: rbd-nbd (the volume healer re-reads each node-stage Secret after a plugin restart), and usage metrics for volumeMode: Block volumes (see Known limitation). For those clusters bootstrap the addon after rook-ceph:

  spec:
    bootstrap:
      - rook-ceph
      - rook-ceph-csi-kms:
          dependsOn: [rook-ceph]

  It binds exactly the five removed rules to the RBD node plugin ServiceAccount, so the effective permissions equal the upstream chart's. Warning: this restores the every-node escalation path. With it, root on any node that runs the plugin can read every Secret and mint a token for any ServiceAccount again. Enable it only for one of the three features. The binding names the RBD ServiceAccount only. A cluster that runs CephFS with encryption adds that ServiceAccount with a patch from its own target (the example is in the addons guide). lo addons --detail shows the config hint.
• The CephFS and NVMe-oF driver names are pinned. The addon's csi-drivers.yaml now sets rook-ceph.cephfs.csi.ceph.com (before: the chart default cephfs.csi.ceph.com) and rook-ceph.nvmeof.csi.ceph.com (before: nvmeof.csi.ceph.com). For CephFS the pinned name is the one Rook writes into its own StorageClass and VolumeSnapshotClass. Both drivers stay disabled by default, so the render of the unchanged addon has the same 96 documents before and after, and exactly one object changes: the RBD node plugin ClusterRole, 10 rules to 5. The least privilege patches target the pinned and the chart default role name of each driver. See Upgrade for a copy of the addon that runs these drivers.

Known limitation

• Usage metrics of block volumes. NodeGetVolumeStats on a volumeMode: Block RBD volume reads the controller-publish Secret to report the used bytes. With the narrowed role that read fails on the volume stats, not on the mount. Kubelet logs the error and the volume keeps working. A cluster that needs these metrics bootstraps rook-ceph-csi-kms.

Upgrade

• An existing project holds an ejected copy of the addon. lo ejects an addon into .lok8s/addons/<name>/ on first use and never overwrites it, and a local copy wins over the embedded one. So a project that bootstrapped rook-ceph with an earlier lo keeps the broad role until you update the copy: lo assets diff addons/rook-ceph shows the changed files, lo assets update addons/rook-ceph applies them when your copy is untouched. Then re-run the bootstrap (lo provision --bootstrap or lo bootstrap --domain <domain>) and run the three kubectl auth can-i lines above. A new project gets the narrowed role on the first bootstrap.
• A cluster with KMS encryption, mounter: rbd-nbd or block-volume metrics adds rook-ceph-csi-kms to spec.bootstrap in the same change. Without it the next encrypted mount, healer run or block-volume stats call fails with a permission error.
• A fork of the addon, or a custom target with its own csi-drivers.yaml, that runs CephFS or NVMe-oF. Inline values: in spec.bootstrap reach chart.yaml (the Rook operator chart), not the driver generator, so only such a copy runs these drivers, and it keeps its own driver names when it rebases. The driver name is the CSI identity: the CSIDriver object, the plugin workloads, the kubelet registration and spec.csi.driver of every PV, which is immutable. Two choices when you rebase:
  • Keep the default name: the patch targets it too, so the role is narrowed and nothing else changes.
  • Adopt the pinned name: the CSI identity changes. Point your StorageClasses and VolumeSnapshotClasses at the new name, migrate the PVs (new PVC + copy), then delete the old Driver and its classes. Do not delete a Driver while a PV still uses it. The patch acts on the render only. A live old Driver that left the render keeps its broad node plugin ClusterRole, and the bootstrap does not prune. After the migration delete that Driver and its RBAC, or narrow the old role by hand with the same patch. The steps are in the upgrade note of the addons guide.
• A project with its own bash tree in .lok8s/ (lo assets eject bash) runs lo assets update bash to get the bash twin of the change (the config hint of the new addon in the bash catalogue).

Follow-up

• #265: feature-specific, namespaced grants for rook-ceph-csi-kms instead of the upstream rule set.
• The controller-side roles (the CSI provisioner, the Rook operator, the ceph-csi operator) are unchanged in this release.

Commits

• a79a82a fix(rook-ceph): least-privilege ClusterRole for the CSI node plugin (#264)