Spec Reference

lok8s uses YAML spec files to define clusters and deployments. These specs double as Kubernetes CRDs when used with the operator.

One YAML document per spec file

A spec file (cluster.lok8s.yaml or deploy.lok8s.yaml) holds one YAML document. The Go readers of lo read only the first document of a file, and the bash readers (yq) read every document. Thus a file with more than one document gives two different specs, and each command that reads the spec refuses the file.

A spec file can have these lines:

one --- line before the spec;
one ... line (the document end marker) after the spec;
blank lines, comment lines and % directives.

lo refuses these layouts:

a --- line after the spec, also a --- at the end of the file with nothing after it;
empty documents before the spec (two or more --- lines before it);
text after a ... line.

The error names the spec file and the lines to remove. For example:

text

[error] clusters/example.dev/cluster.lok8s.yaml holds more than one YAML document. Remove line 8 (a '---' line after the spec).

These commands refuse such a file before they change something: lo provision, lo up, lo bootstrap, lo destroy, lo down, lo recover, lo build, lo deploy (also for the cluster spec that spec.clusterRef names), the lo kubehz commands that read the spec, lo addons and lo lint. lo status shows the error in its Cluster section. lo audit reports the file as one unknown finding and runs no other check on it. lo use and lo doctor show the driver of such a domain as ?.

cluster.lok8s.yaml

Defines a cluster domain. The kind field determines which driver contract handles provisioning.

Common fields (all cluster kinds)

Every cluster.lok8s.yaml shares this base structure regardless of kind:

yaml

apiVersion: cluster.lok8s.dev/v1beta1
kind: Lo | KubeOne | Capi | Kkp | Kubehz  # driver selection
metadata:
  name: my-cluster                        # unique cluster name
spec:
  kubernetes:
    version: "v1.31.12"                   # k8s version (required)
  cluster:
    domain: example.com                   # cluster FQDN (required)
    namespace: default                    # default namespace
  provider:                               # infrastructure provider (optional for Lo)
    name: hetzner                         # provider name (matches .lok8s/providers/<name>/)
    configRef: hetzner.json               # provider config file (relative to cluster dir)
    # OR inline:
    # config: { ... }                     # opaque provider-specific config
  bootstrap:                              # cluster-infra addons (ordered, applied by framework)
    - cilium                              # bare name: .lok8s/addons/<name>/
    - ./targets/networking                # ./path: relative to cluster dir
  kubehz:                                 # kubehz platform integration (optional)
    hosting: self | hosted | shared       # who runs the control plane
    access: none | registered | managed   # kubehz visibility
    agent: cronjob | operator             # which in-cluster agent owns the heartbeat
    apiUrl: https://api.kubehz.cloud      # required when hosting=hosted|shared or access!=none
    space:                                # hosting: shared only (kind: Kubehz)
      slug: acme                          # namespace name; defaults to the domain's first label
      name: Acme Prod                     # display name; defaults to the slug
      nodes: [worker-1]                   # join tickets minted for these names on provision
      limits:                             # the three numbers a space is (whole block optional)
        nodes: 2                          # node ceiling; omit to take the platform's
        namespaces: 1                     # namespace ceiling; omit to take the platform's
        objectCapKiB: 256                 # size cap per Secret/ConfigMap; omit to take the platform's
    upgrades:                             # platform-driven upgrade policy (optional)
      channel: none | patch | minor       # how far the platform may upgrade unasked (default patch)
      defer: window | immediate           # when an allowed upgrade may start
    maintenanceWindow:                    # when platform-driven work may run (optional)
      enabled: true
      daysOfWeek: [Sat]
      startTime: "02:00"                  # HH:MM in the window's timezone
      durationMinutes: 240
      timezone: Europe/Berlin
      exclusions:                         # absolute freezes — a date or a from/to range
        - "2026-12-20/2027-01-06"
  build:                                  # build output shape (optional)
    artifacts: single | split             # split = per-resource files under artifacts/
    encrypt:                              # Secret-encryption policy for split mode
      type: sops                          # default (only value); anything else = hard error
      on: change | always                 # change (default) skips re-encrypting unchanged Secrets
  gitops:                                 # GitOps consumption (optional; implies artifacts: split)
    provider: flux                        # reconciler the split output feeds
    age:                                  # sops age PUBLIC keys for Secret.*.sops.yaml
      - age1reconcilerkey…                #   the reconciler's runtime decryption key
      - age1breakglasskey…                #   operator break-glass

Field	Required	Default	Description
`apiVersion`	yes	—	Always `cluster.lok8s.dev/v1beta1`
`kind`	yes	—	Driver: `Lo`, `KubeOne`, `Capi`, `Kkp`, `Kubehz` (a kubehz Space — `hosting: shared`)
`metadata.name`	yes	—	Unique cluster name
`spec.kubernetes.version`	yes (no for `Lo`)	— (`Lo`: the default node image of the installed kind)	Kubernetes version. For `Lo` it is the tag of the kind node image `kindest/node:<tag>`: any image tag, with an optional `@sha256:<digest>`. See Kubernetes version (Lo)
`spec.cluster.domain`	yes	—	Cluster FQDN (folder name convention)
`spec.cluster.namespace`	no	`default`	Default namespace
`spec.provider.name`	no	—	Infrastructure provider
`spec.provider.configRef`	no	—	Provider config file path
`spec.provider.config`	no	—	Inline provider config (mutually exclusive with configRef)
`spec.bootstrap`	no	`[cilium]`	Ordered list of infra addons (default applied by framework bootstrap when omitted)
`spec.kubehz.hosting`	no	`self`	Who runs the control plane: `self` (you provision and own it), `hosted` (kubehz runs it; needs a kubehz account, because `lo provision` creates the cluster with `KUBEHZ_TOKEN`; with `kind: Lo` also requires `spec.runner`), or `shared` (a kubehz Space — namespaces on a shared plane, nodes you register; requires `kind: Kubehz`)
`spec.kubehz.access`	no	`none`	Platform visibility: `none` (no contact), `registered` (announced + heartbeats → dashboard), or `managed` (adds the management features driven from the dashboard; acting needs `agent: operator`)
`spec.kubehz.agent`	no	`cronjob`	Which in-cluster agent owns the heartbeat: `cronjob` (the bash CronJob — node status, control-plane health, certificate expiry, every 5 minutes) or `operator` (the Go live agent — live view within seconds, plus worker scaling, self-healing and worker upgrades where your tier allows). Exactly one of them beats; `lo kubehz deploy` applies the one you name. Needs `access: registered` or `managed`, and is not valid with `hosting: shared`. See the kubehz guide
`spec.kubehz.apiUrl`	conditional	—	kubehz API endpoint; required when `hosting` is `hosted` or `shared`, or `access != none`; must be HTTPS
`spec.controlPlane.replicas`	no	`1`	For `hosting: hosted`: the apiserver replicas of the hosted control plane (1 to 3), the one shape value `lo provision` sends. There is no plan and no tier: the state limit and its auto-grow step and cap live in the dashboard, and the bill is per started hour, each unit capped at 730 hours a month. See the kubehz guide
`spec.kubehz.space.slug`	no	domain's first label	`hosting: shared` — the namespace name for the Space (a DNS label)
`spec.kubehz.space.name`	no	the slug	`hosting: shared` — display name in the dashboard
`spec.kubehz.space.nodes`	no	`[]`	`hosting: shared` — machine names to mint a single-use join ticket for during `lo provision`. A list of names, not a count: the count is `spec.kubehz.space.limits.nodes`
`spec.kubehz.space.limits.nodes`	no	the platform's	`hosting: shared` — the node ceiling of the space, a whole number of 1 or more. A space has no plan: this and the next two values are the space. The ceiling your account may set is the platform's, and it differs per account: `lo` sends what you write, the api refuses what your account may not have and names its real ceiling. Omit the field and the platform applies its own default. The current ceiling is in the dashboard
`spec.kubehz.space.limits.namespaces`	no	the platform's	`hosting: shared` — the namespace ceiling of the space, a whole number of 1 or more
`spec.kubehz.space.limits.objectCapKiB`	no	the platform's	`hosting: shared` — the size cap for one Secret or ConfigMap in the space's namespaces, a whole number of KiB
`spec.kubehz.upgrades.channel`	no	`patch`	How far the platform may upgrade the cluster unasked: `patch` (patch releases — the default), `minor` (minor releases too), or `none` (no automatic upgrades — not recommended: you forgo automatic security patches). Clusters running releases that fall too far behind are upgraded by the platform regardless of channel (end-of-support floor), with notice. See the kubehz guide
`spec.kubehz.upgrades.defer`	no	`window`	When an allowed upgrade may start: `window` (wait for the next maintenance window) or `immediate` (as soon as it is available)
`spec.kubehz.maintenanceWindow`	no	—	When platform-driven work may run: `enabled`, `daysOfWeek`, `startTime` (`HH:MM`), `durationMinutes`, `timezone` (IANA name), `exclusions` (absolute freezes — a `YYYY-MM-DD` date or `from/to` range). `lo` shape-checks the two `upgrades` enums and the exclusion dates; full semantics are validated where the block is consumed — the platform API for hosted payloads, the managed-tier agent for self-hosted (once upgrade execution ships)
`spec.build.artifacts`	no	`single`	`split` additionally emits per-resource files under `clusters/<domain>/artifacts/` — `<Kind>.<namespace>.<name>.yaml`, Secrets as sops-encrypted `Secret.<ns>.<name>.sops.yaml` (committable; per-object git history for GitOps)
`spec.build.encrypt.type`	no	`sops`	How split-mode Secrets are encrypted. `sops` is the only supported value today; anything else is a hard build error (decoupled from the split trigger — `artifacts: split` shapes every resource, `encrypt` governs only the Secret twins)
`spec.build.encrypt.on`	no	`change`	When a Secret's committed `Secret.*.sops.yaml` is re-encrypted. `change` (default) keeps the existing ciphertext byte-for-byte when the prior decrypts to the same (canonical) plaintext — killing the per-build churn from sops's fresh-per-encrypt data key. `always` re-encrypts every Secret every build (the original behavior). See the build guide.
`spec.gitops.provider`	no	—	Declares the reconciler consuming the split output (implies `build.artifacts: split` unless explicitly `single`)
`spec.gitops.age`	conditional	—	age public keys the split mode encrypts Secrets to; required whenever the render contains a Secret (build fails closed otherwise)

Split-mode notes: artifacts.yaml remains the single source lo deploy applies — the split dir is derived from it after every successful build. Jobs in the split output are GitOps-shaped (ttlSecondsAfterFinished stripped — a TTL-reaped Job would be recreated by the reconciler every interval — and annotated kustomize.toolkit.fluxcd.io/force so fixed-name Jobs recreate on spec changes). lo build --split / --single exist as debug overrides and warn when they contradict the spec.

Secret churn — encrypt.on: change (default). sops mints a fresh data key on every encrypt, so re-encrypting an unchanged Secret rewrites its whole ciphertext and dirties the git diff each build. With on: change, split compares each Secret against its committed twin: if the prior decrypts here (using the ambient sops age key lo build already has — the owner key is a recipient) and its decrypted content canonically equals the fresh render, the existing file is kept verbatim (no re-encrypt). If the prior can't be decrypted (no key / corrupt / missing) it re-encrypts — it can never prove "unchanged", so it fails safe. on: always disables the compare. Chart-minted random secrets must still be pinned (chart existingSecret hooks + the secrets plugin) or the rendered value changes every build (a real change → a re-encrypt) — verify with a double-build diff before pointing a reconciler at the output.

CI render — lo build --no-secrets. For the CI path that has no secrets store and no age key but must still regenerate non-Secret artifacts (e.g. after an image-automation pin bump), lo build --no-secrets splits only non-Secret resources. It never renders, re-encrypts, prunes, or even reads a committed Secret.*.sops.yaml — those are left completely inert — and the secret generators are never invoked (so the cache-first passwd/bash generators can't mint fresh values on a store miss). It achieves this by exporting LOK8S_SECRETS_DISABLE=1 to the kustomize build render (the split shaping alone would not stop the render from hitting the store); the swap's prune is guarded to exclude Secret.*.sops.yaml; a non-Secret dropped from the render is still pruned. See the plugin's env contract.

See the kubehz Platform guide for registration, claiming, and the heartbeat agent.

Provider output (standard schema)

All providers produce the same output JSON. Drivers read this — not the cluster spec — for infrastructure details (IPs, SSH access, node topology).

json

{
  "api": { "endpoint": "<lb_ip>", "port": 6443 },
  "access": [
    { "id": "default", "type": "ssh", "user": "root", "port": 22,
      "privateKey": "~/.ssh/key", "publicKey": "~/.ssh/key.pub" },
    { "id": "bastion", "type": "ssh", "host": "bastion.example.com",
      "user": "jump", "privateKey": "~/.ssh/bastion" },
    { "id": "dedicated", "type": "ssh", "user": "root",
      "privateKey": "~/.ssh/robot_key", "bastion": "bastion" }
  ],
  "nodes": [
    { "name": "cp-0", "role": "control-plane", "group": "cloud-cp",
      "public_ip": "1.2.3.4", "private_ip": "10.0.0.1",
      "access": "default", "ssh_user": "root", "ssh_port": 22 }
  ],
  "network": { "id": "12345", "name": "my-net", "cidr": "10.0.0.0/16" }
}

Field	Description
`access[].id`	Unique identifier for this access method
`access[].type`	Access type: `ssh`, `ssm` (AWS), `gcloud` (GCP)
`access[].bastion`	ID of another access entry to use as jump host
`nodes[].access`	References `access[].id` (default: first entry)
`nodes[].ssh_user/ssh_port`	Back-compat fields (derived from access)

Drivers should read access[] for connection details, not spec.ssh. The number of control-plane nodes is derived from nodes | select(.role == "control-plane") | length, not spec.controlPlane.

Kind-specific fields

Each cluster kind adds its own fields to the common base. Kind-specific fields are documented in the sections below.

Lo Spec

Minimal form — the framework derives everything else from the domain (slot-parsed for *.lok8s.dev) and domain-independent defaults.

yaml

apiVersion: cluster.lok8s.dev/v1beta1
kind: Lo
metadata:
  name: local
spec:
  cluster:
    domain: lok8s.dev

Full form — every field explicit, equivalent to the defaults that the minimal form would produce for slot 125.

yaml

apiVersion: cluster.lok8s.dev/v1beta1
kind: Lo
metadata:
  name: local
spec:
  kubernetes:
    version: "v1.31.4"         # kind node image tag (default: the node image of the installed kind)
  cluster:
    domain: lok8s.dev           # Cluster FQDN (required)
    namespace: default          # Default namespace
  network:                       # Docker bridge network config
    name: local                  # Bridge network name (default: metadata.name)
    cidr: "10.125.125.0/24"     # /24 slot — slot-derived from domain for *.lok8s.dev
    dnsSearch: []                # DNS search list of the nodes and pods (default: [] = no host search domains)
  registries:                    # Registry mirror configuration
    tls: true                    # HTTPS registries (cert: generator-minted; default: true)
    shared:
      enabled: false             # Pull-through mirrors on a shared network (opt-in; default: false)
      network:                   # Only read when enabled: true
        name: lok8s-registries   # Shared network name (default)
        cidr: "10.125.200.0/24"  # Shared network CIDR (default)
    mirrors:                     # Pull-through mirror list (default: io-* set below)
      - name: io-docker
        url: https://registry-1.docker.io
      - name: io-quay
        url: https://quay.io
      - name: io-k8s
        url: https://registry.k8s.io
      - name: io-ghcr
        url: https://ghcr.io
  nodes:                         # Kind cluster node topology
    controlPlane: 1              # Control-plane nodes (default: 1)
    workers: 0                   # Worker nodes (default: 0)
    hostPorts: true              # Bind 80/443/8080 on host (default: true for slot 125, false otherwise)
    maxConcurrentDownloads: 3    # containerd parallel image pulls (default: 3)
  loadBalancer:                  # MetalLB L2 load balancer (optional)
    pool: "10.125.125.125-10.125.125.150"  # 26 IPs; slot-derived by default for *.lok8s.dev
  runtime: kind                  # Runtime: "kind" (default; future: k3s, k0s)
  bootstrap:                     # Cluster-infra addons (ordered; default: [cilium])
    - cilium                     # framework addon: .lok8s/addons/cilium/
    - metallb                    # framework addon: .lok8s/addons/metallb/
  provider:                      # Cloud provider for remote VMs (optional, requires --remote)
    name: hetzner                # Provider name (matches .lok8s/providers/<name>/main)
    configRef: hetzner.json      # Provider config file (relative to cluster dir)
  remote:                        # Remote mode config (optional, requires --remote)
    mode: docker                 # "docker" (local orchestration, remote Docker) | "ci" (everything on VM)
    expose: true                 # Run nginx reverse proxy on remote (default: true when provider is set)
    sync:                        # Repo sync settings (CI mode only)
      path: .                   # What to rsync (default: repo root)
      exclude:                  # Rsync exclusions
        - .git
        - node_modules
        - .secrets
        - .kubeconfig
      dest: /workspace           # Remote destination (default: /workspace)
    tilt: true                   # Start Tilt on the VM after provision (CI mode, default: true)

Provider and remote mode

The spec.provider and spec.remote blocks are only activated when lo provision --remote (or lo up --remote) is used. Without --remote, they are ignored — the same spec works for both local and remote.

Docker mode (default): The provider creates a VM, then the local machine runs kind/registries/bootstrap against the remote Docker via DOCKER_HOST=ssh://<ip>. The API is accessed through an SSH tunnel.

CI mode (mode: ci): The repo is rsynced to the VM and lo provision runs entirely on the remote. The remote handles kind, registries, bootstrap, and optionally Tilt. The local machine sets up an SSH tunnel for kubectl and optionally an nginx proxy for HTTPS access.

See CLI reference — Remote clusters for usage.

Default resolution

When a field is missing, the Lo driver fills it in via two layers:

Domain-independent defaults — apply to any Lo cluster:

Field	Default
`registries.shared.enabled`	`false`
`registries.shared.network.name`	`lok8s-registries`
`registries.shared.network.cidr`	`10.125.200.0/24`
`registries.mirrors`	`io-{docker,quay,k8s,ghcr}` on the standard upstream URLs
`network.dnsSearch`	`[]` (the host's search domains stay out of the cluster)
`nodes.controlPlane`	`1`
`nodes.workers`	`0`
`nodes.maxConcurrentDownloads`	`3`
`kubernetes.version`	none: the kind config names no node image, so kind uses the default node image of its release (`kind version` in `.bin` names that release)
`runtime`	`kind`
`bootstrap`	`[cilium]`

Slot-derived defaults — *.lok8s.dev only. The slot is parsed from spec.cluster.domain: lok8s.dev → 125, <n>.lok8s.dev → n.
Field Default
network.name metadata.name
network.cidr 10.125.<slot>.0/24
loadBalancer.pool 10.125.<slot>.125-10.125.<slot>.150
nodes.hostPorts true for slot 125, false for every other slot
Non-*.lok8s.dev domains must supply network.name and network.cidr explicitly.

Field	Default
`network.name`	`metadata.name`
`network.cidr`	`10.125.<slot>.0/24`
`loadBalancer.pool`	`10.125.<slot>.125-10.125.<slot>.150`
`nodes.hostPorts`	`true` for slot 125, `false` for every other slot

Build and cache registries are framework-private — they always live on the project subnet at .101 and .102 and must not be listed in spec.registries.mirrors.

Kubernetes version (Lo)

For a Lo cluster, spec.kubernetes.version is the tag of the kind node image: every node of the kind config gets image: "kindest/node:<tag>".

Unset: the key is missing or null (~), or the value is "" or the string "null". The kind config then names no image, and kind uses the default node image of its release.
Valid: any OCI image tag, 1 to 128 letters, digits, _, . and -, that does not start with . or -. An optional @sha256:<64 hex> digest can follow. Examples: v1.31.4, v1.31.4@sha256:<digest>, latest, or the tag of an image that you built with kind build node-image.
Refused: a value that is not an image tag, for example kindest/node:v1.31.4 (the full image name).

lo lint, lo provision and lo audit read the version with the same rules in both implementations:

The spec file holds one YAML document (One YAML document per spec file).
YAML aliases and merge keys (<<: *anchor) are resolved: a key of the map wins over a merged key, the last duplicate key wins, and in <<: [*a, *b] the first map wins. An anchor can be on a key. An alias that is used as a key is not resolved (yq reads it by its own name). The read follows only the path to the value, so an alias bomb or an alias cycle in another part of the file costs nothing. The bash implementation needs yq v4.47.1 or later (the flag --yaml-fix-merge-anchor-to-spec); b install installs the pinned yq.
Trailing newlines of the value (a block scalar) are removed.
These shapes are refused, never read as an unset version: a merge key anywhere in the file that does not refer to a map or a list of maps; merge keys on the path to the value that form a cycle, or that need more than 10000 map reads (yq reads a merged map again for each merge key that names it); a spec or spec.kubernetes that is not a map; a version that is a map or a list; a value with a control character (a NUL byte, a tab, an escape sequence, U+0085, U+2028, a bidi override). A problem shows each control character as ? and at most 80 characters of the value.
Each yq call of the bash implementation has a time limit of 10 s and a memory limit of 256 MiB (ulimit -d) where the operating system enforces it. The time limit uses a background sleep, not GNU timeout, so it holds on macOS too. A call that passes a limit is reported with that limit.

lo provision refuses before it creates a VM, a docker network or a container. lo audit reports a refused spec as unknown with the same problem text.

Network Configuration

The spec.network section configures the Docker bridge network used by kind clusters. For *.lok8s.dev domains, both fields are derived from the domain and metadata.name; for other domains, both are required. See SUBNETS.md for the 10.125.0.0/16 slot plan.

Field	Required	Description
`name`	no (for `*.lok8s.dev`)	Docker bridge network name (default: `metadata.name`)
`cidr`	no (for `*.lok8s.dev`)	`/24` CIDR for the cluster's slot (default: `10.125.<slot>.0/24`)
`dnsSearch`	no (default `[]`)	DNS search list of the kind nodes and pods (kind `networking.dnsSearch`, kind v0.18.0 or later). `[]` keeps the host's search domains out of the cluster; a list of at most 32 DNS names (any case, an optional trailing dot) replaces it, as written. `lo lint` checks it. See Host DNS search domains.

Example for an alternate slot:

yaml

spec:
  cluster:
    domain: 50.lok8s.dev
  # network.name defaults to metadata.name
  # network.cidr defaults to "10.125.50.0/24"

Or fully explicit:

yaml

spec:
  network:
    name: lok8s-50
    cidr: "10.125.50.0/24"

Registries Configuration

The spec.registries section configures registry mirrors. Every lok8s cluster ships with two framework-private registries plus four default pull-through mirrors:

Framework-private (build, cache) — plain filesystem registries owned by the framework. Always live on the project subnet at fixed offsets .101 (build) and .102 (cache), even in shared mode, because they hold per-project content (locally-built images for build, credentialed pre-pulls for cache). They get the canonical hostnames lok8s.local and lok8s.cache via containerd mirror config. Do not list them in spec.registries.mirrors — they ship implicitly.
Pull-through mirrors (io-docker, io-quay, io-k8s, io-ghcr by default) — transparent caches for public upstreams. They live on the project subnet at .103+. With shared.enabled: true (opt-in) they move to the shared registry network at 10.125.200.0/24 so multiple kind clusters reuse one cache — see Shared registries for the trade-off.

Field	Required	Description
`tls`	no (default `true`)	HTTPS registries with a cert minted by the Secret plugin; set `false` for plain HTTP (see TLS below)
`shared.enabled`	no (default `false`)	Opt in to put pull-through mirrors on the shared network
`shared.network.name`	no (default `lok8s-registries`)	Shared registry network name
`shared.network.cidr`	no (default `10.125.200.0/24`)	Shared registry network CIDR
`mirrors[].name`	yes (per entry)	Registry name (must not be `build` or `cache`)
`mirrors[].url`	yes (per entry)	Upstream registry URL
`prefix`	no (default `lok8s.local`)	Image registry prefix used for locally-built image references

Registries are reached via raw IP (no host port mapping). The listen port depends on the mode: :80 in the default plain-HTTP mode, or :443 when tls: true (so a bare-IP docker push <ip>/..., which the Docker client resolves to the HTTPS default port, reaches the registry without an explicit port in the ref).

Registry TLS

TLS is the default (spec.registries.tls: true; set false for plain HTTP). Every registry (framework-private and pull-through mirrors) serves HTTPS with a certificate minted by the Secret plugin (the same cert: generator used for application TLS), signed by your shared dev CA at CAROOT. This removes the need for an insecure-registries entry in the host Docker daemon configuration:

The cert is minted into the docker volume <network>-registry-tls with every registry's IP and (for framework registries) hostname (lok8s.local, lok8s.cache) as Subject Alternative Names. Every registry container mounts the volume at /etc/registry/certs. lo up mints again when the IP/hostname set changes. No mkcert binary is involved (the CA at CAROOT is created on demand). See TLS registries.
Containerd inside the kind nodes trusts the cert via an explicit CA file: each hosts.toml references a copy of the dev rootCA.pem (CAROOT) mounted into the node's certs.d tree (no skip_verify). This needs no mkcert -install — in-cluster pulls work out of the box.
Host docker push validates against the system trust store, so the dev CA must be installed there with lo trust; then pushes to lok8s.local/lok8s.cache (and raw IPs) succeed over HTTPS with no daemon configuration.

Prerequisite: the cert is minted by the Secret plugin (no mkcert), but the host Docker client and curl only trust it once the dev CA is in the system trust store — run lo trust once (it wraps mkcert -install; mkcert is managed by b, b install mkcert). $CAROOT/rootCA.pem (default ~/.local/share/mkcert/rootCA.pem) is the trust anchor. If tls: true but the Secret plugin isn't built, lo provision/lo up fail fast; if the CA isn't trusted, the cluster still comes up but docker push fails cert verification until you run lo trust (or another option — see Host push trust options).

Opting out with tls: false keeps plain-HTTP registries, which instead require the registry IP range in the host's insecure-registries.

Registry set for slot 125 (the default cluster) with shared.enabled: true (opt-in):

Name	IP	Hostname	Purpose
`build`	`10.125.125.101`	`lok8s.local`	Tilt push target for locally-built images
`cache`	`10.125.125.102`	`lok8s.cache`	Pre-pull target for `build: false` services with a configured registry
`io-docker`	`10.125.200.2`	`docker.io`	Pull-through mirror
`io-quay`	`10.125.200.3`	`quay.io`	Pull-through mirror
`io-k8s`	`10.125.200.4`	`registry.k8s.io`	Pull-through mirror
`io-ghcr`	`10.125.200.5`	`ghcr.io`	Pull-through mirror

For other slots, replace 125.125 with 125.<slot>. In the default per-project mode, pull-throughs live on the project subnet at .103+ instead.

Containerd wiring: hostname → IP resolution happens via per-host hosts.toml files written by lo::write_certs_d to clusters/<domain>/.containerd/certs.d/. That directory is bind-mounted into every kind node via extraMounts, so containerd reads it on startup. Each host gets BOTH a hostname-keyed entry AND a raw-IP-keyed entry, covering both naming conventions.

Nodes Configuration

The spec.nodes section controls kind cluster node topology.

Field	Default	Description
`controlPlane`	`1`	Number of control-plane nodes (1-9)
`workers`	`0`	Number of worker nodes (0-99)
`hostPorts`	`true`	Bind host ports 80/443/8080 on the first control-plane node
`maxConcurrentDownloads`	`3`	containerd `max_concurrent_downloads` — parallel image layer pulls per node. Matches containerd's own default. Lower to `1` if a flaky pull-through mirror returns "unexpected commit digest" under concurrent pulls; raise for faster cold starts on a reliable mirror.

When absent, a single control-plane node is created (the kind default).

The first control-plane node gets port mappings (80, 443, 8080), the ingress-ready=true label, and a docker socket mount. Additional control-plane and worker nodes get the base image only.

Load Balancer Configuration

The spec.loadBalancer section installs and configures MetalLB for LoadBalancer service support. When absent, MetalLB is not installed.

Field	Description
`pool`	IP range for MetalLB L2 pool (e.g. `10.125.125.125-10.125.125.150` for slot 125)

The pool range must be within spec.network.cidr. IP validation is performed at provisioning time.

MetalLB is pinned to Helm chart version 0.16.1. An IPAddressPool and L2Advertisement are created automatically.

Custom DNS (`spec.coredns`)

spec.coredns injects per-cluster CoreDNS config. The Lo driver renders it into a coredns-custom ConfigMap that the base Corefile imports from /etc/coredns/custom (import custom/*.server + import custom/*.override). Declarative and committed, so it survives a recreate — unlike a runtime kubectl patch. All fields are optional and compose; when spec.coredns is absent the import is a no-op. Full guide: Custom in-cluster DNS.

Field	Type	Description
`hosts[]`	list of `{name, target}`	Driver generates a `name:53 { … }` server block resolving the zone `name` (its apex + every `.name`): `A → target`, `AAAA → NODATA`, other types forwarded. `target: gateway` resolves to the first `spec.loadBalancer.pool` IP* (where the gateway pins via the metallb annotation) — no hardcoded IP to keep in sync; otherwise a literal IP.
`servers`	string	Raw CoreDNS server block(s), inline (a `*.server` file).
`overrides`	string	Raw directives merged into the default `.:53` block (a `*.override` file).
`import`	string	Path (relative to the cluster dir; default `./coredns`) to a directory of raw `.server` / `.override` files.

yaml

spec:
  coredns:
    hosts:
      - name: kubehz.dev      # apex + *.kubehz.dev
        target: gateway       # = loadBalancer.pool[0], or a literal IP

Don't define the same zone via both hosts and a raw servers/import block — CoreDNS rejects duplicate zone definitions.

Bootstrap (Cluster-Infra Addons)

The spec.bootstrap field is an ordered list of addons the framework applies at provision time, before Tilt starts and before any workloads land. This is the framework's one ordering primitive — each entry is applied, then lok8s waits for Deployments/DaemonSets to become ready before moving to the next.

The framework bootstrap in the lo binary applies the list. It runs the same way for all drivers (Lo, KubeOne, Capi, Kkp).

Use this for anything the cluster needs to be healthy before workloads can run: CNI, CSI, CCM, MetalLB, cert-manager, etc. It is not for application workloads — those live under targets/ and have no framework-level ordering.

yaml

spec:
  bootstrap:
    - cilium            # bare name → framework addon
    - metallb
    - cert-manager
    - ./custom/         # relative path → clusters/<domain>/custom/
    - /shared-base/     # absolute path → repo root /shared-base/

Entry resolution:

Form	Resolves to
Bare name (`cilium`)	`.lok8s/addons/<name>/`
`./path` or `../path`	Relative to `clusters/<domain>/`
`/path`	Relative to the repo root

Each entry must point at a kustomize-buildable directory. Framework addons ship as khelm ChartRenderers (chart.yaml + values.yaml), but any kustomization works. If spec.bootstrap is omitted entirely, the framework defaults to [cilium] (every cluster needs a CNI).

Framework addons support driver- and provider-specific values files plus inline overrides. See the Addons guide for the full precedence chain (base < driver < provider < inline) and authoring guidance.

Failure modes:

Missing entry → bootstrap exits non-zero.
kustomize build failure → exits non-zero.
kubectl apply failure → exits non-zero.
Values merge failure → exits non-zero.
Health-wait timeout → logged as warning, non-fatal (the addon might not ship Deployments).

Running lo lint validates that every spec.bootstrap entry resolves to an existing directory.

IP Validation

During provisioning, all computed IPs are validated against the subnet:

Registry IPs (sequential offsets from the project or shared subnet base)
MetalLB pool start and end IPs (if configured)

If any IP falls outside the subnet CIDR, provisioning fails with a clear error message.

Registry Config (`.registries.json`)

During provisioning, the Lo driver generates a .registries.json file in the cluster directory (e.g. clusters/lok8s.dev/.registries.json). This JSON file is the single source of truth for all registry config — IPs, upstream URLs, domains, container hostnames, and shared/project mode.

json

{
  "shared": true,
  "network": { "name": "lok8s-registries", "cidr": "10.125.200.0/24" },
  "project_network": "local",
  "registries": [
    { "name": "build", "ip": "10.125.125.101", "url": "", "domain": "", "host": "lok8s.local", "type": "framework" },
    { "name": "cache", "ip": "10.125.125.102", "url": "", "domain": "", "host": "lok8s.cache", "type": "framework" },
    { "name": "io-docker", "ip": "10.125.200.2", "url": "https://registry-1.docker.io", "domain": "docker.io", "host": "", "type": "mirror" }
  ]
}

All registry consumers (lo registry status, lo::registries, lo::write_certs_d, lo::registry_configmap) read from this JSON via the registry::* helpers in drivers/lo/utils/config-registry.sh. No global bash arrays are used.

The file is deterministic (derived from the cluster spec + slot) and can be committed to the repo.

Registry ConfigMap

A lok8s-registries ConfigMap is created in kube-system during provisioning. It contains the IP, port, and (for mirrors) remote URL of each registry. This allows in-cluster tooling to discover registry mirrors programmatically.

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: lok8s-registries
  namespace: kube-system
data:
  build.ip: "10.125.125.101"
  build.port: "80"
  cache.ip: "10.125.125.102"
  cache.port: "80"
  io-docker.ip: "10.125.200.2"           # shared mode
  io-docker.port: "80"
  io-docker.url: "https://registry-1.docker.io"
  # ... etc

Additionally, a local-registry-hosting ConfigMap is applied in kube-public per KEP-1755. Tilt reads this to auto-configure default_registry() so docker_build('foo', ...) transparently pushes to the build registry without per-user setup:

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: local-registry-hosting
  namespace: kube-public
data:
  localRegistryHosting.v1: |
    host: "10.125.125.101"
    hostFromClusterNetwork: "lok8s.local"

Lo Status

yaml

status:
  phase: Provisioned            # Pending | Provisioning | Provisioned | Failed
  ready: true
  conditions:
    - type: Ready
      status: "True"
      reason: Provisioned
      message: Cluster is ready
      lastTransitionTime: "2025-01-01T00:00:00Z"

KubeOne Spec

KubeOne clusters use the common fields plus spec.provider for infrastructure. SSH config and node topology are defined in the provider config (e.g. hetzner.json), NOT in the cluster spec. The driver reads them from provider::output.

yaml

apiVersion: cluster.lok8s.dev/v1beta1
kind: KubeOne
metadata:
  name: my-prod
spec:
  kubernetes:
    version: "v1.31.12"
  cluster:
    domain: my-cluster.example.com
  provider:
    name: hetzner
    configRef: hetzner.json
  auditLog:                          # optional: apiserver audit log
    policy: audit-policy.yaml        # required in auditLog: relative to this file
    maxAge: 30                       # optional: days (KubeOne default 30)
    maxBackup: 10                    # optional: old files to keep (KubeOne default 3)
    maxSize: 100                     # optional: MB before rotation (KubeOne default 100)
  bootstrap:
    - ./targets/networking
    - ./targets/cnpg-operator

Field	Required	Default	Description
`spec.auditLog.policy`	when `auditLog` is set	—	Path of an audit Policy file (`apiVersion: audit.k8s.io/v1`, `kind: Policy`), relative to the cluster file. The driver gives KubeOne the absolute path (`features.staticAuditLog.config.policyFilePath`)
`spec.auditLog.maxAge`	no	KubeOne: `30`	Days to keep old audit log files (`logMaxAge`), a whole number from 1 to 999999999
`spec.auditLog.maxBackup`	no	KubeOne: `3`	Number of old audit log files to keep (`logMaxBackup`), a whole number from 1 to 999999999
`spec.auditLog.maxSize`	no	KubeOne: `100`	Size in MB at which the audit log file rotates (`logMaxSize`), a whole number from 1 to 999999999

spec.auditLog turns on KubeOne's features.staticAuditLog. The driver checks the spec and the policy file at lo provision, before the provider creates or changes a server and before it writes the manifest. See Security: apiserver audit log for the checks, an example policy, and the forced apply that a running cluster needs.

Deprecated fields (use provider config instead):

~~spec.ssh~~ → provider config sshUser, sshPrivateKey, sshPublicKey
~~spec.controlPlane.replicas~~ → derived from provider output node count. This holds for a cluster you provision yourself. With spec.kubehz.hosting: hosted the same key is not deprecated: it carries the apiserver replicas of the hosted control plane, which the platform cannot derive from a provider.
~~spec.workers~~ → defined in provider config (hetzner.json server entries)

These fields are still read as fallbacks for backward compatibility but should not be used in new cluster specs.

Capi Spec

Note: Capi keeps controlPlane and workers in the cluster spec (unlike KubeOne, where the provider owns node topology) because CAPI generates the Kubernetes resources (KubeadmControlPlane, MachineDeployments) directly. Manifest generation targets Hetzner Cloud only (spec.provider.name: hetzner). See the CAPI guide and the validated examples/capi / examples/capi-ha specs.

yaml

apiVersion: cluster.lok8s.dev/v1beta1
kind: Capi
metadata:
  name: prod
spec:
  kubernetes:
    version: "v1.31.12"
  cluster:
    domain: prod.example.com        # Cluster FQDN (required)
    namespace: default
  managementCluster:
    domain: prod-mgmt.example.com    # CAPI management cluster domain
    local: true                      # run it as a local kind cluster
  provider:
    name: hetzner                    # only hetzner (CAPH) is generated
    config:
      region: fsn1                   # Hetzner Cloud region
      sshKeyName: my-key             # SSH key registered in Hetzner
      image: ubuntu-24.04            # stock image; k8s installed via cloud-init
      network:
        enabled: false               # true → private hcloud network + CCM networking
      placementGroups: false         # true → spread anti-affinity (≤ 10 servers/group)
    credentials:
      envVars: [HCLOUD_TOKEN]
      secretRef: prod-credentials    # Secret on the mgmt cluster (default <name>-credentials)
  controlPlane:
    replicas: 3                      # odd for etcd quorum (default 1)
    type: cpx22                      # Hetzner server type
  workers:                           # each key = its own MachineDeployment pool
    general:
      replicas: 3
      type: cpx22
    apps:
      replicas: 1
      type: cpx22
  oidc:                              # optional
    enabled: false
    issuerUrl: https://auth.example.com
  etcd:
    encryptionSecretName: etcd-encryption
  bootstrap:                         # cluster-infra addons (ordered)
    - cilium                         # CNI
    - ccm                            # Hetzner cloud-controller-manager
  gitops:                            # (deferred — currently a no-op)
    provider: flux
    repo: https://github.com/myorg/infra.git
    path: clusters/prod.example.com/artifacts

Capi Status

yaml

status:
  phase: Provisioned
  ready: true
  provider: hetzner
  controlPlaneEndpoint:
    host: 10.0.0.1
    port: 6443
  kubeconfig:
    secretRef: prod-kubeconfig
  targets:
    networking: Applied
    app: Applied
  gitops:
    provider: flux
    status: Bootstrapped
  conditions:
    - type: InfrastructureReady
      status: "True"
    - type: ControlPlaneReady
      status: "True"

deploy.lok8s.yaml

Defines a deployment domain that targets an existing cluster.

yaml

apiVersion: cluster.lok8s.dev/v1beta1
kind: Deploy
metadata:
  name: api
spec:
  domain: api.example.com       # Deployment domain FQDN
  clusterRef:
    domain: prod.example.com    # Cluster to deploy to (required)
  namespace: api                # Target namespace on the cluster

Note: Deploy CRD workload selection is being reworked. Target selection for Deploy specs will land alongside the services.yaml targets-map design. Until then, a Deploy spec carries only clusterRef + namespace.

Deploy Status

yaml

status:
  phase: Deployed               # Pending | Deploying | Deployed | Failed
  targets:
    app: Applied
    jobs: Applied
  conditions:
    - type: Ready
      status: "True"

services.yaml

Defines services for Tilt local development. Not a CRD -- consumed by the Tilt extension (through the framework's internal env-render step).

yaml

registry:
  prefix: lok8s.local           # Image registry prefix
  branch: builds                # Image path component
  tag: latest                   # Image tag

defaults:
  build: true                   # Default per-service build flag (true = build locally + Tilt live-update)
  dockerfile: service           # "service" (lok8s.Dockerfile) or "production" (Dockerfile)

services:
  my-api:
    enabled: true               # Whether this service is active (default true)
    build: true                 # Whether to docker_build in Tilt (default: defaults.build)
    path: ./my-api              # Service source directory (default: ./<name>)
    namespace: api              # Optional: inject namespace
    dockerfile: production      # Optional: override defaults.dockerfile
    registry:                   # Optional: per-service registry override
      branch: pr-1234
      tag: abc123
    image: ghcr.io/org/img:v1   # Optional: pin to a specific image (mutually exclusive with registry:)
    watch:                      # Optional: extra files to watch
      - package.json

Per-Service lok8s.yaml

Per-service Tilt build configuration. Located at <service-path>/lok8s.yaml.

yaml

build:
  dockerfile: lok8s.Dockerfile  # Dockerfile path (relative to service dir)
  context: .                    # Build context (relative to service dir)
  build_args:                   # Env var names to pass as build args
    - NODE_ENV
  live_update:
    fall_back_on:
      files:
        - package.json
        - yarn.lock
    sync:
      - local_path: ./src
        remote_path: /app/src     # NOTE: remote_path (not container_path)
    run:
      cmd: npm run dev            # NOTE: cmd (not command)
    restart_container: {}       # Optional: restart after sync

ports:
  - from: 3000                  # Host port
    to: 3000                    # Container port

links:
  - url: http://localhost:3000
    name: API

workloads:                      # Optional: override detected workload names
  - my-api-deployment

tilt:
  resource_deps:                # Tilt resource dependencies
    - postgres
  labels:                       # Tilt UI labels
    - backend
  extra_resources:              # Additional k8s_resource groupings
    - name: my-api-migrations
      objects:
        - "my-api-migrate:job"
      resource_deps:
        - postgres
      labels:
        - backend

Spec Reference ​

One YAML document per spec file ​

cluster.lok8s.yaml ​

Common fields (all cluster kinds) ​

Provider output (standard schema) ​

Kind-specific fields ​

Lo Spec ​

Provider and remote mode ​

Default resolution ​

Kubernetes version (Lo) ​

Network Configuration ​

Registries Configuration ​

Registry TLS ​

Nodes Configuration ​

Load Balancer Configuration ​

Custom DNS (spec.coredns) ​

Bootstrap (Cluster-Infra Addons) ​

IP Validation ​

Registry Config (.registries.json) ​

Registry ConfigMap ​

Lo Status ​

KubeOne Spec ​

Capi Spec ​

Capi Status ​

deploy.lok8s.yaml ​

Deploy Status ​

services.yaml ​

Per-Service lok8s.yaml ​

Spec Reference

One YAML document per spec file

cluster.lok8s.yaml

Common fields (all cluster kinds)

Provider output (standard schema)

Kind-specific fields

Lo Spec

Provider and remote mode

Default resolution

Kubernetes version (Lo)

Network Configuration

Registries Configuration

Registry TLS

Nodes Configuration

Load Balancer Configuration

Custom DNS (`spec.coredns`)

Bootstrap (Cluster-Infra Addons)

IP Validation

Registry Config (`.registries.json`)

Registry ConfigMap

Lo Status

KubeOne Spec

Capi Spec

Capi Status

deploy.lok8s.yaml

Deploy Status

services.yaml

Per-Service lok8s.yaml