Cluster changes

Create a separate Kubernetes login, the changes login, that is used only to carry out cluster changes a person approved, with only the rights those changes need.

In the appPlatformSettingsLive systemKubernetesChanges login

The agent's own Kubernetes login only reads. If you switch Cluster changes on, the agent may also propose changes to your cluster, such as restarting a deployment or deleting a stuck pod. Nothing happens unless a person approves it.

When someone approves a change, Cerberius Agents carries it out with a second, separate login, the changes login. The agent never sees this login. It's used for nothing else.

Before you start#

Set up the read-only login first. See Kubernetes access.
Switch Cluster changes on under Settings › Approved changes. See Allowing changes. Until you do, the Changes login row under Kubernetes on Settings › Live system only says that cluster changes are off, and this login can't be added.

What changes are possible#

Change	Works on	Kubernetes rights it needs
Restart (a rolling restart)	deployments, statefulsets, daemonsets	`patch`
Scale to between 0 and 20 replicas	deployments, statefulsets, replicasets	`patch`
Delete	any kind below	`delete`
Patch some fields	any kind below	`patch`
Create from a manifest	any kind below	`create`
Run a CronJob now	cronjobs	`get` on cronjobs, `create` on jobs

These kinds can be changed: deployments, statefulsets, daemonsets, replicasets, pods, jobs, cronjobs, configmaps, services, ingresses, horizontalpodautoscalers and persistentvolumeclaims.

These are never changed, whatever the login allows: Secrets, ServiceAccounts, roles and role bindings, namespaces, nodes, network policies and custom resources. A patch or manifest that asks for privileges, the host's network or files, or a different ServiceAccount is refused before anyone sees it.

Changes stay within the Namespaces it may look at, set in the Cluster login row. Every change is tried first as a dry run, which checks it without changing anything. If any part fails the dry run, nothing runs.

This creates a ServiceAccount called support-agent-changer, a token for it, and a role with only the rights in the table above. It reuses the support-access namespace from Kubernetes access.

Save this as support-agent-changer.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: support-agent-changer
  namespace: support-access
automountServiceAccountToken: false
---
apiVersion: v1
kind: Secret
metadata:
  name: support-agent-changer-token
  namespace: support-access
  annotations:
    kubernetes.io/service-account.name: support-agent-changer
type: kubernetes.io/service-account-token
---
# Only the kinds Cerberius Agents may change. No Secrets, ServiceAccounts or RBAC.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: support-agent-change
rules:
  - apiGroups: ["apps"]
    resources: [deployments, statefulsets, daemonsets, replicasets]
    verbs: [get, list, create, patch, delete]
  - apiGroups: [""]
    resources: [pods, configmaps, services, persistentvolumeclaims]
    verbs: [get, list, create, patch, delete]
  - apiGroups: ["batch"]
    resources: [jobs, cronjobs]
    verbs: [get, list, create, patch, delete]
  - apiGroups: ["networking.k8s.io"]
    resources: [ingresses]
    verbs: [get, list, create, patch, delete]
  - apiGroups: ["autoscaling"]
    resources: [horizontalpodautoscalers]
    verbs: [get, list, create, patch, delete]
---
# Grant it per namespace. Copy this block for each namespace changes may touch.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: support-agent-change
  namespace: shop-prod
subjects:
  - kind: ServiceAccount
    name: support-agent-changer
    namespace: support-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: support-agent-change

kubectl apply -f support-agent-changer.yaml

Or: restarts and scaling only#

If you only want the agent to be able to restart or scale your apps, give the login just that. Replace the ClusterRole above with this one:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: support-agent-change
rules:
  - apiGroups: ["apps"]
    resources: [deployments, statefulsets]
    verbs: [get, patch]

The agent may still propose other changes. If someone approves one, the dry run fails with a forbidden message, and nothing changes.

Add rules as you need them. For example, to delete stuck pods:

  - apiGroups: [""]
    resources: [pods]
    verbs: [get, delete]

To run a CronJob straight away:

  - apiGroups: ["batch"]
    resources: [cronjobs]
    verbs: [get]
  - apiGroups: ["batch"]
    resources: [jobs]
    verbs: [create]

Step 2: Build the kubeconfig#

This is the same script as for the read-only login, with the changer's names.

#!/bin/sh
set -e

SA_NAMESPACE=support-access
SECRET=support-agent-changer-token
DEFAULT_NAMESPACE=shop-prod

SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
CA=$(kubectl -n "$SA_NAMESPACE" get secret "$SECRET" -o jsonpath='{.data.ca\.crt}')
TOKEN=$(kubectl -n "$SA_NAMESPACE" get secret "$SECRET" -o jsonpath='{.data.token}' | base64 --decode)

cat > support-agent-changer.kubeconfig <<EOF
apiVersion: v1
kind: Config
clusters:
  - name: my-cluster
    cluster:
      server: $SERVER
      certificate-authority-data: $CA
users:
  - name: support-agent-changer
    user:
      token: $TOKEN
contexts:
  - name: support-agent-changer
    context:
      cluster: my-cluster
      user: support-agent-changer
      namespace: $DEFAULT_NAMESPACE
current-context: support-agent-changer
EOF

echo "Wrote support-agent-changer.kubeconfig"

Check it can't do more than you meant:

# Should say "yes":
kubectl --kubeconfig support-agent-changer.kubeconfig -n shop-prod auth can-i patch deployments

# Should all say "no":
kubectl --kubeconfig support-agent-changer.kubeconfig -n shop-prod auth can-i get secrets
kubectl --kubeconfig support-agent-changer.kubeconfig -n shop-prod auth can-i create rolebindings
kubectl --kubeconfig support-agent-changer.kubeconfig -n kube-system auth can-i delete pods

Step 3: Add it in Cerberius Agents#

1
Go to Settings › Live system.
2
Under the Kubernetes card, in the Changes login row, click Add changes login….
3
Paste the whole kubeconfig into the Kubeconfig box.
4
Click Save. It's saved straight away, and the row shows Saved.

Field	What it does
Kubeconfig	The changes login. Stored encrypted, never shown again, and never given to the agent. Used only when someone approves a cluster change. Its ServiceAccount should not be able to read Secrets or change permissions.

The same rules apply as for the read-only kubeconfig: it must contain its token and CA certificate, and can't use a login program.

With your own runner

When the agent runs on your own runner, the changes login is kept by you. In the Changes login row, choose how: Scrambled for your runner (you type it here, and your browser scrambles it so only your runner can unscramble it) or In your store (you put it on your runner's side and give us only its name). Your runner then makes the cluster changes you approve, as its own rules allow (see Changes made by your runner). See Two ways to keep your keys.

When it is used#

Only when all of these are true:

Cluster changes are available to your organisation, and switched on for this platform.
The agent proposed a change, and it passed the checks above.
A person approved it. See Approving a change.

If cluster changes are switched off, this login is never used, even though it stays saved. The Changes login row then says Cluster changes are off, so this login is never used, and you can still replace or remove it.

Replace or remove it#

Once a login is saved, the Changes login row has Replace… to paste a new kubeconfig, and Remove to delete it, whether cluster changes are on or off. Removing asks you to confirm. Adding the first one needs cluster changes switched on.

To cut it off on the cluster side, delete its token:

kubectl -n support-access delete secret support-agent-changer-token