Kubernetes access

Create a read-only Kubernetes login for the agent, step by step, and add it as the cluster login under Settings › Live system.

In the appPlatformSettingsLive systemKubernetes

With Kubernetes access, the agent can look at what actually happened in your cluster, not only at the code. It can list and read pods, deployments and other objects, read pod logs and look at events.

It can only read. It can never read Secrets, and never change, run commands in, or connect to anything, whatever the login allows.

You give it access with a kubeconfig: a small file that says which cluster to talk to and how to log in. You'll create a dedicated, read-only login for it below.

The Live system page#

Settings › Live system connects two things: Kubernetes and Grafana Loki. A platform has at most one of each.

When nothing is connected, the page shows one card. Click Connect a cluster or logs, and choose what to connect. Once one is there, Connect another at the top adds the other.

Each one shows as a card with its name and a label: Connected, Not finished, or Not connected yet. Its rows are under the card:

Card	Row	What it is
Kubernetes	Cluster login	The read-only login to your Kubernetes cluster, and the namespaces the agent may look at. This article.
Kubernetes	Changes login	Optional. A second cluster login, used only for changes a person approves. See Cluster changes.
Grafana Loki	Logs	Your Loki logs. See Loki logs.

The label at the top of the page says what is connected: Cluster, Logs, Cluster and logs, or Not connected.

What the agent can read#

What	Kubernetes resources
Pods and their logs	`pods`, `pods/log`
Events	`events`
Workloads	`deployments`, `statefulsets`, `daemonsets`, `replicasets`
Jobs	`jobs`, `cronjobs`
Networking	`services`, `ingresses`
Configuration and storage	`configmaps`, `persistentvolumeclaims`
Autoscaling	`horizontalpodautoscalers`
Cluster	`nodes` and the list of `namespaces` (only with cluster-wide access)
Argo CD (optional)	`applications.argoproj.io`: what is deployed, and whether it is synced and healthy

Secrets and ServiceAccounts are never readable through the agent, even if the login allows them. Anything not in this table isn't readable either. The role below grants exactly these.

The kubeconfig must be self-contained

The agent can't run programs or read files from your computer. So the kubeconfig must hold its token and CA certificate inside it. Kubeconfigs that log in through a program, such as doctl (DigitalOcean), gcloud (Google Cloud), aws (Amazon EKS) or kubelogin (Azure), are refused. So are kubeconfigs that point to certificate or token files. Use the steps below to build one that works.

You'll create:

a namespace to hold the login, called support-access
a ServiceAccount (a login for a program) called support-agent-reader
a token for it that doesn't expire, stored in a Secret
a ClusterRole that lists what it may read (no Secrets)
a RoleBinding in each namespace the agent may look at

In these examples, your apps run in the namespaces shop-prod and payments-prod. Replace them with yours.

Save this as support-agent-reader.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: support-access
---
# The login. It can do nothing until the bindings below give it rights.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: support-agent-reader
  namespace: support-access
automountServiceAccountToken: false
---
# A long-lived token for the login. Kubernetes fills it in.
apiVersion: v1
kind: Secret
metadata:
  name: support-agent-reader-token
  namespace: support-access
  annotations:
    kubernetes.io/service-account.name: support-agent-reader
type: kubernetes.io/service-account-token
---
# What it may read. No Secrets, and only get, list and watch.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: support-agent-read
rules:
  - apiGroups: [""]
    resources: [pods, pods/log, events, services, configmaps, persistentvolumeclaims]
    verbs: [get, list, watch]
  - apiGroups: ["apps"]
    resources: [deployments, statefulsets, daemonsets, replicasets]
    verbs: [get, list, watch]
  - apiGroups: ["batch"]
    resources: [jobs, cronjobs]
    verbs: [get, list, watch]
  - apiGroups: ["networking.k8s.io"]
    resources: [ingresses]
    verbs: [get, list, watch]
  - apiGroups: ["autoscaling"]
    resources: [horizontalpodautoscalers]
    verbs: [get, list, watch]
---
# Grant it in each namespace the agent may look at. Copy this block per namespace.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: support-agent-read
  namespace: shop-prod
subjects:
  - kind: ServiceAccount
    name: support-agent-reader
    namespace: support-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: support-agent-read
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: support-agent-read
  namespace: payments-prod
subjects:
  - kind: ServiceAccount
    name: support-agent-reader
    namespace: support-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: support-agent-read

Apply it with your usual admin access:

kubectl apply -f support-agent-reader.yaml

A ClusterRole is only a list of rights. It grants nothing on its own. Each RoleBinding grants those rights in one namespace only, so the agent sees shop-prod and payments-prod and nothing else.

Leaving out ConfigMaps

ConfigMaps often help the agent spot a wrong setting. If yours hold sensitive values, remove configmaps from the first rule. The agent is then told it isn't allowed, and carries on. Values that look like secrets are removed from everything the agent reads anyway.

Optional: let the agent see Argo CD#

If you deploy with Argo CD, the agent can tell you whether a change has synced and is healthy. Add this to the file. It lets the agent read Argo's applications and nothing else in Argo's namespace. The agent only shows applications that deploy to the namespaces it may see.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: support-agent-argo-read
  namespace: argocd
rules:
  - apiGroups: ["argoproj.io"]
    resources: [applications]
    verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: support-agent-argo-read
  namespace: argocd
subjects:
  - kind: ServiceAccount
    name: support-agent-reader
    namespace: support-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: support-agent-argo-read

The agent looks for Argo CD in the argocd namespace. If yours runs somewhere else, contact us.

Or: read the whole cluster#

If the agent should see every namespace, and nodes too, use a ClusterRoleBinding instead of the RoleBindings. Add this to the file, and leave the RoleBindings out:

# Extra cluster-wide rights: the list of namespaces, and nodes.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: support-agent-read-cluster
rules:
  - apiGroups: [""]
    resources: [namespaces, nodes]
    verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: support-agent-read
subjects:
  - kind: ServiceAccount
    name: support-agent-reader
    namespace: support-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: support-agent-read
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: support-agent-read-cluster
subjects:
  - kind: ServiceAccount
    name: support-agent-reader
    namespace: support-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: support-agent-read-cluster

Per-namespace access is safer. Start there, and widen it only if you need to.

Step 2: Build the kubeconfig#

This script reads the token and the cluster's CA certificate from the Secret you just created, and writes a self-contained kubeconfig. Run it with your usual admin access, pointed at the right cluster.

#!/bin/sh
set -e

SA_NAMESPACE=support-access
SECRET=support-agent-reader-token
DEFAULT_NAMESPACE=shop-prod   # where the agent looks when no namespace is named

SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
CA=$(kubectl -n "$SA_NAMESPACE" get secret "$SECRET" -o jsonpath='{.data.ca\.crt}')
TOKEN=$(kubectl -n "$SA_NAMESPACE" get secret "$SECRET" -o jsonpath='{.data.token}' | base64 --decode)

cat > support-agent-reader.kubeconfig <<EOF
apiVersion: v1
kind: Config
clusters:
  - name: my-cluster
    cluster:
      server: $SERVER
      certificate-authority-data: $CA
users:
  - name: support-agent-reader
    user:
      token: $TOKEN
contexts:
  - name: support-agent-reader
    context:
      cluster: my-cluster
      user: support-agent-reader
      namespace: $DEFAULT_NAMESPACE
current-context: support-agent-reader
EOF

echo "Wrote support-agent-reader.kubeconfig"

If TOKEN comes out empty, wait a few seconds for Kubernetes to fill in the Secret, and run the script again.

With your own runner in this cluster

The agent then works inside the cluster, so it reaches Kubernetes at the cluster's own address. The address in your login often doesn't work from inside. In the script, set SERVER=https://kubernetes.default.svc instead of reading it from your login. To try the file from your computer, add your usual address: --server <your cluster's address>. With a runner, the file is kept by you: Scrambled for your runner as you paste it here (only your runner can unscramble it), or In your store on your runner's side. See Two ways to keep your keys.

Try it before you paste it#

# Should list your pods:
kubectl --kubeconfig support-agent-reader.kubeconfig -n shop-prod get pods

# Should say "no":
kubectl --kubeconfig support-agent-reader.kubeconfig -n shop-prod auth can-i get secrets
kubectl --kubeconfig support-agent-reader.kubeconfig -n shop-prod auth can-i delete pods

Treat the file like a password

Anyone with this file can read your cluster as the agent can. Paste it into Cerberius Agents, then delete the file from your computer.

Step 3: Add it in Cerberius Agents#

1
Go to Settings › Live system. Click Connect a cluster or logs (or Connect another, if Grafana Loki is already there), and choose Kubernetes.
2
In the Cluster login row, click Add cluster login….
3
Paste the whole kubeconfig into the Kubeconfig box, and click Save. It's saved straight away. The row now shows Saved, with the server, the context and how it logs in.
4
In Namespaces it may look at, enter the namespaces, for example shop-prod, payments-prod.
5
Click the Save button under the namespaces.
6
Click Check connection.

Field	What it does
Kubeconfig	The cluster login. It is stored encrypted, and can never be shown again, here or through the API: only replaced or removed. Only its current context is kept, so the agent can't switch to another cluster.
Namespaces it may look at	Shown once a kubeconfig is saved. Separate them with commas. The agent may only look in these namespaces, on top of what the login allows. Leave it empty for everything the login can read. The log watcher also watches these namespaces.

The namespaces have their own Save button, under the field. Check connection checks what is saved, so save first.

The Cluster login row also has a How to make one link. It opens a shorter version of these steps.

What Check connection tells you#

Result	Meaning
Connected, with shop-prod: 12 pods visible	It works. With namespaces set, it lists the pods in each one.
Connected, with N namespaces visible	It works, and no namespaces are set, so it listed every namespace.
Connected, but not allowed to list …	The login works, but has no rights there. Check the RoleBinding for that namespace. If no namespaces are set, set them, or use cluster-wide access.
Note: these credentials can read Secrets	The login is broader than needed. The agent still never reads Secrets, but a narrower login is safer. Use the role above.
This kubeconfig logs in by running …	It uses a login program. Build one with the script above. This also shows in the pop-up when you try to save it.
Could not reach the cluster	The address is wrong, or the cluster's API server isn't reachable from Cerberius Agents.
… points to a private or internal network, which is not allowed	The API server's address is private. Cerberius Agents must reach it on a public address.

Replace or remove it#

Click Replace… in the Cluster login row to paste a new kubeconfig.
Click Remove, and confirm, to stop the agent using your cluster.

To cut off access on the cluster side, delete the token Secret. The token stops working at once:

kubectl -n support-access delete secret support-agent-reader-token

To issue a new token, apply the Secret again, run the script again, and click Replace….

Letting the agent change things#

This login only reads. To let the agent propose cluster changes, such as restarting a deployment, you need a second, separate login. See Cluster changes.