Loki logs

With your logs, the agent can see what happened when the requester hit the problem: the error, the request, the time. It searches them in Loki, the log store from Grafana, using LogQL, Loki's query language.

The agent only reads logs. It can search up to the last 7 days, and gets at most 1,000 lines per search. Values that look like secrets are removed from every line before the agent sees it.

The log watcher uses the same connection.

Connect Loki#

1. 1

   Go to Settings › Live system. Click Connect a cluster or logs (or Connect another, if Kubernetes is already there), and choose Grafana Loki. Its Logs row shows under the card.

2. 2

   Enter the Loki address.

3. 3

   Choose the Login, and fill in the username, password or token it asks for.

4. 4

   If your Loki is multi-tenant, enter the Tenant / org ID.

5. 5

   Describe your labels in How your logs are labelled.

6. 6

   Click Save, under the fields.

7. 7

   Click Check connection.

This Save button saves only the logs settings. The cluster's namespaces have their own, on the Kubernetes card.

Check connection appears once an address is saved, and checks what is saved. It asks Loki for its labels and shows them, for example Labels: app, container, namespace, pod.

Give the agent a login#

Loki itself has no users. Access is usually controlled by whatever sits in front of it. Pick the case that matches yours.

Grafana Cloud#

1. 1

   In Grafana Cloud, open your stack and find the Loki details. Copy the URL and the User (a number).

2. 2

   Go to Administration › Users and access › Cloud access policies, and create a policy with only the logs:read scope, limited to this stack.

3. 3

   Add a token to the policy, and copy it.

4. 4

   In Cerberius Agents, choose Username and password. Enter the user number as Username, and the token as Password.

Leave Tenant / org ID empty for Grafana Cloud.

Your own Loki, behind a login#

If your Loki runs in your own cluster, put it behind an address Cerberius Agents can reach, with a login. For example, with the ingress-nginx controller and basic authentication:

# Create a user name and password (you'll be asked for the password):
htpasswd -c auth support-agent

# Store them in the cluster, next to Loki:
kubectl -n monitoring create secret generic loki-basic-auth --from-file=auth

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: loki-read
  namespace: monitoring
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: loki-basic-auth
    nginx.ingress.kubernetes.io/auth-realm: "Loki"
spec:
  ingressClassName: nginx
  tls:
    - hosts: [loki.example.com]
      secretName: loki-example-tls
  rules:
    - host: loki.example.com
      http:
        paths:
          # Only the read paths the agent uses: queries and labels.
          - path: /loki/api/v1/query_range
            pathType: Exact
            backend:
              service: { name: loki-gateway, port: { number: 80 } }
          - path: /loki/api/v1/labels
            pathType: Exact
            backend:
              service: { name: loki-gateway, port: { number: 80 } }
          - path: /loki/api/v1/label
            pathType: Prefix
            backend:
              service: { name: loki-gateway, port: { number: 80 } }

Change loki-gateway and the port to your Loki's read service. Then choose Username and password in Cerberius Agents, with support-agent and the password you chose.

If your gateway checks a token instead, choose Bearer token. It is sent as Authorization: Bearer <token>.

Describe your labels#

Labels are how Loki groups log lines, for example namespace="shop-prod" or app="api". The agent reads your notes before it searches, so it can go straight to the right logs. Say:

• which label names each app, and what the apps are called
• which label holds the namespace
• how errors are marked
• which details appear in log lines, such as user emails or order IDs

For example:

namespace=shop-prod for the API and workers, namespace=web-prod for the dashboard.
app=<service name>: api, worker, web. Errors contain level=error.
Requests are logged with the merchant's account ID as account=acc_….

The agent can also ask Loki for its labels and their values when it needs them.

How the agent searches#

During an investigation, the agent:

• lists your labels, or the values of one label, to find the right logs
• runs LogQL searches over a time range it chooses, up to the last 7 days, and up to 1,000 lines at a time
• combines what it finds with the code and, if connected, the Kubernetes and database evidence

Every search shows in the agent's steps on the reply page.

Change or remove it#

Change any field and click Save. To stop the agent using Loki, clear Loki address and save.