Log watcher

The log watcher finds errors in your logs before anyone reports them. Every 5 minutes it reads this platform's new log lines from Loki and looks for errors: crashes, exceptions, stack traces, null-pointer errors. Each distinct error becomes one ticket, and the agent investigates it at once.

It never replies to anyone outside your team. The outcome is the agent's findings and, if you approve one, a fix.

The label at the top of the page says how it stands: Watching, Off, or Can't read the logs when the last check failed.

What you need first#

• Loki connected. Loki is the store your logs are kept in. Connect it under Settings › Live system › Grafana Loki. See Loki logs. Until it is, the page shows Your logs are not connected.
• The namespaces to watch. By default the watcher reads the namespaces listed in Namespaces it may look at, under Settings › Live system › Kubernetes. See Kubernetes access. If you don't use Kubernetes, write your own search instead (see Your own search, below).

Switch it on#

1. 1

   Go to Settings › Log watcher.

2. 2

   Under Watching, switch on Watching this platform's logs.

3. 3

   Under Daily limit, set New errors per day.

4. 4

   Choose what happens When an error goes quiet.

5. 5

   Click Save at the bottom.

The first check runs within 5 minutes.

Only owners and admins can change these settings.

How it decides what is an error#

A line counts as an error when it:

• is marked with the level error, fatal or critical, as level=error or as JSON "level": "error"
• contains the word ERROR, FATAL or CRITICAL in capitals
• starts a Python traceback (Traceback (most recent call last))
• names an exception or error, such as ValueError: or NullPointerException
• is a Go panic (panic:)

Lines that say they are INFO, DEBUG or TRACE are ignored, even if they mention an error. For example, a logged web request whose address contains the word "error".

Lines from the same pod within 3 seconds of each other count as one occurrence. That keeps a stack trace together as one error.

One ticket per error#

The watcher recognises the same error when it happens again, even if the time, IDs or numbers in the message differ. It matches on where the error came from (the container or app) and the error message.

The ticket shows the log lines, with secrets removed, and where they came from.

Held tickets#

When the daily limit is reached, new errors still get a ticket, but no investigation. The ticket is Not started, and its Now says Start the agent, with Held back: the log watcher's daily limit was reached. Click Start the agent on the ticket when you're ready, or start several at once from Tickets (status Not started).

Approving the results#

Log-watcher tickets have no requester. On a platform with no ticket system, approving the agent's reply only records your decision and marks the ticket done. Nothing is sent anywhere. The useful part is the agent's findings, and any fix it proposes. See Approving a fix.

Your own search#

By default the watcher searches this platform's namespaces for the error lines described above. Under What to watch, click Use your own search (LogQL). While the box is empty, What it searches now shows exactly what it searches.

To search something else, write your own search in Your own search (LogQL), up to 2,000 characters. LogQL is Loki's search language. Your search replaces the default completely, including the error filter. For example:

{namespace="shop-prod", app=~"api|worker"} |~ "(?i)error|exception|panic"

Leave it empty to go back to the default. Hide closes the box again while it is empty.

Last check#

The strip under the title shows when the watcher Last checked, and what it found then:

Before the first check, it says Not checked yet. The first check runs within 5 minutes.

If the last check failed, The last check failed shows under the strip with the reason. Common reasons are that Loki isn't connected yet, or that no namespaces are chosen to watch. If the reason is about Loki, the box links to Settings › Live system.