Graylog logs
Let the agent search your logs in Graylog or Graylog Cloud, with an access token for a user that can only read the streams it needs.
With your logs, the agent can see what happened when the requester hit the problem: the error, the request, the time. If your logs are in Graylog (5.1 or later, your own or Graylog Cloud), it searches them there.
The agent only reads logs. It writes only the words to search for. Cerberius Agents writes the whole Graylog query itself, so the agent's words never become part of it. It can search up to the last 7 days, and gets at most 1,000 entries per search. Values that look like secrets are removed from every entry before the agent sees it.
The log watcher uses the same connection.
One log source per platform
A platform reads its logs from one place: Loki, Elasticsearch, OpenSearch, Datadog, Better Stack, ClickHouse, Graylog, CloudWatch, Azure Monitor or Google Cloud Logging. To switch, click Disconnect under the one that is connected, then choose the other.
Connect Graylog#
- 1
In Graylog, make a user for Cerberius Agents with the Reader role, and share with it only the streams the agent should read. Open System › Users and Teams, that user, then Edit tokens, and create a token. Copy it: it is shown only once.
- 2
Go to Settings › Live system. Click Connect a cluster or logs (or Connect another), and choose Graylog. Its Logs row shows under the card.
- 3
Enter Graylog's address: the one you open it at, for example
https://graylog.example.com. If Graylog is served under a path, add it, for examplehttps://example.com/graylog. - 4
Leave Sign in with on Access token, and paste the token.
- 5
Optional: enter the Streams to search, one ID per line. Empty: every stream the user can read.
- 6
Describe your logs in How your logs are labelled.
- 7
Click Save, under the fields.
- 8
Click Check connection. It says Graylog's version, the streams, and the newest entry.
| Field | What it does |
|---|---|
| Graylog address | Graylog's own address, with its path if it has one. It must be reachable from the internet, unless your organisation may use internal addresses. |
| Sign in with | Access token (best), or Username and password. Graylog's access token is sent as Graylog asks: the token as the username, the word token as the password. It is stored encrypted and never shown again. |
| Streams | Optional. The IDs of the streams to search: the part after /streams/ in a stream's address in Graylog. Empty: every stream the user can read. Check connection lists them, with their names, when none are chosen. |
| Field names | Optional. Which field holds the message, level, namespace, pod, container, service and host. The defaults fit Graylog's own fields and Fluent Bit's Kubernetes ones: message, level, kubernetes_namespace_name, kubernetes_pod_name, kubernetes_container_name, application_name and source. |
| How your logs are labelled | Notes that help the agent write good searches. |
Field names can hold only letters, digits, _, ., @ and -. Anything else is refused when you
save.
Namespaces#
If the platform lists namespaces (on the Kubernetes card of the same page), every search, and the check, is also limited to them. Several platforms can then share one Graylog: only entries of these namespaces are read, and an entry that names no namespace is left out.
What it can and can't do#
Every search is one request to Graylog's search API (POST /api/search/messages, its Search
Scripting API). Cerberius Agents writes the query: each word, and each value it narrows by, is a
quoted phrase, so no word can change the query. The streams are sent beside the query, never in
it, so a search can only narrow them. Check connection also reads Graylog's version
(/api/) and the streams the user can read (/api/streams). Nothing else is ever called, and
nothing is ever changed.
An error is a level of 3 or less, as syslog and GELF count them (3 is error; 0, 1 and 2 are worse).
An entry whose level isn't such a number is found by words like "error" or "exception" in its
message. If it has a level written as a word, that word then decides: ERROR stays, INFO is
left out.
With your own runner
When the agent runs on your own runner, the token is kept by
you. Save the settings first. Then choose how it is kept: Scrambled for your runner or In
your store. In your store, give the whole header: Basic and the token and :token in base64.
It needs runner 0.18.0 or later.
How the agent searches#
During an investigation, the agent searches with plain words or "quoted phrases". An entry's
message must have all of them, as whole words, in any case. To keep to one source, it adds
pod="api-1" (or namespace, container, service_name, host): these are matched with your
fields, not as words. A ticket from the log watcher suggests a search written that way. It picks
the time range, up to the last 7 days, and up to 1,000 entries at a time. Every search shows in
the agent's steps on the reply page.
Change or remove it#
Change any field and click Save. Another address removes the saved token when you save: enter it again. To stop using it, click Disconnect under it: its settings and token are removed from this platform.