Datadog logs

Connect your Datadog account so the agent can search your logs, with an API key and an application key that can only read logs.

In the appPlatformSettingsLive systemDatadog

With your logs, the agent can see what happened when the requester hit the problem: the error, the request, the time. If your logs are in Datadog, it searches them there.

The agent only reads logs. It can search up to the last 7 days, and gets at most 1,000 entries per search. Values that look like secrets are removed from every entry before the agent sees it.

The log watcher uses the same connection.

One log source per platform

A platform reads its logs from one place: Loki, Elasticsearch, OpenSearch or Datadog. To switch, click Disconnect under the one that is connected, then choose the other.

Connect it#

1
Go to Settings › Live system. Click Connect a cluster or logs (or Connect another), and choose Datadog. Its Logs row shows under the card.
2
Choose the Site your account is on. The address you sign in to Datadog at says which.
3
Enter the API key and the Application key.
4
If the account holds other systems' logs too, enter a filter in Only this platform's logs.
5
Describe your logs in How your logs are labelled.
6
Click Save, under the fields.
7
Click Check connection.

Field	What it does
Site	Where your account is: US1, US3, US5, EU, AP1, AP2 or US1-FED. Searches go only to that site's address.
Only this platform's logs	A Datadog search every search gets, for example `service:booking env:prod`. Empty: all of the account's logs.
API key	From Organization settings › API keys. Stored encrypted and never shown again.
Application key	From Organization settings › Application keys. Stored encrypted and never shown again.
Indexes	Optional. Which log indexes to search, separated with commas. Empty: all.
How your logs are labelled	Notes that help the agent write good searches.

If the platform lists namespaces (on the Kubernetes card of the same page), every search is also limited to them, with Datadog's kube_namespace tag.

Every search is put in brackets next to your filter and namespaces, so it can only narrow them. So a filter or search can't have a bracket it doesn't close (or close one it didn't open), an unclosed quote, or a backslash before a quote or bracket. Saving one is refused and says why.

Check connection checks the API key, then searches the last 15 minutes with the application key, and says when the newest entry was. It says which key Datadog refused, if one is wrong.

With your own runner

When the agent runs on your own runner, the two keys are kept by you. Save the settings first. Then choose how each is kept: Scrambled for your runner or In your store. It needs runner 0.12.0 or later.

Make keys that can only read logs#

1
In Datadog, open Organization settings › Service accounts, and make one for Cerberius Agents with a role that can read logs (the Datadog Read Only Role, or your own role with logs_read_data).
2
Under the service account, create an Application key. Limit its scopes to logs_read_data if you want.
3
Create an API key under Organization settings › API keys, or use an existing one.

What it can and can't do#

The agent writes only the words to search for. Cerberius Agents adds the time range, your filter and your namespaces. It only ever calls Datadog's key check (/api/v1/validate) and its log search (/api/v2/logs/events/search). Nothing is changed in Datadog.

Datadog limits how often logs can be searched, so the log watcher makes one search each time it looks, and takes each error as Datadog has it. Logs with a stack trace usually arrive as one entry, so nothing is lost.

How the agent searches#

During an investigation, the agent searches with Datadog's syntax, for example service:booking-api status:error "invoice" or @http.status_code:>=500. It picks the time range, up to the last 7 days, and up to 1,000 entries at a time. Every search shows in the agent's steps on the reply page.

Change or remove it#

Change any field and click Save. Another Site removes the saved keys when you save: enter them again. To stop using it, click Disconnect under it: its settings and keys are removed from this platform.