Create a Datadog monitor with `pup`

You can create Datadog monitors in the Datadog UI or with the pup CLI. This guide shows how to create one with pup. By the end, your Slack channel gets a message when your service logs errors.

Before you begin

Make sure you have:

Completed the getting started guide to set up basic Datadog integration, so that the data from your service arrives in Datadog.
Installed pup, logged in without --read-only, and set the EU site (datadoghq.eu) in the pup configuration file.
Invited the Datadog app to the Slack channel that should receive the alerts.

Step 1: Write the monitor definition

A monitor definition is the JSON body of the Datadog monitor API. This one alerts when your service logs more than ten errors in five minutes:

error-logs.json

{
  "name": "[<env>] High error log count on <service>", // (1)!
  "type": "log alert",
  "query": "logs(\"env:<env> service:<service> status:error\").index(\"*\").rollup(\"count\").last(\"5m\") > 10",
  "message": "<service> logged more than 10 errors in the last 5 minutes. @slack-<channel>", // (2)!
  "tags": ["env:<env>", "service:<service>"],
  "options": {
    "thresholds": { "critical": 10 },
    "notify_no_data": false
  }
}

Replace <env> with the environment name and <service> with the application name. Your telemetry carries them in the env and service tags.
Replace <channel> with the Slack channel the Datadog app is invited to, without the #.

To start from one of your team's existing monitors instead, for example one created in the Datadog UI, fetch its definition and remove the fields Datadog sets itself, such as id, created, modified, creator, overall_state, and deleted:

pup --no-agent monitors get <id> > error-logs.json

Step 2: Create the monitor

pup monitors create --file error-logs.json --jq .id

The command prints the id of the new monitor. Every later command needs it.

Step 3: Verify

Get the new monitor and list your service's monitors:

pup monitors get <id>
pup monitors list --tags env:<env>,service:<service>

Open https://app.datadoghq.eu/monitors/<id> to see the monitor's status and the error count it watches.

Step 4: Update the monitor

Edit error-logs.json, compare it with the live monitor, and push the change:

pup monitors diff <id> error-logs.json
pup monitors update --file error-logs.json <id>

Read the diff before you push. It shows the changes made in the UI since the file was last applied, and the update replaces them.

Create a Datadog monitor with pup