Respond to a Datadog outage

Reference for keeping visibility into your services when Datadog is unavailable. Your services most likely keep running during a Datadog outage—what needs restoring is visibility.

Before you begin

This page assumes your services are deployed using the Golden Path. Log and metric availability depends on the default configuration it provides.

What to expect during an outage

Your services most likely keep running. When the datadog exporter cannot reach its backend, the collector retries for a few minutes and then drops the telemetry. It never arrives later: traces and custom metrics from the outage window are gone for good, and your dashboards keep a permanent gap for that period.

Nothing tears the task down. The ECS task definition marks the OpenTelemetry Collector container essential, so ECS replaces the task only if the collector process exits. An unreachable backend does not make it exit. The container has no health check that could tell ECS to stop it, and no persistent queue that could grow until the container runs out of memory. Your app's stdout keeps flowing to CloudWatch throughout.

What you lose during the outage

CloudWatch does not replace Datadog. For the duration of the outage you lose:

• Traces. The Golden Path has no AWS-native tracing fallback.
• Custom and business metrics your app sends through the collector.
• Monitors, alerting, and dashboards. Nothing pages you during the outage.

Datadog status

Check status.datadoghq.eu for an active incident on the EU site.

Signal availability

Logs

ECS application logs

Why ECS logs survive the outage

Logs only reach CloudWatch if your app is configured to write to stdout. The ECS awslogs log driver forwards stdout to CloudWatch without involving Datadog—the Datadog copy is a downstream subscriber, so the CloudWatch copy survives the outage. If your app sends logs only over OTLP and not to stdout, there is no CloudWatch copy to fall back to.

Search the ECS logs in CloudWatch

In CloudWatch > Logs > Logs Analytics, run:

SOURCE logGroups(namePrefix: ["/<env>/ecs/<service>"], class: "STANDARD") START=-604800s END=0s |
fields @timestamp, @message
| filter @logStream like /ecs\/<service>/
| sort @timestamp desc

For example, for the too-tikki service in pirates-dev:

SOURCE logGroups(namePrefix: ["/pirates-dev/ecs/too-tikki"], class: "STANDARD") START=-604800s END=0s |
fields @timestamp, @message
| filter @logStream like /ecs\/too-tikki/
| sort @timestamp desc

env:pirates-dev service:too-tikki

Lambda logs

Why Lambda logs survive the outage

Lambda functions write logs to CloudWatch by default, independently of Datadog.

Search the Lambda logs in CloudWatch

In CloudWatch > Logs > Logs Analytics, run:

SOURCE logGroups(logGroupPattern: "/aws/lambda/<function-name>", class: "STANDARD") START=-604800s END=0s |
fields @timestamp, @message
| sort @timestamp desc

service:<function-name>

ALB and CloudFront access logs

The Golden Path writes ALB and CloudFront access logs to S3. Logging is on by default and no template variable turns it off. Datadog receives its copy from a forwarder Lambda subscribed to the bucket, so the S3 copy is unaffected by the outage.

Find the log bucket

Bucket names end in a suffix Terraform generates, so look the name up instead of typing it. Replace <env> with your environment, such as pirates-dev.

For an ALB, <alb-name> is the load balancer name without the environment prefix, such as main-public:

aws ssm get-parameter \
  --name "/<env>/alb/<alb-name>/access_logs_s3_bucket_id" \
  --query Parameter.Value --output text

For CloudFront, <name> is the site name, such as zebra. The parameter holds the bucket domain name, so drop the .s3.amazonaws.com suffix to get the bucket name:

aws ssm get-parameter \
  --name "/<env>/cloudfront-static-website/<name>/logs-s3-bucket-domain-name" \
  --query Parameter.Value --output text

If you would rather browse, the names look like pirates-dev-alb-main-public-logs-<suffix> and pirates-dev-cloudfront-zebra-logs-<suffix>.

Find the logs in the bucket

Infrastructure metrics

CloudWatch collects these metrics independently of Datadog—they are always available, even during a Datadog outage. Navigate to CloudWatch > Metrics > Classic metrics, select a namespace such as AWS namespaces > ECS, and click View automatic dashboard to get a pre-built view of the most useful metrics.

Disable telemetry collection

Only do this if ECS tasks are cycling because the collector container itself is failing, not because Datadog is unreachable.

A Datadog outage by itself is not a reason to disable the OTel Collector. However, the collector container is marked essential in the ECS task definition. If the collector is crash-looping (for example due to a bad configuration or a bug), ECS stops and replaces the entire task, taking your application down with it. Disabling telemetry collection requires a template reinstall, a terraform apply, and a redeploy mid-incident, and it improves neither log nor metric availability.

1. Turn off telemetry collection in package-config.yml:

TelemetryCollection:
  Enable: false

1. Reinstall the template and apply:

ok pkg install
terraform plan && terraform apply

Re-enable telemetry collection the same way once Datadog recovers.