# Respond to a Datadog outage

Reference for keeping visibility into your services when Datadog is unavailable. Your services most likely keep running during a Datadog outage—what needs restoring is visibility.

## Before you begin

This page assumes your services are deployed using the Golden Path. Log and metric availability depends on the default configuration it provides.

## What to expect during an outage

Your services most likely keep running. When the `datadog` exporter cannot reach its backend, the collector retries for a few minutes and then drops the telemetry. It never arrives later: traces and custom metrics from the outage window are gone for good, and your dashboards keep a permanent gap for that period.

Nothing tears the task down. The ECS task definition marks the OpenTelemetry Collector container `essential`, so ECS replaces the task only if the collector process exits. An unreachable backend does not make it exit. The container has no health check that could tell ECS to stop it, and no persistent queue that could grow until the container runs out of memory. Your app's `stdout` keeps flowing to CloudWatch throughout.

## What you lose during the outage

CloudWatch does not replace Datadog. For the duration of the outage you lose:

- **Traces.** The Golden Path has no AWS-native tracing fallback.
- **Custom and business metrics** your app sends through the collector.
- **Monitors, alerting, and dashboards.** Nothing pages you during the outage.

## Datadog status

Check [status.datadoghq.eu](https://status.datadoghq.eu/){:target="_blank"} for an active incident on the EU site.

## Signal availability

| Signal                       | In Datadog                                       | In AWS                                     | Available during outage? |
| ---------------------------- | ------------------------------------------------ | ------------------------------------------ | :----------------------: |
| Application logs             | **Logs > Log Explorer**                          | **CloudWatch > Logs > Logs Analytics**     |           Yes            |
| Lambda logs                  | **Logs > Log Explorer**                          | **CloudWatch > Logs > Logs Analytics**     |           Yes            |
| ALB / CloudFront access logs | **Logs > Log Explorer**                          | **S3**—access log bucket                   |           Yes            |
| Infrastructure metrics       | **Dashboards** or **Metrics > Metrics Explorer** | **CloudWatch > Metrics > Classic metrics** |           Yes            |
| Business / custom metrics    | **Dashboards** or **Metrics > Metrics Explorer** | No AWS equivalent                          |          **No**          |
| Traces                       | **APM > Trace Explorer**                         | No AWS equivalent                          |          **No**          |

## Logs

### ECS application logs

#### Why ECS logs survive the outage

Logs only reach CloudWatch if your app is configured to write to `stdout`. The ECS `awslogs` log driver forwards `stdout` to CloudWatch without involving Datadog—the Datadog copy is a downstream subscriber, so the CloudWatch copy survives the outage. If your app sends logs only over OTLP and not to `stdout`, there is no CloudWatch copy to fall back to.

#### Search the ECS logs in CloudWatch

In **CloudWatch > Logs > Logs Analytics**, run:

```text
SOURCE logGroups(namePrefix: ["/<env>/ecs/<service>"], class: "STANDARD") START=-604800s END=0s |
fields @timestamp, @message
| filter @logStream like /ecs\/<service>/
| sort @timestamp desc
```

For example, for the `too-tikki` service in `pirates-dev`:

```text
SOURCE logGroups(namePrefix: ["/pirates-dev/ecs/too-tikki"], class: "STANDARD") START=-604800s END=0s |
fields @timestamp, @message
| filter @logStream like /ecs\/too-tikki/
| sort @timestamp desc
```

<!-- prettier-ignore-start -->
!!! info "The equivalent search in Datadog"
    In **Datadog > Logs > Log Explorer**, when Datadog is available:

    ```text
    env:pirates-dev service:too-tikki
    ```
<!-- prettier-ignore-end -->

### Lambda logs

#### Why Lambda logs survive the outage

Lambda functions write logs to CloudWatch by default, independently of Datadog.

#### Search the Lambda logs in CloudWatch

In **CloudWatch > Logs > Logs Analytics**, run:

```text
SOURCE logGroups(logGroupPattern: "/aws/lambda/<function-name>", class: "STANDARD") START=-604800s END=0s |
fields @timestamp, @message
| sort @timestamp desc
```

<!-- prettier-ignore-start -->
!!! info "The equivalent search in Datadog"
    In **Datadog > Logs > Log Explorer**:

    ```text
    service:<function-name>
    ```

    Or filter by `aws.function_name:<function-name>` in the search bar.
<!-- prettier-ignore-end -->

### ALB and CloudFront access logs

The Golden Path writes ALB and CloudFront access logs to S3. Logging is on by default and no template variable turns it off. Datadog receives its copy from a forwarder Lambda subscribed to the bucket, so the S3 copy is unaffected by the outage.

#### Find the log bucket

Bucket names end in a suffix Terraform generates, so look the name up instead of typing it. Replace `<env>` with your environment, such as `pirates-dev`.

For an ALB, `<alb-name>` is the load balancer name without the environment prefix, such as `main-public`:

```bash
aws ssm get-parameter \
  --name "/<env>/alb/<alb-name>/access_logs_s3_bucket_id" \
  --query Parameter.Value --output text
```

For CloudFront, `<name>` is the site name, such as `zebra`. The parameter holds the bucket domain name, so drop the `.s3.amazonaws.com` suffix to get the bucket name:

```bash
aws ssm get-parameter \
  --name "/<env>/cloudfront-static-website/<name>/logs-s3-bucket-domain-name" \
  --query Parameter.Value --output text
```

If you would rather browse, the names look like `pirates-dev-alb-main-public-logs-<suffix>` and `pirates-dev-cloudfront-zebra-logs-<suffix>`.

#### Find the logs in the bucket

| Source     | Key prefix                                                             | Format                                                                                 |
| ---------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| ALB        | `AWSLogs/<account-id>/elasticloadbalancing/<region>/<yyyy>/<mm>/<dd>/` | Compressed with `gzip`, space-delimited. AWS delivers every 5 minutes.                 |
| CloudFront | `cloudfront-logs/`                                                     | Compressed with `gzip`, tab-separated. AWS delivers with a delay of up to a few hours. |

## Infrastructure metrics

CloudWatch collects these metrics independently of Datadog—they are always available, even during a Datadog outage. Navigate to **CloudWatch > Metrics > Classic metrics**, select a namespace such as **AWS namespaces > ECS**, and click **View automatic dashboard** to get a pre-built view of the most useful metrics.

| Namespace                           | Metrics                                           | Answers                                           |
| ----------------------------------- | ------------------------------------------------- | ------------------------------------------------- |
| **AWS namespaces > ECS**            | `CPUUtilization`, `MemoryUtilization`             | Is the service saturated?                         |
| **AWS namespaces > ApplicationELB** | `HTTPCode_Target_5XX_Count`, `TargetResponseTime` | Does it fail? Is it slow?                         |
| **AWS namespaces > ApplicationELB** | `HealthyHostCount`, `RequestCount`                | Is it up and taking traffic?                      |
| **AWS namespaces > Lambda**         | `Invocations`, `Errors`, `Duration`, `Throttles`  | Are the functions running, failing, or throttled? |
| **AWS namespaces > CloudFront**     | `Requests`, `4xxErrorRate`, `5xxErrorRate`        | Is the edge serving traffic?                      |

## Disable telemetry collection

Only do this if ECS tasks are cycling because the collector container itself is failing, not because Datadog is unreachable.

A Datadog outage by itself is not a reason to disable the OTel Collector, but the collector container is marked `essential` in the ECS task definition. If the collector is crash-looping (for example due to a bad configuration or a bug), ECS stops and replaces the entire task, taking your application down with it. Disabling telemetry collection requires a template reinstall, a `terraform apply`, and a redeploy mid-incident, and it improves neither log nor metric availability.

1. Turn off telemetry collection in `package-config.yml`:

   ```yml title="package-config.yml"
   TelemetryCollection:
     Enable: false
   ```

1. Reinstall the template and apply:

   ```sh
   ok pkg install
   terraform plan && terraform apply
   ```

Re-enable telemetry collection the same way once Datadog recovers.