Forward CloudWatch and S3 logs to Datadog

This guide will help you set up log forwarding from AWS CloudWatch and S3 (e.g., ALB logs) to Datadog.

Things to consider

Recommended practice

Set up log forwarding in each stack where a log source exists rather than creating a centralized forwarding setup. This approach reduces coupling between stacks and makes it easier to manage permissions and dependencies.

All bucket or log group tags are forwarded to Datadog

As an example, if a log group or S3 bucket logically belongs to a specific application, you might want to add a service tag to it in AWS. The same tag will then make its way to Datadog and make it easier to query across all relevant logs for a given application.

ALB logs and load-balancing-alb template

If you're using the load-balancing-alb template, ALB log forwarding is automatically configured for you in newer versions. Check if you need to update your template version and apply the update (both in the ALB stack and associated -data stack).

Before you begin

Make sure that you have:

Completed the getting started guide to set up basic Datadog integration.
A CloudWatch log group or S3 bucket with logs that you want to forward to Datadog.

Step 1: Forward CloudWatch logs

Use the datadog-log-subscription module to forward CloudWatch logs to Datadog.

Add the following module to your Terraform configuration:

dev/stacks/example-cloudwatch-data/datadog.tf

module "forward_to_datadog" {
  source              = "git@github.com:oslokommune/golden-path-iac//terraform/modules/datadog-log-subscription?ref=datadog-log-subscription-v0.1.1"
  environment         = local.environment
  cloudwatch_sources  = [
    {
      log_group_name  = aws_cloudwatch_log_group.application.name
    }
  ]
}

Step 2: Forward S3 logs

For S3, add S3 sources to the same module:

dev/stacks/example-s3-data/datadog.tf

module "forward_to_datadog" {
  source              = "git@github.com:oslokommune/golden-path-iac//terraform/modules/datadog-log-subscription?ref=datadog-log-subscription-v0.1.1"
  environment         = local.environment
  s3_sources = [
    {
      bucket_name = aws_s3_bucket.logs.name
    }
  ]
}

Step 3: Apply the configuration

Run Terraform to create the log forwarding resources:
```
terraform plan
terraform apply
```

Step 4: Verify log forwarding

Check the Log Explorer in Datadog for incoming log entries. Use the env and source tags to filter the results. Example search: env:pirates-dev source:(elb OR cloudwatch).

Existing log entries are not forwarded

Only log entries that have been created in CloudWatch or S3 after the forwarding was enabled will be sent to Datadog.