Skip to content

Forward logs from CloudWatch and S3 to Datadog

This guide shows you how to forward logs from AWS CloudWatch and S3 (like ALB logs) to Datadog.

Before you begin

Make sure that you have:

  • Completed the getting started guide to set up basic Datadog integration.
  • A CloudWatch log group or S3 bucket with logs that you want to forward to Datadog.

Things to consider

Recommended practice

Set up log forwarding in each stack where a log source exists rather than creating a centralized forwarding setup. This approach reduces coupling between stacks and makes it easier to manage permissions and dependencies.

All bucket or log group tags are forwarded to Datadog

As an example, if a log group or S3 bucket logically belongs to a specific application, you might want to add a service tag to it in AWS. The same tag will then make its way to Datadog and make it easier to query across all relevant logs for a given application.

ALB logs and load-balancing-alb template

If you're using the load-balancing-alb template, ALB log forwarding is automatically configured in newer versions. Check if you need to update your template version and apply the update (both in the ALB stack and associated -data stack).

Step 1: Forward CloudWatch logs

Use the datadog-log-subscription module to forward CloudWatch logs. Add it to your Terraform configuration:

dev/stacks/example-cloudwatch-data/datadog.tf
module "forward_to_datadog" {
  source              = "git@github.com:oslokommune/golden-path-iac//terraform/modules/datadog-log-subscription?ref=datadog-log-subscription-v0.1.1"
  environment         = local.environment
  cloudwatch_sources  = [
    {
      log_group_name  = aws_cloudwatch_log_group.application.name
    }
  ]
}

Step 2: Forward S3 logs

For S3, add S3 sources to the same module:

dev/stacks/example-s3-data/datadog.tf
module "forward_to_datadog" {
  source              = "git@github.com:oslokommune/golden-path-iac//terraform/modules/datadog-log-subscription?ref=datadog-log-subscription-v0.1.1"
  environment         = local.environment
  s3_sources = [
    {
      bucket_name = aws_s3_bucket.logs.name
    }
  ]
}

Step 3: Apply the configuration

Run Terraform to create the log forwarding resources:

terraform plan
terraform apply

Step 4: Verify log forwarding

Check the Log Explorer in Datadog for incoming logs. Use the env and source tags to filter results:

env:pirates-dev source:(elb OR cloudwatch)

Existing log entries aren't forwarded

Only log entries created in CloudWatch or S3 after enabling forwarding will be sent to Datadog.