Forward logs from CloudWatch and S3 to Datadog
This guide shows you how to forward logs from AWS CloudWatch and S3 (like ALB logs) to Datadog.
Before you begin
Make sure that you have:
- Completed the getting started guide to set up basic Datadog integration.
- A CloudWatch log group or S3 bucket with logs that you want to forward to Datadog.
Things to consider
Recommended practice
Set up log forwarding in each stack where a log source exists rather than creating a centralized forwarding setup. This approach reduces coupling between stacks and makes it easier to manage permissions and dependencies.
All bucket or log group tags are forwarded to Datadog
As an example, if a log group or S3 bucket logically belongs to a specific application, you might want to add a service
tag to it in AWS. The same tag will then make its way to Datadog and make it easier to query across all relevant logs for a given application.
ALB logs and load-balancing-alb
template
If you're using the load-balancing-alb
template, ALB log forwarding is automatically configured in newer versions. Check if you need to update your template version and apply the update (both in the ALB stack and associated -data
stack).
Step 1: Forward CloudWatch logs
Use the datadog-log-subscription
module to forward CloudWatch logs. Add it to your Terraform configuration:
module "forward_to_datadog" {
source = "git@github.com:oslokommune/golden-path-iac//terraform/modules/datadog-log-subscription?ref=datadog-log-subscription-v0.1.1"
environment = local.environment
cloudwatch_sources = [
{
log_group_name = aws_cloudwatch_log_group.application.name
}
]
}
Step 2: Forward S3 logs
For S3, add S3 sources to the same module:
module "forward_to_datadog" {
source = "git@github.com:oslokommune/golden-path-iac//terraform/modules/datadog-log-subscription?ref=datadog-log-subscription-v0.1.1"
environment = local.environment
s3_sources = [
{
bucket_name = aws_s3_bucket.logs.name
}
]
}
Step 3: Apply the configuration
Run Terraform to create the log forwarding resources:
Step 4: Verify log forwarding
Check the Log Explorer in Datadog for incoming logs. Use the env
and source
tags to filter results:
Existing log entries aren't forwarded
Only log entries created in CloudWatch or S3 after enabling forwarding will be sent to Datadog.