Forward CloudWatch and S3 logs to Datadog
This guide will help you set up log forwarding from AWS CloudWatch and S3 (e.g., ALB logs) to Datadog.
Things to consider
Recommended practice
Set up log forwarding in each stack where a log source exists rather than creating a centralized forwarding setup. This approach reduces coupling between stacks and makes it easier to manage permissions and dependencies.
All bucket or log group tags are forwarded to Datadog
As an example, if a log group or S3 bucket logically belongs to a specific application, you might want to add a service
tag to it in AWS. The same tag will then make its way to Datadog and make it easier to query across all relevant logs for a given application.
ALB logs and load-balancing-alb template
If you're using the load-balancing-alb
template, ALB log forwarding is automatically configured for you in newer versions. Check if you need to update your template version and apply the update (both in the ALB stack and associated -data
stack).
Before you begin
Make sure that you have:
- Completed the getting started guide to set up basic Datadog integration.
- A CloudWatch log group or S3 bucket with logs that you want to forward to Datadog.
Step 1: Forward CloudWatch logs
Use the datadog-log-subscription
module to forward CloudWatch logs to Datadog.
Add the following module to your Terraform configuration:
module "forward_to_datadog" {
source = "git@github.com:oslokommune/golden-path-iac//terraform/modules/datadog-log-subscription?ref=datadog-log-subscription-v0.1.1"
environment = local.environment
cloudwatch_sources = [
{
log_group_name = aws_cloudwatch_log_group.application.name
}
]
}
Step 2: Forward S3 logs
For S3, add S3 sources to the same module:
module "forward_to_datadog" {
source = "git@github.com:oslokommune/golden-path-iac//terraform/modules/datadog-log-subscription?ref=datadog-log-subscription-v0.1.1"
environment = local.environment
s3_sources = [
{
bucket_name = aws_s3_bucket.logs.name
}
]
}
Step 3: Apply the configuration
-
Run Terraform to create the log forwarding resources:
Step 4: Verify log forwarding
- Check the Log Explorer in Datadog for incoming log entries. Use the
env
andsource
tags to filter the results. Example search:env:pirates-dev source:(elb OR cloudwatch)
.
Existing log entries are not forwarded
Only log entries that have been created in CloudWatch or S3 after the forwarding was enabled will be sent to Datadog.