Collect Prometheus metrics from ECS services

This guide shows you how to set up AWS OpenTelemetry Collector as a sidecar container² in an Amazon ECS task definition.

What's a sidecar container?

It's a helper container that runs alongside the main application container in the same ECS task.² Just like a sidecar on a motorcycle.

Reference implementation

Here's how the OpenTelemetry Collector is running alongside the too-tikki application in pirates-iac.

Before you begin

Your application must expose metrics in the Prometheus format. Most teams use the Micrometer library to instrument their applications, which has built-in support for Prometheus.

You need to have completed the guide Create a Prometheus workspace.

Reference implementation

Here's how Amazon Managed Service for Prometheus is set up in pirates-iac.

Step 1: Set up the module

Navigate to the stack of the application you want to collect metrics for and download the module template:

ok get-template ecs_otel_collector_sidecar

Step 2: Configure the module

Go through the module template and consider the different variables. The module template is pre-filled with the most common variables, but you can add more if needed. Take a look at the module documentation for a full list of available variables.

Step 3: Apply the configuration

The module itself is non-intrusive. Apply the configuration to see how it minimally affects the stack:

terraform init
terraform apply

Step 4: Add the container definition to your ECS service

The module outputs a container definition that you can use in your ECS service (ecs_service.tf):

container_definitions = {
  main_container = local.main_container
  otel_container = module.aws_opentelemetry_collector.otel_container_definition_terraform_aws_modules_format
}

Step 5: Define container dependencies

Add a dependencies configuration block in the container definition of your main application (ecs_container_definition_main.tf):

dependencies = [
  {
    containerName = module.aws_opentelemetry_collector.otel_container_definition_terraform_aws_modules_format.name
    condition     = "START"
  }
]

This makes OpenTelemetry Collector start before the main application.

Step 6: Add the IAM policies to your ECS task execution role

In ecs_service.tf, add:

task_exec_iam_role_policies = {
  opentelemetry_collector = module.aws_opentelemetry_collector.task_exec_policy_arn
}

Step 7: Add the IAM policies to your ECS task role

In ecs_service.tf, add:

tasks_iam_role_policies = {
  opentelemetry_collector = module.aws_opentelemetry_collector.task_policy_arn
}

Step 8: Apply the final configuration

terraform apply

Verify

1. Sign in to the AWS Management Console.
2. Open Amazon Elastic Container Service.
3. Choose the relevant cluster and service.
4. Go to the Tasks tab.
5. Select a task ID to view its details.
6. Switch to the Logs tab.
7. In the Filter container dropdown, pick aws-otel-collector.

You should see the following log message:

Everything is ready. Begin running and processing data.

Expected error message

This error message usually appears once or twice:

Failed to scrape Prometheus endpoint

Don't worry. This happens because the OpenTelemetry Collector starts up before the application.

---

1. Amazon uses Cortex for its Prometheus Service, unlike Grafana Labs which uses Mimir. Consider Mimir for self-hosting. ↩

2. Microsoft explains the sidecar pattern in their Architecture Center. ↩↩