Skip to content

Customize OTel configuration

Customize the OTel configuration to control which telemetry your application sends to Datadog.

Things to consider

Custom metric pricing

Custom metrics can help you understand your business and systems, but they can be expensive. Datadog and most observability vendors treat each unique combination of metric attribute values as a separate metric.

Be conscious of the number of custom metrics you emit. In particular, avoid metric attributes that take on a high number of values (also called high-dimensionality), such as user IDs, IP addresses, and timestamps.

If removing certain metrics or metric attributes at the application layer is difficult, you can have the OTel collector filter them out by customizing the OTel configuration.

Before you begin

Make sure you have:

Filter out specific metrics

stacks/dev/app-example/config_override.tf
locals {
  otel_metric_config = {
    ignored_metrics = [
      # NOTE: We always want to remove the metrics below
      "^http\\.server\\.request\\.duration$",
      "^http\\.client\\.request\\.duration$",
      "^otlp\\.exporter\\.exported$",
      "^otlp\\.exporter\\.seen$",
      "^queueSize$",
      "^processedLogs$",
      "^processedSpans$",
      # TODO: Add the name of the metric you want to remove here
    ]
  }
}

Remove metric attributes

stacks/dev/app-example/config_override.tf
locals {
  # We add a new OTel processor to the OTel configuration
  otel_additional_processors = {
    "transform/remove_metric_attributes" = {
      metric_statements = [
        # Remove an attribute from all metrics
        "delete_key(datapoint.attributes, \"example.attribute\")",
        # Remove attribute only from specific metric
        "delete_key(datapoint.attributes, \"example.attribute\") where metric.name == \"system.memory.usage\""
      ]
    }
  }

  # We add our new processor to the OTel pipeline that contains metrics
  otel_pipeline_processors = {
    "metrics" = {
      processors = ["transform/remove_metric_attributes"]
    }
  }
}

Only keep traces from a specific endpoint

stacks/dev/app-example/config_override.tf
locals {
  # Configure tail sampling to only keep traces from a specific endpoint
  otel_tail_sampling_config = {
    additional_policies = [
      {
        name = "drop-everything-but-one-route"
        type = "drop"
        drop = {
          drop_sub_policy = [
            {
              name = "inverted-match"
              type = "string_attribute"
              string_attribute = {
                key                    = "http.route"
                values                 = ["^/my-route$"]
                enabled_regex_matching = true
                invert_match           = true
              }
            }
          ]
        }
      }
    ]
  }
}

Bring your own OTel Collector configuration

Warning

When you bring your own OTel configuration, you'll lose useful behavior and telemetry tags that the Golden Path provides out-of-the-box. Consider using the otel-config-generator Terraform module to extend the default configuration rather than replacing it, or ask Utviklerflyt to support your use case.

stacks/dev/app-example/config_override.tf
locals {
  otel_config = yamlencode(file("my-config.yml"))
}