Skip to content

Customize OTel configuration

Customize the OTel configuration to control which telemetry your application sends to Datadog.

Things to consider

Custom metric pricing

While custom metrics that you emit from your application can be valuable for business and system insights, they can be somewhat expensive as Datadog and most observability vendors treat each unique combination of metric attribute values as separate metrics. For this reason you should be concious of the number of custom metrics you emit, and especially avoid metric attributes that take on a high number of values (also called high-dimensionality) such as user IDs, IP addresses and timestamps. If it's difficult to remove certain metrics or metric attributes at the application layer, you can instead have the OTel collector filter them out by customizing the OTel configuration.

Before you begin

Make sure you have:

Filter out specific metrics

stacks/dev/app-example/config_override.tf
locals {
  otel_metric_config = {
    ignored_metrics = [
      # NOTE: We always want to remove the metrics below
      "^http\\.server\\.request\\.duration$",
      "^http\\.client\\.request\\.duration$",
      "^otlp\\.exporter\\.exported$",
      "^otlp\\.exporter\\.seen$",
      "^queueSize$",
      "^processedLogs$",
      "^processedSpans$",
      # TODO: Add the name of the metric you want to remove here
    ]
  }
}

Remove metric attributes

stacks/dev/app-example/config_override.tf
locals {
  # We add a new OTel processor to the OTel configuration
  otel_additional_processors = {
    "transform/remove_metric_attributes" = {
      metric_statements = [
        # Remove an attribute from all metrics
        "delete_key(datapoint.attributes, \"example.attribute\")",
        # Remove attribute only from specific metric
        "delete_key(datapoint.attributes, \"example.attribute\") where metric.name == \"system.memory.usage\""
      ]
    }
  }

  # We add our new processor to the OTel pipeline that contains metrics
  otel_pipeline_processors = {
    "metrics" = {
      processors = ["transform/remove_metric_attributes"]
    }
  }
}

Only keep traces from a specific endpoint

stacks/dev/app-example/config_override.tf
locals {
  # Configure tail sampling to only keep traces from a specific endpoint
  otel_tail_sampling_config = {
    additional_policies = [
      {
        name = "drop-everything-but-one-route"
        type = "drop"
        drop = {
          drop_sub_policy = [
            {
              name = "inverted-match"
              type = "string_attribute"
              string_attribute = {
                key                    = "http.route"
                values                 = ["^/my-route$"]
                enabled_regex_matching = true
                invert_match           = true
              }
            }
          ]
        }
      }
    ]
  }
}

Bring your own OTel Collector configuration

Warning

When you bring your own OTel configuration, you'll lose useful behavior and telemetry tags that the Golden Path provides out-of-the-box. Consider using the otel-config-generator Terraform module to extend the default configuration rather than replacing it completely, or ask Kjøremiljø to support your use case.

stacks/dev/app-example/config_override.tf
locals {
  otel_config = yamlencode(file("my-config.yml"))
}