Skip to content

Common issues

Below are some common errors or issues along with their solutions. If you can't find a solution here the team can help you at #origo-kjøremiljø-support on Slack.

otter-help-small

ACM certificate issuance timeout error

This error arises if DNS validation fails to resolve as the certificate issuance process unfolds. Consequently, the certificate cannot transition to the ISSUED state, resulting in a timeout.

Error message
Error: waiting for ACM Certificate (arn:aws:acm:eu-west-1:1234567890:certificate/some-unique-id) to be issued: timeout while waiting for state to become 'ISSUED' (last state: 'PENDING_VALIDATION', timeout: 1h15m0s)

  with module.ecs_service.aws_acm_certificate_validation.cert_validation,
  on .terraform/modules/ecs_service/terraform/modules/ecs_service/dns.tf line 40, in resource "aws_acm_certificate_validation" "cert_validation":
  40: resource "aws_acm_certificate_validation" "cert_validation" {

Solution

Register the name server DNS records in origo-aws-infrastructure, as described in the setup guide.

Docker Hub rate limits

Docker Hub imposes pull rate limits, which can cause issues when using ECS1.

Error message
Stopped reason Cannotpullcontainererror: pull image manifest has been retried 5 time(s): httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/busybox/manifests/sha256:abc: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Solution

To prevent this issue, you can opt to use the Amazon ECR public gallery, which is not subject to rate limits and is seamlessly integrated with various other AWS services.

Error when updating ok

Earlier versions of ok required a GitHub token for updating itself to a new version. This could result in the following error:

Error message
Could not download version GITHUB_TOKEN

Solution

  • Create a new classic GitHub token, export to your environment and re-run ok update
  • Make sure to delete your GitHub token once you have the latest version of ok

Note: the ok tool have moved away from using GITHUB_TOKEN and now rely on gh for authenticating your session when updating ok. The documentation on ok will take you through how to do this.

Expired Grafana API key

When running terraform apply in the stack created in the guide Configure Grafana, you get an error message similar to the following:

Error message
Error: error reading datasource with ID `1:2`: [GET /datasources/{id}][401] getDataSourceByIdUnauthorized {"message":"Expired API key"}

  with grafana_data_source.prometheus,
  on grafana_settings.tf line 1, in resource "grafana_data_source" "prometheus":
   1: resource "grafana_data_source" "prometheus" {

Solution

Run Terraform to update the API key:

cd observability
terraform apply

Example output (output truncated for brevity):

  # module.grafana_api_keys.time_static.rotate must be replaced
-/+ resource "time_static" "rotate" {
      ~ day     = 7 -> (known after apply)
      ~ hour    = 11 -> (known after apply)
      ~ id      = "2024-02-07T11:37:31Z" -> (known after apply)
      ~ minute  = 37 -> (known after apply)
      ~ month   = 2 -> (known after apply)
      ~ rfc3339 = "2024-02-07T11:37:31Z" # forces replacement -> (known after apply) # forces replacement
      ~ second  = 31 -> (known after apply)
      ~ unix    = 1707305851 -> (known after apply)
      ~ year    = 2024 -> (known after apply)
    }

You can now go to the stack observability-grafana-settings and run terraform apply without encountering this error.

  1. Advice for customers dealing with Docker Hub rate limits. ↩