Skip to content

Common issues

Below are some common errors or issues along with their solutions. If you can't find a solution here the team can help you at #utviklerflyt-support on Slack.

otter-help-small

ACM certificate issuance timeout error

This error happens when DNS validation fails during certificate issuance. The certificate can't reach the ISSUED state, causing a timeout.

Error message
Error: waiting for ACM Certificate (arn:aws:acm:eu-west-1:1234567890:certificate/some-unique-id) to be issued: timeout while waiting for state to become 'ISSUED' (last state: 'PENDING_VALIDATION', timeout: 1h15m0s)

  with module.ecs_service.aws_acm_certificate_validation.cert_validation,
  on .terraform/modules/ecs_service/terraform/modules/ecs_service/dns.tf line 40, in resource "aws_acm_certificate_validation" "cert_validation":
  40: resource "aws_acm_certificate_validation" "cert_validation" {

Solution

Register the name server DNS records in origo-aws-infrastructure, as described in the setup guide.

Docker Hub rate limits

Docker Hub imposes pull rate limits, which can cause issues when using ECS1.

Error message
Stopped reason Cannotpullcontainererror: pull image manifest has been retried 5 time(s): httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/busybox/manifests/sha256:abc: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Solution

Use the Amazon ECR public gallery instead. It has no rate limits and works well with AWS services.

Error when updating ok

Earlier versions of ok required a GitHub token for updating itself to a new version. This could result in the following error:

Error message
Could not download version GITHUB_TOKEN

Solution

  • Create a new classic GitHub token, export to your environment and re-run ok update
  • Make sure to delete your GitHub token once you have the latest version of ok

Note: ok now uses gh for authentication instead of GITHUB_TOKEN. The documentation on ok will take you through how to do this.

Solution

Run Terraform to update the API key:

cd observability
terraform apply

Example output (output truncated for brevity):

  # module.grafana_api_keys.time_static.rotate must be replaced
-/+ resource "time_static" "rotate" {
      ~ day     = 7 -> (known after apply)
      ~ hour    = 11 -> (known after apply)
      ~ id      = "2024-02-07T11:37:31Z" -> (known after apply)
      ~ minute  = 37 -> (known after apply)
      ~ month   = 2 -> (known after apply)
      ~ rfc3339 = "2024-02-07T11:37:31Z" # forces replacement -> (known after apply) # forces replacement
      ~ second  = 31 -> (known after apply)
      ~ unix    = 1707305851 -> (known after apply)
      ~ year    = 2024 -> (known after apply)
    }

You can now go to the stack observability-grafana-settings and run terraform apply without encountering this error.

  1. Advice for customers dealing with Docker Hub rate limits. ↩