Troubleshooting common application errors

This topic describes the errors Terraform Enterprise may report if your deployment is misconfigured and how to resolve them.

Kubernetes Fails to Pull Image

Symptom

Kubernetes pods are failing to pull the container image with a BackOff error.

Signals

kubectl describe pod is stuck in the Waiting state with the ErrImagePull reason.

$ kubectl describe pod terraform-enterprise-7f649f6598-2k79b
...
Containers:
  terraform-enterprise:
    State:          Waiting
      Reason:       ErrImagePull
...

Solution

Update the image pull policy for the deployment to always.

Empty S3 static credentials

Symptom

Application fails to start.

Signals

Logs show the following S3 prefix detection error.

2023-05-10T23:38:18.100Z [ERROR] terraform-enterprise: startup: error="failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, static credentials are empty"

Solution

Set TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE to true when using IAM auth for S3.

Unknown certificate with VCS integration

Symptom

You cannot configure a VCS connection within Terraform Enterprise.

Signals

Setting up VCS fails with unknown certificate issuer error.

Solution

Include the CA certificate for your VCS server in the CA Bundle. Ensure the TFE_TLS_CA_BUNDLE_FILE is set to a path pointing to your CA bundle.

Unknown certificate with failing Terraform runs

Symptom

Terraform plans and applies fail.

Signals

Logs for task worker and archivist show an x509 error.

Solution

Include the CA certificates for all hosts that Terraform must communicate with, including your Terraform Enterprise server itself, in the CA Bundle. Ensure the TFE_TLS_CA_BUNDLE_FILE is set to a path pointing to your CA bundle.

Unable to fetch Terraform binary

Symptom

Terraform plans and applies fail with failed downloading terraform.

Signals

Terraform run logs contain.

Operation failed: failed fetching Terraform: failed downloading terraform: failed downloading "https://releases.hashicorp.com/terraform/1.3.2/terraform_1.3.2_linux_amd64.zip": GET https://releases.hashicorp.com/terraform/1.3.2/terraform_1.3.2_linux_amd64.zip giving up after 5 attempt(s): failed making temp file: open /tmp/terraform/8c23e18ed1846a552fc22ed5ee80eec9.download-67d5219a-aa5c-cd41-3262-2b9d57c1bfe2: read-only file system

Solution

Ensure the TFE_DISK_CACHE_PATH location is properly backed by a writable volume.