✦ TIL

When validating an ACM certificate via DNS, AWS polls for a specific CNAME value. If that record is proxied through Cloudflare (orange cloud), Cloudflare intercepts the request and AWS sees Cloudflare’s IP instead of the record value it’s looking for. Validation hangs indefinitely with no useful error message.

The fix in Terraform:

resource "cloudflare_record" "acm_validation" {
  ...
  proxied = false
}

DNS-only on the validation record. You can proxy the actual API subdomain all you want, but the _acme-challenge CNAME has to be unproxied or the cert never issues.

In Cloudflare’s Terraform provider v4, the value argument on cloudflare_record was deprecated in favor of content. Using value still works but generates warnings on every plan and will break in v5.

# before (v3 style)
resource "cloudflare_record" "api" {
  value = aws_api_gateway_domain_name.this.regional_domain_name
}

# after (v4+)
resource "cloudflare_record" "api" {
  content = aws_api_gateway_domain_name.this.regional_domain_name
}

Easy fix, but easy to miss if you’re copying examples from older blog posts or the provider’s own older docs.

If you try to terraform destroy an API Gateway stack that has a custom domain mapping attached, AWS throws an error and refuses to delete the stage. The custom domain mapping has to be removed first.

This is a good argument for keeping domain infrastructure in a separate Terraform stack with its own state. Destroy the domain stack first, then the app stack, and AWS has no objections.

# correct order
cd infra/envs/dev-domain && terraform destroy
cd infra/envs/dev       && terraform destroy

Trying to do it in a single stack forces you to either manually detach the mapping in the console or add explicit depends_on and destroy ordering hacks to your Terraform config.

Modern Docker Desktop enables BuildKit by default. BuildKit attaches provenance attestations to images, producing an OCI image manifest. Lambda only accepts Docker manifest v2 schema 2 and will silently fail when it tries to pull an OCI manifest.

The failure mode is sneaky: docker push succeeds, the image looks fine in ECR, but Lambda errors on deploy with a cryptic ResourceNotFoundException.

Fix is one flag:

docker build --provenance=false -t my-image .

Worth adding this to any build script that targets Lambda so you don’t chase the same ghost twice.

AWS and HashiCorp stepped it up and now it’s finally time to ditch the DynamoDB table to handle state concurrency. This is officially supported on Terraform 1.11.0 and higher.

Now simply enable versioning on the bucket, then, configure your backend in Terraform to use S3 with native locking enabled.

To enable S3 state locking, use the following optional argument:

• use_lockfile Whether to use a lockfile for locking the state file. Defaults to false.

terraform {
  required_version = "~> 1.11.0"

  backend "s3" {
    bucket       = "s3-native-lock-setup"
    key          = "backend/terraform.tfstate"
    region       = "eu-north-1"
    profile      = "ProfileForAccount-12345678910"
    use_lockfile = true
  }
}

Check out the whole reference at developer.hashicorp.com.

IMDSv2 (Instance Metadata Service v2) requires a session-oriented token before any metadata call — a defense against SSRF attacks that plagued IMDSv1. The two-step is easy to forget when writing quick debug scripts.

# Step 1 — get a token (TTL in seconds)
TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Step 2 — use it
curl -sH "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/

Worth enforcing IMDSv2-only on all instances via instance metadata options — HttpTokens: required. Anything still using v1 is a misconfiguration waiting to be a finding.

Conventional wisdom says systemd boots faster because of parallel service startup. True on bloated installs. On a minimal Devuan box with ~30 services, sysvinit cold-boots to login in under 4 seconds — competitive with systemd on the same hardware because there’s simply less to start.

The real win is predictability. Boot sequence is a shell script you can read top to bottom. No unit dependency graph to debug, no systemctl list-dependencies --reverse spelunking when something breaks at 2am.

# Everything that runs at boot, in order, no surprises
ls /etc/rc2.d/

If you’ve never read an init script, /etc/init.d/networking is a good place to start. It’s just bash.