Terraform state will be the thing that breaks

Terraform configuration is declarative and reviewable. State is a JSON file that records what Terraform believes it has created, so it can work out the difference between desired and actual.

Every serious Terraform problem traces back to that file being wrong, lost, or contended. And because the code is in Git while the state usually is not, teams tend to protect the thing that is already safe and leave the thing that is not.

State is a database, not a file

The single most damaging misconception is treating terraform.tfstate as a config file. It is a serialised database that happens to be JSON, and it is the authoritative record of your infrastructure.

Local state has no locking, no concurrency control, and no history. Two people running apply against the same directory will corrupt it, and they will usually find out afterwards.

Remote state, done properly

Use a shared backend with locking from the first day, even with one person on the project. Solo projects grow, and the migration is far more painful while infrastructure is small.

The standard pattern is object storage for state plus a lock table:

terraform {
  backend "s3" {
    bucket         = "acme-terraform-state"
    key            = "prod/network/terraform.tfstate"
    region         = "eu-west-2"
    dynamodb_table = "acme-terraform-locks"
    encrypt        = true
  }
}

Note the key containing the full path. That is deliberate, and it is the most commonly missed detail.

One state file per environment, per bounded component

Monolithic state — one file describing an entire organisation — fails in three ways at once. It locks during applies, so teams block each other. A single wrong resource blocks every plan. And every apply rewrites the whole file, so a merge conflict is almost certain once two people work concurrently.

Split state along lines that limit blast radius:

• Separate per environment. prod, staging and dev never share a state file.
• Separate per bounded component. Networking, data, and an application estate are different lifecycles.
• Split when blast radius demands it, not before. Three small state files beat thirty tiny ones — every split adds a module boundary to maintain.

Locking is not optional

Concurrent apply is the mechanism behind most corrupted states. Locking prevents it. The historical mechanism is a DynamoDB table; the modern one uses S3 native conditional writes, and Terraform 1.10+ uses that by default for S3 backends without a dynamodb_table block.

Whichever you use, never disable locking because a pipeline hung and you are in a hurry. The locked state is not the problem; the interrupted run is.

Back up state, properly

Object versioning on the state bucket is the minimum, not the whole answer. It gives you rollback within the bucket. It does not help if someone deletes the bucket.

• Versioning enabled on the state bucket, non-negotiable
• Encryption at rest, plus a deliberate decision on who can read it, because state contains plaintext values
• Object lock or a separate backup in another account for the state you would be most upset to lose
• Restores tested. A backup you have never restored is a hypothesis.

Never edit state by hand

Every stack trace in the history of Terraform contains someone who decided a manual edit would be faster. The supported commands exist, and they are safer than they look:

terraform state list
terraform state show aws_instance.web
terraform state rm aws_instance.orphaned
terraform state mv aws_instance.old aws_instance.new
terraform import aws_instance.imported id=i-abc123

state rm is the one to reach for when a resource was deleted outside Terraform and apply proposes recreating it. state mv is the one for renaming or restructuring — and it is far safer than it appears, because Terraform updates the binding in the same operation rather than destroying and recreating.

The expensive mistake: renaming a state bucket

Worth calling out specifically, because it is a recurring pattern. Changing the bucket or key in a backend block looks like a config change. It is not.

If the backend key or bucket changes, Terraform looks in a different location, finds an empty state, and concludes that everything must be created. On the next apply against production, that is duplicate infrastructure, duplicate DNS records, and in the case of a database, a genuinely serious problem.

To rename properly, move the object in the backend first, then update the config:

aws s3 cp s3://old-bucket/old/key s3://new-bucket/new/key

# update the backend block, then:
terraform init -migrate-state
terraform plan   # MUST be empty of create actions before you go further

If that plan proposes creating anything, stop. The migration did not work, and proceeding is how you get two of everything.

Recovering a lost state file

It will happen. In rough order of preference:

1. Object version history. Restore the previous version and inspect it before applying anything.
2. Terraform Cloud or Enterprise state history. Continuous versioning, if you use it.
3. Rebuild from providers. Every provider can enumerate its resources. Painful, but it produces a real inventory, which is more valuable than the state file anyway.
4. Import everything. Slow, and it must be done before the next apply, not after.

The one thing not to do is run apply and see what happens. That converts a state problem into an infrastructure problem, and it is very hard to undo.

A short checklist

• Remote backend with locking, from the first commit
• One state file per environment, split by bounded component
• Versioning on the state bucket, with a restore actually tested
• Encryption at rest, and access restricted deliberately
• Backend key includes the full path from the project root
• No manual state edits — use state rm, mv and import
• Never change a backend bucket or key without init -migrate-state