Terraform Backends Overview: Local, Remote, Configuration

Before Terraform can make changes to your infrastructure, it needs to understand what already exists. It does that through the state file.

The state file keeps a record of the resources Terraform manages. It uses resource IDs and other metadata to map the resource blocks in your configuration to the underlying infrastructure, such as cloud servers, databases, networks, and load balancers.

Backends define where Terraform stores the state file.

Find out how backends work, how to set them up, and why they are so important for teams and automated pipelines.

What Is a Terraform Backend?

A Terraform backend is a setting that tells Terraform where to store and read its state file.

When you run the terraform plan or terraform apply command, Terraform compares the saved state, your configuration, and the actual infrastructure to determine whether to create, modify, or delete resources.

By default, Terraform stores the state file in the local project directory. If you want to use a different storage location, you need to configure a backend in a backend block within the main terraform block.

In production, teams usually do not use local storage. Instead, they store state remotely using services like Amazon S3, Azure Storage, Google Cloud Storage, or HCP Terraform.

Why Are Terraform Backends Important?

Once you start working on team projects, the benefits of storing state in a shared location instead of a single machine quickly become clear:

• Team members and CI/CD pipelines have a single source of truth.
• The risk of two people working from different copies of the same state file is much lower.
• You have better control over who can read or change the state file.
• Onboarding new team members is faster because they can access the latest state from day one.
• Mistakes are less of a problem because the shared state gives everyone access to the same current information.
• It is easier to track infrastructure changes over time, especially if your backend or storage service supports versioning or keeps a history of state changes.

How Do Terraform Backends Work?

Terraform backends have a clear workflow. When you run terraform init, Terraform will:

1. Read the backend block in your Terraform configuration.

2. Connect to the configured backend.

3. Initialize the working directory and download the required providers and modules.

4.  At this point, if you changed the backend or moved from local state to a remote backend, Terraform may ask you to confirm before migrating the existing state.

Terraform can now use the new state location. When you run terraform plan or terraform apply, it follows these steps:

1. Terraform loads the current state from the configured location.

2. If the backend supports locking, Terraform locks the state before making changes.

3. Terraform compares the saved state, your setup, and the actual infrastructure.

4. Terraform determines what needs to be created, modified, or deleted.

5. After the operation finishes, Terraform updates the state to reflect the new infrastructure state.

6. Terraform releases the lock so another user or process can access the state.

Getting Started with Terraform Backends

A local Terraform backend works out of the box and requires no additional setup. If you want to use a remote backend, you need to add a backend block to your configuration.

The Default Backend (Local)

By default, Terraform stores the state file on the machine where you run Terraform commands. The file is named terraform.tfstate and is placed in the current project directory.

This is practical for learning or small personal projects because Terraform manages local state for you. But as more people or automation tools start using the same infrastructure, it becomes more difficult to manage.

The main drawbacks of storing state locally include:

• Multiple users would need access to the same machine or a copy of the same state file.
• Local files are easier to lose, overwrite, or forget to back up.
• Team members might accidentally work with different versions of the state.
• State can contain sensitive data, and storing it on a local computer can be risky.
• You do not have a central location to manage shared access, version history, or recovery.

For these reasons, most organizations move state to a remote backend instead of keeping it on a local machine.

Types of Remote Backends

Terraform supports several popular remote backends, including Amazon S3, Azure Blob Storage, Google Cloud Storage, and HCP Terraform.

The simplest option is to use the same cloud platform where your infrastructure runs. This makes setup and access management easier since your team is already familiar with the environment. You can choose a different backend if you prefer, but it may take longer to set up and involve extra steps.

The following compares four popular remote backends and their main features:

Configure Backends

You can write the backend configuration directly in your Terraform configuration file, inside the top-level terraform block. Each configuration can have only one backend block.

backend Block Syntax

A backend block has two main parts. The backend type tells Terraform which storage system to use, and arguments, which specify where and how to store the state.

For backends such as Amazon S3 (s3), Azure Blob Storage (azurerm), and Google Cloud Storage (gcs), the backend type is the backend block label.

In the following AWS example, s3 is the backend type, while bucket, key, and region are S3-specific arguments:

terraform {
  backend "s3" {
    bucket       = "my-remote-terraform-state"
    key          = "prod/terraform.tfstate"
    region       = "us-west-2"
    use_lockfile = true
  }
}

Each backend has its own set of arguments:

Amazon S3 (s3)

• bucket. S3 bucket name.
• key. Path to the state file in the bucket.
• region. AWS region where the bucket is located.
• use_lockfile. Set to true to enable S3-native state locking.

Azure Blob Storage (azurerm)

• storage_account_name. Azure Storage account name.
• container_name. Name of the storage container.
• key. Name of the blob used for the state file.

Google Cloud Storage (gcs)

• bucket. GCS bucket name.
• prefix. An optional prefix Terraform uses for the state object.

The argument values have to be hardcoded or passed using another supported method, such as -backend-config. You cannot set them using input variables, locals, or data source values.

cloud Block Syntax

HCP Terraform uses a cloud block instead of a standard backend block. The cloud block links your Terraform setup to an HCP Terraform organization and workspace. In this example, Terraform connects the configuration to a specific Terraform workspace:

terraform { 
   cloud { 
      organization = "my-really-cool-company" 
      workspaces { 
            name = "awesome-network" 
        } 
     } 
}

HCP Terraform has its own set of arguments you use inside the cloud block:

• organization. The name of your organization registered with HCP Terraform.
• workspaces { name = …}. Connects your configuration to a single, specific workspace.
• workspaces { tags = …}. Connects the configuration to all workspaces that match the listed tags.

You cannot use a cloud block and a backend block together in the same configuration.

Partial Configuration with -backend-config

Sometimes you need to reuse the same Terraform configuration across environments. The -backend-config option lets you keep the same backend block in your code and pass additional arguments and values during initialization.

For example, you can define only part of the backend in your configuration:

terraform {
  backend "s3" {
    key = "prod/terraform.tfstate"
  }
}

When you run the terraform init command, you can provide additional values with the -backend-config option:

terraform init \
  -backend-config="bucket=my-remote-terraform-state" \
  -backend-config="region=us-west-2"

Backend settings you pass with -backend-config are saved in the .terraform directory and included in plan files. To stay safe, avoid using this method for sensitive values, such as authentication data.

Authentication and Access Security

Terraform needs a way to authenticate before it can read from or write to a remote backend.

You should not hardcode credentials directly in the backend block. Doing so can expose sensitive information in places you might not expect. For example, this is a pattern you should avoid:

terraform {
  backend "s3" {
    bucket     = "my-remote-terraform-state"
    key        = "prod/terraform.tfstate"
    region     = "us-west-2"
    access_key = "DO_NOT_PUT_KEYS_HERE"
    secret_key = "DO_NOT_PUT_SECRETS_HERE"
  }
}

Instead, choose a safer authentication method your platform supports, such as IAM roles, RBAC, or environment variables. This way, sensitive values stay out of your Terraform configuration, making your setup much more secure. For example, if you use the S3 backend, Terraform can read AWS credentials and region settings from environment variables.

The following examples use export, which works in Zsh and similar shells on Linux and macOS. Set these environment variables in your terminal before running Terraform:

export AWS_ACCESS_KEY_ID="your-public-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-access-key"
export AWS_DEFAULT_REGION="us-west-2"

This way, the backend block only needs to specify where the state is stored:

terraform {
  backend "s3" {
    bucket = "my-remote-terraform-state"
    key    = "prod/terraform.tfstate"
  }
}

By keeping sensitive values out of your configuration, it is also easier to reuse your backend setup in different environments.

Managing the Terraform Backend Lifecycle

After you set up a backend, Terraform uses it throughout the normal workflow. However, there are occasions on which you may need to run manual backend or state commands.

Reconfiguring an Existing Backend (terraform init -reconfigure)

If you update backend settings after Terraform has already been initialized, you need to run terraform init again to apply the changes. In some cases, Terraform needs additional instructions on how to handle the configuration.

The -reconfigure option tells Terraform to ignore the previous backend setup and use the current configuration. To do this, enter the following command:

terraform init -reconfigure

This command does not move your existing state to a new backend. It only tells Terraform to reconnect using the modified configuration.

If you need to copy existing state to a new backend, terraform init -migrate-state is usually the better option.

Migrating State from Local to Remote

When you move from local state to a remote backend, Terraform can copy the existing state to the new location. This saves time because Terraform does not need to recreate the infrastructure. To migrate state to a remote backend:

1. Create a backend block for the remote backend or update the existing backend configuration. This example uses the GCS backend:

terraform {
  backend "gcs" {
    bucket       = "my-remote-terraform-state"
    prefix          = "prod/state"
  }
}

The bucket argument specifies which GCS bucket to use, while prefix is the path within the bucket that Terraform uses to organize the state object.

2. Run the following command to initialize the new configuration and start the migration:

terraform init -migrate-state

3. Confirm that you want to proceed with the migration when prompted.

4. Run the following command to verify that Terraform can read the current state:

terraform plan

5. Keep a backup of the old local state file until you confirm that the migration worked.

This method is usually the safest way to move from a local terraform.tfstate file to a remote backend.

Manually Pulling and Pushing State

Terraform includes commands that allow you to pull the current state to a local system or push a local state file to a backend. You can use them to inspect the state, recover a version missing from your local system, or fix a configuration issue, but they are not part of a normal daily Terraform workflow.

Use the following command to print the current state in your terminal:

terraform state pull

To save the output to a local file, run:

terraform state pull > local-backup.tfstate

To upload a local state file to the configured backend, enter:

terraform state push local-backup.tfstate

This command attempts to overwrite the destination state. Only use it if you are sure you need to make a manual repair.

Recovering from Stuck Locks (terraform force-unlock)

Running more than one Terraform operation at once can cause conflicts or corrupt the state. To prevent users and tools from writing to state while it is being updated, Terraform locks the state file. While a lock is active, other operations have to wait or fail.

Usually, you do not need to lock or unlock state manually. Terraform releases locks automatically after a run finishes. However, a lock can get stuck if a run crashes, the network drops, or the process is interrupted.

If this happens, Terraform will show a lock ID in the error message. You can use the ID to unlock the state:

terraform force-unlock LOCK_ID

Before running terraform force-unlock, make sure there are no active Terraform runs.

Only use this command to recover from a stuck lock. Do not use force-unlock just because another run is taking longer than usual.

Terraform Backends in CI/CD Automation

When you run Terraform in a CI/CD pipeline, it needs to initialize the backend and authenticate to the remote state location without human intervention. There are several steps you can take to keep this process fully automated.

Non-Interactive Initialization (-input=false)

In a CI/CD pipeline, there is usually nobody there to respond to prompts in the terminal. This is why automated Terraform runs need to be non-interactive.

You can use the -input=false option to stop Terraform from asking for missing values during initialization:

terraform init -input=false

If a setting or variable is missing, the pipeline will fail instead of waiting for input.

When you run terraform apply, Terraform usually asks for confirmation before applying changes. To skip this step, pipelines often use the -auto-approve option along with -input=false:

terraform apply -input=false -auto-approve

If the pipeline already approved the deployment before Terraform starts, -auto-approve tells Terraform not to ask for additional confirmation.

Using TF_IN_AUTOMATION

TF_IN_AUTOMATION is an environment variable that lets Terraform know it is running inside an automated workflow. When the variable is set, Terraform adjusts part of its output so it is less focused on human instructions. This makes CI/CD logs cleaner and easier to read.

Use the following command to set the TF_IN_AUTOMATION variable:

export TF_IN_AUTOMATION=1

Even though the change is mostly cosmetic, the next time you run Terraform, the output will be easier to follow in CI/CD logs and other automated environments.

Authenticating Pipelines with OIDC

A CI/CD pipeline needs a way to authenticate to remote state storage, such as Amazon S3 or Google Cloud Storage. OIDC is a common method that uses short-lived identity tokens for authentication.

Once you set up a trust relationship between the CI/CD platform and the cloud provider, the pipeline can request temporary access credentials during a run. This is how the process works:

1. The CI/CD platform creates an OIDC identity token for the pipeline run and sends a request to the cloud provider.

2. The cloud provider confirms the token comes from a trusted pipeline and provides temporary access credentials.

3. Terraform uses these temporary credentials to initialize the backend, read state, acquire a lock, and write the updated state after the run.

Access is temporary, and you do not need to store permanent cloud credentials in the pipeline configuration, making your CI/CD pipelines more secure.

Terraform Backends Best Practices

Follow these best practices when working with Terraform backends:

• Use a remote backend for team projects. Local state works for smaller personal projects, but teams in a production environment should always use a remote backend.
• Turn on state locking. Some backends do not enable state locking by default. Check if you need to enable and configure this feature to keep your state consistent.
• Enable versioning or backups. Files get deleted by mistake, updated incorrectly, or become corrupted. To recover from these issues, make sure your backend supports versioning and backups, and enable them if necessary.
• Limit access to backend. Only users who need to run Terraform should have read and write access to the state location.
• Do not hardcode credentials in backend blocks. Keep passwords, access keys, tokens, and client secrets out of .tf files. Use features and tools your provider recommends, such as environment variables.
• Back up state before migrating backends. Save a copy of the current terrafrom.tfstate file before moving from a local state to a remote backend.
• Do not edit state files manually. Terraform state is a readable JSON file, but making changes directly is risky. If you need to make minor changes, use Terraform commands such as terraform state, terraform import, or backend migration options.

Terraform Backends Common Mistakes

If you are new to Terraform backends, it is easy to make assumptions about how they work. Here are some important things to remember when using Terraform backends:

• Backend storage does not work out of the box. Terraform uses the backend to store state, but you usually need to set up the storage location first. For example, if you use an S3 backend, make sure the S3 bucket exists before you run terraform init.
• The provider and backend do not have to match. Even though it is often easier to use the same platform, keep in mind that you can use any supported backend with any provider.
• Terraform must be reinitialized after backend changes. Whenever you update the backend configuration, run terraform init again. If you skip this step, Terraform might not connect to the right state location.
• Remote state storage and remote execution are different concepts. A remote backend stores state remotely, but Terraform still runs on your computer or in a CI/CD pipeline unless you use a service like HCP Terraform that supports remote runs.
• State locking does not work the same everywhere. Locking depends on the backend and its configuration. Some backends support it automatically, while others require additional setup.

Conclusion

This article explained how Terraform backends store and manage state, why remote backends are important for teams, and different ways to keep state safe.

If you plan to provision more infrastructure resources with Terraform, Bare Metal Cloud is a strong choice for dedicated server workloads.