Terraform Backends

Overview
This post will talk about an overview of two types of Terraform backends: Standard and Enhanced. Standard backends, such as AWS S3 and Azure Storage, store state files but do not execute commands. Enhanced backends, like Terraform Cloud, can store state and perform operations such as terraform plan
and terraform apply
.
We also cover some state locking to prevent conflicts, emphasizes the importance of handling sensitive data in state files and discusses encryption options for remote storage of state.
Standard backends
When we're talking about Terraform backends, there are two backend we can use a Standard Backend. With standard it could be something like an AWS S3 bucket or an Azure storage blob container where your state file is stored. This only stores state, it's not running commands like terraform apply
. Your commands are usually executed on a build server or just in the CLI on your local machine.
When you're working with standard backends you're not required to have a Terraform Cloud account. You could just have an Azure account for example in which you are provisioning resources and you could run your configuration on your local machine authenticated to Azure and use an Azure blob storage account as your backend where you still store state.
Example of using AWS S3 as backend:
terraform {
backend "s3" {
bucket = "mybucket"
key = "path/to/my/key"
region = "us-east-1"
}
}
Example of using Azure Storage:
terraform {
backend "azurerm" {
resource_group_name = "StorageAccount-ResourceGroup"
storage_account_name = "abcd1234"
container_name = "tfstate"
key = "prod.terraform.tfstate"
}
}
You can find more examples here
Enhanced backends
You have the enhanced backend type which supports both local and remote state. Local is the default and with remote you have Terraform Cloud that can store state but it can also perform operations and execute commands, like runnin a terraform apply
if you have approved the output of it running terraform plan
- essentially supports running remote operations.
The cloud environment is responsible for the run environment. So it is running commands pretty much like a virtual build server. You have to configure environment variables (which you can mark as sensitive) if you're creating a workspace for a certain project as Terraform Cloud is not aware of your credentials.
Terraform remote_state
Working with data-blocks and using resource "terraform_remote_state" "resource"
is pretty useful if you want to reference a state file in another configuration.
However, when using terraform remote state I only have access to the values that are outputted from the module. This means I have to specifically state outputs in the source module if I want to use them in another configuration using the Terraform Remote State object. In my experience, it is more likely that I would use a data block and refer to my sources directly that way as it allows access to more data just out-of-the-box.
State-locking& versioning
We want State locking to happen on all operations that could write to the state file to prevent multiple people writing when we have a team that is working with the same state file to prevent multiple editors and corrupting the state file. Most backends, but not all, support state-locking so it is important that you read the documentation before starting to use a backend in your environment.
You can add flags to force unlock if the state is in a locked state but be aware of the risks. You would run the command terraform force-unlock
. You would get a unique lock ID that you have to reference, and you have to type -force
If I am to chose a backend to use I would recommend ensuring that it supports both state-locking and versioning of the state-file to ensure you have backups to use.
Some more best practices
The local state file that is stored on your machine is just a plain text JSON file. So when you're working with a local state, it is important that you do not share your state file with others. You do not commit your state file to a version control system such as Git and push it to a remote repository in Github for instance. With remote states, ensure you have encryption at rest and in transit to protect your data.
If you're working with Terraform Cloud you get the benefits of having some of those features enabled for your state handling pretty much out of the box.
However this is also true when we're talking about third-party storage such as Azure Blob Storage or Amazon S3 Bucket but may require some configuration on your end
It basically depends on the service provider you are using. You have to review the capabilities of these services and make sure if they can store sensitive data and what other capabilities they can offer you for logging who accessed state files, versioning of the state file, and such.
You have the option to work with a .terraform ignore, and it pretty much works the same way as a .git ignore, you could say. By default, you already are excluding .terraform and .git.
Conclusion
Main points
- There are two types of Terraform backends: Standard and Enhanced & they come with different feature sets
- Standard backends store state files and do not run commands
- Enhanced backends in remote setting, like Terraform Cloud, store state and perform operations such as
terraform plan
&terraform apply
- State locking to prevent conflicts & versioning allows you to restore state if need be
References

About me
