How to Use Terraform Data Sources to Improve Infrastructure as Code

Infrastructure as Code (IaC) often extends beyond the resources Terraform creates and manages. Many environments already contain virtual networks, resource groups, secrets, operating system images, and other infrastructure components that new deployments must reference rather than recreate. Terraform data sources allow users to retrieve this existing information and use it throughout a configuration.

Using data sources helps create reusable and portable Terraform configurations.

This article explains how to use data sources to build more flexible and maintainable Infrastructure as Code.

What Are Terraform Data Sources?

Terraform data sources allow configurations to retrieve information from infrastructure or external services without managing the underlying resources. They provide read-only access to existing objects, metadata, configuration values, and provider-specific information that Terraform can use to plan and apply infrastructure changes.

Unlike resources, data sources do not create, update, or delete infrastructure. Instead, Terraform queries the provider during execution and makes the returned values available to other resources, variables, outputs, and expressions. This allows a configuration to integrate with existing infrastructure or to obtain information that changes over time, such as available images or network identifiers.

Data sources are available in most Terraform providers, including cloud platforms, DNS providers, secret management systems, and infrastructure services.

Data Sources vs. Terraform Resources

Although both resources and data sources interact with infrastructure providers, they serve different purposes. A common deployment uses both together. For example, a configuration can retrieve an existing virtual network through a data source and then provision new virtual machines inside that network using Terraform resources.

The following table compares the two according to key features:

Anatomy of a Data Block

Terraform defines data sources with the data block. Each block identifies the provider-specific data source, specifies how Terraform should locate the requested object, and exposes attributes that other resources can reference.

Understanding each component of a data block makes it easier to work with different providers because the overall structure remains the same even though the supported arguments and exported attributes vary.

data "<DATA_SOURCE_TYPE>" "<LOCAL_NAME>" {
  argument = value

  filter {
    name   = "..."
    values = ["..."]
  }
}

All Terraform data sources follow the same basic structure. The provider determines which arguments, optional filters, and exported attributes are available for each data source. Some require only a single identifying argument, while others support additional configuration options to refine the query.

The following sections explain each part of the data block and how Terraform uses it during the planning phase.

Provider Type

The provider type specifies which provider and data source Terraform should query.

In the syntax example above, <DATA_SOURCE_TYPE> represents the provider-specific data source Terraform queries. This value depends on the provider.

For example:

data "pnap_private_network" "production"

The provider determines which arguments the data source accepts and which attributes become available after Terraform retrieves the requested information.

Local Labels

The second identifier is a local label that uniquely identifies the data source within the current Terraform configuration.

For example:

data "pnap_private_network" "production"

Here, production is the local label. Terraform uses this identifier when other resources or expressions reference the data source.

For example, the following expression references the ID of the production data source:

data.pnap_private_network.production.id

Use descriptive labels that reflect the purpose of the retrieved resource. Clear naming conventions improve readability, especially in large configurations that contain multiple data sources of the same type.

Argument Configurations

Arguments define which object Terraform should retrieve. Depending on the provider, arguments may include names, IDs, regions, locations, tags, or other identifying properties.

For example:

data "pnap_private_network" "production" {
  name = "production-network"
}

In this example, the name argument tells Terraform which existing private network to retrieve. Other providers may use different arguments, such as resource IDs, regions, or tags, depending on the type of data source.

Terraform validates the supplied arguments before querying the provider.

Filters

Some providers, including many AWS data sources, support dedicated filter blocks, while others expose provider-specific lookup arguments such as names, resource groups, or labels.

For example:

filter {
  name   = "..."
  values = ["..."]
}

When available, filters make configurations more flexible because they continue to work even when resource identifiers differ between environments.

Not all providers implement filters in the same way. Some providers, such as AWS, Azure, and Google Cloud, support filter blocks, while others primarily identify resources using arguments such as names or IDs. Refer to your provider's documentation to determine the supported lookup methods.

Whenever possible, use the provider's native resource lookup mechanisms instead of hardcoded identifiers to improve the portability and maintainability of your Terraform configurations.

Attributes

After Terraform retrieves a data source, it exposes a set of attributes that other configuration blocks can reference.

For example:

data.pnap_private_network.production.id

The example above references the ID attribute of the retrieved private network. Depending on the provider and data source, Terraform may expose additional attributes such as names, IP addresses, associated servers, locations, tags, and other configuration settings.

These attributes can be referenced by resources, variables, outputs, local values, and expressions throughout the Terraform configuration.

How Terraform Evaluates Data Sources?

Terraform normally evaluates data sources during the planning process before creating or modifying managed resources. It queries the provider, retrieves the requested information, and makes the returned values available throughout the execution plan.

Because Terraform resolves data sources before creating dependent resources, infrastructure can automatically adapt to changes in the provider environment. For example, if a deployment selects the latest operating system image using filters, Terraform retrieves the current image before generating the execution plan.

Using Data Sources for Better Infrastructure as Code

Terraform data sources support a wide range of Infrastructure as Code workflows beyond simply retrieving existing resources. They help create reusable configurations, integrate with infrastructure managed outside the current project, and reduce the need for hardcoded values that require manual updates.

The following sections describe common scenarios where data sources improve the flexibility, maintainability, and security of Terraform deployments.

Querying Infrastructure Provider Metadata and Options

Many providers expose metadata through data sources, including available machine images, instance types, regions, availability zones, and supported operating systems.

Instead of hardcoding these values, configurations can dynamically query the provider. This reduces maintenance because deployments automatically use current provider information. It also makes Terraform modules more portable, as they can adapt to different environments without requiring configuration changes.

For example, a deployment can retrieve the latest supported operating system image instead of referencing an image ID that may become outdated over time.

Integrating with Pre-Existing Network and Server Resources

Organizations rarely build every infrastructure component from scratch. Existing virtual networks, subnets, security groups, storage volumes, and resource groups often serve as the foundation for new deployments.

Data sources allow Terraform to retrieve these existing resources and attach newly created infrastructure without importing them into Terraform state. This approach is useful when shared infrastructure is managed by another team or was provisioned outside the current Terraform configuration.

By referencing existing resources instead of recreating them, deployments become easier to maintain and less likely to introduce duplicate infrastructure.

Sharing State Across Configurations via Remote State

Large environments commonly split infrastructure into multiple Terraform projects. For example, one configuration may provision network resources, while another deploys application servers that depend on those resources.

The terraform_remote_state data source allows one configuration to read outputs from another Terraform state file. This enables teams to reuse information such as VPC IDs, subnet IDs, load balancer addresses, or storage resource identifiers without duplicating configuration.

Because remote state may contain sensitive information, restrict access to state files and expose only the outputs that downstream configurations require.

Managing Sensitive Data and API Keys Safely

Applications often require credentials such as API keys, database passwords, certificates, or authentication tokens. Instead of storing these values directly in Terraform variables or configuration files, data sources can retrieve them from dedicated secret management services.

Common examples include:

• API keys.
• Database passwords.
• TLS certificates.
• Authentication tokens.

Using a centralized secrets manager improves security and simplifies credential rotation because secrets can be updated without modifying the Terraform configuration itself.

Keep in mind that sensitive values retrieved through data sources may still be stored in the Terraform state file, so protect the state using appropriate access controls and encryption.

Creating Dynamic Configurations with Filters

Filters allow Terraform to locate resources based on their properties instead of fixed identifiers.

This approach improves portability across environments because deployments continue to work even when resource IDs differ between development, staging, and production. It also reduces maintenance by allowing Terraform to select resources dynamically based on tags, names, or other provider-specific attributes.

For example, a deployment can locate the latest Ubuntu image or select a subnet tagged for a specific environment, rather than relying on hardcoded values.

Managing Resource Dependencies

Data sources frequently provide input values that Terraform resources require during deployment.

For example, a virtual machine may depend on:

• An existing subnet.
• A security group.
• A resource group.
• A storage account.

By retrieving these resources first, Terraform can provision dependent infrastructure without manually supplying identifiers. This helps maintain consistent relationships between existing and newly created infrastructure components.

Validating Input Data

Data sources can also serve as a validation mechanism before Terraform creates infrastructure.

If Terraform cannot locate the requested resource, the planning phase fails before any infrastructure changes occur. This allows administrators to detect missing resources, incorrect configuration values, or provider access issues early in the deployment process.

Early validation reduces the risk of partially completed deployments and simplifies troubleshooting by identifying problems before Terraform begins creating resources.

Cross-Referencing

Many Terraform deployments rely on multiple existing infrastructure components that must work together. Data sources make it possible to retrieve information from several resources and combine their attributes within a single configuration.

For example, a deployment may retrieve:

• A network.
• A DNS zone.
• A TLS certificate.
• A storage bucket.

Terraform can then use these values to provision new infrastructure that integrates with existing services while keeping the configuration organized and easy to maintain.

Step-by-Step Examples: Implementing Data Sources

The following examples illustrate common scenarios in which Terraform data sources simplify Infrastructure as Code. Rather than creating every infrastructure component from scratch, these examples show how to retrieve existing resources, securely access external information, and use the returned values to provision dependent infrastructure. Unless otherwise noted, the examples use the phoenixNAP Terraform provider to demonstrate these concepts.

Together, they illustrate how data sources help build reusable and maintainable Terraform configurations.

Reading an Existing Infrastructure Resource Group

Many organizations provision servers on existing private networks. Instead of creating a new network for each deployment, Terraform can reuse an existing private network when provisioning additional infrastructure. This reduces duplication and allows multiple Terraform projects to share the same networking resources.

The following example retrieves an existing private network from phoenixNAP Bare Metal Cloud:

data "pnap_private_network" "production" {
  name = "production-network"
}

Terraform reads the private network's properties during the planning phase and makes them available to other resources in the configuration without taking ownership of the network itself.

Fetching Secrets from a Secure Cloud Vault

Infrastructure deployments often require sensitive information such as database passwords, API keys, or certificates. Rather than storing these values directly in Terraform variables or configuration files, you can retrieve them from a dedicated secrets management service at deployment time.

The following example retrieves a database password from a secret management service using the AWS provider:

data "aws_secretsmanager_secret_version" "database" {
  secret_id = "database-password"
}

This approach centralizes secret management while allowing Terraform to use the retrieved values during deployment. Even when using a secure vault, remember that sensitive values may still be stored in the Terraform state file, so protect the state appropriately.

Utilizing Data Outputs to Provision Dependent Infrastructure

One of the primary purposes of data sources is to provide the information other resources require. Instead of manually copying resource identifiers between configurations, Terraform can reference attributes returned by a data source and automatically use them when provisioning infrastructure.

The following example references the ID of an existing private network retrieved through a data source:

resource "pnap_server" "web" {
  hostname = "web-01"
  os       = "ubuntu/noble"
  type     = "s1.c1.small"
  location = "PHX"

  private_network_ids = [
    data.pnap_private_network.production.id
  ]
}

During the planning phase, Terraform retrieves the private network information and uses its ID when provisioning the server. This eliminates hardcoded resource identifiers and allows the same configuration to work across multiple environments where network IDs may differ.

The following screenshot shows a Terraform plan with attributes retrieved from data sources:

How Terraform Refreshes Data Sources

Terraform refreshes data sources to ensure they reflect the current state of the infrastructure before generating an execution plan. Because data sources represent information managed outside the current Terraform configuration, Terraform typically refreshes them by querying the provider whenever they are evaluated, ensuring the configuration uses up-to-date information.

Understanding when data sources are refreshed helps explain why changes to existing infrastructure may appear during the planning phase, even if Terraform does not directly manage those resources.

Evaluation During the Plan Phase

Terraform evaluates and refreshes data sources during the planning phase. Each execution queries the provider to obtain the latest information before Terraform calculates the infrastructure changes required to reach the desired state. Unlike managed resources, data sources are not refreshed from the Terraform state alone. Instead, Terraform reads their current values directly from the provider whenever possible, ensuring that resources depending on those values use the most up-to-date information.

Handling Data Sources that Change Frequently

Some data sources return information that changes regularly, such as the latest operating system image, available software versions, or provider metadata.

Although dynamic queries help keep deployments current, they can also produce different execution plans over time. For example, a configuration that always retrieves the latest operating system image may propose changes when a newer image becomes available.

If deployment consistency is more important than always using the latest available resource, consider specifying a fixed version or using more restrictive filters to limit unexpected changes.

Managing Drift Detection and Non-Managed Infrastructure

Data sources allow Terraform to detect changes that affect referenced infrastructure by re-reading the current object during evaluation. Unlike managed resources, Terraform does not track or remediate drift for infrastructure referenced only through data sources.

For example, if an administrator deletes, renames, or modifies an existing resource outside Terraform, the next plan may fail because the data source can no longer retrieve the expected object. This allows administrators to detect missing or changed dependencies before Terraform attempts to provision new infrastructure.

Although Terraform cannot automatically correct changes made to resources outside its management, refreshing data sources helps ensure that deployment decisions are based on the current infrastructure state.

Using Terraform Data Sources with Bare Metal Cloud

Terraform data sources are useful when provisioning infrastructure on phoenixNAP Bare Metal Cloud because they enable deployments to integrate with existing infrastructure rather than relying on hardcoded configuration values. This is particularly beneficial when multiple Terraform projects share networking resources or when infrastructure is deployed across multiple environments.

Data sources also complement the phoenixNAP provider by allowing Terraform configurations to reference existing infrastructure while continuing to provision new servers with the pnap_server resource. This approach improves consistency, reduces duplication, and makes Terraform modules easier to reuse.

Referencing Existing Network Resources

Many production environments already contain private networks that support multiple applications and servers. Instead of recreating these networks, data sources allow Terraform to retrieve information about the existing infrastructure and use it when provisioning additional resources.

This approach helps maintain a consistent network architecture and is especially useful when networking resources are managed separately from compute infrastructure or shared across multiple Terraform projects.

Reusing Existing Public IP Blocks

Organizations often reserve public IP blocks for use across multiple servers or applications. Rather than manually copying network information between Terraform configurations, data sources can retrieve existing IP allocations and make them available to newly provisioned infrastructure.

Reusing existing networking resources simplifies IP address management and helps maintain consistent network configurations across different environments.

Creating Reusable Server Deployment Modules

The pnap_server resource requires several configuration values, including the operating system, server type, deployment location, pricing model, and network configuration. Instead of embedding these values directly into every Terraform configuration, you can combine data sources with variables, local values, or remote state outputs to create reusable deployment modules.

This approach reduces duplication and helps standardize server deployments across development, testing, and production environments while keeping the configuration easier to maintain.

When defining reusable server modules, avoid hardcoding provider credentials in the Terraform configuration. Instead, authenticate using environment variables or a configuration file to simplify credential management and reduce the risk of exposing sensitive information.

Sharing Infrastructure Information Between Terraform Projects

As infrastructure grows, organizations often separate networking, security, and compute resources into different Terraform configurations. The terraform_remote_state data source allows one project to consume outputs generated by another without duplicating configuration.

For example, one Terraform project can provision shared networking resources, while another retrieves the exported network information and uses it when deploying Bare Metal Cloud servers. This approach improves modularity and allows teams to manage different parts of the infrastructure independently.

Use terraform_remote_state only when configurations intentionally share state. For loosely coupled systems, provider APIs or dedicated service discovery mechanisms may provide better separation.

Common Mistakes When Working with Data Sources

Although Terraform data sources are straightforward to use, configuration errors can prevent deployments from succeeding or produce unexpected results. Understanding these common mistakes helps reduce troubleshooting time and improves the reliability of Terraform configurations.

Missing Resources

A data source can retrieve only resources that already exist. If Terraform cannot locate the requested object, the planning phase fails before any infrastructure changes are applied.

This issue commonly occurs when a resource has been deleted, renamed, moved to another region, or exists in a different account or subscription than the provider is configured to access. Incorrect permissions can produce similar errors because Terraform cannot read the requested resource.

To avoid these issues, verify that the resource exists, confirm that the provider configuration targets the correct environment, and ensure the credentials have sufficient permissions to read the resource before running terraform plan.

Filtering Errors

Filters make Terraform configurations more flexible, but incorrectly configured filters often produce unexpected results. Filters that are too broad may return multiple matching resources, while filters that are too restrictive may return none.

Whenever possible, use combinations of tags, names, or other unique attributes that identify a single resource. Testing filters before using them in production helps prevent deployment failures caused by ambiguous or missing results.

If the provider documentation specifies that a data source must return exactly one resource, review your filter criteria carefully to ensure they uniquely identify the intended object.

Cycle Errors

Terraform builds a dependency graph before planning and applying infrastructure changes. If a data source depends on a resource that also depends on the same data source, Terraform cannot determine which object to evaluate first.

These circular dependencies result in cycle errors during the planning phase. Resolving them typically requires restructuring the configuration so that resources and data sources have a clear evaluation order without referencing one another recursively.

When possible, separate resource creation from resource lookups to keep dependency relationships straightforward and easier to maintain.

Dependency Deadlocks

Some data sources rely on values that are unknown until another resource has been created. In these situations, Terraform may postpone reading the data source until the apply phase. If multiple resources depend on one another indirectly, the dependency graph can become difficult to resolve.

Keeping Terraform configurations modular and avoiding unnecessary dependencies between resources and data sources helps prevent these situations. When explicit dependencies are required, use them only when Terraform cannot determine the correct evaluation order automatically.

Simplifying dependency chains also makes Terraform plans easier to understand and reduces the likelihood of deployment failures caused by complex resource relationships.

Sensitive Data Leaks in Outputs and State Files

Data sources frequently retrieve sensitive information such as passwords, API keys, certificates, and authentication tokens. Although Terraform allows outputs to be marked as sensitive, this only hides the values from console output. The underlying data may still be stored in the Terraform state file.

Protect state files by using secure remote backends, encrypting stored state, and limiting access to authorized users. When possible, avoid exposing sensitive values through outputs unless another Terraform configuration requires them.

Review state storage regularly to ensure it follows your organization's security policies and access control requirements.

Best Practices for Using Terraform Data Sources

Following a few best practices can make Terraform configurations easier to maintain, more portable, and less prone to deployment errors. The recommendations below help reduce hardcoded values, improve readability, and protect sensitive information when working with data sources.

Prefer Native Filters Over Hardcoded Identifiers

Whenever possible, use provider-supported filters instead of fixed resource IDs.

Filtering by tags, names, or other resource attributes creates more portable configurations that work across multiple environments with fewer modifications. It also reduces maintenance overhead because Terraform can locate resources dynamically, even when identifiers differ across environments.

Secure Data Handling for Outputs

Only expose sensitive data when absolutely necessary. Mark outputs containing confidential information as sensitive and restrict access to Terraform state files, since the state may contain retrieved secret values even if they are not displayed in the console.

When available, use a secure remote backend with encryption and access controls to protect sensitive state data.

Combine Data Sources with Local Values

Local values improve readability when multiple resources reference the same data source attributes.

Instead of repeating long expressions throughout the configuration, assign frequently used values to local variables and reference those locals consistently across the deployment. This makes configurations easier to read and simplifies future updates if the underlying data source changes.

Conclusion

This article explained how Terraform data sources work, how they differ from resources, and how to use them to build more dynamic, reusable, and maintainable Infrastructure as Code. It also demonstrated practical implementation examples, common pitfalls, and best practices for integrating existing infrastructure into Terraform deployments.

Next, see how Terraform providers work or how to organize reusable Terraform code into modules to simplify infrastructure management.