Skip to main content
Terraform · lesson 10 of 17

Beginner

7

Intermediate

7

Advanced

3

Also part of Terraform

4
Open the Terraform page →
Terraform intermediate Lesson 10 of 17

Data Sources and Dependencies (read existing infrastructure)

Data sources read what Terraform doesn't manage. See when they're read, a hidden dependency that fails, the reference and depends_on fixes, and a cycle.

By SkillByExample · Updated

terraform data-sources depends_on dependency-graph

A data source reads something Terraform doesn’t manage. References between blocks, whether resources or data sources, form the dependency graph that decides what runs first. Examples use Terraform 1.16.5 and hashicorp/local 2.9.1, whose local_file data source reads a file from disk. AWS’s data "aws_vpc" works the same way against an API.

Reading something Terraform doesn’t own

terraform {
  required_providers {
    local  = { source = "hashicorp/local", version = "~> 2.9" }
    random = { source = "hashicorp/random", version = "~> 3.9" }
  }
}

# Something Terraform does not manage: a file another team maintains.
data "local_file" "allowed_ports" {
  filename = "${path.module}/allowed-ports.txt"
}

locals {
  ports = split(",", trimspace(data.local_file.allowed_ports.content))
}

output "first_port" {
  value = local.ports[0]
}
$ terraform plan -no-color
data.local_file.allowed_ports: Reading...

Planning failed. Terraform encountered an error while generating this plan.


Error: Read local file data source error

  with data.local_file.allowed_ports,
  on main.tf line 9, in data "local_file" "allowed_ports":
   9: data "local_file" "allowed_ports" {

The file at given path cannot be read.

+Original Error: open ./allowed-ports.txt: no such file or directory
$ echo "8080,8443" > allowed-ports.txt
$ terraform apply -auto-approve -no-color
data.local_file.allowed_ports: Reading...
data.local_file.allowed_ports: Read complete after 0s [id=a4a29fdd8a2de04ef1d9c730bcaad08ce88e116d]

Changes to Outputs:
  + first_port = "8080"

You can apply this plan to save these new output values to the Terraform
state, without changing any real infrastructure.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

first_port = "8080"
$ terraform state list
data.local_file.allowed_ports

The data source was read during plan. That’s why a missing file failed the plan before anything was applied. A data source never creates the thing it reads, so it must already exist. After the apply, 0 added, yet the data source is in state under the data. prefix. That’s a cached copy of what was read, and destroy won’t touch the real file.

A hidden dependency

This config writes a file and reads it back, but the data source names the path as a plain string:

resource "local_file" "generated" {
  filename = "${path.module}/generated.txt"
  content  = "build=42\n"
}

data "local_file" "readback" {
  filename = "${path.module}/generated.txt" # a plain string, not a reference
}

output "readback" {
  value = data.local_file.readback.content
}
$ terraform apply -auto-approve -no-color
data.local_file.readback: Reading...

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform planned the following actions, but then encountered a problem:

  # local_file.generated will be created
  + resource "local_file" "generated" {
      + content              = <<-EOT
            build=42
        EOT
      + content_base64sha256 = (known after apply)
      + content_base64sha512 = (known after apply)
      + content_md5          = (known after apply)
      + content_sha1         = (known after apply)
      + content_sha256       = (known after apply)
      + content_sha512       = (known after apply)
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "./generated.txt"
      + id                   = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Error: Read local file data source error

  with data.local_file.readback,
  on main.tf line 12, in data "local_file" "readback":
  12: data "local_file" "readback" {

The file at given path cannot be read.

+Original Error: open ./generated.txt: no such file or directory

To you, the two blocks clearly go together, because they name the same file. To Terraform they are unrelated, since neither refers to the other. Every argument of the data source was known, so it was read at plan time, before the file existed. Terraform only knows about dependencies that are written as references.

The fix: reference the resource

data "local_file" "readback" {
  filename = local_file.generated.filename
}
$ terraform apply -auto-approve -no-color | grep -E '^  #|Read|Creat|^Apply'
  # data.local_file.readback will be read during apply
  # (depends on a resource or a module with changes pending)
  # local_file.generated will be created
local_file.generated: Creating...
local_file.generated: Creation complete after 0s [id=860e3a7f85c3d1b2b5107d535c7b53977b53245e]
data.local_file.readback: Reading...
data.local_file.readback: Read complete after 0s [id=860e3a7f85c3d1b2b5107d535c7b53977b53245e]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The reference adds an edge to the graph, so the read is deferred until after the file is created (“will be read during apply”). The value of filename is the same string as before. What changed is that Terraform now knows where it comes from.

A dependency cycle

resource "random_pet" "frontend" {
  prefix = random_pet.backend.id
}

resource "random_pet" "backend" {
  prefix = random_pet.frontend.id
}
$ terraform validate -no-color

Error: Cycle:
  random_pet.backend
  random_pet.frontend

Each one needs the other’s id before it can be created, so neither can go first. Terraform lists every node in the loop. Fix it by removing one direction of the dependency.

Practice

1. Change allowed-ports.txt to 9090,8080 and plan without touching any .tf file.

Solution 
$ echo "9090,8080" > allowed-ports.txt
$ terraform plan -no-color
data.local_file.allowed_ports: Reading...
data.local_file.allowed_ports: Read complete after 0s [id=a06bedbfa19fd3ea9328192e9da70a98044e3864]

Changes to Outputs:
  ~ first_port = "8080" -> "9090"

You can apply this plan to save these new output values to the Terraform
state, without changing any real infrastructure.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Data sources are read again on every plan, so changes made outside Terraform show up automatically. That’s the point of a data source, and also why a plan can change when nobody edited the config.

2. Keep the plain-string filename and fix the hidden dependency with depends_on instead of a reference.

Solution 
data "local_file" "readback" {
  filename   = "${path.module}/generated.txt"
  depends_on = [local_file.generated]
}
$ terraform apply -auto-approve -no-color | grep -E '^  #|Read|Creat|^Apply'
  # data.local_file.readback will be read during apply
  # (depends on a resource or a module with changes pending)
  # local_file.generated will be created
local_file.generated: Creating...
local_file.generated: Creation complete after 0s [id=860e3a7f85c3d1b2b5107d535c7b53977b53245e]
data.local_file.readback: Reading...
data.local_file.readback: Read complete after 0s [id=860e3a7f85c3d1b2b5107d535c7b53977b53245e]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

This gives the same order, but the path is now written twice and can drift apart. Keep depends_on for dependencies that no attribute can express.

3. Break the cycle by removing backend’s prefix, then apply. Which pet is created first?

Solution 
$ terraform apply -auto-approve -no-color | grep Creat
random_pet.backend: Creating...
random_pet.backend: Creation complete after 0s [id=nearby-prawn]
random_pet.frontend: Creating...
random_pet.frontend: Creation complete after 0s [id=nearby-prawn-amused-teal]

backend goes first because frontend still refers to it. The frontend’s name starts with the backend’s name, which shows the value really was passed along.

Next: sensitive values, covering what sensitive = true hides from plan output and what it leaves in plain text.

Frequently Asked Questions

When do I use a data source instead of a resource?
Use a data source when something already exists and Terraform should only read it, such as a VPC another team owns, the latest machine image, or a file maintained elsewhere. Use a resource when Terraform should create, update and destroy the object. Data sources appear in state as data.<type>.<name> but are never destroyed.
When does Terraform read a data source?
During plan, on every run, if all its arguments are known. If an argument depends on a resource that has pending changes, the read is deferred to apply and the plan shows 'will be read during apply', with its attributes as known after apply.
When do I need depends_on?
Only when one object needs another but no expression connects them, for example a data source reading a file path written as a plain string. Prefer a reference, like filename = local_file.generated.filename, because it carries the value and the ordering at once; depends_on carries only the ordering and makes Terraform more conservative about deferring reads.
What does 'Error: Cycle' mean in Terraform?
Two or more objects refer to each other, directly or through a chain, so there is no order in which to create them. Terraform lists the objects in the cycle. Break it by removing one of the references, often by moving a shared value into a variable or local that both can read.