How Terraform Works Internally: HCL Parsing, Dependency Graph Execution, and the Provider Protocol Behind Infrastructure as Code

When engineers describe Terraform to each other they usually say “it’s declarative” and “it builds a dependency graph.” Both are true but neither explains much. The gap between “it reads HCL and calls AWS” and “here is exactly what happens between terraform plan and the API call” is where real operational understanding lives. This article fills that gap.

HCL Parsing: More Than a Config File

HashiCorp Configuration Language is not YAML with extra syntax. It is a full expression language with a parse tree, a type system, and a separate evaluation phase. Terraform processes HCL in two passes.

The first pass is syntactic: the HCL parser (implemented in the hashicorp/hcl library) builds a *hcl.File containing a body of attributes and blocks. At this stage, expressions are not evaluated. They are stored as hcl.Expression values, which are essentially AST fragments.

// Conceptual TypeScript equivalent of what the HCL parser produces
interface HCLBody {
  attributes: Record<string, HCLExpression>;
  blocks: HCLBlock[];
}

interface HCLBlock {
  type: string;       // "resource", "variable", "module", etc.
  labels: string[];   // ["aws_s3_bucket", "uploads"]
  body: HCLBody;
}

interface HCLExpression {
  // Unevaluated AST. Variables are references, not values.
  variables(): TraversalReference[];
  evaluate(ctx: EvalContext): (Value | Diagnostic);
}

The second pass is semantic: Terraform walks the loaded bodies and calls expression.Value(evalCtx) on each attribute, providing an EvalContext that maps variable names and functions. This is important because the two-pass design is what makes forward references in HCL safe. Terraform does not evaluate local.bucket_name when it parses the line. It records that a variable reference exists, builds the dependency graph, determines evaluation order, then calls Value() in topological order.

Expression Types and the Type System

HCL uses a structural type system built on the zclconf/go-cty library. Every value has a concrete type: cty.String, cty.Number, cty.Bool, or a collection type like cty.List(cty.String). When expressions are evaluated, type mismatches surface at plan time rather than apply time. This is why terraform plan can catch a type error in a for_each key before any API call is made.

Template expressions ("${var.region}-${var.env}") go through a separate string template parser that produces a sequence of literal string segments and expression segments, each evaluated and concatenated. Function calls like toset(), merge(), and jsonencode() are resolved from a built-in function registry.

Building the Dependency Graph

After loading all configuration files in a directory (and recursively loading called modules), Terraform constructs a directed acyclic graph where each node is a resource, data source, local value, variable, output, or module call. Edges represent dependencies.

Explicit vs Implicit Dependencies

Terraform recognizes two dependency forms.

Explicit dependencies come from depends_on. These are escape hatches for side-effectful resources where the dependency cannot be expressed through attribute references alone, like an IAM policy that must exist before a Lambda can assume a role even if the Lambda definition does not reference the policy ARN.

Implicit dependencies are the common case. When an expression in resource B references an attribute of resource A, the HCL parser records a traversal: aws_s3_bucket.uploads.id. The graph builder resolves that traversal to the node for aws_s3_bucket.uploads and adds an edge B -> A. This is what makes the graph deterministic: it is derived mechanically from the syntax.

resource "aws_s3_bucket" "uploads" {
  bucket = "myapp-uploads-${var.env}"
}

resource "aws_s3_bucket_versioning" "uploads" {
  bucket = aws_s3_bucket.uploads.id   # implicit dependency edge added here

  versioning_configuration {
    status = "Enabled"
  }
}

Terraform resolves aws_s3_bucket.uploads.id as a traversal, finds the node, and adds a dependency edge. No explicit depends_on needed.

Graph Reduction and Cycle Detection

After the initial graph is built, Terraform runs a transitive reduction to remove redundant edges. If A depends on B and B depends on C, the edge A -> C is redundant because A already depends on C transitively through B. Removing it produces a cleaner graph for visualization (terraform graph) and a faster walk.

Cycles in the graph are a hard error. Terraform uses depth-first search to detect them before any evaluation begins. The error output names the cycle path so you can find the circular reference.

Parallel Execution via DAG Walk

With the DAG in hand, Terraform executes the walk in topological order using a concurrent walker. Nodes with no unmet dependencies become immediately eligible. The walker maintains a pool of goroutines. When a node’s evaluation completes, the walker marks it done and enqueues its dependents if their remaining dependency count reaches zero.

The default parallelism is 10 concurrent operations. This is configurable with terraform apply -parallelism=N. Setting it too high on providers with rate limits (AWS has per-service limits) causes throttling errors. The right value depends on the provider, not the local machine.

The Provider Protocol: gRPC Under the Hood

Every resource operation in Terraform goes through a provider plugin. Providers are separate binaries that Terraform starts as child processes. They communicate over a local gRPC connection.

Plugin Architecture

When terraform init downloads a provider, it places a binary in .terraform/providers/. When Terraform needs to use it, it starts the binary as a subprocess. The subprocess listens on a randomly assigned localhost port and writes the address to stdout in a handshake format. Terraform reads that address and opens a gRPC client connection.

The protocol is defined in Protocol Buffers. The key service definition (simplified) looks like this:

service Provider {
  rpc GetProviderSchema(GetProviderSchema.Request)
    returns (GetProviderSchema.Response);

  rpc ValidateProviderConfig(ValidateProviderConfig.Request)
    returns (ValidateProviderConfig.Response);

  rpc PlanResourceChange(PlanResourceChange.Request)
    returns (PlanResourceChange.Response);

  rpc ApplyResourceChange(ApplyResourceChange.Request)
    returns (ApplyResourceChange.Response);

  rpc ReadResource(ReadResource.Request)
    returns (ReadResource.Response);

  rpc ImportResourceState(ImportResourceState.Request)
    returns (ImportResourceState.Response);
}

The PlanResourceChange / ApplyResourceChange Split

This is the key insight behind how terraform plan produces accurate previews without touching real infrastructure.

During plan, Terraform calls PlanResourceChange with three inputs: the prior state (what the state file says exists), the configuration (what the HCL says should exist), and the provider schema. The provider returns a planned state: the set of attributes that will result from the operation. Some attributes are marked as “computed” and their values are represented as unknown. This is why plan output shows (known after apply) for things like auto-generated IDs.

During apply, Terraform calls ApplyResourceChange with the same prior state and the planned state from the previous call. The provider performs the actual API calls and returns the final state with all unknowns resolved. Terraform writes that final state to the state file.

// Conceptual representation of what flows through the provider protocol
interface PlanResourceChangeRequest {
  typeName: string;
  priorState: DynamicValue;      // from state file, null if new resource
  proposedNewState: DynamicValue; // merged from config + prior state
  config: DynamicValue;           // raw configuration values
  priorPrivate: Uint8Array;       // provider-private data from prior state
}

interface PlanResourceChangeResponse {
  plannedState: DynamicValue;     // may contain unknown values
  requiresReplace: AttributePath[]; // which changes force replacement
  plannedPrivate: Uint8Array;
}

The requiresReplace field in the plan response is what drives the forces replacement annotation. The provider, not Terraform core, knows which attribute changes require destroying and recreating a resource. The aws provider marks bucket on aws_s3_bucket as requiring replacement because S3 does not support renaming buckets. Terraform core merely reads that signal and renders it.

Provider Schema Negotiation

On startup, Terraform calls GetProviderSchema to retrieve the full schema for every resource and data source the provider manages. The schema includes attribute types, optionality, sensitivity flags (for masking in plan output), and deprecation notices. Terraform validates all configuration against this schema before ever calling PlanResourceChange. This is the second layer of type checking, after HCL’s own type evaluation.

State Mechanics: What the JSON Actually Contains

The state file is a JSON document managed entirely by Terraform core. It is not a format providers control. Each resource instance gets an entry containing:

• schema_version: an integer the provider uses to migrate attributes during upgrades
• attributes: the full set of output attributes from the last successful apply
• private: a base64-encoded blob that the provider uses for its own bookkeeping (pagination cursors, ETag values, etc.) that are not user-visible
• dependencies: the list of resource addresses this instance depended on at the time of the last apply

Drift Detection

When you run terraform plan on an existing configuration, Terraform calls ReadResource for each resource in the state file. The provider queries the actual API and returns the current attributes. Terraform compares those against the state file. If they differ, the resource has drifted. The diff is folded into the plan alongside any configuration changes.

If ReadResource returns null, the resource no longer exists in the provider. Terraform plans to recreate it.

Remote Backends and Locking

Local state is a file at terraform.tfstate. Remote backends (S3 with DynamoDB locking, Terraform Cloud, Google Cloud Storage) move that file to a shared store and add a locking primitive. The locking protocol is: before writing, acquire a lock (DynamoDB conditional write on a lock item, or HCP Terraform’s API lock endpoint). After writing, release the lock. If a process crashes holding a lock, operators must manually remove the lock record.

State files contain sensitive values (database passwords, private keys, API tokens) in plaintext. Encryption at rest is a backend concern: S3 SSE-KMS, Terraform Cloud’s built-in encryption, or the native state encryption introduced in Terraform 1.7 using AES-256-GCM with a provider-managed key.

The Plan/Apply Lifecycle End to End

Putting it all together, the sequence for a terraform apply on a configuration with three resources (A, B depends on A, C depends on B) runs as follows.

terraform init downloads providers, initializes the backend, and locks provider versions into .terraform.lock.hcl.

terraform plan loads and parses all HCL files, evaluates expressions in dependency order, constructs the DAG, reads the remote state, calls ReadResource for each existing resource to detect drift, calls PlanResourceChange for each resource that needs to change, and serializes the plan to a binary file (using a protobuf encoding, not the state JSON format).

terraform apply (with the saved plan file) deserializes the plan, re-validates that it was produced from the same state version it is about to write, acquires the state lock, walks the DAG and calls ApplyResourceChange for each node in topological order using the concurrent walker, writes each result to the in-memory state as it completes, and flushes the full state to the backend when the walk finishes. If the process is interrupted mid-apply, the partial state flush that happens after each resource apply limits the blast radius.

Production Considerations

Plan output is a contract, not a preview. Saved plan files include a checksum of the state version they were produced against. If state changes between plan and apply (another engineer applied something), Terraform rejects the plan. Always use saved plan files in CI rather than running terraform apply -auto-approve without a plan step.

Provider version pinning matters more than most teams realize. A minor provider version bump can change which attributes mark requiresReplace. An unreviewed provider upgrade in terraform init has triggered unintended database replacements in production. Pin provider versions in .terraform.lock.hcl and review diffs on upgrades.

Large state files degrade plan performance. ReadResource is called for every resource in state during each plan. A workspace with 500 resources calls the provider 500 times before any change evaluation starts. Split large deployments across workspaces by lifecycle (network, data, application) rather than by environment alone.

Unknown values propagate. If resource A has a computed output that resource B uses as an input, and A is being created in the same plan, B’s plan will show (known after apply) for that attribute. If C depends on B’s computed output, C also shows unknowns. Chains of unknown propagation make plan output harder to review and occasionally cause apply-time failures when providers reject unknown values they expected to be concrete. Minimize cross-resource unknown dependencies where possible.

State encryption is not optional at scale. Default S3 backend configurations store Terraform state with bucket-level SSE. That encrypts data at rest but still exposes plaintext to anyone with S3 read access to the bucket. Use Terraform 1.7+ state encryption with a KMS-managed key, or Terraform Cloud’s server-side encryption, so secrets in state require an explicit KMS grant to read.

Import before you manage. Resources created outside Terraform (console, CLI, other tools) that you want Terraform to manage must be imported first. Running terraform import calls ImportResourceState on the provider, which queries the resource by its external ID and writes the result to state. Without import, a subsequent terraform apply will try to create a duplicate and either fail or create orphaned infrastructure.

Tool Tradeoffs

Tool	State model	Language	Provider ecosystem	Abstraction level	Drift detection	Notable tradeoff
Terraform	JSON file, pluggable backends	HCL	Largest (thousands of providers)	Resource-level	Built-in via ReadResource	HCL limits programmatic flexibility
OpenTofu	Same as Terraform (fork at 1.6)	HCL	Terraform-compatible	Resource-level	Built-in	Community-maintained; provider compatibility needs verification per release
Pulumi	Like Terraform state, Pulumi Cloud or S3	TypeScript / Python / Go / C#	Smaller, wraps Terraform providers via bridge	Resource-level, plus higher abstractions	Built-in	General-purpose language means more flexibility and more rope to hang yourself with
AWS CDK	CloudFormation stack state	TypeScript / Python / Java	AWS-only (natively), CDKTF for multi-cloud	Construct library abstractions	CloudFormation change sets	Tightly coupled to CloudFormation limits and deployment model
SST	CloudFormation or Pulumi backend	TypeScript	AWS-focused, growing	Application-level (functions, tables, buckets as first-class)	Via Pulumi or CloudFormation	High abstraction is fast but obscures what is actually deployed; less suitable for network and IAM heavy work

OpenTofu diverged from Terraform at version 1.6 after HashiCorp changed Terraform’s license from MPL to BSL. The wire-level provider protocol is compatible, so the same provider binaries work with both runtimes. The gaps widen as each project ships new features independently, so check release notes before assuming compatibility.

Pulumi’s model compiles your TypeScript or Python program to a sequence of resource registrations using a language host RPC. The engine then performs the same DAG walk and provider protocol calls Terraform uses. The difference is that the graph is constructed at runtime by executing your program rather than by statically analyzing HCL. This makes Pulumi more flexible for conditional resources and dynamic configurations, but it means a logic bug in your program can produce a graph that looks correct until apply destroys something unintentionally.

Closing

Terraform’s architecture is a clean separation of concerns: HCL parsing and expression evaluation in core, resource lifecycle in provider plugins over gRPC, state as a serialized snapshot of provider-returned attributes. The DAG walk and the plan/apply split are the two ideas that make it reliable at scale. Understanding them converts confusing plan output and cryptic error messages into debuggable, predictable behavior.