Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

The Four Deliveries: File, Env, Secret, Volume

One engine, four ways a document reaches a workload — because real software disagrees about how it wants to be configured. Grafana re-reads files; Airflow reads environment variables at boot and nothing else; a Strimzi-shaped operator watches Kubernetes Secrets. Picking the delivery is a one-line decision here; what follows is the map.

File (webhook + agent)Env (env-inject)Secret (operator target)Volume (CSI, node agent)
The consumerreads/watches a filereads environ at startreads/watches a k8s Secret, or envFromreads/watches a file
Freshnesslive — atomic rename, watcher cadencefrozen at container start (Kubernetes' rule); env-restart opts into a kubelet container-restart on change — seconds, no pod recreationlive object; watchers react, envFrom at next startlive, same renames from a shared watch
Touches etcd?never — tmpfs emptyDirnever — same tmpfs file, sourcedyes — a Secret lives in etcd, stated out loudnever
Restart to update?noyes (next pod start)no for Secret-watchers; yes for envFromno
Containers per renderoneonenone in the workloadnone — one per node, shared
Set up bypod annotationspod annotations (+ explicit command)DynamicConfigRender CRa csi: volume, no annotations at all
Real exampleGrafanaAirflowKafka clientthe node agent's page

The decision procedure, in order: can the consumer read a file? File delivery — it is the only fully live one and the only one that never touches etcd. Does something else own the consumer (an operator that only reads Secrets)? The Secret target. Env-only software? env-inject, with the start-freeze stated rather than papered over.

And the fourth is a scale answer rather than a shape answer. The volume delivers the same bytes as the file does, from one process per node instead of one beside every render — 10,000 pods at 2.5 renders each is 25,000 sidecar containers, and that number is the only reason to reach for it. It costs the isolation the other three have: one process holds the store credentials of every pod on its node. Choose it from a measurement, not a preference, and read its page before you do.

Against the Vault Agent Injector

The closest relative of the webhook+agent half — same architecture (mutating webhook, injected init/sidecar, shared memory volume, pod service-account auth, file permissions and run-as knobs), different center of gravity:

Vault Agent Injectordynamic-config-k8s
BackendsVaultnine stores — Vault among them, plus Consul, etcd, git, Redis, NATS, S3, Firestore, config-server
Pod authKubernetes auth (SA token)the same, wherever the store speaks it (Vault, Consul); IRSA/Workload Identity for S3/Firestore; secret-based stays first-class where a store never will
What is deliveredrendered secrets, Consul-Template languagethe resolved configuration document — precedence, validation, provenance — in json/toml/yaml/ini/properties or a minijinja template
File perms / ownershipannotationsannotations (file-mode, agent-run-as-user/group), root refused at admission
Env variablesyou rewrite the command by hand to source the fileenv-inject writes the wrap for you, refuses the impossible cases by name
k8s Secret objectsnothe operator's secret: target, when the consumer requires one
Lease renewalrenews renewable leases, re-fetches non-renewable onesthe same, since 0.3.0: dynamic: "true" reads a dynamic engine, renews a renewable lease at 65% of its TTL, never sends a renewal to one the store marked non-renewable, re-issues at 90% instead, and hands the lease back on SIGTERM
PKI certificatestracks the certificate's own lifetimethe same: whichever expires first binds — the lease, or the certificate's notAfter
Failure semanticslast-known-good, retrieslast-known-good with a startup policy, a deletion policy, a staleness ceiling wired to readiness, and drift detection on the rendered file
Knowing it arrivedthe file existsthe file exists and the application said it applied it — require-ack makes that readiness
Rolling a changeall pods at oncecanary-configmap: a deterministic cohort takes it first, widened by editing a ConfigMap with no restart
Delivery shapesfile (and env by hand)file, env, k8s Secret, and a CSI volume from a per-node agent
Scopesecrets deliveryconfiguration delivery that treats secrets as first-class fields

What it still does that this does not, and what this declines on purpose, are written down rather than left to a table's silence: seven items in VAULT-PARITY-GAPS.md and twelve in VAULT-PARITY-REFUSED.md, both at the root of this repository. The short version is that the remaining gaps are ergonomics — a render-spec ConfigMap, two exit-on-failure policies, log format — and the refusals are the proxy, the cache, arbitrary commands, and anything that would let a pod rewrite the container this webhook injects.

The injector's template idiom, translated

The Vault injector spells "render me a connection string" as a per-secret annotation pair; here the same result is one source and one template, because the whole pod has one resolved document:

# Vault Agent Injector:
#   vault.hashicorp.com/agent-inject-secret-db-creds: "secret/data/db-app"
#   vault.hashicorp.com/agent-inject-template-db-creds: |
#     {{- with secret "secret/data/db-app" -}}
#     postgres://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/appdb
#     {{- end }}

# dynamic-config-k8s, the same string from the same KV secret:
dynamic-config.rs/source: "vault"
dynamic-config.rs/endpoint: "https://vault.vault.svc:8200"
dynamic-config.rs/key: "secret/db-app"
dynamic-config.rs/auth: "kubernetes"
dynamic-config.rs/auth-role: "db-app"
dynamic-config.rs/path: "/config/db.env"
dynamic-config.rs/template: |
  DATABASE_URL=postgres://{{ username }}:{{ password }}@postgres:5432/appdb

The template owns the bytes (minijinja, strict-undefined: a typo is an error, not an empty string), so any shape works — a URL, an .env, a whole config file. What does NOT translate is the database/creds/… path in the injector's example: that is Vault's dynamic secrets engine, credentials minted per-request with leases — the boundary the next paragraph prices. This agent reads KV; for minted-with-TTL credentials, run the Vault Agent beside it.

One more idiom, matched: the injector's several -secret-<name> pairs per pod are this webhook's named renders — source.db, key.db, path.db beside the default, one agent and one file per name, all in one shared directory. When the pod wants them MERGED into a single document instead, the config server composes sections and the pod reads one endpoint. What has no counterpart is agent-inject-command (a post-render hook): env-restart covers the restart case, and anything richer belongs to the app.

Honest edge the other way: for dynamic Vault secrets with leases (database credentials minted per-pod, TTL renewal mid-life), the Vault Agent is the purpose-built tool and this is not — this agent re-fetches documents; it does not manage leases.

Against External Secrets Operator

The closest relative of the operator half — same split between a namespaced store and a platform-owned cluster store, different product:

External Secrets Operatordynamic-config-k8s
Store definitionSecretStore / ClusterSecretStoreDynamicConfigClass / ClusterDynamicConfigClass — the same two scopes, allowlist included
Outputa Kubernetes Secret, alwaysa ConfigMap, a Secret, or a file no etcd ever sees
The etcd tradeevery delivered secret lives in etcdonly the Secret target does, and choosing it is explicit — the file path exists precisely to avoid it
Data modelkey-by-key secret mappingwhole configuration documents: precedence, validation, formats, provenance
Env deliveryenvFrom the Secretthe same via envEntries — or env-inject, which needs no Secret at all
Backendsvery many secret managersnine configuration stores
TemplatingSecret templatesminijinja over the resolved document

Honest edge the other way: as a secret-synchronisation fleet tool across dozens of managers (AWS/GCP/Azure SM, Doppler, 1Password…), ESO has breadth this project does not chase — the store list here grows by demand, not by roadmap.

Use cases, mapped

  • Airflow / env-only software → env-inject over a rendered dotenv (example); add env-restart: "true" and a changed document restarts just that container in seconds — otherwise changes wait for the next pod start, and that limit is stated instead of hidden.
  • Grafana / anything that re-reads files → the sidecar; live updates, zero etcd, tmpfs only (example).
  • Strimzi-shaped operators / JVM client.properties → the Secret target, file or envEntries shape (example).
  • Multi-tenant platforms → ClusterDynamicConfigClass with a namespaces allowlist; tenants never see a credential (example).
  • A chart's existingSecret / an operator's secretName: → the Secret target with shape: entries — leaf keys verbatim, so the names some other chart already chose are met exactly (example).
  • All of it at once → the four-component shop stack: three secrets injected three ways (a chart's existingSecret, a secretKeyRef env, a mounted-and-live file), the API on env-inject + env-restart, the worker on live files — credentials existing only in the platform namespace.
  • Vault dynamic database credentials with TTLs → the Vault Agent Injector, genuinely. Use both: it owns the lease, this owns the configuration.