dynamic-config on Kubernetes

Annotate a pod; an agent appears in it that renders configuration from a remote store to a file the application watches. The agent-injector shape, for configuration.

metadata:
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "consul"
    dynamic-config.rs/endpoint: "http://consul:8500"
    dynamic-config.rs/key: "myapp/config.json"
    dynamic-config.rs/path: "/config/rendered.toml"

The application needs no store client, no store credential, and no code change: it reads a file, and if it reads it with any dynamic-config binding it also reloads on every re-render — the agent writes atomically, exactly the whole-file event a watcher wants.

When not to use this. The engine runs in-process everywhere; a Rust, Python or Node service that can hold a store credential should usually use its own binding's remote support and skip the sidecar entirely. This integration exists for the pods that want files rendered for them: Java services reading .properties, anything that must not carry store credentials in-process, and fleets standardising one injection pattern.

The three pieces, staged

Install

helm install dynamic-config deploy/helm

That is the whole install: no dependencies, nothing to pre-create. The chart mints a CA and a ten-year serving certificate at install time, embeds the caBundle in the webhook configuration, and reuses the Secret on upgrades so the trust does not silently rotate. The webhook terminates that TLS in-process — the API server speaks HTTPS to admission webhooks and nothing else.

The cert-manager mode

helm install dynamic-config deploy/helm \
  --set webhook.certManager.enabled=true \
  --set webhook.certManager.issuerRef.name=<your-issuer>

With cert-manager the certificate is renewed — cainjector maintains the caBundle, and the webhook picks up the renewed pair from disk without a restart (it polls the mounted files for a changed modification time). The trade between the two:

Both mount the same Secret shape at the same path; the webhook cannot tell them apart, which is what makes switching later a values change.

failurePolicy

Ignore by default, and the book owns the trade: Fail would make the webhook a single point of failure for every pod creation in selected namespaces, while Ignore means an annotated pod created during a webhook outage starts without its agent — loudly, because the file its application waits for never appears. Flip it with --set webhook.failurePolicy=Fail once the webhook has earned it in your cluster; the security page carries the full argument.

Namespace gating

Clusters that prefer opt-in injection (the Istio shape) set webhook.namespaceGating=true and label the namespaces that want it:

kubectl label namespace payments dynamic-config.rs/injection=enabled

Everything else is invisible to the webhook — the security page explains what that buys.

What the chart hardens for you

The security page is the complete inventory; the short list: both deployments run as non-root with the restricted-PSS container posture, the webhook's ServiceAccount mounts no API token, kube-system and the release namespace are excluded from injection, two replicas ride a PodDisruptionBudget, and tag: latest fails the render. An optional NetworkPolicy writes down that the webhook accepts the API server and calls nobody.

One namespace of its own

Install into a dedicated namespace, always:

helm install dynamic-config deploy/helm -n dynamic-config --create-namespace

The webhook configuration excludes its own namespace by name — the self-deadlock guard — so a release installed into default silently excludes every workload sharing default with it. The chart's NOTES print a warning when that happens; the e2e smoke installs the dedicated way for the same reason.

Values, all of them

The chart README is the full values reference — naming overrides, common labels, per-component images and pull policy, service accounts, probe and rollout tuning, extraEnv/extraVolumes escape hatches, namespace gating, the operator's RBAC toggle. Everything the templates read is in that table.

Without helm: kustomize

deploy/kustomize/ carries the same resources as a base plus two TLS overlays — cert-manager, and bring-your-own-PEMs via secretGenerator. Its README is the three-step walkthrough, including the one caBundle patch kustomize cannot express. The CRDs ship inside the base, drift-gated against the operator's --crds output like every other copy.

The smoke test

The e2e smoke (e2e/smoke.sh) is the install, end to end, against a kind cluster: the chart in its zero-dependency default, a live Consul, one annotated pod, the rendered file read back out of it, and the injected container's security posture asserted on the running pod. CERT_MANAGER=1 e2e/smoke.sh runs the same flow through the other TLS mode.

The Annotation Contract

v1, and it is the API: a change here is a breaking change of this integration, whatever the binaries think. Two golden files in the webhook's test suite byte-compare the full admission response, so the contract cannot move without a reviewed diff saying it moved.

The core seven

Behaviour

Authentication

Each store takes its own methods; the store's page spells every one of them out with full manifests. The webhook forwards these to the agent verbatim, and the agent refuses a wrong combination at startup — in the pod's events, not as a store error twenty minutes later.

Secrets and certificates

Secret material never rides an annotation — kubectl describe pod prints annotations and arguments to anyone with pod read access. These four name Kubernetes objects instead; the webhook mounts them and the agent reads them. The geography is fixed.

Value forms, source by source

What endpoint, key and auth take for each value of source — the store pages carry the full manifests, this table is the lookup:

The prefix is claimed territory

Every dynamic-config.rs/* annotation must be a key this page lists — an unknown one fails the admission. The rule exists for the typo: tokne-secret silently ignored would be a pod running without the authentication it declared, and nobody would know until the audit. Annotations outside the prefix are none of this webhook's business and pass untouched.

The one liberty the strictness buys back: because every key is validated, template and template-configmap could ship later without a migration — pods that used them early were refused, not silently ignored. They shipped; the Rendering page owns them.

What fails the admission

A wrong ask fails the admission. A pod that says inject: "true" and misspells the rest is refused with the reason, not started without its configuration — silence there is how an outage begins. The refusals, verbatim from the tests:

• inject set to anything but "true"/"false"
• a missing required annotation, named in the message
• mode outside init | sidecar | both
• watch-seconds that does not parse as whole seconds
• a *-secret value without the <secret-name>/<key> slash
• endpoint and endpoint-secret both set — one address, one place
• ssh-secret alongside an auth other than ssh-key
• volume-medium outside memory | disk; native-sidecar outside true | false
• a resource annotation that is not a Kubernetes quantity
• any dynamic-config.rs/* key the contract does not list — including source: etcd|nats|s3, refused with a message naming the version that takes them
• template and template-configmap both set — one template, one place

Whatever passes the webhook is validated again by the agent, which knows the store-by-store rules (vault kubernetes needs auth-role, consul kubernetes needs auth-mount, a certificate needs its key). Those refusals land in the injected container's log and the pod's events.

DynamicConfigClass (0.3.0) shrinks all of this to a class reference — the operator page carries the shape.

Secrets, Certificates, and What Shows Where

Before wiring any store's authentication, one map of where things are visible. Kubernetes shows different fields to different eyes:

That table decides the whole contract:

• Names travel in annotations — a role name, an auth method's mount, a username. Reading them tells an attacker nothing they could not guess.
• Secrets travel as environment variables drawn from Secrets — the agent reads three, and there is no flag for the second one on purpose:

• Key material travels as mounts, read-only, into the agent container alone — the application containers never see them:

A private CA, end to end

Most internal Vaults, Consuls and git hosts serve TLS from an internal PKI. The chain is one ConfigMap and one annotation:

kubectl create configmap vault-ca --from-file=ca.crt=./internal-ca.pem

metadata:
  annotations:
    dynamic-config.rs/endpoint: "https://vault.vault.svc:8200"
    dynamic-config.rs/ca-configmap: "vault-ca"

The agent gets --ca /etc/dynamic-config/ca/ca.crt, and the store crate under it adds the CA to its trust roots — the same TlsConfig every store crate takes, so the spelling is identical for all six stores.

There is no way to turn verification off. The store crates refuse that setting by design, and the agent adds no flag for it: a configuration channel that skips TLS verification is a configuration channel anyone on the path can write to.

A client certificate

Some stores authenticate with the certificate (vault's cert method), some merely allow mTLS in front. Either way it is one kubernetes.io/tls Secret:

kubectl create secret tls vault-client \
  --cert=./client.pem --key=./client-key.pem

    dynamic-config.rs/tls-secret: "vault-client"

The certificate and key must come together; the agent refuses one without the other before any byte leaves the pod.

Why the pod's own identity beats all of this

Three of the six stores can authenticate a pod with no distributed secret at all — the pod's service-account token or the node's cloud identity:

• vault: auth: kubernetes
• consul: auth: kubernetes
• firestore: auth: metadata-server

Where one of those is available, prefer it: nothing to rotate, nothing to leak, and revocation is the platform's own. The token-shaped methods on every page exist for the stores and shops where it is not.

The Security Posture

Everything this integration does to a cluster, listed where an auditor can find it. The one-line summary: injection never relaxes a pod's posture, secrets never appear where kubectl describe reaches, and the webhook holds no credentials at all.

The injected agent complies with restricted PSS

Every injected container — init, sidecar, or native sidecar — carries the full restricted Pod Security Standard posture, so injection works in namespaces that enforce pod-security.kubernetes.io/enforce: restricted and passes the audit in ones that only warn:

securityContext:
  runAsNonRoot: true
  runAsUser: 65532          # distroless nonroot
  runAsGroup: 65532
  allowPrivilegeEscalation: false
  capabilities: { drop: ["ALL"] }
  readOnlyRootFilesystem: true
  seccompProfile: { type: RuntimeDefault }

The root filesystem is read-only because the agent writes exactly one place: the shared volume. Which is —

The rendered file lives in memory

The shared emptyDir is medium: Memory by default: rendered configuration regularly carries credentials, and tmpfs keeps them off the node's disk and out of its backups. The file is gone when the pod is. A pod that prefers disk (a giant document, a memory-tight node) says so:

    dynamic-config.rs/volume-medium: "disk"

Note the accounting: tmpfs pages count against the pod's memory. Configuration documents are small; the default agent memory limit below leaves room.

The agent's resource ask

Injected with requests and limits so it can never be the reason the node evicts the app:

resources:
  requests: { cpu: 10m, memory: 32Mi }
  limits: { memory: 64Mi }        # no CPU limit: throttling a config
                                  # agent buys nothing, delays reloads

Four annotations move them per pod: agent-cpu-request, agent-memory-request, agent-cpu-limit, agent-memory-limit — each a Kubernetes quantity, refused at admission when it is not.

Native sidecars

On Kubernetes 1.29+, ask for the sidecar as the platform now spells it:

    dynamic-config.rs/native-sidecar: "true"

The watching agent becomes an init container with restartPolicy: Always — started before the app containers, stopped after them, and a Job with one finishes, where a classic sidecar would hold it in Running forever. With mode: "both" the one-shot init still lands first, so the file-exists-before-the-app guarantee survives the move.

Secrets: where each thing is allowed to appear

The contract's rule: names in annotations, secrets in Secret-backed environment variables, key material in read-only mounts into the agent container alone. The webhook enforces it — there is no annotation that accepts a token value, and the password slot has no flag on the agent at all.

The webhook holds nothing

• Its ServiceAccount sets automountServiceAccountToken: false, and the deployment repeats it. The webhook reads the AdmissionReview it is handed and answers; it never calls the API server, so it carries no credential to steal.
• It terminates TLS in-process with the certificate the chart issued; the private key never leaves its mount. Renewals are picked up from disk without a restart.
• The optional NetworkPolicy (networkPolicy.enabled=true) writes both facts down for the CNI: ingress only on 8443, egress empty.

The webhook cannot select itself

The webhook configuration excludes kube-system, kube-node-lease and the release's own namespace by name — a mutating webhook that can select its own pods can deadlock its own rollout, and one that can mutate the control plane is a cluster risk with no matching reward. Add more with webhook.excludeNamespaces.

failurePolicy: the whole trade

Ignore (default): an unreachable webhook lets pods through un-injected. The failure is visible where it matters — the annotated pod's application waits for a file that never comes — and invisible where it does not: un-annotated pods, which are most pods, never notice.

Fail: no annotated pod can start un-injected, and no pod at all can start in selected namespaces while the webhook is down. Two replicas, a PodDisruptionBudget and topology spread are the chart's mitigations; they shrink the window, they do not close it.

Start on Ignore, alert on the webhook's availability, and flip to Fail when the alert has been quiet long enough to trust.

Every admission leaves a line

The webhook logs one structured line per decision that matters — namespace, pod name, source, and patched or refused — and no annotation values: endpoints and role names belong in the cluster, not in every log aggregator downstream. Pods that never asked are counted but not logged.

GET /metrics on the serving port exposes the counters in Prometheus text format:

dynamic_config_admissions_total{outcome="skipped"} 1042
dynamic_config_admissions_total{outcome="patched"} 63
dynamic_config_admissions_total{outcome="refused"} 2

A rising refused is somebody fighting the contract; alert on it.

Typos cannot pass

An unknown dynamic-config.rs/* annotation fails the admission — the reference explains the rule. The enterprise version of the argument: a misspelled token-secret that is silently ignored produces a pod that runs, connects anonymously, and reads whatever the store's anonymous policy allows. Refusing at admission turns a quiet posture downgrade into a loud create-time error.

Templates are code, and scoped like data

A template renders only the resolved document — the same value the application reads. There is no file access, no environment access, no network in the template language; a hostile template can misrender the config file it owns and nothing else. Undefined keys are strict errors, so a template cannot silently swallow a value either. Keeping templates in ConfigMaps puts them through the same review as the code they effectively are.

Namespace gating

webhook.namespaceGating=true flips injection to Istio-style opt-in: only namespaces labeled dynamic-config.rs/injection: enabled are selected at all. Two things follow:

• the blast radius of the webhook is exactly the namespaces that asked;
• failurePolicy: Fail becomes a per-namespace promise — a platform team can fail closed for its opted-in tenants without coupling every pod CREATE in the cluster to this webhook.

Fleet-wide agent defaults

The injected container's resource defaults come from the chart (agent.defaults.*), not from a constant in a binary — platform teams set the fleet's floor once, and the per-pod annotations still override it. The same values file pins the agent image the webhook injects.

Supply chain

• Images are distroless, run as nonroot, and the chart refuses tag: latest at render time; a digest value pins harder than a tag can.
• The release workflow signs images with cosign and attaches SBOMs; ghcr.io and Docker Hub carry the same digests.
• The agent binary embeds the engine and the store crates from crates.io — the same audited path every other binding uses; there is no k8s-only fork of anything.

What this integration never does

No hostPath, no privileged, no capabilities added, no writes outside the shared volume, no API server calls from the webhook, no credentials in flags or annotations, and no way — flag or annotation — to turn TLS verification off toward any store.

Rendering

The agent does one resolution, through the same engine every binding uses, and writes the resolved document — so the file on disk is what an in-process consumer would have computed, not a second dialect.

The output format follows --out's extension: .json, .toml, .yaml, .ini, .properties.

The flat formats are legal here and refused by the engine's save — both on purpose. save's contract is a typed round trip, which a string-widening format cannot keep. A rendered file for a consumer is a different contract, and the agent owns it, stated:

• Nested tables become dotted keys (properties) or sections (INI).
• A string that would widen on the way back in — "1.10", "true" — is double-quoted in INI, so the round trip through the engine's own parser answers the same document. There is a test that holds exactly this.
• Arrays are refused, by path. Neither format has them; inventing an encoding would be a dialect of one. Render to json/toml/yaml when the document has lists.

Writes are write-then-rename, so a watching application sees whole files — the same courtesy an atomic-save editor pays, and the reason the engine's own watcher tolerates a 25ms grace.

On a fetch failure the sidecar keeps the last rendered file and says so in its log — keep-last-good, the organisation's standing behaviour. An init run with nothing yet rendered fails instead, which fails the pod, which is what an init container is for.

Templates

Without one, the agent renders the resolved document verbatim — same keys, same shapes, only the format changes with the extension. That is the right default and it stays the default.

A template takes over when verbatim cannot serve: an application that wants DATABASE_URL=postgres://… assembled from three keys, a framework with its own nesting, a file with a header. The template owns the output bytes, which also frees the extension — .env and .conf become legal exactly there.

apiVersion: v1
kind: ConfigMap
metadata:
  name: billing-template
data:
  template: |
    DATABASE_URL=postgres://{{ db.user }}@{{ db.host }}:{{ db.port }}/billing
    BETA={{ flags.beta }}

    dynamic-config.rs/path: "/config/app.env"
    dynamic-config.rs/template-configmap: "billing-template"
    # or, for one-liners:
    dynamic-config.rs/template: "db={{ db.host }}:{{ db.port }}"

The syntax is minijinja's — Jinja2: {{ value }}, {% for %}, {% if %}, filters. The template's context is the resolved document, the same value every binding reads, so a template cannot see anything the application could not.

The semantics that matter in production:

• Undefined is strict. {{ db.hots }} is a render error, not an empty string — a typo that silently renders nothing would ship a broken file with a clean exit code. At startup the error is fatal and lands in the pod's events; during a watch, the running pod keeps its last good file, like any fetch failure.
• Booleans render as true/false, not Python's True — a template writes config files, and every format this agent speaks spells them lowercase.
• The trailing newline survives. Env files want one; what the template author wrote is what lands.
• The ConfigMap is re-read at every render, so editing the template takes effect on the next tick — no rollout. It is also why the template belongs in a ConfigMap: it is code, and it gets reviewed and versioned like code.
• template and template-configmap together are refused at admission: one template, one place.

The Six, and the Three Waiting

One page per store the agent speaks, each with the full pod YAML for every authentication method the store takes — copy, adjust names, apply. Everything is runnable; the consul flow is what the e2e harness runs on every pull request.

Common to all six:

• The store's address rides dynamic-config.rs/endpoint — or endpoint-secret when the address itself carries a password.
• The document's key rides dynamic-config.rs/key; the per-store syntax (mount/path, application/profile, a file path) is on the store's page.
• Secret material rides Secrets, never annotations; the geography page has the one diagram.
• A private CA is the same one annotation everywhere: dynamic-config.rs/ca-configmap.

Every pairing on these pages also exists as a ready-to-apply manifest in the repository's examples/ directory — thirteen files, each self-contained with its Secret placeholders.

etcd, NATS, S3 — 0.2.0

The other three store crates exist and work — from the engine, from the bindings — but their clients are async, and the 0.1 agent drives the blocking RemoteSource trait. The agent refuses them by name today:

--source etcd lands in 0.2.0 (its client is async); consul, vault,
config-server, firestore, git and redis are the 0.1 stores

Until then, the pattern that works today: put a config server in front. The server side speaks all nine stores including the async three, and the agent speaks the server. That indirection is also the answer when a fleet of pods should not each hold store credentials — the server holds them once.

Consul

A KV path over HTTP. Four ways in, ordered from development to production.

The key is a KV path with an extension — myapp/config.json — and the extension names the stored format; the rendered format is the path annotation's extension, and the two need not agree.

Anonymous

Correct for a Consul with ACLs disabled, and for a default policy that allows reads — both ordinary in development. No auth annotation at all:

apiVersion: v1
kind: Pod
metadata:
  name: billing
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "consul"
    dynamic-config.rs/endpoint: "http://consul.infra.svc:8500"
    dynamic-config.rs/key: "myapp/config.json"
    dynamic-config.rs/path: "/config/rendered.toml"
spec:
  containers:
    - name: app
      image: myapp:1

This is the exact flow the e2e harness runs on every pull request.

An ACL token

The CONSUL_HTTP_TOKEN you already have, moved into a Secret:

kubectl create secret generic consul-token --from-literal=token=b3a7…

metadata:
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "consul"
    dynamic-config.rs/endpoint: "http://consul.infra.svc:8500"
    dynamic-config.rs/key: "myapp/config.json"
    dynamic-config.rs/path: "/config/rendered.toml"
    dynamic-config.rs/token-secret: "consul-token/token"

The webhook wires the Secret into the agent as DYNAMIC_CONFIG_AGENT_TOKEN; nothing token-shaped appears in the pod spec. A static token is also the one method that cannot recover on its own — when it expires or is revoked, the fetch fails until the Secret is updated. The two methods below fix that.

Kubernetes: login with the pod's identity

Consul's auth methods issue a token in exchange for a bearer the method trusts — for the kubernetes type, the pod's own service-account JWT. Nothing is distributed; the token is minted per login.

Consul side, once (the Consul docs on auth methods carry the full story):

consul acl auth-method create -type kubernetes -name k8s-pods \
  -kubernetes-host https://kubernetes.default.svc \
  -kubernetes-ca-cert @/path/to/cluster-ca.crt \
  -kubernetes-service-account-jwt "$REVIEWER_JWT"

consul acl binding-rule create -method k8s-pods \
  -bind-type role -bind-name 'config-reader' \
  -selector 'serviceaccount.namespace==default'

Pod side, two annotations — auth-mount carries the auth method's name, because that is the coordinate Consul's login endpoint wants:

metadata:
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "consul"
    dynamic-config.rs/endpoint: "http://consul.infra.svc:8500"
    dynamic-config.rs/key: "myapp/config.json"
    dynamic-config.rs/path: "/config/rendered.toml"
    dynamic-config.rs/auth: "kubernetes"
    dynamic-config.rs/auth-mount: "k8s-pods"

The JWT is read from disk at every login rather than once: the kubelet rotates projected service-account tokens, and a copy taken at startup expires with the pod still running. If a projected volume moved the token off the conventional path, say where:

    dynamic-config.rs/auth-token-path: "/var/run/secrets/tokens/consul"

JWT: any bearer the method trusts

The same login endpoint, with the bearer supplied instead of read from the service-account mount — an OIDC id token, a JWT signed by something Consul trusts. The bearer is a secret, so it rides a Secret:

    dynamic-config.rs/auth: "jwt"
    dynamic-config.rs/auth-mount: "oidc-ci"
    dynamic-config.rs/token-secret: "ci-idtoken/jwt"

TLS

An internal Consul serving from a private PKI needs its CA trusted; one ConfigMap, one annotation:

    dynamic-config.rs/endpoint: "https://consul.infra.svc:8501"
    dynamic-config.rs/ca-configmap: "consul-ca"

A cluster fronting Consul with mTLS adds tls-secret.

When it fails

The agent keeps the last good render on any fetch failure — the rendering page spells out that guarantee.

Beyond the agent's flags, the consul crate can also scope reads to a datacenter and tune blocking-query waits; those knobs live on the store crate itself for applications embedding the engine directly.

Vault

KV v2, and the widest auth surface of the six — all seven of the vault crate's methods work through annotations. Ordered from development to production; if the pod runs on Kubernetes (it does — you are reading the k8s book), start with kubernetes.

The key is <mount>/<path>, the way vault CLI users write it: secret/myapp is the myapp path on the secret KV mount. What the agent renders is the secret's data — the fields under data.data in vault's own JSON.

A token

The method every tutorial starts with, and the only one that cannot recover on its own: there are no credentials behind it to log in again with. A renewable token is still renewed; a revoked one is the 3 a.m. page.

vault policy write myapp-read - <<'HCL'
path "secret/data/myapp" { capabilities = ["read"] }
HCL
vault token create -policy=myapp-read -ttl=768h -format=json \
  | jq -r .auth.client_token \
  | xargs -I{} kubectl create secret generic vault-token --from-literal=token={}

apiVersion: v1
kind: Pod
metadata:
  name: billing
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "vault"
    dynamic-config.rs/endpoint: "https://vault.vault.svc:8200"
    dynamic-config.rs/key: "secret/myapp"
    dynamic-config.rs/path: "/config/rendered.yaml"
    dynamic-config.rs/token-secret: "vault-token/token"
    dynamic-config.rs/ca-configmap: "vault-ca"
spec:
  containers:
    - name: app
      image: myapp:1

Kubernetes: the pod's own identity

No secret distributed anywhere: the agent presents the pod's service-account JWT to vault's kubernetes auth method and gets a token scoped to a role. This is the pod YAML the webhook's second golden file locks:

metadata:
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "vault"
    dynamic-config.rs/endpoint: "https://vault.vault.svc:8200"
    dynamic-config.rs/key: "secret/myapp"
    dynamic-config.rs/path: "/config/rendered.yaml"
    dynamic-config.rs/auth: "kubernetes"
    dynamic-config.rs/auth-role: "myapp"
    dynamic-config.rs/ca-configmap: "vault-ca"

Vault side, once:

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host=https://kubernetes.default.svc

vault write auth/kubernetes/role/myapp \
  bound_service_account_names=billing \
  bound_service_account_namespaces=default \
  policies=myapp-read ttl=1h

Three details that bite:

• The JWT is re-read from disk at every login, not cached: the kubelet rotates projected tokens, and a copy taken at startup expires with the pod still running. Nothing to configure — stated here so a security review can check the box.

• A projected token with a custom audience lives off the conventional path; point at it:

      dynamic-config.rs/auth-token-path: "/var/run/secrets/tokens/vault"
  

• A second kubernetes mount (multi-cluster Vaults mount one per cluster) is auth-mount:

      dynamic-config.rs/auth-mount: "kubernetes-prod-eu"
  

AppRole

The usual choice for a service outside Kubernetes, and for shops that want an identity Vault owns rather than the cluster. Two halves: the role id is public and rides an annotation; the secret id is a secret and rides a Secret.

vault auth enable approle
vault write auth/approle/role/myapp policies=myapp-read \
  secret_id_ttl=90d token_ttl=1h

vault read -field=role_id auth/approle/role/myapp/role-id
vault write -f -field=secret_id auth/approle/role/myapp/secret-id \
  | xargs -I{} kubectl create secret generic vault-approle --from-literal=secret-id={}

    dynamic-config.rs/auth: "approle"
    dynamic-config.rs/auth-role: "<the role id>"
    dynamic-config.rs/password-secret: "vault-approle/secret-id"

The secret id travels as DYNAMIC_CONFIG_AGENT_PASSWORD — the second secret slot, which has no flag equivalent on purpose.

JWT / OIDC

Any JWT a jwt mount trusts — a CI job's id token, a workload identity from another platform. The JWT itself is the credential, so it rides the token Secret:

    dynamic-config.rs/auth: "jwt"
    dynamic-config.rs/token-secret: "workload-jwt/jwt"
    dynamic-config.rs/auth-role: "myapp"      # when the mount has no default
    dynamic-config.rs/auth-mount: "jwt-ci"    # when not mounted at "jwt"

Userpass and LDAP

Same shape, different directory. The username is a name and rides an annotation; the password rides a Secret:

kubectl create secret generic vault-ldap --from-literal=password=…

    dynamic-config.rs/auth: "ldap"            # or "userpass"
    dynamic-config.rs/auth-username: "svc-billing"
    dynamic-config.rs/password-secret: "vault-ldap/password"

Cert: a TLS client certificate

The certificate IS the credential: vault's cert method authenticates the TLS handshake itself. One kubernetes.io/tls Secret carries the pair:

kubectl create secret tls vault-client --cert=client.pem --key=client-key.pem

    dynamic-config.rs/auth: "cert"
    dynamic-config.rs/tls-secret: "vault-client"
    dynamic-config.rs/auth-role: "myapp"      # when the mount does not pick by subject
    dynamic-config.rs/ca-configmap: "vault-ca"

Vault side:

vault auth enable cert
vault write auth/cert/certs/myapp certificate=@client-ca.pem \
  allowed_common_names=billing.default policies=myapp-read

Vault Enterprise namespaces

One annotation, passed through to the X-Vault-Namespace header:

    dynamic-config.rs/namespace: "team-payments"

How the token lifecycle behaves

Worth knowing before the first incident, whatever the method:

• Close to expiry, the token is renewed — or, if it cannot be, replaced by a fresh login. This is the path that normally fires.
• After a request, a 403 is treated as the token stopped working and triggers exactly one fresh login and retry. Clocks skew, Vault revokes, a lease is shorter than it said.
• If a fresh token also gets 403, the problem is the policy rather than the lease, and the error says so instead of hanging in a retry loop.

When it fails

Config Server

This project's own server — the aggregation answer. The server side speaks all nine store crates (including the async three the agent cannot drive yet) and merges files, profiles and remote stores into one document per <application>/<profile>; the agent then speaks the server. The remote book owns the server's full story; what follows is the k8s-side wiring.

Two reasons to put it between the pods and the stores:

• Credential concentration. A fleet of pods each holding a vault token is a fleet of tokens to rotate. The server holds the store credentials once; the pods hold short client tokens that grant exactly one application's sections.
• The async stores today. etcd, NATS and S3 reach the agent in 0.2.0 — through the server they work now.

The wiring

The key is <application>/<profile>:

apiVersion: v1
kind: Pod
metadata:
  name: billing
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "config-server"
    dynamic-config.rs/endpoint: "http://config-server.infra.svc:8888"
    dynamic-config.rs/key: "billing/prod"
    dynamic-config.rs/path: "/config/rendered.json"
spec:
  containers:
    - name: app
      image: myapp:1

The bearer token

The server's [[server.clients]] blocks name each client and the applications it may read. The token is a secret; it rides a Secret:

# server.toml, server side
[[server.clients]]
name = "billing-pods"
token = "…at least 32 bytes…"
applications = ["billing"]

kubectl create secret generic config-server-token --from-literal=token=…

    dynamic-config.rs/token-secret: "config-server-token/token"

There is no other method on this store — one bearer, scoped server-side, is the whole model. Asking for auth: anything is refused by the agent with a sentence saying exactly that.

TLS

The server behind TLS from an internal PKI is the same one annotation as everywhere:

    dynamic-config.rs/endpoint: "https://config-server.infra.svc:8443"
    dynamic-config.rs/ca-configmap: "internal-ca"

A server requiring client certificates takes tls-secret alongside.

When it fails

Firestore

A Google Cloud document read as configuration. The endpoint is not a url: it is <project> or <project>/<database>, and the key is collection/document (nest deeper as environments/prod/config/db).

On GKE the right method is the first one, and it involves no secret at all.

Metadata-server (Workload Identity)

The workload's own identity, from the metadata server — reachable from GKE, Cloud Run, GCE, and nowhere else, which is the security property that makes it the default. The agent asks for a token, gets a short-lived one, and renews it as it approaches expiry.

auth can be omitted entirely — metadata-server is what the agent does for firestore when nothing else is asked:

apiVersion: v1
kind: Pod
metadata:
  name: billing
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "firestore"
    dynamic-config.rs/endpoint: "acme-prod"
    dynamic-config.rs/key: "config/billing"
    dynamic-config.rs/path: "/config/rendered.json"
spec:
  serviceAccountName: billing
  containers:
    - name: app
      image: myapp:1

GKE side, the Workload Identity pairing (the GKE docs own the full ceremony):

gcloud iam service-accounts create billing-reader
gcloud projects add-iam-policy-binding acme-prod \
  --member serviceAccount:billing-reader@acme-prod.iam.gserviceaccount.com \
  --role roles/datastore.viewer

gcloud iam service-accounts add-iam-policy-binding \
  billing-reader@acme-prod.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:acme-prod.svc.id.goog[default/billing]"

kubectl annotate serviceaccount billing \
  iam.gke.io/gcp-service-account=billing-reader@acme-prod.iam.gserviceaccount.com

Access token

A token somebody already obtained — gcloud auth print-access-token produces one. It expires within the hour and the agent cannot renew it, so this is a debugging method, not a deployment method; the honest use is a one-shot init container in a test cluster:

    dynamic-config.rs/mode: "init"
    dynamic-config.rs/auth: "access-token"
    dynamic-config.rs/token-secret: "gcp-token/token"

Emulator

The Firestore emulator wants no credential and a different endpoint; api-url points the API somewhere other than Google's:

    dynamic-config.rs/auth: "emulator"
    dynamic-config.rs/api-url: "http://firestore-emulator.test.svc:8080"
    dynamic-config.rs/endpoint: "demo-project"

The named database

The second database in a project is the endpoint's second segment:

    dynamic-config.rs/endpoint: "acme-prod/eu-config"

When it fails

Git

Configuration that lives where its reviews live. The agent polls the ref's advertisement — one handshake per tick, a transfer only when the ref actually moved — so a 15-second watch does not hammer the host.

The endpoint is anything git understands (https://…, ssh://…, git@host:org/repo.git); the key is the file's path inside the repository; the ref defaults to branch main.

Anonymous

A public repository over HTTPS — no auth annotation:

apiVersion: v1
kind: Pod
metadata:
  name: billing
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "git"
    dynamic-config.rs/endpoint: "https://github.com/acme/config.git"
    dynamic-config.rs/key: "billing/prod.yaml"
    dynamic-config.rs/path: "/config/rendered.yaml"
    dynamic-config.rs/ref: "main"
spec:
  containers:
    - name: app
      image: myapp:1

A token over HTTPS

How every host takes a token: HTTP basic auth with the token in the password half. GitHub PATs and App installation tokens, GitLab deploy and project tokens, Azure DevOps PATs — all the same shape. The username half is filler that these hosts ignore; the agent sends x-access-token, the value GitHub documents:

kubectl create secret generic config-repo-token --from-literal=token=ghp_…

    dynamic-config.rs/auth: "token"
    dynamic-config.rs/token-secret: "config-repo-token/token"

For the rare host that does read the username, name it:

    dynamic-config.rs/auth-username: "deploy"

A GitLab deploy token is the least-privilege pick on that platform: scope read_repository, one repository, its own expiry.

An SSH deploy key

One kubernetes.io/ssh-auth Secret; its conventional key name is ssh-privatekey, and the webhook mounts it 0400 because ssh refuses group-readable keys:

ssh-keygen -t ed25519 -f deploy_key -N ""
# register deploy_key.pub as a read-only deploy key on the host
kubectl create secret generic config-deploy-key \
  --type=kubernetes.io/ssh-auth --from-file=ssh-privatekey=deploy_key

    dynamic-config.rs/endpoint: "git@github.com:acme/config.git"
    dynamic-config.rs/ssh-secret: "config-deploy-key"

auth: ssh-key is implied by ssh-secret when no auth is named. The key is offered with IdentitiesOnly=yes, so an agent holding other keys cannot exhaust the server's auth tries before the right one.

The stock image caveat, out loud: git-over-SSH is carried by the ssh program, exactly as git itself does it — and the distroless agent image does not contain one. HTTPS works from the stock image; SSH needs an image with an ssh client:

FROM ghcr.io/dynamic-config-rs/dynamic-config-agent:0.1.0 AS agent
FROM alpine:3.20
RUN apk add --no-cache openssh-client ca-certificates \
 && printf 'github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl\n' \
    >> /etc/ssh/ssh_known_hosts
COPY --from=agent /dynamic-config-agent /dynamic-config-agent
ENTRYPOINT ["/dynamic-config-agent"]

…and the chart's agent.image value points at it. The known_hosts line is the second half ssh insists on; pin your own host's key, not a copy of this one.

Branch, tag, commit

ref takes four spellings:

    dynamic-config.rs/ref: "main"           # a branch, plainly
    dynamic-config.rs/ref: "branch:release" # the same, spelled out
    dynamic-config.rs/ref: "tag:v1.4"       # a tag
    dynamic-config.rs/ref: "commit:8f3a…"   # one exact tree, forever

A tag or commit still ticks the watch loop, and still never transfers — useful with mode: init for a pinned, reproducible render.

A self-hosted host with a private CA

The same annotation as every other store:

    dynamic-config.rs/endpoint: "https://git.internal.acme/config.git"
    dynamic-config.rs/ca-configmap: "internal-ca"

When it fails

Redis

A key read as a document, watched by polling. The key carries its format in its extension (myapp/config.json); a key without one needs the document format to be guessable, so give it one.

Redis is the one store whose credentials travel in the url — requirepass and ACL users have no other place. That shapes the whole page: the moment the url grows a password, it stops being an annotation and becomes a Secret.

An open Redis

Development, a sidecar cache, a cluster-internal instance behind a NetworkPolicy:

apiVersion: v1
kind: Pod
metadata:
  name: billing
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "redis"
    dynamic-config.rs/endpoint: "redis://redis.infra.svc:6379/0"
    dynamic-config.rs/key: "myapp/config.json"
    dynamic-config.rs/path: "/config/rendered.toml"
spec:
  containers:
    - name: app
      image: myapp:1

The trailing /0 is the database index; omit it for 0.

requirepass

The password goes into the url, and the url goes into a Secret — endpoint-secret replaces endpoint entirely, and the agent reads the address from DYNAMIC_CONFIG_AGENT_ENDPOINT:

kubectl create secret generic redis-url \
  --from-literal=url='redis://:s3cr3t@redis.infra.svc:6379/0'

metadata:
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "redis"
    dynamic-config.rs/endpoint-secret: "redis-url/url"
    dynamic-config.rs/key: "myapp/config.json"
    dynamic-config.rs/path: "/config/rendered.toml"

Setting both endpoint and endpoint-secret fails the admission — one address, one place. Error messages redact the password even when the url cannot be parsed, because a parse error is the error most likely to be pasted somewhere.

An ACL user

Redis 6+ ACLs put a username before the password. Server side:

ACL SETUSER config-reader on >s3cr3t ~myapp/* +get resetchannels

The url names the user, read-only on exactly the config prefix:

kubectl create secret generic redis-url \
  --from-literal=url='redis://config-reader:s3cr3t@redis.infra.svc:6379/0'

TLS

rediss:// (two esses) plus the CA:

    dynamic-config.rs/endpoint: "rediss://redis.infra.svc:6380/0"
    dynamic-config.rs/ca-configmap: "redis-ca"

A redis:// url with TLS material is refused — a deployment that believes it is encrypted and is not — and there is no way to turn verification off. A client certificate is tls-secret, as everywhere.

With a password too, the whole rediss://user:pass@… url rides endpoint-secret and the CA annotation stays as it is.

When it fails

The Operator (0.3.0)

The CRDs are settled and generated today (deploy/crds.json, kept in sync with the Rust types by a CI gate); the reconcilers are 0.3.0's work. What follows is the contract they will honour, written first so the annotation users of 0.1/0.2 can see where their YAML is going.

DynamicConfigClass — name the store once

The class bundles source, endpoint and the token Secret, so pods stop repeating them:

apiVersion: dynamic-config.rs/v1alpha1
kind: DynamicConfigClass
metadata:
  name: infra-consul
  namespace: billing
spec:
  source: consul
  endpoint: http://consul.infra.svc:8500
  tokenSecret: consul-agent-token     # optional; its `token` key travels

A pod then says only what is its own:

metadata:
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/class: "infra-consul"      # 0.3.0
    dynamic-config.rs/key: "myapp/config.json"
    dynamic-config.rs/path: "/config/rendered.toml"

The long-form annotations keep working forever; the class is sugar over them, not a replacement.

DynamicConfigRender — a ConfigMap instead of a sidecar

For workloads that cannot take an injected container — third-party charts, jobs, anything sidecar-averse — the operator renders into a ConfigMap on a cadence, and the workload mounts it like any other:

apiVersion: dynamic-config.rs/v1alpha1
kind: DynamicConfigRender
metadata:
  name: billing-config
  namespace: billing
spec:
  class: infra-consul
  key: myapp/config.json
  target:
    configMap: billing-rendered
    file: config.properties        # the extension picks the format here too
  intervalSeconds: 30
status:                             # written by the operator
  renderedAt: "2026-08-18T09:00:00Z"
  lastError: null                   # kind and path only, never a value

Two honesty notes, stated now so 0.3.0 cannot soften them:

• ConfigMap propagation is slow — kubelet syncs mounted ConfigMaps on its own cadence (up to a minute-plus). The sidecar's emptyDir is the low-latency path; DynamicConfigRender trades latency for no-sidecar. The page will carry measured numbers when the reconciler ships.
• A ConfigMap is not a Secret. Rendering a Vault document into a ConfigMap is a downgrade the operator will refuse unless the target says secret: instead — that field arrives with the reconciler.

What the operator will not do

Own lifecycles inside pods, restart workloads on render, or template documents. It renders and it reports; reacting is the workload's business, and the whole engine exists so reacting is cheap.

Troubleshooting

Each symptom, its first command, and the usual cause. The agent's logs are JSON; kubectl logs <pod> -c dynamic-config-agent | jq . is the readable form.

The file never appears

kubectl get pod <name> -o jsonpath='{.spec.containers[*].name}'

No dynamic-config-agent in the list → the webhook never mutated. In order of likelihood:

1. The pod predates the webhook install — admission only sees CREATE.
2. failurePolicy: Ignore swallowed a webhook outage: kubectl get events -n <ns> | grep dynamic-config and the webhook deployment's own logs say which.
3. The annotation said inject: "false" or misspelled the prefix — dynamic-config.rs/, with the dot and the slash.

Agent present, file absent → the agent is failing. Its log carries the store error verbatim minus values:

kubectl logs <pod> -c dynamic-config-agent | jq -r '.fields.error // .fields.message'

The admission was denied

That is the contract working: inject: "true" with a missing or malformed companion annotation fails the pod's creation, and the reason names the annotation:

Error creating: admission webhook "inject.dynamic-config.rs" denied the
request: dynamic-config.rs/inject is true, so dynamic-config.rs/path is required

Silently starting without configuration is the failure mode this refusal exists to prevent; add the named annotation.

401/403 from the store

The token travels in DYNAMIC_CONFIG_AGENT_TOKEN, not in annotations.

kubectl exec <pod> -c dynamic-config-agent -- env | grep -c DYNAMIC_CONFIG

0 means the Secret was never mounted onto the agent container. Config-server 401s specifically: the bearer must belong to a [[server.clients]] block whose applications list names the application in your --key — the server's audit log line for the refusal names the client it matched.

The rendered file is stale

The sidecar keeps the last good render on fetch failure, on purpose — staleness with a warning beats an empty file. The log says so at warn level. --watch-seconds too high is the boring cause; a store ACL that started refusing is the interesting one, and the 401 section above applies.

mode: init never refreshes by design: rotation there is a pod restart, which is the trade the Vault page states.

--out refused at startup

The extension picks the format, and only five are legal: .json .toml .yaml .ini .properties. The error lists them; .conf and .cfg are nobody's format and stay refused.

Arrays in the document, flat output requested

`hosts` is an array, and neither flat format has one; render to json,
toml or yaml instead

Exactly what it says: pick a structured output, or reshape the document. The Rendering page owns the reasoning.

Reading what the webhook actually did

The golden test's fixture is the contract, and a live pod can be compared against it:

kubectl get pod <name> -o json | jq '.spec.containers[].volumeMounts'
kubectl get pod <name> -o json | jq '.spec.volumes[] | select(.name == "dynamic-config")'

The pod was created, nothing was injected

In order of likelihood:

1. The namespace is excluded. kube-system, kube-node-lease and the chart's own namespace never get injection, plus anything in webhook.excludeNamespaces:

   kubectl get mutatingwebhookconfiguration dynamic-config \
     -o jsonpath='{.webhooks[0].namespaceSelector}'
   

2. The webhook was down and failurePolicy: Ignore let the pod through. The API server records exactly that:

   kubectl get events --field-selector reason=FailedAdmissionWebhook -A
   kubectl -n <chart-namespace> get pods -l app.kubernetes.io/component=webhook
   

3. TLS trust is broken — the caBundle does not match what the webhook serves. With the self-signed default this happens when the Secret was deleted but the webhook configuration was not re-rendered; helm upgrade heals both sides. The API server's opinion:

   kubectl logs -n <chart-namespace> deploy/dynamic-config-webhook | tail
   

The webhook pod refuses to start: "no TLS material"

The exact message names the two paths it looked at. The Secret dynamic-config-webhook-tls is missing or empty — with cert-manager enabled, check the Certificate:

kubectl describe certificate dynamic-config-webhook

An injected pod is rejected by Pod Security admission

It should not be: the injected container carries the full restricted posture. If a namespace enforces something stricter than restricted (an OPA/Kyverno policy), read the denial message — the security page lists every field the injection sets, so the diff is one screen.

The template refuses to render

Two shapes, two places:

• At startup (a parse error, an undefined key on the first render): the agent exits and the reason is in the injected container's log — strict on purpose, so a typo'd {{ db.hots }} cannot ship an empty string with a clean exit code.

• During a watch (the ConfigMap was edited into an error): the pod keeps its last good file and the agent logs the render error every tick until the template is fixed. Check with:

  kubectl logs <pod> -c dynamic-config-agent | tail
  kubectl get configmap billing-template -o jsonpath='{.data.template}'
  

Stability & Versioning

Experimental, stated plainly: this is the youngest repository in the organisation, the annotation contract is v1, and the operator's reconcilers have not shipped. It stays 0.x until the operator has soak history, whatever the rest of the family does.

• The annotation contract is the API; a breaking change to it bumps the minor and regenerates the golden file in the same commit.
• The three components version together; images are the artefacts.
• The agent's store list grows additively (etcd, nats and s3 in 0.2.0).
• The engine dependency is a caret: an engine patch reaches the images on rebuild.

The repository's ROADMAP carries the ladder in full — the async stores and etcd's two-methods-forever answer, the self-rotating webhook TLS mode and the one narrow RBAC it will cost, the operator's reconcilers.