The Annotation Contract

v1, and it is the API: a change here is a breaking change of this integration, whatever the binaries think. Two golden files in the webhook's test suite byte-compare the full admission response, so the contract cannot move without a reviewed diff saying it moved.

The core seven

Naming a store once

    dynamic-config.rs/class: "team-vault"
    dynamic-config.rs/key: "billing/config.json"
    dynamic-config.rs/path: "/config/app.yaml"

A DynamicConfigClass — the object the operator has read since 0.1.1 — holds the source, the endpoint and the credential's Secret. A pod that names one says only what is its own.

The class supplies defaults: a pod that names both a class and its own endpoint keeps its own, which is the rule the installation document already follows. Its own namespace is looked in first, then the cluster scope, so a namespace can override a platform default without asking the platform.

Off unless an administrator turns it on (webhook.classes.enabled), and the reason is worth reading. The admission path calls the API server nowhere — that is what keeps a busy API server from failing every pod creation in the cluster — and enabling this does not change it: the classes are listed on a background timer into a map in memory, and admission reads the map. What it does change is that the webhook holds a credential and a cluster-wide read of two custom resources, which is worth having only if pods actually name classes.

The cost is a synchronisation delay a mounted ConfigMap already has. A class created seconds ago may not have been polled yet, and a pod naming it is refused with that sentence rather than admitted without the store the class was to supply.

Two refusals are about who may use what:

• a ClusterDynamicConfigClass whose namespaces list does not include the pod's — the list is what keeps cluster-scoped from meaning anyone
• a cluster class whose credential Secret lives in another namespace. A pod mounts Secrets from its own namespace and nowhere else, so that class is usable by a DynamicConfigRender, which reads the Secret itself, and not by an injected agent

The namespace gate, before any pod is read

Every key below is per-pod. One optional guard sits a level above: with webhook.namespaceGating=true the webhook selects only namespaces labeled dynamic-config.rs/injection: enabled — a label, not an annotation, because the gate lives in the webhook configuration's namespaceSelector and Kubernetes selectors cannot see annotations. Inside a gated namespace the per-pod dynamic-config.rs/inject: "true" is still required; the security page owns the trade (blast radius, per-namespace failurePolicy: Fail), and examples/namespace-gating.yaml is the ready-to-apply shape.

What the webhook writes back

Written by the webhook, never by a pod. A mutating webhook is not called once — reinvocationPolicy: IfNeeded asks the API server to call it again whenever a later webhook changes the pod, and some controllers resubmit a spec that has already been admitted. A marked pod is passed through untouched; without the mark, the second pass would add the agent again, and two containers with one name is a pod the API server refuses.

A pod that sets this annotation to anything else is refused, with a message saying so. Setting it to injected by hand is a way of saying "do not inject me", which inject: "false" already says more clearly.

Behaviour

Freshness, and what happens when a store is not there

The store is not always reachable, and the document does not always still exist. Five annotations say what the agent does about it; every default is the one that keeps a running application running.

Dynamic secrets

For a Vault path that mints a credential rather than storing one — database/creds/…, pki/issue/…, aws/creds/….

Vault revokes an expired lease on its own eventually; revoking on the way out is what makes eventually into now. See Vault.

Reaching the store over TLS

Two settings beside ca-configmap and tls-secret, and they are not the same kind of thing: one moves which name is checked, the other stops checking.

Not every store can express either. The clients differ, and a store that cannot say something refuses the whole configuration by name rather than ignoring it:

The two columns are almost disjoint, and that is the clients rather than a choice: ureq can turn verification off and cannot override a name, tonic can override a name and cannot turn verification off.

What tls-skip-verify costs

It is not a weaker TLS. It is TLS without the part that makes it mean anything: any party on the network path can present any certificate, read what is sent, and rewrite what comes back — and what comes back from a store is the configuration and the credentials this pod is about to run on.

So it is the one annotation in this contract a workload cannot reach on its own. Four things gate it:

• an administrator must set webhook.allowTlsSkipVerify: true, or the annotation is refused with that value named
• it is refused alongside ca-configmap: naming an authority and then not checking it are two answers to one question
• every pod using it earns an admission warning, which kubectl prints
• the agent logs it at start and reports dynamic_config_agent_tls_verification_skipped 1, so one alert finds every pod doing it

Rotation, without a restart

The kubelet rewrites a mounted ConfigMap or Secret in place: the file the container sees changes and nothing tells the process. Every store client in this family reads its trust material once, when it builds — so before 0.3.0 a rotated CA meant a pod restart, and a rotation nobody restarted for meant a store that stopped answering at a moment unrelated to the rotation.

The agent now watches the files it was given and rebuilds the client when they move. A rebuild is not a restart: the process stays up, the rendered file never leaves the volume, the last-known-good stays, and every counter keeps counting — only the client is new. dynamic_config_agent_tls_reloads_total counts them, and a fleet where that stays at zero through a CA rotation is a fleet that did not notice.

A rotation writes more than one file, so a change is confirmed by reading twice a quarter-second apart: rebuilding a TLS client from a half-written certificate and key is a failure that reads as a bad certificate.

Tokens needed none of this and never did — the projected service-account token and the config server's bearer file are re-read on every use, so the rotation that actually happens hourly was always picked up.

Try ca-configmap first. The two situations people usually reach for skip-verify in — a development server with a self-signed certificate, an enterprise private CA — are both one more certificate to trust, which is one annotation and keeps the server authenticated.

Telling the application what it is running

Telling the application the file moved

    dynamic-config.rs/notify-http: "http://127.0.0.1:8080/-/reload"

The rename is atomic, so a consumer never sees half a document — but nobody tells it the document changed. nginx, Prometheus and most legacy daemons reload on a request and on nothing else.

After the rename, never before: the whole promise of the notification is that the document is already there when it arrives. One attempt, a two second deadline, and never fatal — the file is already correct, so a notification that did not land has undone nothing. notifications_total and notification_failures_total say how it went.

Localhost only, by construction. http://127.0.0.1, http://localhost or http://[::1], and nothing else — an agent that will POST to an arbitrary URL is an SSRF primitive holding this pod's store credential. The address is checked at admission and in the agent, because the two run in different places. Refused with mode: init, where the container writes once and exits before the application it would notify has started.

There is no signal form. Signalling a sibling container needs shareProcessNamespace: true — a pod-wide change to the process boundary between containers, which the webhook will not make on a workload's behalf.

Failures where somebody is already looking

    dynamic-config.rs/events: "true"

kubectl describe pod is where an operator looks first, and a render failure was not there — it was in the sidecar's log, one kubectl logs -c dynamic-config-agent away from the question. With this the agent writes a Warning Event on its own pod for a render that failed (RenderFailed) and for a document that vanished from the store (DocumentAbsent).

Off, and twice opt-in. The sidecar carries no API credential in any other configuration — that is a property the rest of this design leans on — so writing Events means mounting the pod's service-account token beside the application. An administrator has to create the Role (agent.events.enabled with the namespaces, which grants create on events and nothing else) and set webhook.allowEvents, and only then may a pod ask. Asking without the installation offering it is refused, with the chart value named, rather than admitted into a 403 at the first failure.

An Event is commentary on work that already happened, so it is written off the loop: a render never waits on the API server, and an Event that could not be written is logged once and dropped.

Some of the fleet before all of it

    dynamic-config.rs/canary-configmap: "rollout"    # or "rollout/percent"

kubectl create configmap rollout --from-literal=percent=5
# look at the five per cent
kubectl patch configmap rollout --type merge -p '{"data":{"percent":"100"}}'

A change published to every pod at once either works or is an incident. This buys the third outcome: a few pods take it, somebody looks, and the rest follow or do not.

Which pods is the pod's own name hashed into a bucket from 0 to 99, so the cohort is deterministic and stable — a cohort that reshuffled as it widened would put every pod through the new document eventually and prove nothing about any of them. No coordination and no leader: five thousand agents each answer for themselves.

Who widens it is whoever edits the ConfigMap. That is why the percentage is a mounted file and not an annotation: the kubelet rewrites a mount in place, so the cohort grows with no pod restart — and a restart would discard the very state a canary exists to watch. A pod outside the cohort holds what it fetched and publishes it the moment the number passes its bucket, so nothing has to be re-fetched and the store need not say anything again.

0 holds everybody and 100 holds nobody; both ends read as what they mean. A file that is missing or does not parse is no canary at all rather than zero — a typo must not freeze the fleet on its current document.

Two series say what is happening: canary_holding is 1 while this pod is outside the cohort, and canary_percent is the number it last read. Together with applied from below they are the question somebody has to answer before widening: did the pods that took it actually run it?

Refused with mode: init, where the agent publishes once and exits.

One interaction to know. A held pod's document is by definition not the newest, so max-staleness will eventually call it stale. That is correct — it is stale — but a long-running canary under a short staleness ceiling will make held pods unready, and the two numbers should be chosen together.

Knowing the application actually applied it

    dynamic-config.rs/meta: "true"          # so the app can read the digest
    dynamic-config.rs/require-ack: "true"   # and readiness waits for it

renders_total says a document reached disk. It says nothing about whether the application read it, and an application still running the previous one while every dashboard reports success is the outage this closes.

The application POSTs the fingerprint it is running to /applied on the metrics port:

curl -fsS -XPOST --data-binary @- http://127.0.0.1:9110/applied   <<< "$(jq -r .fingerprint /config/.app.yaml.meta)"

200 means that is the current document, 409 means a different one is published and names it — a restarted application acknowledges what it read before the restart, and a slow one acknowledges a generation the store has already replaced. Neither is an error the application caused, and neither is convergence.

A Rust, Python or Node application does not need the meta file: fingerprint() is the same string, from the same digest.

Three series come out of it, and the third is the one to alert on:

require-ack makes that readiness. The pod stays 0/2 until the application says it applied the document, so a Service sends no traffic to a pod running configuration nobody confirmed. Off by default and refused without a readiness probe or on mode: init: it needs the application's cooperation, and one that never acknowledges would never become ready.

An application that never acknowledges is not penalised. It leaves applied at zero and nothing else changes.

What the file was before

    dynamic-config.rs/history: "3"

The rename that publishes a new document is the same rename that destroys the old one, so what was the file before? — the question an incident starts with — has no answer by the time anybody asks it. This keeps the replaced generation beside the render, under /config/.app.yaml.history/<when>-<digest>.yaml, newest kept and oldest pruned past the count.

At most ten, and off unless asked: the rendered volume is the pod's own memory by default, so every kept generation is charged to a limit that is 64Mi by default. Each copy takes the mode the render had, so a history entry cannot be read by anything the document itself could not be.

Refused with volume-medium: disk, where a replaced secret would sit on node-backed storage, outlive the pod that held it, and survive a reboot.

This is not a rollback. Putting an old document back needs the application to say the new one is bad, and nothing here can hear that — half a feature that implies the other half is worse than neither. This keeps files; a person reads them.

When something else writes to the rendered file

    dynamic-config.rs/on-drift: "repair"     # or warn (default), fail

The agent owns the file; the volume is shared. A debug session, an init container with an opinion, an application that rewrites its own configuration — any of them can change it, and until the next change arrives from the store, nothing notices.

Checked on the same cadence the store is read. warn is the default because the agent cannot know whether the write was a mistake or the point; what it can do is stop the difference being invisible. repair writes the rendered document back, for a file whose contents are the store's and nobody else's. fail ends the agent, so the pod restarts and renders again.

Either way dynamic_config_agent_drift goes to 1 and drift_total counts it. An unreadable file counts as drifted too: one that has been deleted, or has become a directory, is not the file this agent wrote either.

Several files, one fetch

also.<name> cuts more files from the same fetched document, and publishes them all or none of them:

dynamic-config.rs/source: "vault"
dynamic-config.rs/key: "secret/myapp"
dynamic-config.rs/path: "/config/app.yaml"
# the same document, a section of it, as a second file:
dynamic-config.rs/also.db: "/config/db.env"
dynamic-config.rs/also-section.db: "database"

One fetch, one generation, one refusal if any of them cannot be written — so a failure in the third does not leave the first two published. Each write is still its own atomic rename: a reader can catch the microseconds between two of them, which is a rename apart rather than a fetch apart.

This is within one document. Several stores cannot share a generation — two stores have no common instant and no protocol between them can say "these two reads are the same" — so a second store stays a named render, below.

Several documents, one pod

Every store-shaped key accepts a .<name> suffix, and each name is one more injected agent writing one more file into the same directory:

# the default render:
dynamic-config.rs/source: "vault"
dynamic-config.rs/key: "secret/myapp"
dynamic-config.rs/path: "/config/app.yaml"
dynamic-config.rs/auth: "kubernetes"
# a second, named `cache`:
dynamic-config.rs/source.cache: "redis"
dynamic-config.rs/endpoint-secret.cache: "redis-url/url"
dynamic-config.rs/key.cache: "myapp/cache.json"
dynamic-config.rs/path.cache: "/config/cache.toml"

Per-name: everything a store needs — source, endpoint(+secret), key, path, section, watch cadence, every auth key, CA/TLS/ssh material (mounted under suffixed paths), templates, file-mode. Pod-wide, on purpose: mode, volume medium, resources, run-as identity, env-inject (the default render's file is the one that can become the environment). Two rules, refused with the fix named: every path lives in the default path's directory (one shared volume), and each name's source/key/path are required exactly like the default's. Container names follow the suffix — dynamic-config-agent-cache in kubectl get pod, so a broken render says which one it is.

Authentication

Each store takes its own methods; the store's page spells every one of them out with full manifests. The webhook forwards these to the agent verbatim, and the agent refuses a wrong combination at startup — in the pod's events, not as a store error twenty minutes later.

Secrets and certificates

Secret material never rides an annotation — kubectl describe pod prints annotations and arguments to anyone with pod read access. These four name Kubernetes objects instead; the webhook mounts them and the agent reads them. The geography is fixed.

Value forms, source by source

What endpoint, key and auth take for each value of source — the store pages carry the full manifests, this table is the lookup:

The prefix is claimed territory

Every dynamic-config.rs/* annotation must be a key this page lists — an unknown one fails the admission. The rule exists for the typo: tokne-secret silently ignored would be a pod running without the authentication it declared, and nobody would know until the audit. Annotations outside the prefix are none of this webhook's business and pass untouched.

Retiring a key

Nothing here is deprecated. When something is, the contract has somewhere to say so: every key is one row of a registry inside the webhook, carrying the release that retired it and what to write instead. A pod that sets a retired key is admitted with a warning naming its replacement, rather than refused — a contract that breaks a working pod to make a point is a contract people pin an old version of.

Two properties come from the key list being one table rather than several. Whether a key may take a .name suffix is read off the same row, so the two can no longer disagree; and the documentation is checked against that table by a test, so a key cannot be accepted and left undocumented.

The one liberty the strictness buys back: because every key is validated, template and template-configmap could ship later without a migration — pods that used them early were refused, not silently ignored. They shipped; the Rendering page owns them.

What fails the admission

A wrong ask fails the admission. A pod that says inject: "true" and misspells the rest is refused with the reason, not started without its configuration — silence there is how an outage begins. The refusals, verbatim from the tests:

• inject set to anything but "true"/"false"
• a missing required annotation, named in the message
• mode outside init | sidecar | both
• watch-seconds that does not parse as whole seconds
• a *-secret value without the <secret-name>/<key> slash
• endpoint and endpoint-secret both set — one address, one place
• ssh-secret alongside an auth other than ssh-key
• volume-medium outside memory | disk; native-sidecar outside true | false
• a resource annotation that is not a Kubernetes quantity
• file-mode outside octal 0400–0777 — setuid bits answer no question, and an owner-unreadable file is write-only noise
• agent-run-as-user/-group of 0 — the agent stays nonroot in every configuration
• inject-containers naming a container the pod does not have, or set to nothing at all — a render nothing can read is a render nobody asked for
• init-first with mode: sidecar, where there is no init container
• agent-run-as-same-user with no runAsUser to inherit, alongside agent-run-as-user, or on an application that runs as root
• revoke-grace past the pod's terminationGracePeriodSeconds, at "0", or without dynamic
• notify-http at anything but a localhost address, or with mode: init
• on-drift outside warn | repair | fail
• history at "0", past ten, or alongside volume-medium: disk
• agent-ephemeral-request larger than agent-ephemeral-limit
• timeout at "0", which is no deadline rather than the store's own
• agent-image naming an image no webhook.agentImageAllow prefix admits
• require-ack without a readiness probe, or with mode: init
• canary-configmap with mode: init
• events where the installation does not offer it — the refusal names the chart value an administrator sets
• class naming one that is not visible, one whose namespaces list excludes this pod, or one whose credential is in another namespace
• tls-skip-verify where the installation does not offer it, or alongside ca-configmap
• env-inject naming a container that inject-containers leaves out: the wrapper sources the rendered file, so it has to be able to read it
• env-inject with mode: sidecar, a container the pod does not have, or a container with no explicit command
• env-restart without env-inject, without mode: both, or on a container that already owns a livenessProbe
• agent-env entries that are not NAME=value, names that are not UPPER_SNAKE, a name set twice, or a name that shadows what aws-secret already sets
• an agent-env name outside the installation's allowlist for the pod's namespace — the refusal names the chart value that opens it
• a source sourceDeny turns off in the pod's namespace, or one missing from a non-empty sourceAllow — checked on every render, named suffixes included
• an annotation that differs from a PINNED installation value (!-marked, or any set value under overridable: "false") — the refusal names both values; restating the pinned value passes
• any dynamic-config.rs/* key the contract does not list
• template and template-configmap both set — one template, one place

What passes with a warning

Not every misconfiguration earns a refusal. Kubernetes lets an admission response carry warnings, which kubectl prints and a controller records, and four configurations earn one:

The bar is high on purpose: a warning on every admission is a warning nobody reads. Each of these is a configuration that works and is probably not what was meant.

Checking a manifest before the cluster does

The webhook is a pure function of a pod and an installation, so the same decision is available without a cluster:

$ dynamic-config-webhook validate pod.yaml deployment.yaml
pod.yaml: allowed, and an agent would be injected
deployment.yaml: refused (InvalidAnnotation)
  dynamic-config.rs/auth-role is required for vault kubernetes auth

JSON or YAML, one exit code for the lot, installation defaults from the process's own environment — so a CI job that runs the webhook's image with the chart's environment gets the answer the cluster would give. The point of catching it here is that the alternative is catching it from a rollout that will not start.

Whatever passes the webhook is validated again by the agent, which knows the store-by-store rules (vault kubernetes needs auth-role, consul kubernetes needs auth-mount, a certificate needs its key). Those refusals land in the injected container's log and the pod's events.

DynamicConfigClass shrinks all of this to a class reference — the operator page carries the shape.

The agent-env gate

agent-env puts variables on the container that holds store credentials, and environment steers SDKs — HTTPS_PROXY reroutes the agent's traffic, AWS_CA_BUNDLE and SSL_CERT_FILE swap its trust roots. So the names that may pass are the installer's decision, declared once:

# values.yaml
webhook:
  agentEnvAllow: "payments: HTTPS_PROXY, AWS_*; *: RUST_LOG"

Semicolons separate groups; each group takes an optional namespace: head (absent or * means every namespace); a trailing * on a name is a prefix glob. Empty — the default — refuses the annotation everywhere. Kustomize installs set the same grammar in the DYNAMIC_CONFIG_WEBHOOK_AGENT_ENV_ALLOW variable, and either way the webhook validates it at startup and refuses to serve on a typo.

The gate is NOT a namespace annotation, by design: whoever edits a namespace is usually the tenant being gated, and a gate its subject can open is not a gate. It is also not read from the namespace object at admission time — the webhook holds no RBAC and asks the API server for nothing; the pod's namespace arrives inside the AdmissionReview, and the ruling comes from the webhook's own configuration.

The source gates

The same authority model gates which STORES a pod may use:

webhook:
  sourceAllow: "payments: vault, s3; *: consul"   # empty = every store
  sourceDeny: "sandbox: git"                       # subtractive, wins

sourceAllow empty means every store everywhere — the safe default for an upgrade; non-empty means ONLY the listed, per namespace. sourceDeny turns stores off outright and outranks the allowlist. Both cover every render on the pod — a denied store cannot ride in on a named suffix — and both refusals name the namespace and the value that opens the gate. Kustomize sets DYNAMIC_CONFIG_WEBHOOK_SOURCE_ALLOW / _SOURCE_DENY; a name that is not a real store fails webhook startup, so a typo cannot silently gate nothing.

Defaults come in tiers

Every default in the table above is only the LAST tier of four:

annotation  >  per-store default  >  fleet default  >  built-in

The middle tiers are the installation's (the full list): fleet defaults cover every knob — resources, file-mode, watch-seconds, mode, volume-medium, native-sidecar, agent-run-as-user/-group, metrics-port, path, source, and a fleet-wide agent environment — and agent.defaults.perStore sets, one store at a time, the same knobs PLUS every store-shaped annotation: endpoint, key, auth and its friends, the credential Secrets, the templates. A pod can deploy carrying nothing but inject: "true". Pod-wide knobs (mode, volume, resources, identity) take the DEFAULT render's store tier; per-render knobs resolve against each render's own store. Every tier is validated with the SAME rules as the annotation it stands in for, at webhook startup — and any value can be PINNED (!, or overridable: "false"), refusing a differing annotation instead of being overridden by it.

Installation Defaults and Gates carries the full knob table, a per-store example for all nine stores, and the gates in depth.