Vault

KV v2, and the widest auth surface of the six — all seven of the vault crate's methods work through annotations. Ordered from development to production; if the pod runs on Kubernetes (it does — you are reading the k8s book), start with kubernetes.

The key is <mount>/<path>, the way vault CLI users write it: secret/myapp is the myapp path on the secret KV mount. What the agent renders is the secret's data — the fields under data.data in vault's own JSON.

A token

The method every tutorial starts with, and the only one that cannot recover on its own: there are no credentials behind it to log in again with. A renewable token is still renewed; a revoked one is the 3 a.m. page.

vault policy write myapp-read - <<'HCL'
path "secret/data/myapp" { capabilities = ["read"] }
HCL
vault token create -policy=myapp-read -ttl=768h -format=json \
  | jq -r .auth.client_token \
  | xargs -I{} kubectl create secret generic vault-token --from-literal=token={}

apiVersion: v1
kind: Pod
metadata:
  name: billing
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "vault"
    dynamic-config.rs/endpoint: "https://vault.vault.svc:8200"
    dynamic-config.rs/key: "secret/myapp"
    dynamic-config.rs/path: "/config/rendered.yaml"
    dynamic-config.rs/token-secret: "vault-token/token"
    dynamic-config.rs/ca-configmap: "vault-ca"
spec:
  containers:
    - name: app
      image: myapp:1

Kubernetes: the pod's own identity

No secret distributed anywhere: the agent presents the pod's service-account JWT to vault's kubernetes auth method and gets a token scoped to a role. This is the pod YAML the webhook's second golden file locks:

metadata:
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "vault"
    dynamic-config.rs/endpoint: "https://vault.vault.svc:8200"
    dynamic-config.rs/key: "secret/myapp"
    dynamic-config.rs/path: "/config/rendered.yaml"
    dynamic-config.rs/auth: "kubernetes"
    dynamic-config.rs/auth-role: "myapp"
    dynamic-config.rs/ca-configmap: "vault-ca"

Vault side, once:

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host=https://kubernetes.default.svc

vault write auth/kubernetes/role/myapp \
  bound_service_account_names=billing \
  bound_service_account_namespaces=default \
  policies=myapp-read ttl=1h

Three details that bite:

• The JWT is re-read from disk at every login, not cached: the kubelet rotates projected tokens, and a copy taken at startup expires with the pod still running. Nothing to configure — stated here so a security review can check the box.

• A projected token with a custom audience lives off the conventional path; point at it:

      dynamic-config.rs/auth-token-path: "/var/run/secrets/tokens/vault"
  

• A second kubernetes mount (multi-cluster Vaults mount one per cluster) is auth-mount:

      dynamic-config.rs/auth-mount: "kubernetes-prod-eu"
  

AppRole

The usual choice for a service outside Kubernetes, and for shops that want an identity Vault owns rather than the cluster. Two halves: the role id is public and rides an annotation; the secret id is a secret and rides a Secret.

vault auth enable approle
vault write auth/approle/role/myapp policies=myapp-read \
  secret_id_ttl=90d token_ttl=1h

vault read -field=role_id auth/approle/role/myapp/role-id
vault write -f -field=secret_id auth/approle/role/myapp/secret-id \
  | xargs -I{} kubectl create secret generic vault-approle --from-literal=secret-id={}

    dynamic-config.rs/auth: "approle"
    dynamic-config.rs/auth-role: "<the role id>"
    dynamic-config.rs/password-secret: "vault-approle/secret-id"

The secret id travels as DYNAMIC_CONFIG_AGENT_PASSWORD — the second secret slot, which has no flag equivalent on purpose.

JWT / OIDC

Any JWT a jwt mount trusts — a CI job's id token, a workload identity from another platform. The JWT itself is the credential, so it rides the token Secret:

    dynamic-config.rs/auth: "jwt"
    dynamic-config.rs/token-secret: "workload-jwt/jwt"
    dynamic-config.rs/auth-role: "myapp"      # when the mount has no default
    dynamic-config.rs/auth-mount: "jwt-ci"    # when not mounted at "jwt"

Userpass and LDAP

Same shape, different directory. The username is a name and rides an annotation; the password rides a Secret:

kubectl create secret generic vault-ldap --from-literal=password=…

    dynamic-config.rs/auth: "ldap"            # or "userpass"
    dynamic-config.rs/auth-username: "svc-billing"
    dynamic-config.rs/password-secret: "vault-ldap/password"

Cert: a TLS client certificate

The certificate IS the credential: vault's cert method authenticates the TLS handshake itself. One kubernetes.io/tls Secret carries the pair:

kubectl create secret tls vault-client --cert=client.pem --key=client-key.pem

    dynamic-config.rs/auth: "cert"
    dynamic-config.rs/tls-secret: "vault-client"
    dynamic-config.rs/auth-role: "myapp"      # when the mount does not pick by subject
    dynamic-config.rs/ca-configmap: "vault-ca"

Vault side:

vault auth enable cert
vault write auth/cert/certs/myapp certificate=@client-ca.pem \
  allowed_common_names=billing.default policies=myapp-read

Vault Enterprise namespaces

One annotation, passed through to the X-Vault-Namespace header:

    dynamic-config.rs/namespace: "team-payments"

How the token lifecycle behaves

Worth knowing before the first incident, whatever the method:

• Close to expiry, the token is renewed — or, if it cannot be, replaced by a fresh login. This is the path that normally fires.
• After a request, a 403 is treated as the token stopped working and triggers exactly one fresh login and retry. Clocks skew, Vault revokes, a lease is shorter than it said.
• If a fresh token also gets 403, the problem is the policy rather than the lease, and the error says so instead of hanging in a retry loop.

When it fails