Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Consul

A KV path over HTTP. Four ways in, ordered from development to production.

The key is a KV path with an extension — myapp/config.json — and the extension names the stored format; the rendered format is the path annotation's extension, and the two need not agree.

Anonymous

Correct for a Consul with ACLs disabled, and for a default policy that allows reads — both ordinary in development. No auth annotation at all:

apiVersion: v1
kind: Pod
metadata:
  name: billing
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "consul"
    dynamic-config.rs/endpoint: "http://consul.infra.svc:8500"
    dynamic-config.rs/key: "myapp/config.json"
    dynamic-config.rs/path: "/config/rendered.toml"
spec:
  containers:
    - name: app
      image: myapp:1

This is the exact flow the e2e harness runs on every pull request.

An ACL token

The CONSUL_HTTP_TOKEN you already have, moved into a Secret:

kubectl create secret generic consul-token --from-literal=token=b3a7…
metadata:
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "consul"
    dynamic-config.rs/endpoint: "http://consul.infra.svc:8500"
    dynamic-config.rs/key: "myapp/config.json"
    dynamic-config.rs/path: "/config/rendered.toml"
    dynamic-config.rs/token-secret: "consul-token/token"

The webhook wires the Secret into the agent as DYNAMIC_CONFIG_AGENT_TOKEN; nothing token-shaped appears in the pod spec. A static token is also the one method that cannot recover on its own — when it expires or is revoked, the fetch fails until the Secret is updated. The two methods below fix that.

Kubernetes: login with the pod's identity

Consul's auth methods issue a token in exchange for a bearer the method trusts — for the kubernetes type, the pod's own service-account JWT. Nothing is distributed; the token is minted per login.

Consul side, once (the Consul docs on auth methods carry the full story):

consul acl auth-method create -type kubernetes -name k8s-pods \
  -kubernetes-host https://kubernetes.default.svc \
  -kubernetes-ca-cert @/path/to/cluster-ca.crt \
  -kubernetes-service-account-jwt "$REVIEWER_JWT"

consul acl binding-rule create -method k8s-pods \
  -bind-type role -bind-name 'config-reader' \
  -selector 'serviceaccount.namespace==default'

Pod side, two annotations — auth-mount carries the auth method's name, because that is the coordinate Consul's login endpoint wants:

metadata:
  annotations:
    dynamic-config.rs/inject: "true"
    dynamic-config.rs/source: "consul"
    dynamic-config.rs/endpoint: "http://consul.infra.svc:8500"
    dynamic-config.rs/key: "myapp/config.json"
    dynamic-config.rs/path: "/config/rendered.toml"
    dynamic-config.rs/auth: "kubernetes"
    dynamic-config.rs/auth-mount: "k8s-pods"

The JWT is read from disk at every login rather than once: the kubelet rotates projected service-account tokens, and a copy taken at startup expires with the pod still running. If a projected volume moved the token off the conventional path, say where:

    dynamic-config.rs/auth-token-path: "/var/run/secrets/tokens/consul"

JWT: any bearer the method trusts

The same login endpoint, with the bearer supplied instead of read from the service-account mount — an OIDC id token, a JWT signed by something Consul trusts. The bearer is a secret, so it rides a Secret:

    dynamic-config.rs/auth: "jwt"
    dynamic-config.rs/auth-mount: "oidc-ci"
    dynamic-config.rs/token-secret: "ci-idtoken/jwt"

TLS

An internal Consul serving from a private PKI needs its CA trusted; one ConfigMap, one annotation:

    dynamic-config.rs/endpoint: "https://consul.infra.svc:8501"
    dynamic-config.rs/ca-configmap: "consul-ca"

A cluster fronting Consul with mTLS adds tls-secret.

When it fails

symptomlook atusual cause
agent log: 403consul acl token read -self with the same tokenthe token lacks key_prefix read on the path
agent log: login refusedconsul acl auth-method read -name k8s-podsbinding rule selector does not match the pod's namespace/SA
agent starts, file never updatesconsul kv get myapp/config.jsonthe key moved, or the watch interval is long — check watch-seconds

The agent keeps the last good render on any fetch failure — the rendering page spells out that guarantee.

Beyond the agent's flags, the consul crate can also scope reads to a datacenter and tune blocking-query waits; those knobs live on the store crate itself for applications embedding the engine directly.