Externalizing Configuration
John Harris
Application configuration is anything that varies between environments. For example, stateful applications depend on different database endpoints in testing and production environments. A best practice in cloud-native development is to decouple configuration from code. This means keeping database endpoints and credentials separate from the application’s source code. If your application has environment-specific configuration hard-coded into its repository, VMware recommends refactoring your application to decouple source code from configuration.
Runtime Injection
Once configuration has been decoupled from source code, configure your application to consume it at runtime by presenting environment variables or mounting a file in the container.
Environment Variables
Most programming languages support reading environment variables from the local environment. The following is an example of this in Go.
package main
import "os"
import "fmt"
func main() {
fmt.Println("DATABASE_ENDPOINT:", os.Getenv("DATABASE_ENDPOINT"))
}
This program produces the following output:
$ export DATABASE_ENDPOINT=foo.example.com
$ go run environment-variables-example.go
DATABASE_ENDPOINT: foo.example.com
Once your application is configured to consume configuration from environment variables, configure your container orchestrator to inject them at runtime. The following example shows how to configure a Kubernetes pod with environment variables.
apiVersion: v1
kind: Pod
metadata:
name: environment-variable-example
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "echo $DATABASE_ENDPOINT" ]
env:
- name: DATABASE_ENDPOINT
value: "foo.example.com"
Mounted Files
Another way to inject configuration is through a local file. Most programming languages support reading from a local file. The following is an example of this in Go.
package main
import (
"fmt"
"io/ioutil"
"os"
)
func main() {
dat, err := ioutil.ReadFile("/etc/example-configuration")
if err != nil {
panic(e)
}
fmt.Print(string(dat))
}
This program produces the following output:
$ echo "DATABASE_ENDPOINT=foo.example.com" > /etc/example-configuration
$ go run read-file-example.go
DATABASE_ENDPOINT=foo.example.com
Once your application is configured to consume configuration from a local file,
configure your container orchestrator to mount it at runtime. In Kubernetes, use
a
ConfigMap
to mount a configuration file inside a pod. The following example shows how to
configure a Kubernetes pod with a ConfigMap mounted at /etc/configuration
.
---
apiVersion: v1
kind: ConfigMap
metadata:
name: example-configuration
namespace: default
data:
DATABASE_ENDPOINT: foo.example.com
---
apiVersion: v1
kind: Pod
metadata:
name: example-configmap
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "cat /etc/configuration" ]
volumeMounts:
- name: config-volume
mountPath: /etc/configuration
volumes:
- name: config-volume
configMap:
name: example-configuration
restartPolicy: Never
These examples cover basic use cases within the context of Kubernetes. For more details, refer to the Kubernetes documentation on environment variables and ConfigMaps.
Sensitive Information
Some configuration, such as passwords and OAuth tokens, are sensitive
information and should be handled with extra care. This section shows how to
handle sensitive information in the context of Kubernetes using
Secrets. The
biggest difference between a Secret and ConfigMap is that Secret data is stored
in base64
encoding, whereas ConfigMap data is stored in plain text.
Secret Creation
Create a secret using kubectl
with the following command:
kubectl create secret generic prod-db-secret --from-literal=username=produser --from-literal=password=Y4nys7f11
This produces the following object in Kubernetes:
apiVersion: v1
kind: Secret
metadata:
name: database-credentials
type: Opaque
data:
username: YWRtaW4=
password: MWYyZDFlMmU2N2Rm
Environment Variables
Secrets may be mounted in a pod as an environment variable, just like configuration.
apiVersion: v1
kind: Pod
metadata:
name: example-secret-environment-variable
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "echo $SECRET_USERNAME && echo $SECRET_PASSWORD" ]
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: database-credentials
key: username
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: database-credentials
key: password
restartPolicy: Never
Mounted Files
Similarly, secrets may also be mounted in a pod as a file.
apiVersion: v1
kind: Pod
metadata:
name: example-secret-file
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "cat /etc/credentials" ]
volumeMounts:
- name: credential-mount
mountPath: "/etc/credentials"
readOnly: true
volumes:
- name: credential-mount
secret:
secretName: database-credentials
Credential Managers
Your operations team may use a dedicated service to manage the lifecycle of
credentials, such as HashiCorp’s Vault.
Consequently, credentials would not be sourced from Kubernetes. Your deployment
manifest would pull application credentials using an
initContainer.
An initContainer
is a job that runs in a container before your application is
deployed and uses the same underlying filesystem. This can be used to pull
credential information from outside your Kubernetes cluster and write it to a
location that will later be mounted and read by your application. The
vault-kubernetes-authenticator
project implements this idea by using an initContainer
to pull credentials
from an instance of HashiCorp’s Vault. Below is an example of how you can
configure your Deployment manifest with this tool to integrate your application
with Vault. This example is modified from the project’s GitHub page to align
with examples in this document.
apiVersion: v1
kind: Pod
metadata:
name: example-secret-vault
spec:
securityContext:
runAsUser: 1001
fsGroup: 1001
volumes:
- name: vault-auth
emptyDir:
medium: Memory
- name: vault-secrets
emptyDir:
medium: Memory
initContainers:
- name: vault-authenticator
image: sethvargo/vault-kubernetes-authenticator:0.2.0
imagePullPolicy: Always
volumeMounts:
- name: vault-auth
mountPath: /var/run/secrets/vaultproject.io
env:
- name: VAULT_ROLE
value: myapp-role
securityContext:
allowPrivilegeEscalation: false
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "ls /home/vault && cat /var/run/secrets/vaultproject.io" ]
volumeMounts:
- name: vault-auth
mountPath: /home/vault
- name: vault-secrets
mountPath: /var/run/secrets/vaultproject.io
env:
- name: HOME
value: /home/vault
This manifest describes a deployment in which the vault-authenticator
container runs as an initContainer
, pulls a credential from vault and writes
it to /var/run/secrets/vaultproject.io
. The stateful-application
container
then mounts this directory and consumes its contents for use as credentials.
GitOps Credential Management
GitOps refers to the practice of using a git
repository as a single source
of truth for application source code and configuration. Although storing
application source code and configuration in version control is a best practice,
you should avoid storing sensitive information in version control as plain text.
Bitnami’s Sealed Secrets and
Soluto’s Kamus projects are two GitOps
solutions for encrypting sensitive information, storing it in version control
and making it available to your application. The best path forward will depend
on your specific requirements. VMware recommends application developers and
platform operators work together to discuss the best solution given their
business requirements and constraints.