Deploying Remote Concourse CI Workers
Luke Short
There are many use-cases for having remote Concourse worker hosts deployed across different sites. This fits into the hybrid cloud model, helps to distribute resources, and can be used for failover scenarios. This guide walks through setting up local and remote worker hosts in a Concourse cluster that is deployed on Kubernetes.
The first step is to create a new Concourse CI cluster with at least the web hosts. The full Helm chart values.yaml can be found here.
Generate the required SSH keys for production use. The upcoming examples will assume these have been generated. For testing purposes, this can be skipped as the Helm chart provides default public and private SSH keys.
$ ssh-keygen -t rsa -b 4096 -m PEM -f ./concourse_sessionSigningKey
$ ssh-keygen -t rsa -b 4096 -m PEM -f ./concourse_hostKey
$ ssh-keygen -t rsa -b 4096 -m PEM -f ./concourse_workerKey
$ ls -1
concourse_hostKey
concourse_hostKey.pub
concourse_sessionSigningKey
concourse_sessionSigningKey.pub
concourse_workerKey
concourse_workerKey.pub
If no local worker hosts are required, set worker.enabled: false. In the example below, they are enabled. For the Secrets, define all of the required values for SSH keys (secrets.hostKeyPub, secrets.hostKey, secrets.sessionSigningKey, secrets.workerKeyPub, and secrets.workerKey). This will allow cross-communication between web and worker hosts.
---
# values.yaml file.
concourse:
web:
# Define the URL to use for Concourse.
# Use "http://" for simplified testing purposes.
# This needs to use the domain defined in "web.ingress.hosts".
externalUrl: http://concourse.example.com
auth:
mainTeam:
# The default user to create.
localUser: admin
# Automatically create Secret objects for the default user account and SSH keys.
secrets:
# Define the password for the "admin" user as "password123".
localUsers: admin:password123
# Public and private SSH keys for the web hosts.
hostKeyPub: |-
ssh-rsa <OMITTED>
hostKey: |-
-----BEGIN RSA PRIVATE KEY-----
<OMITTED>
-----END RSA PRIVATE KEY----
# Private SSH key for the web hosts to securely sign HTTP session tokens.
# No public key is required.
sessionSigningKey: |-
-----BEGIN RSA PRIVATE KEY-----
<OMITTED>
-----END RSA PRIVATE KEY----
# Public and private SSH keys for the worker hosts.
workerKeyPub: |-
ssh-rsa <OMITTED>
workerKey: |-
-----BEGIN RSA PRIVATE KEY-----
<OMITTED>
-----END RSA PRIVATE KEY----
web:
service:
workerGateway:
type: NodePort
# The web hosts port to use for remote worker hosts to SSH into.
NodePort: 32222
ingress:
enabled: true
hosts:
# Define the domain name for Concourse.
# This needs to match what is defined for 'concourse.web.externalURL'.
- concourse.example.com
# Disable PersistentVolumeClaims for testing purposes.
persistence:
enabled: false
postgresql:
persistence:
enabled: false
$ helm install -f values.yaml --namespace concourse --create-namespace concourse concourse/concourse
Find the Ingress external IP address that is in use. A valid DNS A record needs to be setup for the domain used which is concourse.example.com in our example.
$ kubectl get ingress --namespace concourse
Login into http://concourse.example.com with the username admin and password password123 to verify it is up and functional.
Switch Kubernetes context to a different/remote cluster.
$ kubectl config get-contexts
$ kubectl config set-context <CONTEXT>
Disable components that are not required for a workers-only deployment with web.enabled: false and postgresql.enabled: false. For the Secrets, only define secrets.hostKeyPub and secrets.workerKey (do not define secrets.hostKey, secrets.sessionSigningKey, or secrets.workerKeyPub). The remote worker hosts will SSH into the web hosts to add themselves to the cluster. Along with this, a SSH tunnel back from the web to remote worker hosts is created. This way, the web hosts can delegate pipelines to the hosts.
---
# values-remote.yaml file.
# Disable the web and PostgreSQL components
# since this is a workers-only deployment.
web:
enabled: false
postgresql:
enabled: false
# Automatically create Secret objects for the SSH keys.
secrets:
create: true
concourse:
worker:
tsa:
hosts:
# This needs to be the same domain and NodePort used for the original Concourse deployment.
# The worker hosts will SSH into original deployment to add itself to the Concourse cluster.
- concourse.exampe.com:32222
# This tag is used in Concourse pipelines for running on specified work hosts.
tag: remote_worker
env:
# The networking values can be changed to anything.
# All traffic will be tunneled by the Kubernetes CNI plugin.
- name: CONCOURSE_GARDEN_NETWORK_POOL
value: "10.254.0.0/16"
- name: CONCOURSE_GARDEN_DENY_NETWORK
value: "169.254.169.254/32"
- name: CONCOURSE_GARDEN_MAX_CONTAINERS
value: "50"
secrets:
hostKeyPub: |-
ssh-rsa <OMITTED>
workerKey: |-
-----BEGIN RSA PRIVATE KEY-----
<OMITTED>
-----END RSA PRIVATE KEY----
# Disable PersistentVolumeClaims for testing purposes.
persistence:
enabled: false
Deploy a remote worker in a different namespace. The remote worker hosts listen on a non-standard SSH port 2222/TCP. The remote workers need to allow the web hosts via providing their public SSH key.
$ helm install -f values-remote.yaml --namespace concourse-remote --create-namespace concourse concourse/concourse
Log into the Concourse web dashboard by going to http://concourse.example.com. Then download the fly binary for the operating system of the workstation.
Verify that the local and remote worker hosts are being listed.
$ fly login --insecure -c http://concourse.example.com -u admin -p password123 -t concourse
$ fly -t concourse workers
name containers platform tags team state version age
concourse-remote-worker-0 0 linux remote_worker none running 2.3 4m16s
concourse-remote-worker-1 0 linux remote_worker none running 2.3 4m14s
concourse-worker-0 0 linux none none running 2.3 7m21s
concourse-worker-1 0 linux none none running 2.3 7m21s
Run a demo pipeline that only runs on the remote workers. It has two jobs: one to record the time and the second to read the time. Notice how the pipeline has various places where a list of tags associated with worker hosts need to be specified. The pipeline will run on any worker host matching at least one of the tags. Without these tags, the pipeline could be scheduled on any worker host in the Concourse cluster.
$ cat <<EOF > /tmp/hello-world-remote-pipeline.yaml
resources:
- name: time
tags:
- remote_worker
type: time
source:
location: America/Denver
interval: 60s
jobs:
- name: sli-test
serial: true
build_log_retention:
builds: 100
plan:
- get: time
tags:
- remote_worker
trigger: true
- task: thing-one
tags:
- remote_worker
config:
platform: linux
image_resource:
type: registry-image
source:
repository: alpine
inputs:
- name: time
outputs:
- name: some-output
run:
path: sh
args:
- -ec
- |
echo "hello-world" > some-output/hi-everybody.txt
echo ""
[ -f time/input ] && echo "Time is reporting"
- task: thing-two
tags:
- remote_worker
config:
platform: linux
image_resource:
type: registry-image
source:
repository: alpine
inputs:
- name: some-output
run:
path: sh
args:
- -ec
- |
cat some-output/hi-everybody.txt
EOF
$ fly -t concourse set-pipeline -p hello-world-remote -c /tmp/hello-world-remote-pipeline.yaml
$ fly -t concourse unpause-pipeline -p /tmp/hello-world-remote
Find the latest build for one of the pipeline tasks.
$ fly -t concourse builds | grep "hello-world-remote" | head -n 1
2427 hello-world-remote/sli-test/13 succeeded 2021-08-04@13:03:14-0600 2021-08-04@13:03:31-0600 17s main system
Verify that it is only running on the remote worker hosts.
$ fly -t concourse watch --job hello-world-remote/sli-test --build 13 | grep "selected worker:"
selected worker: concourse-remote-worker-0
selected worker: concourse-remote-worker-0
selected worker: concourse-remote-worker-1
selected worker: concourse-remote-worker-0
selected worker: concourse-remote-worker-1
selected worker: concourse-remote-worker-0