Kubernetes Runner

The Kubernetes Runner is GitLab Runner configured with the Kubernetes executor. Instead of running jobs on a single host (shell executor) or in containers on the runner's Docker daemon (docker executor), it runs each job as a pod in a Kubernetes cluster. This makes the runner cloud-native: it integrates with Kubernetes' scheduling, resource management, and — most importantly — auto-scaling.

When a job arrives, the runner talks to the Kubernetes API server and requests a new pod for that job. Kubernetes schedules the pod onto a node, pulls the image, starts the pod, and runs the job's script. When the job finishes, the pod is deleted. Because pods are ephemeral, the environment is fresh every time — the same guarantee the Docker executor provides, but now delivered by Kubernetes itself.

The real power of the Kubernetes Runner is that it inherits Kubernetes' auto-scaling. If you run a Cluster Autoscaler (or Karpenter), then when there are more jobs than the cluster has capacity for, the cluster will add nodes automatically. When the jobs finish, the cluster shrinks back down. This means your CI/CD infrastructure scales with demand — you pay for compute only when jobs are running, and you never have to manually provision build servers.

For teams already operating Kubernetes, the Kubernetes Runner is usually the right choice: it reuses the same cluster, the same observability stack, and the same operational practices. For teams without Kubernetes, the setup is more involved but the auto-scaling benefits can justify the investment.

Understanding the Kubernetes Runner's architecture is essential for configuring and troubleshooting it. The runner is itself deployed as a pod in the cluster, and it creates additional pods for each job. Three types of pods work together to run a single job: the helper pod, the build pod, and any service pods.

The helper pod is a small container that GitLab Runner uses for internal coordination — it handles things like cloning the repository into a shared volume, executing commands, and streaming logs back to GitLab. The build pod is where your job's script actually runs; its image is the one specified in .gitlab-ci.yml. The service pods are created for the services keyword in your job — databases, caches, and other dependencies that the build pod needs.

All three pods share volumes and network namespaces so they can communicate with each other. The helper pod can execute commands in the build pod via an exec-like mechanism, and services are reachable from the build pod by their alias hostname. Understanding this three-pod architecture explains why jobs sometimes need a little extra memory — the helper and service pods consume some of the requested resources too.

There are two ways to install the Kubernetes Runner: using the official Helm chart (recommended) or using a manual Deployment manifest. The Helm chart is recommended because it handles the RBAC, ConfigMap, Deployment, and Secret resources for you, and makes upgrades easy.

Before installing, you need to decide how the runner will authenticate to GitLab. There are two approaches: using a runner authentication token (obtained by registering a runner manually and reusing its token) or using the newer runner authentication tokens created via the GitLab UI or API. The Helm chart supports both approaches. The newer approach is recommended because it doesn't require a registration token to be present in the cluster.

You also need to decide on the namespace, service account, and RBAC permissions. The runner needs permission to create, get, list, watch, and delete pods in its namespace, plus permission to create secrets and configmaps for cache and artifacts. The Helm chart creates an appropriately scoped service account and role for you.

The Kubernetes Runner is configured via config.toml, which is mounted into the runner pod. The most important section is [runners.kubernetes], which controls the pod template used for each job: the image, CPU and memory limits, node selectors, service account, and security context. Getting this section right is the key to a performant and secure runner.

One of the most important settings is cpu_limit and memory_limit. These determine the resources requested for the build pod, which in turn determine which nodes the job can be scheduled on. If you set these too low, jobs may be OOM-killed or starved for CPU. If you set them too high, jobs may not fit on smaller nodes, or you may waste resources. The right values depend on your workload — start conservative and tune based on observed usage.

The namespace setting determines where job pods are created. Typically this is the same namespace as the runner itself. The service_account setting determines the identity used by the pods — by default, the default service account of the namespace, which is fine unless your jobs need special permissions. The node_selector and tolerations settings let you control which nodes jobs run on — useful for dedicating specific nodes to CI/CD or for using spot instances.

The most compelling feature of the Kubernetes Runner is auto-scaling — and it comes essentially for free if you already use the Cluster Autoscaler. The runner doesn't need to know anything about auto-scaling: it just requests pods. When there aren't enough nodes to schedule those pods, the Cluster Autoscaler notices the pending pods and adds nodes. When the jobs finish and nodes are idle, the Cluster Autoscaler removes them.

There are two distinct scaling mechanisms to understand. First, Kubernetes itself schedules as many job pods as it can fit on available nodes. Second, the Cluster Autoscaler adds nodes when pods can't be scheduled. Together, these give you the effect of a runner that scales from zero to hundreds of concurrent jobs and back, all without manual intervention. This is particularly powerful for spiky workloads — a big release might create 50 jobs at once, but the cluster can grow to handle them and shrink back within minutes.

There are some caveats. Auto-scaling takes time — new nodes take 30 seconds to a few minutes to join the cluster, so jobs may be pending for a while during a spike. And there are limits: node pools have max sizes, and large clusters cost money. But the flexibility is enormous, and for most teams the auto-scaling savings (both time and money) far outweigh the complexity.

Individual jobs can override many of the runner's Kubernetes settings using CI/CD variables. This is powerful because different jobs have different resource needs — a small lint job doesn't need 4GB of memory, while a large build might need 8GB. By setting these variables in .gitlab-ci.yml, you can right-size each job's pod and help the scheduler place it optimally.

The most commonly used variables are KUBERNETES_CPU_REQUEST, KUBERNETES_MEMORY_REQUEST, KUBERNETES_CPU_LIMIT, and KUBERNETES_MEMORY_LIMIT. These directly map to the pod's resource requests and limits. Setting requests is especially important for auto-scaling: if a job's request is too small, it may be scheduled on a node that can't actually run it efficiently; if it's too large, the job may not fit anywhere.

Beyond resources, jobs can override the service account, node selector, annotations, and even the entire pod spec. The KUBERNETES_OVERWRITE_CONTAINER_SPEC variable lets a job specify its own build container spec, giving it full control. These features let sophisticated jobs customize their environment while keeping the runner's defaults sensible for the common case.

Security is especially important for the Kubernetes Runner because it runs untrusted code inside your cluster. If a job pod is compromised, an attacker could try to attack other workloads or the cluster itself. The good news is that Kubernetes provides excellent security primitives — you just need to enable them.

The most important controls are pod security contexts (run as non-root, drop capabilities, read-only filesystem), network policies (isolate job pods from production), RBAC (minimal permissions for the runner service account), and namespace isolation (run the runner in its own namespace). Combining these creates strong defense in depth: even if a job is compromised, its ability to affect anything else is limited.

For untrusted code (public repositories, contributed MRs), consider additional controls like gVisor or Kata Containers for stronger isolation, and ephemeral runners that are destroyed after each job. The most security-sensitive teams run one runner per project, or use the "runners autoscaling" feature to run each job in a fresh pod with no persistence.