Secure CI/CD Runners: Best Practices & Hardening Guide

Modern continuous delivery pipelines rely on Continuous Integration (CI) runners to execute shell scripts, build container images, and package release artifacts, placing these compute workers directly at the crossroads of unvetted source code and privileged infrastructure. When an attacker submits a malicious pull request or compromises a build step on an unhardened runner, they can extract ambient cloud credentials, escape containers via misconfigured Docker sockets, pivot into internal VPCs, or inject backdoors into upstream software binaries before cryptographic signing. Because runners process arbitrary developer inputs while simultaneously holding deployment permissions, treating them as unmonitored utilities creates a severe supply chain vulnerability. Securing this boundary requires treating runners as zero-trust compute nodes—a core architectural discipline emphasized by training platforms like DevOpsSchool to help engineering teams master pipeline defense, granular workload isolation, and practical CI/CD infrastructure hardening.

Understand the Different Runner Models Before Securing Them

Securing build execution requires understanding where, how, and for how long the runner process lives. Different runner deployment architectures introduce distinct operational models, isolation capabilities, and risk profiles.

+-------------------+-----------------------------------+-----------------------------------+
| Runner Model      | Primary Isolation Mechanism       | Key Security Trade-offs           |
+-------------------+-----------------------------------+-----------------------------------+
| SaaS / Hosted     | Hypervisor or microVM isolation   | Managed infrastructure; untrusted |
|                   | per job run                       | network ingress to private clouds |
+-------------------+-----------------------------------+-----------------------------------+
| Self-Hosted       | Bare metal or private VM managed  | Full network access to internal   |
| (Persistent)      | by the organization               | networks; high contamination risk |
+-------------------+-----------------------------------+-----------------------------------+
| Self-Hosted       | Ephemeral VM or pod destroyed     | Clean state per job; requires     |
| (Ephemeral)       | immediately after execution       | orchestration & auto-scaling      |
+-------------------+-----------------------------------+-----------------------------------+
| Container-Based   | Linux namespaces, cgroups,        | Lightweight; vulnerable to shared |
|                   | capability drops, seccomp         | kernel bugs & Docker socket risks |
+-------------------+-----------------------------------+-----------------------------------+
| VM-Based          | Hardware-assisted hypervisor      | Strong isolation boundary; higher |
|                   | boundaries (KVM, Firecracker)     | cold-start provisioning latency   |
+-------------------+-----------------------------------+-----------------------------------+
| Kubernetes-Based  | Pod sandboxes, runtime classes    | Highly elastic; shared node       |
|                   | (gVisor, Kata Containers)         | attack vectors if misconfigured   |
+-------------------+-----------------------------------+-----------------------------------+

Shared / Hosted Runners

Hosted runners (such as standard GitHub-hosted runners or GitLab.com shared runners) run on infrastructure owned and maintained by the platform vendor.

• Operational Reality: Each job executes inside a clean, pre-configured virtual machine that is provisioned on demand and decommissioned immediately upon completion.
• Isolation: Hypervisor-level virtualization isolates each customer’s workloads.
• Security Posture: You outsource host operating system hardening, physical security, and base image patching to the vendor. However, because these systems reside outside your corporate network perimeter, granting them access to private registries or VPC resources requires inbound firewalls, VPN tunnels, or dynamic identity federation.

Self-Hosted Persistent Runners

Self-hosted runners operate on dedicated bare-metal servers, persistent virtual machines, or long-running cloud instances managed entirely by internal platform teams.

• Operational Reality: A single host runs continuously, executing jobs sequentially or concurrently across different repositories and branches.
• Isolation: Often poor. Multiple jobs frequently share the same underlying operating system, local disk storage, tools cache, and system daemons.
• Security Posture: Persistent hosts create a high risk of cross-job contamination. If job $A$ downloads malware or changes a system binary, job $B$ (potentially building a release artifact for a different repository) inherits the modified environment. Persistent self-hosted runners should never be used for untrusted pull-request execution.

Self-Hosted Ephemeral Runners

Ephemeral runners leverage autoscaling APIs (such as the GitHub Actions Actions Runner Controller or GitLab Runner autoscaling drivers) to spawn compute nodes on demand.

• Operational Reality: An orchestrator launches a fresh virtual machine or container instance to process exactly one job. Once the job concludes—whether it succeeds, fails, or times out—the compute resource is destroyed completely.
• Isolation: State cannot survive between jobs. Any local files, compromised packages, or modified environmental variables are wiped out with the instance lifecycle.
• Security Posture: This represents the gold standard for self-hosted execution. It eliminates persistent malware and workspace contamination while preserving the flexibility of private network integration.

Container-Based vs. Virtual-Machine-Based vs. Kubernetes Runners

• Container-Based Runners: Jobs execute as containers sharing a single underlying Linux kernel. While lightweight and fast to start, container boundaries rely on kernel namespaces and control groups (cgroups). A kernel vulnerability or a misconfigured capability (e.g., executing with elevated privileges) can compromise the host node.
• Virtual-Machine-Based Runners: Jobs execute within a full hardware-virtualized boundary (e.g., AWS EC2, GCP Compute Engine, or microVMs like Firecracker). The hypervisor enforces a strict boundary between virtual machines. Compromising the kernel inside a guest VM does not grant access to the host or neighbor machines.
• Kubernetes-Based Runners: The CI orchestrator spawns dynamic Pods inside a Kubernetes cluster. While operationally efficient, Kubernetes clusters introduce complex attack surfaces, including node-level credential access, API server tokens, and internal container network meshes. Hardening requires strict Pod Security Standards, RuntimeClasses, and NetworkPolicies.

Identify the Main Attack Paths Inside a CI Runner

Understanding how attackers compromise runners is essential for designing resilient controls.

       [ Attack Paths Inside an Unhardened CI Runner ]
       
    1. Untrusted PR Execution  --->  Runs arbitrary code in build step
    2. Docker Socket Mount     --->  Container escape to host root
    3. Memory / Env Scraping   --->  Harvests long-lived tokens / secrets
    4. Cache Poisoning         --->  Injects trojans into shared build caches
    5. Network Pivoting        --->  Scans 169.254.169.254 & internal VPC
    6. Artifact Modification   --->  Tampers with binaries before upload

Malicious Pull Requests and Untrusted Pipeline Code

• Scenario: An open-source repository or multi-tenant enterprise repo triggers automatic CI runs on pull requests submitted by external or unprivileged users. The submitter includes an altered build step or test script (npm test, make build, or direct shell injection in the workflow YAML).
• Impact: The runner executes untrusted commands with the privileges of the runner process, exposing environment variables, checking out private parent code, and interacting with local storage.
• Defense: Never run untrusted pull-request workflows on persistent self-hosted runners. Require manual approval for runs triggered by non-collaborators. Disable secret inheritance for forks.

Credential and Token Exposure

• Scenario: A pipeline job exposes runtime variables through unmasked console output, core dumps, or third-party diagnostic utilities. An attacker writes an automated step that reads /proc/$PID/environ or prints all variables using env | base64.
• Impact: Build tokens, personal access tokens (PATs), deployment keys, and third-party API secrets leak into build logs, which are accessible to read-only project viewers.
• Defense: Use automated secret masking in logs, avoid passing static credentials via environment variables, inject secrets via transient files or memory mounts, and switch to short-lived OpenID Connect (OIDC) identity tokens.

Privilege Escalation and Host Compromise via the Docker Socket

• Scenario: To facilitate container builds within containers, an administrator mounts the host’s /var/run/docker.sock directly into the runner job container (-v /var/run/docker.sock:/var/run/docker.sock).
• Impact: Docker socket access is functionally equivalent to root access on the host. Any process inside the container can communicate directly with the host Docker daemon, launch a privileged container that mounts the host root filesystem (-v /:/host), chroot into /host, and gain full control over the underlying operating system.
• Defense: Ban Docker socket mounts. Implement unprivileged, daemonless container builders such as Kaniko, Buildah, or rootless BuildKit.

Persistent Malware and Workspace Contamination

• Scenario: A pipeline executes on a long-running, shared VM runner. A compromised job modifies a globally installed binary (e.g., overwriting /usr/local/bin/curl or modifying python packages in the global site-packages directory) or deposits an executable in /tmp.
• Impact: Subsequent jobs running on the same host execute the trojanized utility, poisoning downstream builds across different teams or environments.
• Defense: Use strictly ephemeral runner infrastructure. If persistent hosts must be used, enforce non-root execution, make system partitions read-only, and wipe workspaces completely between job runs.

Dependency and Build-Tool Compromise (Cache Poisoning)

• Scenario: Runners share a common local directory for package caches (~/.m2, ~/.npm, ~/.cargo) to speed up build execution. An attacker updates their project build file to overwrite a popular library package inside the shared cache with a backdoored variant.
• Impact: Unrelated pipelines building on the same agent pick up the poisoned library from the shared local cache without fetching the authentic upstream version.
• Defense: Isolate cache stores by repository, branch, and security boundary. Store remote build caches in immutable, signed cloud object storage buckets.

Network-Based Attacks and Lateral Movement

• Scenario: A runner executes in a default cloud VPC subnet with open outbound internet connectivity and unblocked routing to internal subnets. The runner script queries the cloud Instance Metadata Service (IMDS) at [http://169.254.169.254](http://169.254.169.254) to extract instance profile credentials, then scans internal database ports.
• Impact: The runner becomes a launchpad for deep lateral movement within the corporate network, bypassing border perimeter firewalls.
• Defense: Block access to the IMDS IP using host firewall rules or enforce IMDSv2 with zero-hop limits. Isolate runners in egress-restricted subnets using security groups and egress proxies.

Design Runner Isolation Around Trust Boundaries

Security architecture begins with trust segmentation. Pipelines do not all share the same security profile: compiling an unvetted pull request from an external fork carries a radically different risk profile than deploying verified master-branch artifacts to a production Kubernetes cluster.

+-----------------------------------------------------------------------------------+
|                        RUNNER TRUST BOUNDARY ARCHITECTURE                         |
+-----------------------------------------------------------------------------------+

 [ TIER 1: Untrusted Workloads ] 
   - Public PRs, Untrusted Forks, Experimental Branches
   - Infrastructure: Ephemeral MicroVMs / SaaS Hosted Runners
   - Network: Strict Outbound Egress; NO Internal Subnet Access
   - Credentials: NO Secrets, NO Production IAM Access
                                  |
                                  v (PR Merged via Code Review)
 [ TIER 2: Internal Build & Test ]
   - Verified Mainline Commits, Internal Feature Branches
   - Infrastructure: Ephemeral Containers / Dedicated Self-Hosted Pools
   - Network: Restricted to Internal Package Mirrors & Code Repos
   - Credentials: Read-Only Internal Registry Tokens, Ephemeral Test Secrets
                                  |
                                  v (Artifact Tagged & Cryptographically Signed)
 [ TIER 3: Sensitive Deployment ]
   - Release Branches, Production Deployments, Cryptographic Signing
   - Infrastructure: Hardened, Isolated Bare-Metal / Dedicated Single-Tenant VMs
   - Network: Direct Outbound to Production API Endpoints via Private Link
   - Credentials: OIDC Short-Lived Deployment Roles Only

Isolating Jobs by Workload Type

To construct resilient runner pools:

1. Tier 1 (Untrusted Workloads): Run pull requests from external forks or low-trust teams exclusively on vendor-hosted runners or ephemeral compute instances in an isolated, non-routable VPC. These environments must receive zero secrets and have no access to private resources.
2. Tier 2 (Internal Build and Test): Run internal team development on isolated container or VM pools. Grant read access to internal artifact registries and package repositories.
3. Tier 3 (Production Deployment and Signing): Dedicate a hardened, highly restricted pool of runners exclusively for release tagging, production deployments, and artifact signing. These runners must never process arbitrary feature branches or unvetted pull requests.

Preventing Cross-Job Workspace Contamination

When persistent runners are unavoidable, never allow jobs to run in uncleaned directories.

• Configure clean checkout options (git clean -ffdx) before and after job execution.
• Maintain dedicated filesystem roots per project (e.g., /opt/runners/work/{project-id}/{job-id}).
• Prevent cross-mounting of user home directories where tools deposit sensitive configuration files (~/.docker/config.json, ~/.aws/credentials, ~/.netrc). Store these configs on transient, in-memory RAM drives (tmpfs) that are unmounted and scrubbed when the job process terminates.

Harden the Runner Host and Its Execution Environment

The host operating system provides the bedrock for runner security. Any compromise of the host OS invalidates the security controls of every container running on top of it.

Minimal Operating Systems and Package Hygiene

Run runners on purpose-built, minimal operating system distributions (such as Alpine Linux, Debian Minimal, or immutable operating systems like Flatcar Container Linux or AWS Bottlerocket).

• Remove compilers (gcc, clang), network diagnostics (nmap, netcat, tcpdump), and unnecessary scripting runtimes from persistent runner hosts unless specifically required by the build process. If a build requires compilation tools, package those tools inside dedicated build container images rather than installing them globally on the runner host.
• Disable unused system services and daemons:Bashsudo systemctl disable --now bluetooth.service cups.service avahi-daemon.service

Non-Root Execution and Privilege Restrictions

Never run the CI agent process as root. Create a dedicated, unprivileged system user with a disabled shell and no administrative rights:

Bash

# Create an unprivileged system user without interactive login
sudo useradd -r -s /bin/false -d /opt/ci-runner -m ci-runner

# Restrict ownership of runner working directories
sudo chown -R ci-runner:ci-runner /opt/ci-runner
sudo chmod 750 /opt/ci-runner

Ensure that ci-runner has no entries in /etc/sudoers or /etc/sudoers.d/. If a pipeline step attempts to run sudo, it must fail immediately.

Filesystem Permissions and Mount Hardening

Apply strict mount options to temporary and shared storage locations within /etc/fstab:

# Restrict execution and device creation on shared temporary filesystems
tmpfs    /tmp        tmpfs    defaults,noexec,nosuid,nodev    0 0
tmpfs    /var/tmp    tmpfs    defaults,noexec,nosuid,nodev    0 0

Setting noexec on /tmp stops an entire class of automated exploit payloads from running out of temporary scratch space.

Host-Level Auditing and Remote Access Restrictions

• Disable Interactive SSH: CI runners are compute workers, not developer workstations. Disable SSH entirely, or restrict access via bastion hosts and mutual TLS (mTLS) session managers (e.g., AWS Systems Manager Session Manager or Cloudflare Access) with mandatory multi-factor authentication (MFA).
• Configure Linux Audit Logging (auditd): Monitor sensitive file access and process execution. Add rules to /etc/audit/rules.d/ci-runner.rules:-w /etc/passwd -p wa -k identity_changes -w /etc/shadow -p wa -k identity_changes -w /opt/ci-runner -p wa -k runner_binaries -a always,exit -F arch=b64 -S execve -F uid=ci-runner -k runner_exec

Protect Credentials, Tokens, and Secrets During Job Execution

CI pipelines require credentials to fetch dependencies, authenticate with artifact registries, and deploy infrastructure. However, static, long-lived secrets represent an unacceptable security liability.

Eliminate Static Secrets via OpenID Connect (OIDC)

Modern CI platforms support OpenID Connect (OIDC) federation with major cloud providers (AWS, Google Cloud, Microsoft Azure, HashiCorp Vault). OIDC allows a runner to exchange a cryptographically signed JSON Web Token (JWT) issued by the CI provider for short-lived, scoped cloud credentials, eliminating the need to store static AWS_ACCESS_KEY_ID or service account keys in repository secrets.

+------------------------------------------------------------------------------------+
|                         OIDC AUTHENTICATION SEQUENCE                               |
+------------------------------------------------------------------------------------+

[ CI Runner ]               [ CI/CD Platform (OIDC) ]             [ Cloud IAM (AWS/GCP) ]
      |                                  |                                   |
      |-- 1. Requests OIDC Token ------->|                                   |
      |                                  |-- 2. Signs JWT with repo claims ->|
      |<-- 3. Returns Signed JWT --------|                                   |
      |                                                                      |
      |-- 4. Presents Signed JWT to Assume Role ---------------------------->|
      |                                                                      | Validates Issuer,
      |                                                                      | Audience & Claims
      |<-- 5. Returns Temporary Credentials (valid for 15-60 mins) ----------|
      |                                                                      
 [ Executes Deployment ]

GitHub Actions to AWS via OIDC Example:

YAML

name: Secure Production Deployment
on:
  push:
    branches:
      - main

permissions:
  id-token: write   # Required to request the OIDC JWT token
  contents: read    # Minimal read access to source code

jobs:
  deploy:
    runs-on: [self-hosted, production-deploy]
    steps:
      - name: Checkout Source Code
        uses: actions/checkout@v4

      - name: Configure AWS Credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/ProductionDeploymentRole
          aws-region: us-east-1
          audience: sts.amazonaws.com

      - name: Deploy Application
        run: |
          aws s3 sync ./dist s3://production-application-bucket/ --delete

Cloud IAM Trust Policy Condition: Ensure the cloud IAM role enforces strict claims validation so that only the designated repository, environment, and branch can assume the role:

JSON

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:my-org/production-app:ref:refs/heads/main"
        }
      }
    }
  ]
}

Log Masking and Secret Injection Pitfalls

While CI systems automatically mask known secret variables in build logs, simple transformations can bypass masking filters:

• Base64 Encoding: Running echo $API_KEY | base64 produces an encoded string that the log scrubber will not recognize, exposing the plain credential to anyone who decodes the logs.
• Diagnostic Dumps: Commands such as set -x in bash cause the shell to print every expanded variable directly to stderr before running the line. Always pair secret-handling scripts with set +x.
• Filesystem Leaks: If a script writes a secret to a file (e.g., /tmp/credentials.json), ensure it writes to a mounted tmpfs volume with chmod 600 permissions, and securely shred the file immediately after use:Bashshred -u /tmp/credentials.json

Secure Containers, Images, and Runner Dependencies

Containerization is central to modern pipelines. However, running containers inside CI/CD requires strict controls to maintain isolation.

The Fatal Danger of /var/run/docker.sock

Mounting the host Docker socket into a runner container gives that container unconstrained control over the host daemon. Consider this real attack path:

Bash

# A compromised CI step running inside a container with /var/run/docker.sock mounted
docker run --rm -v /:/host-fs alpine:latest \
  sh -c "echo 'ci-attacker ALL=(ALL) NOPASSWD:ALL' >> /host-fs/etc/sudoers"

Within one second, the pipeline step escapes the container sandbox and grants itself administrative access across the host operating system.

Safer Alternatives for Building Container Images

To build container images inside CI pipelines safely:

1. Rootless Buildah / Podman: Executes image builds inside a user namespace without requiring root privileges or a background daemon.
2. Kaniko: An open-source tool developed to build container images from a Dockerfile inside a container or Kubernetes cluster without relying on a Docker daemon. Kaniko executes entirely in userspace, unpacking the base image into a snapshot directory, running instructions, and pushing layers directly to the target registry.
3. Dedicated Ephemeral VM Builders: If full Docker engine compatibility is required, provision dedicated single-tenant virtual machines that are destroyed immediately after the image build finishes.

+----------------------------------------------------------------------------+
|                     CONTAINER BUILD ENGINE ISOLATION                       |
+----------------------------------------------------------------------------+

  UNSAFE PATTERN:
  [ Runner Container ] --(Mounts /var/run/docker.sock)--> [ Host Docker Daemon ]
                                                            (Host Root Access)

  SAFE PATTERN (DAEMONLESS USERSPACE BUILD):
  +-------------------------------------------------------+
  | [ Ephemeral Runner Sandbox ]                          |
  |   +-------------------------------------------------+ |
  |   | Kaniko / Rootless Buildah (Userspace Execution) | |
  |   +-----------------------+-------------------------+ |
  +---------------------------|---------------------------+
                              v (Pushes Layer Artifacts over HTTPS)
                    [ Container Registry ]

Base Image Verification and Dependency Scanning

• Image Pinning: Never use mutable tags such as :latest for build tools and base environments. Pin container images by their immutable cryptographic digest:Dockerfile# Unsafe: Target layer can change upstream at any time FROM node:20-alpine # Safe: Cryptographically locked to an exact image manifest FROM node:20-alpine@sha256:d840dd67323b7849e8a719c2c2f42a19b88241f5e88849b28be937b8d43c2c77
• Automated Container Scanning: Scan all base build images before introducing them into the runner infrastructure using tools like Trivy or Grype.
• Generate SBOMs During the Build: Use tools such as Syft to generate a Software Bill of Materials (SBOM) for every compiled artifact, making all dependencies and library versions auditable across the supply chain.

Network Controls and Access Restrictions for Runners

A CI runner should never have unbounded network privileges. By default, runners should only be able to communicate with the services strictly necessary to complete their build tasks.

       [ Runner Host: 10.10.4.15 ]
                   |
                   +--- [ ALLOW ] ---> Git Repositories (Port 443)
                   +--- [ ALLOW ] ---> Internal Artifact Registry (Port 443)
                   +--- [ ALLOW ] ---> Cloud Provider STS / OIDC (Port 443)
                   |
                   x--- [ BLOCK ] ---> Cloud Metadata API (169.254.169.254)
                   x--- [ BLOCK ] ---> Internal Production Databases (Port 5432/3306)
                   x--- [ BLOCK ] ---> Corporate Staging Subnets (10.20.0.0/16)
                   x--- [ BLOCK ] ---> Arbitrary Internet Outbound (Port 22/25/8080)

Blocking Cloud Metadata Services (IMDS)

Any runner operating inside AWS, GCP, Azure, or DigitalOcean can easily be exploited to query the link-local metadata address (169.254.169.254). If the underlying VM has an associated IAM instance profile, a compromised build step can extract temporary security credentials and assume the role of the host.

Mitigate this risk on the host using iptables or local packet filters:

Bash

# Block the ci-runner system user from reaching the cloud metadata service
sudo iptables -A OUTPUT -m owner --uid-owner ci-runner -d 169.254.169.254 -j REJECT

In AWS environments, enforce IMDSv2 and configure the instance metadata response hop limit to 1. This network hop limit prevents containers running on the host from retrieving metadata credentials:

Bash

aws ec2 modify-instance-metadata-options \
    --instance-id i-0123456789abcdef0 \
    --http-tokens required \
    --http-put-response-hop-limit 1 \
    --http-endpoint enabled

Outbound Network Filtering and Egress Proxies

• Place self-hosted runners within private subnets without direct internet gateways.
• Route outbound traffic through a filtering forward proxy (such as Squid or an enterprise egress gateway). Configure an explicit allowlist containing only the domains required for operations:
  • Source control hosts (github.com, gitlab.com, internal git servers)
  • Authorized package repositories (registry.npmjs.org, pypi.org, internal Artifactory/Nexus endpoints)
  • Security vulnerability databases and OIDC token endpoints
• Drop outbound traffic to arbitrary IP addresses and non-standard egress ports (e.g., ports 21, 22, 23, 25, 445).

Platform-Specific Runner Security Configurations

Each continuous integration orchestrator approaches runner registration, execution boundaries, and permissions differently.

GitHub Actions

• Ephemeral Self-Hosted Runners: Pass the --ephemeral flag during runner registration so that the runner processes exactly one job before terminating:Bash./config.sh --url https://github.com/my-org/my-repo \ --token AEXAMPLETOKEN12345 \ --ephemeral \ --unattended
• Restrict Workflow Default Permissions: In repository settings, set default GITHUB_TOKEN permissions to read-only. Avoid assigning broad write privileges at the repository level; instead, grant granular permissions within specific jobs:YAMLpermissions: contents: read pull-requests: read issues: none
• Runner Groups: Group enterprise runners into logical pools and restrict them so they cannot be accessed by public repositories or unapproved development teams.

GitLab CI/CD

• Executor Selection: Avoid the shell executor on multi-tenant or production-critical runners, as shell executors run scripts directly in the host OS environment. Use the docker or kubernetes executors, or leverage the virtualbox or docker+machine autoscaling drivers for strong isolation.
• Protected Runners and Protected Branches: Mark runners that handle deployment credentials as Protected Runners. Protected runners will only process jobs originating from protected branches (e.g., main or release tags) and protected environments, preventing feature-branch pipelines from stealing production credentials.
• Disable Privileged Mode: Inside /etc/gitlab-runner/config.toml, ensure privileged = false:Ini, TOML[[runners]] name = "secure-docker-runner" url = "https://gitlab.com/" token = "GLRT-xxxxxxxxx" executor = "docker" [runners.docker] image = "alpine:3.19" privileged = false disable_entrypoint_overwrite = true volumes = ["/cache"]

Jenkins

• Never Build on the Controller: Running build steps on the Jenkins controller node gives pipelines direct access to the Jenkins home directory (JENKINS_HOME), allowing them to read stored master credentials, API tokens, and server configurations. Configure the controller with zero build executors.
• Enable Agent-to-Controller Access Control: Turn on the Agent-to-Controller Subsystem security boundary. This prevents a compromised agent from requesting unauthorized data or executing arbitrary commands on the Jenkins controller.
• Use Ephemeral Kubernetes Agents: Deploy the Jenkins Kubernetes plugin to provision single-use agent Pods that are created on demand and torn down when the pipeline terminates.

Azure DevOps

• Agent Pool Segregation: Separate Azure Pipelines agent pools into distinct tiers. Restrict production deployment pipelines so they can only query dedicated, isolated deployment pools.
• Service Connection Approvals: Avoid granting open access to Service Connections across all pipelines. Require explicit administrative approvals and environment-level checks before a pipeline can use a connection to cloud infrastructure.
• Maintenance Jobs: Schedule automated maintenance routines to purge build directories, local caches, and temporary artifacts on persistent agent hosts:Project Settings -> Agent Pools -> [Your Pool] -> Settings -> Maintenance Job

Practical Lab: Build a More Secure CI Runner

This hands-on lab walks through configuring a hardened, non-root self-hosted Linux runner with restricted workspace permissions, strict firewall rules, and cleanup hooks.

       [ Lab Architecture: Secure Host Sandbox ]
       +-------------------------------------------------------+
       | Host System (Ubuntu / Debian minimal VM)              |
       |                                                       |
       |  +-------------------------------------------------+  |
       |  | System User: 'ci-runner' (UID: 1500)            |  |
       |  | Home / Workspace: /opt/ci-runner (chmod 750)    |  |
       |  +-------------------------------------------------+  |
       |                           |                           |
       |  +-------------------------------------------------+  |
       |  | iptables: Blocks 169.254.169.254                |  |
       |  | systemd Service: NoNewPrivileges=true           |  |
       |  | Environment: In-memory tmpfs /opt/ci-runner/tmp |  |
       |  +-------------------------------------------------+  |
       +-------------------------------------------------------+

Lab Objective

Deploy a self-hosted runner instance adhering to least-privilege principles: running as an unprivileged user, blocking access to cloud metadata services, restricting workspace permissions, and securing runtime process privileges with systemd.

Prerequisites

• A freshly deployed virtual machine running Ubuntu 22.04 or Debian 12 (use a disposable test instance).
• Root administrative access (sudo) for initial configuration.

Step 1: Create a Dedicated Low-Privilege Service Account

Never reuse standard user accounts for CI execution.

Bash

# Create a dedicated system user with an explicit UID and a restricted home directory
sudo useradd -u 1500 -m -d /opt/ci-runner -s /bin/bash ci-runner

# Lock the password to disable interactive password-based logins
sudo passwd -l ci-runner

# Restrict permissions on the home directory
sudo chmod 750 /opt/ci-runner

Step 2: Configure Hardened Filesystem and Workspace Structure

Set up a clean directory layout for runner binaries, working directories, and temporary storage:

Bash

# Create operational directories
sudo mkdir -p /opt/ci-runner/bin /opt/ci-runner/work /opt/ci-runner/tmp

# Mount a secure in-memory filesystem for temporary build files
sudo mount -t tmpfs -o size=1G,noexec,nosuid,nodev tmpfs /opt/ci-runner/tmp

# Ensure permissions belong strictly to the runner account
sudo chown -R ci-runner:ci-runner /opt/ci-runner

Step 3: Implement Host-Level Metadata and Network Egress Filtering

Block the runner user from reaching cloud metadata services while allowing necessary HTTPS outbound connections:

Bash

# Flush existing rules on the lab instance (use caution on production systems)
sudo iptables -F OUTPUT

# 1. Allow local loopback traffic
sudo iptables -A OUTPUT -o lo -j ACCEPT

# 2. Block the runner user from accessing the cloud metadata service
sudo iptables -A OUTPUT -m owner --uid-owner 1500 -d 169.254.169.254 -j REJECT

# 3. Allow established and related connections
sudo iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# 4. Allow outbound DNS (port 53 UDP/TCP) for the runner user
sudo iptables -A OUTPUT -p udp --dport 53 -m owner --uid-owner 1500 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 53 -m owner --uid-owner 1500 -j ACCEPT

# 5. Allow outbound HTTPS (port 443) to reach GitHub/GitLab and registries
sudo iptables -A OUTPUT -p tcp --dport 443 -m owner --uid-owner 1500 -j ACCEPT

# 6. Reject all other outbound traffic initiated by the runner user
sudo iptables -A OUTPUT -m owner --uid-owner 1500 -j REJECT

Step 4: Create a Sandboxed Systemd Service Unit

Run the runner daemon under systemd with kernel security flags enabled:

Ini, TOML

# Write to /etc/systemd/system/hardened-ci-runner.service
[Unit]
Description=Hardened Self-Hosted CI Runner
After=network.target

[Service]
Type=simple
User=ci-runner
Group=ci-runner
WorkingDirectory=/opt/ci-runner
ExecStart=/opt/ci-runner/bin/run.sh
Restart=always
RestartSec=10

# Security and Isolation Flags
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/opt/ci-runner/work /opt/ci-runner/tmp
PrivateTmp=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictRealtime=true

[Install]
WantedBy=multi-user.target

Reload systemd to register the new unit:

Bash

sudo systemctl daemon-reload

Step 5: Validate Security Restrictions

Log in as the test runner user and confirm that the defensive controls are functioning as expected:

Bash

# Drop into the runner user account
sudo -u ci-runner -i

# Test 1: Verify the cloud metadata service is completely unreachable
curl --connect-timeout 3 http://169.254.169.254/latest/meta-data/
# EXPECTED OUTPUT: curl: (7) Failed to connect to 169.254.169.254: Connection refused

# Test 2: Verify outbound HTTPS connectivity to public repositories functions
curl -I --connect-timeout 5 https://github.com
# EXPECTED OUTPUT: HTTP/2 200 (or HTTP/1.1 200 OK)

# Test 3: Verify administrative privilege escalation is denied
sudo -l
# EXPECTED OUTPUT: ci-runner is not in the sudoers file. This incident will be reported.

# Test 4: Verify write protection on sensitive system paths
touch /etc/test-compromise
# EXPECTED OUTPUT: touch: cannot touch '/etc/test-compromise': Permission denied

Monitor Runner Activity and Detect Suspicious Behavior

Hardening reduces the attack surface, but monitoring ensures that abnormal activities are detected and contained before they can cause widespread damage.

+-----------------------------------------------------------------------------------+
|                        RUNNER MONITORING & TELEMETRY HUB                          |
+-----------------------------------------------------------------------------------+

   [ Host System Calls ]       [ Process Spawns ]       [ Network Telemetry ]
   (auditd / eBPF / Falco)     (ps, systemd logs)      (iptables, VPC flow logs)
              |                        |                            |
              +------------------------+----------------------------+
                                       |
                                       v
                     [ Central Security Information & ]
                     [    Event Management (SIEM)     ]
                                       |
                   +-------------------+-------------------+
                   |                                       |
                   v                                       v
         [ Alert: IMDS Sweep ]                 [ Alert: Shell Invocation ]
         - Rule: Query 169.254.169.254         - Rule: Spawning netcat / socat
         - Severity: Critical                  - Action: Terminate Runner Job

Key Indicators of Runner Compromise

Security teams should alert on the following abnormal patterns:

• Process Anomalies: Spawning unexpected network utilities or debuggers (nmap, nc, socat, tcpdump, strace, gdb) from pipeline steps.
• Crypto-Mining Signatures: Sustained 100% CPU and GPU utilization accompanied by outbound connections to known mining pool ports (e.g., ports 3333, 4444, 8333).
• Interactive Shell Spawns: A pipeline step launching an interactive reverse shell (bash -i >& /dev/tcp/...).
• Accessing Sensitive Local Files: Pipeline processes attempting to read files outside the designated workspace (/etc/shadow, /root, other user profiles).
• Repeated Metadata Queries: Bursts of HTTP requests targeted at the cloud link-local address (169.254.169.254).

Runtime Security Detection with Falco

Falco (a CNCF runtime security tool) leverages extended Berkeley Packet Filters (eBPF) to monitor kernel-level system calls in real time. Deploying Falco on runner hosts allows you to flag suspicious activity immediately:

YAML

# Sample Falco rule to detect reverse shells spawned inside CI jobs
- rule: Shell Spawned by Build Runner
  desc: Detects an unexpected interactive shell spawned within a runner workspace
  condition: >
    spawned_process and
    proc.name in (bash, sh, zsh, ksh) and
    user.name = "ci-runner" and
    not proc.pname in (run.sh, runner-engine)
  output: >
    Suspicious shell spawned on runner host (user=%user.name command=%proc.cmdline parent=%proc.pname)
  priority: WARNING
  tags: [ci_security, attack_detection]

Centralized Log Retention

Never store security-relevant audit logs exclusively on the runner host. Compromised runners can have their local log records modified or scrubbed (/var/log/syslog or /var/log/audit/audit.log). Ship telemetry, system audit events, container logs, and job output streams to a centralized, write-once SIEM platform (such as OpenSearch, Splunk, or cloud logging buckets) in real time.

Create a Runner Security Maintenance and Governance Plan

Runner security is not a one-time configuration task. Teams need ongoing operational hygiene and policy enforcement to maintain a strong security posture over time.

Automated Patching and Continuous Decommissioning

• Maximum Lifespan Policies: Never allow a self-hosted runner virtual machine to live indefinitely. Enforce a mandatory maximum lifespan (e.g., terminate and reprovision every 24 to 72 hours). Rebuilding runner nodes from updated, golden base images ensures OS-level security patches are systematically applied.
• Automated Base Image Pipelines: Build hardened VM images (AMIs, GCP images) or runner base containers weekly using tools like HashiCorp Packer, running vulnerability scans prior to deployment.
• Runner Registration Hygiene: Regularly audit registered runners in your CI platform settings. Automatically delete runners that have been offline or inactive for more than seven days.

Audit Checklist for CI Runner Infrastructure

Use this practical checklist during operational reviews and security audits:

[ ] Compute Model: Ephemeral compute instances used for all untrusted/forked PRs.
[ ] Privilege Boundary: The CI daemon and pipeline jobs run as a non-root user.
[ ] Docker Isolation: Host /var/run/docker.sock is never mounted into untrusted containers.
[ ] Container Builds: Unprivileged daemonless tools (Kaniko/Buildah) used for image builds.
[ ] Secret Architecture: Short-lived OIDC tokens replace static, long-lived cloud keys.
[ ] Cloud Metadata: Link-local metadata service (169.254.169.254) blocked by host firewall.
[ ] Network Egress: Private runner subnets restrict outbound traffic via an allowlist.
[ ] Filesystem Protections: /tmp mounted as tmpfs with 'noexec,nosuid,nodev' flags set.
[ ] Segmented Pools: High-privilege deployment runners separated from general build runners.
[ ] Runtime Monitoring: eBPF, auditd, or Falco rules active and shipping alerts to a SIEM.

Common Runner Security Mistakes and Their Safer Alternatives

+-----------------------------------+-----------------------------------+-----------------------------------+
| Insecure Practice                 | Why It Is Dangerous               | Recommended Safer Alternative     |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Mounting /var/run/docker.sock     | Grants containerized pipeline     | Use rootless builders like        |
| into the build container          | processes root escape to the host | Kaniko, Buildah, or isolated VMs  |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Running untrusted pull requests   | Allows external attackers to run  | Use vendor-hosted or ephemeral    |
| on persistent self-hosted runners | code, steal secrets, pivot inside | runners; isolate forks completely |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Storing long-lived cloud API keys | Static keys leak easily in logs,  | Use OpenID Connect (OIDC) for     |
| in repository secret settings     | environment dumps, and traces     | short-lived, scoped credentials   |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Reusing one runner pool for all   | Low-trust test jobs can access    | Create tiered runner pools:       |
| builds, tests, and deployments    | or poison deployment credentials  | Untrusted, Internal, Production   |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Allowing open outbound internet   | Compromised runners can fetch     | Enforce outbound firewall rules,  |
| connectivity from build subnets   | exploit toolkits or exfiltrate data| egress proxies, and IMDS blocks   |
+-----------------------------------+-----------------------------------+-----------------------------------+
| Running the runner daemon as the  | Any shell breakout immediately    | Create an unprivileged user       |
| root administrative user          | grants complete control of host   | without sudo permissions          |
+-----------------------------------+-----------------------------------+-----------------------------------+

Trade-Offs of Strong Isolation

Implementing strict security controls introduces engineering trade-offs:

• Build Speed vs. Clean Isolation: Ephemeral virtual machines eliminate cross-job contamination, but provisioning a fresh VM introduces startup latency (often 30 to 90 seconds). This can be mitigated by pre-baking large dependencies into golden VM images or using warm pools of ephemeral runners.
• Developer Convenience vs. Security: Denying root privileges and Docker socket access prevents developers from running quick sudo apt-get commands inside their pipeline workflows. To resolve this without sacrificing security, provide pre-approved, curated container images containing the necessary development toolchains.

Frequently Asked Questions

1. What is CI runner security?

CI runner security refers to the architectural patterns, hardening configurations, network boundaries, and access controls used to protect the compute workers that execute Continuous Integration and Continuous Delivery pipelines. It ensures that running build scripts cannot compromise source code integrity, leak infrastructure credentials, expose internal networks, or modify software release artifacts.

2. Why are self-hosted CI runners risky?

Self-hosted runners often live inside private enterprise networks and have access to internal systems, databases, and private registries. If an organization runs untrusted pull-request jobs on persistent self-hosted runners without hypervisor-level isolation, attackers can run arbitrary code, escape containers, steal cached environment credentials, and pivot across the internal network.

3. How can GitHub Actions runners be secured?

To secure GitHub Actions:

• Use standard GitHub-hosted runners or ephemeral self-hosted instances.
• Never use self-hosted runners for public repositories or untrusted forks.
• Require manual maintainer approval before running workflows from outside contributors.
• Set default GITHUB_TOKEN permissions to read-only.
• Authenticate to cloud providers via OIDC instead of long-lived secrets.

4. What is the difference between persistent and ephemeral runners?

A persistent runner is a long-running virtual machine or bare-metal host that executes multiple pipeline jobs sequentially over days, weeks, or months. An ephemeral runner is created dynamically to execute a single job and is completely destroyed immediately after that job finishes, ensuring that no state, files, malware, or environment variables survive between runs.

5. Should CI runners run as root?

No. Running runner daemons or pipeline jobs as root creates a critical privilege escalation risk. If a pipeline job runs as root, any arbitrary command execution or container escape immediately grants full control over the underlying host. Runners should execute under dedicated, unprivileged system accounts without sudo access.

6. How can secrets be protected during pipeline execution?

Secrets can be protected by:

• Replacing static credentials with dynamic, short-lived OIDC federation.
• Storing required static secrets in centralized secret managers rather than hardcoded pipeline files.
• Injecting secrets at runtime into memory-backed filesystems (tmpfs) rather than environment variables.
• Using built-in CI log-masking capabilities.
• Preventing secrets from flowing down to fork-based pull request runs.

7. Why is Docker socket access dangerous on CI runners?

Mounting /var/run/docker.sock exposes the host’s Docker engine control socket to the container. Because the Docker daemon runs with root privileges, any process inside the container can command the host daemon to spawn a new container that mounts the host’s root filesystem, completely bypassing container boundaries and escalating privileges to root on the host machine.

8. How should runners handle untrusted pull requests?

Untrusted pull requests should run in isolated environments:

• Execute untrusted workflows on ephemeral, disposable infrastructure with no access to internal networks.
• Strip all pipeline access to production secrets, API keys, and deployment credentials.
• Require manual approval from a trusted repository maintainer before triggering builds for new or non-collaborator pull requests.

9. How often should CI runners be patched?

Ephemeral runners built from automated base images should be updated and redeployed continuously, ideally on a weekly cycle or whenever critical upstream CVEs are released. Persistent runners must receive automated operating system security patches regularly, have their lifespans capped at 24 to 72 hours, and be recycled continuously to avoid configuration drift.

10. What should teams do if a CI runner is compromised?

If a runner compromise is suspected:

1. Immediately revoke all credentials and access tokens exposed to or stored on that runner, including cloud access keys, repository tokens, and deployment roles.
2. Isolate the runner node from the network by revoking its security group rules or severing its network interface.
3. Review audit logs (auditd, eBPF, SIEM) to map any lateral movement or unauthorized network calls made by the runner.
4. Quarantine any build artifacts, packages, or container images produced by the compromised runner during the incident window to prevent supply chain contamination.
5. Decommission the host entirely and reprovision from a clean, verified golden image.

Make Runner Security Part of Pipeline Design

CI/CD runners represent one of the most critical security boundaries in modern engineering organizations. Because they operate at the intersection of developer code, privileged cloud credentials, and internal networks, insecure runners provide adversaries with an attractive launchpad for supply chain attacks, lateral movement, and data exfiltration.

Treating runner hosts as unhardened, shared utilities is an architectural anti-pattern. Building a resilient delivery pipeline requires designing runner infrastructure with the same zero-trust principles applied to production environments:

• Enforce strict workload isolation across different repositories, branches, and trust tiers.
• Prioritize ephemeral compute models that destroy instances after every run to eliminate persistent contamination.
• Harden host environments by enforcing non-root execution, minimal operating systems, and strict filesystem protections.
• Eliminate static, long-lived credentials in favor of short-lived, verifiable OIDC authentication.
• Restrict network egress to block access to cloud metadata services, internal corporate subnets, and unapproved internet domains.
• Continuously monitor and audit runtime behavior using system call inspection and centralized telemetry.

A secure CI/CD pipeline depends not just on writing vulnerability-free application code, but also on maintaining complete control over the infrastructure that compiles, tests, signs, and delivers that code. Auditing your organization’s runner trust boundaries today is an essential step toward building a truly defensible software supply chain.