How to install custom Gitlab runners with libvirt executor

Introduction

I have been recently discovering Gitlab CI and quite impressed by its capabilities. While going through Gitlab CI's, I wonder how can I run my pipelines at my lab. It would give me better build capabilities as some of my tests require VMs rather than containers. Then I started diving into on-premise Gitlab runners and managed to install a KVM based runner in my lab. Let me share my work:

I have been working with Gitlab CI only for a day while writing that article and there will be a lot to discover. Feel free to comment if there is a better way to handle a task than how I handled it

Please refer to official documentation as a reference if you face any issues. I also used the official documentation page to write the below post.

Prerequisites

We need to have KVM installed on the server that we will be using as a Gitlab runner. That server will provision VMs based on your pipeline in gitlab-ci.yml. If your server does not have KVM installed, you can use my Ansible role to install it quickly. Let me give you an example of how to use it:

You need to install the role first on your server where you will trigger ansible. You can install the role by ansible-galaxy role install yasarlaro.kvm command.

---
- name: Install KVM
  hosts: kvmserver
  become: true
  roles:
    - role: yasarlaro.kvm

Once the KVM is installed, we need to create a base image to be consumed by Gitlab CI. I prefer to use CentOS 7 but you can choose a different one based on your project.

You will hit an error with your e2fsck version. You will need to download the latest source binary of e2fsprogs from your OS distributor and install it.

[  17.0] Resizing (using virt-resize) to expand the disk to 8.0G
virt-resize: error: libguestfs error: resize2fs: e2fsck 1.42.9 
(28-Dec-2013)
/dev/sda1 has unsupported feature(s): metadata_csum
e2fsck: Get a newer version of e2fsck!

If reporting bugs, run virt-resize with debugging enabled and include the 
complete output:

To install it from on CentOS 7:

$ wget https://mirrors.edge.kernel.org/pub/linux/kernel/people/tytso/e2fsprogs/v1.45.6/
$ tar -xf e2fsprogs-1.45.6.tar.xz
$ cd e2fsprogs-1.45.6
$ ./configure
$ make
$ make install

You will also need to install libguestfs-xfs on CentOS 7 to handle XFS file system with virt-builder utility. Otherwise you will hit below error:

[  22.7] Resizing (using virt-resize) to expand the disk to 8.0G
virt-resize: error: unknown/unavailable method for expanding the xfs 
filesystem on /dev/sda4

Now it is time to install the image so each new build request will use that base image:

virt-builder centos-7.8 \
    --size 8G \
    --output /var/lib/libvirt/images/gitlab-runner-centos7.qcow2 \
    --format qcow2 \
    --hostname gitlab-runner-centos7 \
    --network \
    --install curl \
    --run-command 'curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | bash' \
    --run-command 'curl -s "https://packagecloud.io/install/repositories/github/git-lfs/script.rpm.sh" | bash' \
    --run-command 'useradd -m -p "" gitlab-runner -s /bin/bash' \
    --install gitlab-runner,git,git-lfs,openssh-server \
    --run-command "git lfs install --skip-repo" \
    --ssh-inject gitlab-runner:file:/root/.ssh/id_rsa.pub \
    --run-command "echo 'gitlab-runner ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers" \
    --run-command "sed -E 's/GRUB_CMDLINE_LINUX=\"\"/GRUB_CMDLINE_LINUX=\"net.ifnames=0 biosdevname=0\"/' -i /etc/default/grub" \
    --run-command "grub2-mkconfig -o /boot/grub2/grub.cfg" \
    --run-command "sed 's/ONBOOT=no/ONBOOT=yes/g' -i /etc/sysconfig/network-scripts/ifcfg-eth0" \
    --run-command "sed 's/SELINUX=enforcing/SELINUX=disabled/g' -i /etc/selinux/config"

To see available operating systems for virt-builder utility: osinfo-query os command.

Please note that the image disk to be used is gitlab-runner-centos7.qcow2.

Install gitlab-runner binary

To install GitLab Runner:

Add the official GitLab repository:

# For Debian/Ubuntu/Mint
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

# For RHEL/CentOS/Fedora
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash

2. Install Binary

# For Debian/Ubuntu/Mint
apt update && apt install gitlab-runner

# For RHEL/CentOS/Fedora
yum install gitlab-runner

Register Gitlab runner

Now it is time to register the runner and configure it.

[root@runner ~]# gitlab-runner register
Runtime platform                                    arch=amd64 os=linux pid=24725 revision=943fc252 version=13.7.0
Running in system-mode.                            
                                                   
Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab.com/
Enter the registration token:
xXxXxXxXxXxXxX
Enter a description for the runner:
[runner.onur.lab]: 
Enter tags for the runner (comma-separated):
custom,libvirt,kvm                        
Registering runner... succeeded                     runner=p9njk8fx
Enter an executor: docker, parallels, shell, ssh, custom, docker-ssh, virtualbox, docker+machine, docker-ssh+machine, kubernetes:
custom
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! 
[root@runner ~]#

There is an option to use a template configuration file which will make it easier but I will edit the post later once I test it properly

Configure Gitlab Runner

Now we need to tell Github runner how to spin up a VM once needed and how to terminate it once it is not needed anymore.

First thing is to edit config.toml

/etc/gitlab-runner/config.toml

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "lab-runner1"
  url = "https://gitlab.com/"
  token = "xxxxx"
  executor = "custom"
  builds_dir = "/home/gitlab-runner/builds"
  cache_dir = "/home/gitlab-runner/cache"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.custom]
    prepare_exec = "/opt/libvirt-driver/prepare.sh" # Path to a bash script to create VM.
    run_exec = "/opt/libvirt-driver/run.sh" # Path to a bash script to run script inside of VM over ssh.
    cleanup_exec = "/opt/libvirt-driver/cleanup.sh" # Path to a bash script to delete VM and disks.

As you can see here, we need some scripts to be placed under /opt/libvirt-driver path:

[root@runner libvirt-driver]# ls -l
total 16
-rwxr-xr-x 1 root root  437 Jan 15 14:42 base.sh
-rwxr-xr-x 1 root root  363 Jan 15 13:58 cleanup.sh
-rwxr-xr-x 1 root root 1561 Jan 15 15:20 prepare.sh
-rwxr-xr-x 1 root root  451 Jan 15 13:58 run.sh

Let me also provide you the content of those scripts below:

#!/usr/bin/env bash

# /opt/libvirt-driver/run.sh

currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base script.

VM_IP=$(_get_vm_ip)

ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no gitlab-runner@"$VM_IP" /bin/bash < "${1}"
if [ $? -ne 0 ]; then
    # Exit using the variable, to make the build as failure in GitLab
    # CI.
    exit "$BUILD_FAILURE_EXIT_CODE"
fi

#!/usr/bin/env bash

# /opt/libvirt-driver/prepare.sh

currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base script.

set -eo pipefail

# trap any error, and mark it as a system failure.
trap "exit $SYSTEM_FAILURE_EXIT_CODE" ERR

# Copy base disk to use for Job.
qemu-img create -f qcow2 -b "$BASE_VM_IMAGE" "$VM_IMAGE"

# Install the VM
virt-install \
    --name "$VM_ID" \
    --os-variant centos7.0 \
    --disk "$VM_IMAGE" \
    --import \
    --vcpus=2 \
    --ram=2048 \
    --network default \
    --graphics none \
    --noautoconsole

# Wait for VM to get IP
echo 'Waiting for VM to get IP'
for i in $(seq 1 30); do
    VM_IP=$(_get_vm_ip)

    if [ -n "$VM_IP" ]; then
        echo "VM got IP: $VM_IP"
        break
    fi

    if [ "$i" == "30" ]; then
        echo 'Waited 30 seconds for VM to start, exiting...'
        # Inform GitLab Runner that this is a system failure, so it
        # should be retried.
        exit "$SYSTEM_FAILURE_EXIT_CODE"
    fi

    sleep 1s
done

# Wait for ssh to become available
echo "Waiting for sshd to be available"
for i in $(seq 1 30); do
    if ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no gitlab-runner@"$VM_IP" >/dev/null 2>/dev/null; then
        break
    fi

    if [ "$i" == "30" ]; then
        echo 'Waited 30 seconds for sshd to start, exiting...'
        # Inform GitLab Runner that this is a system failure, so it
        # should be retried.
        exit "$SYSTEM_FAILURE_EXIT_CODE"
    fi

    sleep 1s
done

#!/usr/bin/env bash

# /opt/libvirt-driver/cleanup.sh

currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base script.

set -eo pipefail

# Destroy VM.
virsh shutdown "$VM_ID"

# Undefine VM.
virsh undefine "$VM_ID"

# Delete VM disk.
if [ -f "$VM_IMAGE" ]; then
    rm "$VM_IMAGE"
fi

#!/usr/bin/env bash

# /opt/libvirt-driver/base.sh

VM_IMAGES_PATH="/var/lib/libvirt/images"
BASE_VM_IMAGE="$VM_IMAGES_PATH/gitlab-runner-centos7.qcow2"
VM_ID="runner-$CUSTOM_ENV_CI_RUNNER_ID-project-$CUSTOM_ENV_CI_PROJECT_ID-concurrent-$CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID-job-$CUSTOM_ENV_CI_JOB_ID"
VM_IMAGE="$VM_IMAGES_PATH/$VM_ID.qcow2"

_get_vm_ip() {
    virsh -q domifaddr "$VM_ID" | awk '{print $4}' | sed -E 's|/([0-9]+)?$||'
}

Then we will need to restart the service so the new configuration can be loaded

$ systemctl restart gitlab-runner

How to use custom Gitlab runner

Now you should be able to see your custom runner registered under Gitlab Project --> Settings --> CI / CD

You can write a simple gitlab-ci.yml and push it to test your build:

.gitlab-ci.yml

stages:
  - test

build:
  stage: test
  script:
    - /usr/sbin/ip addr show
    - /usr/bin/sudo yum install wget -y
    - /usr/bin/cat /etc/*release
    - /usr/bin/hostname
    - /usr/bin/date
    - /usr/bin/pwd
  tags:
      - custom

PreviousQuick Introduction to GitLab CICD Pipelines NextAzure Infrastructure Automation with Gitlab CI

Last updated 3 years ago