How to install custom Gitlab runners with libvirt executor
I have been recently discovering Gitlab CI and quite impressed by its capabilities. While going through Gitlab CI's, I wonder how can I run my pipelines at my lab. It would give me better build capabilities as some of my tests require VMs rather than containers. Then I started diving into on-premise Gitlab runners and managed to install a KVM based runner in my lab. Let me share my work:
I have been working with Gitlab CI only for a day while writing that article and there will be a lot to discover. Feel free to comment if there is a better way to handle a task than how I handled it
Please refer to official documentation as a reference if you face any issues. I also used the official documentation page to write the below post.
We need to have KVM installed on the server that we will be using as a Gitlab runner. That server will provision VMs based on your pipeline in
gitlab-ci.yml. If your server does not have KVM installed, you can use my Ansible role to install it quickly. Let me give you an example of how to use it:You need to install the role first on your server where you will trigger ansible. You can install the role by
ansible-galaxy role install yasarlaro.kvm command.---
- name: Install KVM
hosts: kvmserver
become: true
roles:
- role: yasarlaro.kvm
Once the KVM is installed, we need to create a base image to be consumed by Gitlab CI. I prefer to use CentOS 7 but you can choose a different one based on your project.
You will hit an error with your e2fsck version. You will need to download the latest source binary of
e2fsprogs from your OS distributor and install it.[ 17.0] Resizing (using virt-resize) to expand the disk to 8.0G
virt-resize: error: libguestfs error: resize2fs: e2fsck 1.42.9
(28-Dec-2013)
/dev/sda1 has unsupported feature(s): metadata_csum
e2fsck: Get a newer version of e2fsck!
If reporting bugs, run virt-resize with debugging enabled and include the
complete output:
To install it from on CentOS 7:
$ wget https://mirrors.edge.kernel.org/pub/linux/kernel/people/tytso/e2fsprogs/v1.45.6/
$ tar -xf e2fsprogs-1.45.6.tar.xz
$ cd e2fsprogs-1.45.6
$ ./configure
$ make
$ make install
You will also need to install
libguestfs-xfs on CentOS 7 to handle XFS file system with virt-builder utility. Otherwise you will hit below error:[ 22.7] Resizing (using virt-resize) to expand the disk to 8.0G
virt-resize: error: unknown/unavailable method for expanding the xfs
filesystem on /dev/sda4
Now it is time to install the image so each new build request will use that base image:
virt-builder centos-7.8 \
--size 8G \
--output /var/lib/libvirt/images/gitlab-runner-centos7.qcow2 \
--format qcow2 \
--hostname gitlab-runner-centos7 \
--network \
--install curl \
--run-command 'curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | bash' \
--run-command 'curl -s "https://packagecloud.io/install/repositories/github/git-lfs/script.rpm.sh" | bash' \
--run-command 'useradd -m -p "" gitlab-runner -s /bin/bash' \
--install gitlab-runner,git,git-lfs,openssh-server \
--run-command "git lfs install --skip-repo" \
--ssh-inject gitlab-runner:file:/root/.ssh/id_rsa.pub \
--run-command "echo 'gitlab-runner ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers" \
--run-command "sed -E 's/GRUB_CMDLINE_LINUX=\"\"/GRUB_CMDLINE_LINUX=\"net.ifnames=0 biosdevname=0\"/' -i /etc/default/grub" \
--run-command "grub2-mkconfig -o /boot/grub2/grub.cfg" \
--run-command "sed 's/ONBOOT=no/ONBOOT=yes/g' -i /etc/sysconfig/network-scripts/ifcfg-eth0" \
--run-command "sed 's/SELINUX=enforcing/SELINUX=disabled/g' -i /etc/selinux/config"
To see available operating systems for
virt-builder utility: osinfo-query os command.Please note that the image disk to be used is gitlab-runner-centos7.qcow2.
To install GitLab Runner:
- 1.Add the official GitLab repository:
# For Debian/Ubuntu/Mint
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
# For RHEL/CentOS/Fedora
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash
2. Install Binary
# For Debian/Ubuntu/Mint
apt update && apt install gitlab-runner
# For RHEL/CentOS/Fedora
yum install gitlab-runner
Now it is time to register the runner and configure it.
[root@runner ~]# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=24725 revision=943fc252 version=13.7.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab.com/
Enter the registration token:
xXxXxXxXxXxXxX
Enter a description for the runner:
[runner.onur.lab]:
Enter tags for the runner (comma-separated):
custom,libvirt,kvm
Registering runner... succeeded runner=p9njk8fx
Enter an executor: docker, parallels, shell, ssh, custom, docker-ssh, virtualbox, docker+machine, docker-ssh+machine, kubernetes:
custom
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
[root@runner ~]#
There is an option to use a template configuration file which will make it easier but I will edit the post later once I test it properly
Now we need to tell Github runner how to spin up a VM once needed and how to terminate it once it is not needed anymore.
First thing is to edit
config.toml/etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "lab-runner1"
url = "https://gitlab.com/"
token = "xxxxx"
executor = "custom"
builds_dir = "/home/gitlab-runner/builds"
cache_dir = "/home/gitlab-runner/cache"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.custom]
prepare_exec = "/opt/libvirt-driver/prepare.sh" # Path to a bash script to create VM.
run_exec = "/opt/libvirt-driver/run.sh" # Path to a bash script to run script inside of VM over ssh.
cleanup_exec = "/opt/libvirt-driver/cleanup.sh" # Path to a bash script to delete VM and disks.
As you can see here, we need some scripts to be placed under /opt/libvirt-driver path:
[root@runner libvirt-driver]# ls -l
total 16
-rwxr-xr-x 1 root root 437 Jan 15 14:42 base.sh
-rwxr-xr-x 1 root root 363 Jan 15 13:58 cleanup.sh
-rwxr-xr-x 1 root root 1561 Jan 15 15:20 prepare.sh
-rwxr-xr-x 1 root root 451 Jan 15 13:58 run.sh
Let me also provide you the content of those scripts below:
run.sh
prepare.sh
cleanup.sh
base.sh
#!/usr/bin/env bash
# /opt/libvirt-driver/run.sh
currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base script.
VM_IP=$(_get_vm_ip)
ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no gitlab-runner@"$VM_IP" /bin/bash < "${1}"
if [ $? -ne 0 ]; then
# Exit using the variable, to make the build as failure in GitLab
# CI.
exit "$BUILD_FAILURE_EXIT_CODE"
fi
#!/usr/bin/env bash
# /opt/libvirt-driver/prepare.sh
currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base script.
set -eo pipefail
# trap any error, and mark it as a system failure.
trap "exit $SYSTEM_FAILURE_EXIT_CODE" ERR
# Copy base disk to use for Job.
qemu-img create -f qcow2 -b "$BASE_VM_IMAGE" "$VM_IMAGE"
# Install the VM
virt-install \
--name "$VM_ID" \
--os-variant centos7.0 \
--disk "$VM_IMAGE" \
--import \
--vcpus=2 \
--ram=2048 \
--network default \
--graphics none \
--noautoconsole
# Wait for VM to get IP
echo 'Waiting for VM to get IP'
for i in $(seq 1 30); do
VM_IP=$(_get_vm_ip)
if [ -n "$VM_IP" ]; then
echo "VM got IP: $VM_IP"
break
fi
if [ "$i" == "30" ]; then
echo 'Waited 30 seconds for VM to start, exiting...'
# Inform GitLab Runner that this is a system failure, so it
# should be retried.
exit "$SYSTEM_FAILURE_EXIT_CODE"
fi
sleep 1s
done
# Wait for ssh to become available
echo "Waiting for sshd to be available"
for i in $(seq 1 30); do
if ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no gitlab-runner@"$VM_IP" >/dev/null 2>/dev/null; then
break
fi
if [ "$i" == "30" ]; then
echo 'Waited 30 seconds for sshd to start, exiting...'
# Inform GitLab Runner that this is a system failure, so it
# should be retried.
exit "$SYSTEM_FAILURE_EXIT_CODE"
fi
sleep 1s
done
#!/usr/bin/env bash
# /opt/libvirt-driver/cleanup.sh
currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base script.
set -eo pipefail
# Destroy VM.
virsh shutdown "$VM_ID"
# Undefine VM.
virsh undefine "$VM_ID"
# Delete VM disk.
if [ -f "$VM_IMAGE" ]; then
rm "$VM_IMAGE"
fi
#!/usr/bin/env bash
# /opt/libvirt-driver/base.sh
VM_IMAGES_PATH="/var/lib/libvirt/images"
BASE_VM_IMAGE="$VM_IMAGES_PATH/gitlab-runner-centos7.qcow2"
VM_ID="runner-$CUSTOM_ENV_CI_RUNNER_ID-project-$CUSTOM_ENV_CI_PROJECT_ID-concurrent-$CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID-job-$CUSTOM_ENV_CI_JOB_ID"
VM_IMAGE="$VM_IMAGES_PATH/$VM_ID.qcow2"
_get_vm_ip() {
virsh -q domifaddr "$VM_ID" | awk '{print $4}' | sed -E 's|/([0-9]+)?$||'
}
Then we will need to restart the service so the new configuration can be loaded
$ systemctl restart gitlab-runner
Now you should be able to see your custom runner registered under Gitlab Project --> Settings --> CI / CD
You can write a simple
gitlab-ci.yml and push it to test your build:.gitlab-ci.yml
stages:
- test
build:
stage: test
script:
- /usr/sbin/ip addr show
- /usr/bin/sudo yum install wget -y
- /usr/bin/cat /etc/*release
- /usr/bin/hostname
- /usr/bin/date
- /usr/bin/pwd
tags:
- custom