Onur Yasarlar
Search…
About Me
Linux
How to Install KVM on CentOS 7/8 and Ubuntu 18/20
How to Install VirtualBox on CentOS 7/8 and Ubuntu 18
How to Install Vagrant on Centos 7/8 and Ubuntu 18
Using vim Effectively
Kubernetes
How to Install a Kubernetes Cluster
How to Install and Use K3D
CI/CD
How to install custom Gitlab runners with libvirt executor
Azure
Azure Infrastructure Automation with Gitlab CI
Powered By GitBook
How to install custom Gitlab runners with libvirt executor

Introduction

I have been recently discovering Gitlab CI and quite impressed by its capabilities. While going through Gitlab CI's, I wonder how can I run my pipelines at my lab. It would give me better build capabilities as some of my tests require VMs rather than containers. Then I started diving into on-premise Gitlab runners and managed to install a KVM based runner in my lab. Let me share my work:
I have been working with Gitlab CI only for a day while writing that article and there will be a lot to discover. Feel free to comment if there is a better way to handle a task than how I handled it
Please refer to official documentation as a reference if you face any issues. I also used the official documentation page to write the below post.

Prerequisites

We need to have KVM installed on the server that we will be using as a Gitlab runner. That server will provision VMs based on your pipeline in gitlab-ci.yml. If your server does not have KVM installed, you can use my Ansible role to install it quickly. Let me give you an example of how to use it:
You need to install the role first on your server where you will trigger ansible. You can install the role by ansible-galaxy role install yasarlaro.kvm command.
---
- name: Install KVM
hosts: kvmserver
become: true
roles:
- role: yasarlaro.kvm
Once the KVM is installed, we need to create a base image to be consumed by Gitlab CI. I prefer to use CentOS 7 but you can choose a different one based on your project.
You will hit an error with your e2fsck version. You will need to download the latest source binary of e2fsprogs from your OS distributor and install it.
[ 17.0] Resizing (using virt-resize) to expand the disk to 8.0G
virt-resize: error: libguestfs error: resize2fs: e2fsck 1.42.9
(28-Dec-2013)
/dev/sda1 has unsupported feature(s): metadata_csum
e2fsck: Get a newer version of e2fsck!
​
If reporting bugs, run virt-resize with debugging enabled and include the
complete output:
​
To install it from on CentOS 7:
$ wget https://mirrors.edge.kernel.org/pub/linux/kernel/people/tytso/e2fsprogs/v1.45.6/
$ tar -xf e2fsprogs-1.45.6.tar.xz
$ cd e2fsprogs-1.45.6
$ ./configure
$ make
$ make install
You will also need to install libguestfs-xfs on CentOS 7 to handle XFS file system with virt-builder utility. Otherwise you will hit below error:
[ 22.7] Resizing (using virt-resize) to expand the disk to 8.0G
virt-resize: error: unknown/unavailable method for expanding the xfs
filesystem on /dev/sda4
Now it is time to install the image so each new build request will use that base image:
virt-builder centos-7.8 \
--size 8G \
--output /var/lib/libvirt/images/gitlab-runner-centos7.qcow2 \
--format qcow2 \
--hostname gitlab-runner-centos7 \
--network \
--install curl \
--run-command 'curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | bash' \
--run-command 'curl -s "https://packagecloud.io/install/repositories/github/git-lfs/script.rpm.sh" | bash' \
--run-command 'useradd -m -p "" gitlab-runner -s /bin/bash' \
--install gitlab-runner,git,git-lfs,openssh-server \
--run-command "git lfs install --skip-repo" \
--ssh-inject gitlab-runner:file:/root/.ssh/id_rsa.pub \
--run-command "echo 'gitlab-runner ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers" \
--run-command "sed -E 's/GRUB_CMDLINE_LINUX=\"\"/GRUB_CMDLINE_LINUX=\"net.ifnames=0 biosdevname=0\"/' -i /etc/default/grub" \
--run-command "grub2-mkconfig -o /boot/grub2/grub.cfg" \
--run-command "sed 's/ONBOOT=no/ONBOOT=yes/g' -i /etc/sysconfig/network-scripts/ifcfg-eth0" \
--run-command "sed 's/SELINUX=enforcing/SELINUX=disabled/g' -i /etc/selinux/config"
To see available operating systems for virt-builder utility: osinfo-query os command.
Please note that the image disk to be used is gitlab-runner-centos7.qcow2.

Install gitlab-runner binary

To install GitLab Runner:
  1. 1.
    Add the official GitLab repository:
# For Debian/Ubuntu/Mint
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
​
# For RHEL/CentOS/Fedora
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash
2. Install Binary
# For Debian/Ubuntu/Mint
apt update && apt install gitlab-runner
​
# For RHEL/CentOS/Fedora
yum install gitlab-runner

Register Gitlab runner

Now it is time to register the runner and configure it.
[[email protected] ~]# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=24725 revision=943fc252 version=13.7.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab.com/
Enter the registration token:
xXxXxXxXxXxXxX
Enter a description for the runner:
[runner.onur.lab]:
Enter tags for the runner (comma-separated):
custom,libvirt,kvm
Registering runner... succeeded runner=p9njk8fx
Enter an executor: docker, parallels, shell, ssh, custom, docker-ssh, virtualbox, docker+machine, docker-ssh+machine, kubernetes:
custom
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
[[email protected] ~]#
​
There is an option to use a template configuration file which will make it easier but I will edit the post later once I test it properly

Configure Gitlab Runner

Now we need to tell Github runner how to spin up a VM once needed and how to terminate it once it is not needed anymore.
First thing is to edit config.toml
/etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
​
[session_server]
session_timeout = 1800
​
[[runners]]
name = "lab-runner1"
url = "https://gitlab.com/"
token = "xxxxx"
executor = "custom"
builds_dir = "/home/gitlab-runner/builds"
cache_dir = "/home/gitlab-runner/cache"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.custom]
prepare_exec = "/opt/libvirt-driver/prepare.sh" # Path to a bash script to create VM.
run_exec = "/opt/libvirt-driver/run.sh" # Path to a bash script to run script inside of VM over ssh.
cleanup_exec = "/opt/libvirt-driver/cleanup.sh" # Path to a bash script to delete VM and disks.
As you can see here, we need some scripts to be placed under /opt/libvirt-driver path:
[[email protected] libvirt-driver]# ls -l
total 16
-rwxr-xr-x 1 root root 437 Jan 15 14:42 base.sh
-rwxr-xr-x 1 root root 363 Jan 15 13:58 cleanup.sh
-rwxr-xr-x 1 root root 1561 Jan 15 15:20 prepare.sh
-rwxr-xr-x 1 root root 451 Jan 15 13:58 run.sh
Let me also provide you the content of those scripts below:
run.sh
prepare.sh
cleanup.sh
base.sh
#!/usr/bin/env bash
​
# /opt/libvirt-driver/run.sh
​
currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base script.
​
VM_IP=$(_get_vm_ip)
​
ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no [email protected]"$VM_IP" /bin/bash < "${1}"
if [ $? -ne 0 ]; then
# Exit using the variable, to make the build as failure in GitLab
# CI.
exit "$BUILD_FAILURE_EXIT_CODE"
fi
#!/usr/bin/env bash
​
# /opt/libvirt-driver/prepare.sh
​
currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base script.
​
set -eo pipefail
​
# trap any error, and mark it as a system failure.
trap "exit $SYSTEM_FAILURE_EXIT_CODE" ERR
​
# Copy base disk to use for Job.
qemu-img create -f qcow2 -b "$BASE_VM_IMAGE" "$VM_IMAGE"
​
# Install the VM
virt-install \
--name "$VM_ID" \
--os-variant centos7.0 \
--disk "$VM_IMAGE" \
--import \
--vcpus=2 \
--ram=2048 \
--network default \
--graphics none \
--noautoconsole
​
# Wait for VM to get IP
echo 'Waiting for VM to get IP'
for i in $(seq 1 30); do
VM_IP=$(_get_vm_ip)
​
if [ -n "$VM_IP" ]; then
echo "VM got IP: $VM_IP"
break
fi
​
if [ "$i" == "30" ]; then
echo 'Waited 30 seconds for VM to start, exiting...'
# Inform GitLab Runner that this is a system failure, so it
# should be retried.
exit "$SYSTEM_FAILURE_EXIT_CODE"
fi
​
sleep 1s
done
​
# Wait for ssh to become available
echo "Waiting for sshd to be available"
for i in $(seq 1 30); do
if ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no [email protected]"$VM_IP" >/dev/null 2>/dev/null; then
break
fi
​
if [ "$i" == "30" ]; then
echo 'Waited 30 seconds for sshd to start, exiting...'
# Inform GitLab Runner that this is a system failure, so it
# should be retried.
exit "$SYSTEM_FAILURE_EXIT_CODE"
fi
​
sleep 1s
done
​
#!/usr/bin/env bash
​
# /opt/libvirt-driver/cleanup.sh
​
currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base script.
​
set -eo pipefail
​
# Destroy VM.
virsh shutdown "$VM_ID"
​
# Undefine VM.
virsh undefine "$VM_ID"
​
# Delete VM disk.
if [ -f "$VM_IMAGE" ]; then
rm "$VM_IMAGE"
fi
#!/usr/bin/env bash
​
# /opt/libvirt-driver/base.sh
​
VM_IMAGES_PATH="/var/lib/libvirt/images"
BASE_VM_IMAGE="$VM_IMAGES_PATH/gitlab-runner-centos7.qcow2"
VM_ID="runner-$CUSTOM_ENV_CI_RUNNER_ID-project-$CUSTOM_ENV_CI_PROJECT_ID-concurrent-$CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID-job-$CUSTOM_ENV_CI_JOB_ID"
VM_IMAGE="$VM_IMAGES_PATH/$VM_ID.qcow2"
​
_get_vm_ip() {
virsh -q domifaddr "$VM_ID" | awk '{print $4}' | sed -E 's|/([0-9]+)?$||'
}
Then we will need to restart the service so the new configuration can be loaded
$ systemctl restart gitlab-runner

How to use custom Gitlab runner

Now you should be able to see your custom runner registered under Gitlab Project --> Settings --> CI / CD
You can write a simple gitlab-ci.yml and push it to test your build:
.gitlab-ci.yml
stages:
- test
​
build:
stage: test
script:
- /usr/sbin/ip addr show
- /usr/bin/sudo yum install wget -y
- /usr/bin/cat /etc/*release
- /usr/bin/hostname
- /usr/bin/date
- /usr/bin/pwd
tags:
- custom
​
​
Kubernetes - Previous
How to Install and Use K3D
Next - Azure
Azure Infrastructure Automation with Gitlab CI
Last modified 1yr ago
Copy link
On this page
Introduction
Prerequisites
Install gitlab-runner binary
Register Gitlab runner
Configure Gitlab Runner
How to use custom Gitlab runner