Onur Yasarlar
Search…
Linux
How to install custom Gitlab runners with libvirt executor
Introduction
I have been recently discovering Gitlab CI and quite impressed by its capabilities. While going through Gitlab CI's, I wonder how can I run my pipelines at my lab. It would give me better build capabilities as some of my tests require VMs rather than containers. Then I started diving into on-premise Gitlab runners and managed to install a KVM based runner in my lab. Let me share my work:
I have been working with Gitlab CI only for a day while writing that article and there will be a lot to discover. Feel free to comment if there is a better way to handle a task than how I handled it
Please refer to official documentation as a reference if you face any issues. I also used the official documentation page to write the below post.
Prerequisites
We need to have KVM installed on the server that we will be using as a Gitlab runner. That server will provision VMs based on your pipeline in
gitlab-ci.yml. If your server does not have KVM installed, you can use my Ansible role to install it quickly. Let me give you an example of how to use it:You need to install the role first on your server where you will trigger ansible. You can install the role by
ansible-galaxy role install yasarlaro.kvm command.1
---
2
- name: Install KVM
3
hosts: kvmserver
4
become: true
5
roles:
6
- role: yasarlaro.kvm
Copied!
Once the KVM is installed, we need to create a base image to be consumed by Gitlab CI. I prefer to use CentOS 7 but you can choose a different one based on your project.
You will hit an error with your e2fsck version. You will need to download the latest source binary of
e2fsprogs from your OS distributor and install it.1
[ 17.0] Resizing (using virt-resize) to expand the disk to 8.0G
2
virt-resize: error: libguestfs error: resize2fs: e2fsck 1.42.9
3
(28-Dec-2013)
4
/dev/sda1 has unsupported feature(s): metadata_csum
5
e2fsck: Get a newer version of e2fsck!
6
​
7
If reporting bugs, run virt-resize with debugging enabled and include the
8
complete output:
9
​
Copied!
To install it from on CentOS 7:
1
$ wget https://mirrors.edge.kernel.org/pub/linux/kernel/people/tytso/e2fsprogs/v1.45.6/
2
$ tar -xf e2fsprogs-1.45.6.tar.xz
3
$ cd e2fsprogs-1.45.6
4
$ ./configure
5
$ make
6
$ make install
Copied!
You will also need to install
libguestfs-xfs on CentOS 7 to handle XFS file system with virt-builder utility. Otherwise you will hit below error:1
[ 22.7] Resizing (using virt-resize) to expand the disk to 8.0G
2
virt-resize: error: unknown/unavailable method for expanding the xfs
3
filesystem on /dev/sda4
Copied!
Now it is time to install the image so each new build request will use that base image:
1
virt-builder centos-7.8 \
2
--size 8G \
3
--output /var/lib/libvirt/images/gitlab-runner-centos7.qcow2 \
4
--format qcow2 \
5
--hostname gitlab-runner-centos7 \
6
--network \
7
--install curl \
8
--run-command 'curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | bash' \
9
--run-command 'curl -s "https://packagecloud.io/install/repositories/github/git-lfs/script.rpm.sh" | bash' \
10
--run-command 'useradd -m -p "" gitlab-runner -s /bin/bash' \
11
--install gitlab-runner,git,git-lfs,openssh-server \
12
--run-command "git lfs install --skip-repo" \
13
--ssh-inject gitlab-runner:file:/root/.ssh/id_rsa.pub \
14
--run-command "echo 'gitlab-runner ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers" \
15
--run-command "sed -E 's/GRUB_CMDLINE_LINUX=\"\"/GRUB_CMDLINE_LINUX=\"net.ifnames=0 biosdevname=0\"/' -i /etc/default/grub" \
16
--run-command "grub2-mkconfig -o /boot/grub2/grub.cfg" \
17
--run-command "sed 's/ONBOOT=no/ONBOOT=yes/g' -i /etc/sysconfig/network-scripts/ifcfg-eth0" \
18
--run-command "sed 's/SELINUX=enforcing/SELINUX=disabled/g' -i /etc/selinux/config"
Copied!
To see available operating systems for
virt-builder utility: osinfo-query os command.Please note that the image disk to be used is gitlab-runner-centos7.qcow2.
Install gitlab-runner binary
To install GitLab Runner:
- 1.Add the official GitLab repository:
1
# For Debian/Ubuntu/Mint
2
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
3
​
4
# For RHEL/CentOS/Fedora
5
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash
Copied!
2. Install Binary
1
# For Debian/Ubuntu/Mint
2
apt update && apt install gitlab-runner
3
​
4
# For RHEL/CentOS/Fedora
5
yum install gitlab-runner
Copied!
Register Gitlab runner
Now it is time to register the runner and configure it.
1
[[email protected] ~]# gitlab-runner register
2
Runtime platform arch=amd64 os=linux pid=24725 revision=943fc252 version=13.7.0
3
Running in system-mode.
4
5
Enter the GitLab instance URL (for example, https://gitlab.com/):
6
https://gitlab.com/
7
Enter the registration token:
8
xXxXxXxXxXxXxX
9
Enter a description for the runner:
10
[runner.onur.lab]:
11
Enter tags for the runner (comma-separated):
12
custom,libvirt,kvm
13
Registering runner... succeeded runner=p9njk8fx
14
Enter an executor: docker, parallels, shell, ssh, custom, docker-ssh, virtualbox, docker+machine, docker-ssh+machine, kubernetes:
15
custom
16
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
17
[[email protected] ~]#
18
​
Copied!
There is an option to use a template configuration file which will make it easier but I will edit the post later once I test it properly
Configure Gitlab Runner
Now we need to tell Github runner how to spin up a VM once needed and how to terminate it once it is not needed anymore.
First thing is to edit
config.toml/etc/gitlab-runner/config.toml
1
concurrent = 1
2
check_interval = 0
3
​
4
[session_server]
5
session_timeout = 1800
6
​
7
[[runners]]
8
name = "lab-runner1"
9
url = "https://gitlab.com/"
10
token = "xxxxx"
11
executor = "custom"
12
builds_dir = "/home/gitlab-runner/builds"
13
cache_dir = "/home/gitlab-runner/cache"
14
[runners.custom_build_dir]
15
[runners.cache]
16
[runners.cache.s3]
17
[runners.cache.gcs]
18
[runners.custom]
19
prepare_exec = "/opt/libvirt-driver/prepare.sh" # Path to a bash script to create VM.
20
run_exec = "/opt/libvirt-driver/run.sh" # Path to a bash script to run script inside of VM over ssh.
21
cleanup_exec = "/opt/libvirt-driver/cleanup.sh" # Path to a bash script to delete VM and disks.
Copied!
As you can see here, we need some scripts to be placed under /opt/libvirt-driver path:
1
[[email protected] libvirt-driver]# ls -l
2
total 16
3
-rwxr-xr-x 1 root root 437 Jan 15 14:42 base.sh
4
-rwxr-xr-x 1 root root 363 Jan 15 13:58 cleanup.sh
5
-rwxr-xr-x 1 root root 1561 Jan 15 15:20 prepare.sh
6
-rwxr-xr-x 1 root root 451 Jan 15 13:58 run.sh
Copied!
Let me also provide you the content of those scripts below:
run.sh
prepare.sh
cleanup.sh
base.sh
1
#!/usr/bin/env bash
2
​
3
# /opt/libvirt-driver/run.sh
4
​
5
currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
6
source ${currentDir}/base.sh # Get variables from base script.
7
​
8
VM_IP=$(_get_vm_ip)
9
​
10
ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no [email protected]"$VM_IP" /bin/bash < "${1}"
11
if [ $? -ne 0 ]; then
12
# Exit using the variable, to make the build as failure in GitLab
13
# CI.
14
exit "$BUILD_FAILURE_EXIT_CODE"
15
fi
Copied!
1
#!/usr/bin/env bash
2
​
3
# /opt/libvirt-driver/prepare.sh
4
​
5
currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
6
source ${currentDir}/base.sh # Get variables from base script.
7
​
8
set -eo pipefail
9
​
10
# trap any error, and mark it as a system failure.
11
trap "exit $SYSTEM_FAILURE_EXIT_CODE" ERR
12
​
13
# Copy base disk to use for Job.
14
qemu-img create -f qcow2 -b "$BASE_VM_IMAGE" "$VM_IMAGE"
15
​
16
# Install the VM
17
virt-install \
18
--name "$VM_ID" \
19
--os-variant centos7.0 \
20
--disk "$VM_IMAGE" \
21
--import \
22
--vcpus=2 \
23
--ram=2048 \
24
--network default \
25
--graphics none \
26
--noautoconsole
27
​
28
# Wait for VM to get IP
29
echo 'Waiting for VM to get IP'
30
for i in $(seq 1 30); do
31
VM_IP=$(_get_vm_ip)
32
​
33
if [ -n "$VM_IP" ]; then
34
echo "VM got IP: $VM_IP"
35
break
36
fi
37
​
38
if [ "$i" == "30" ]; then
39
echo 'Waited 30 seconds for VM to start, exiting...'
40
# Inform GitLab Runner that this is a system failure, so it
41
# should be retried.
42
exit "$SYSTEM_FAILURE_EXIT_CODE"
43
fi
44
​
45
sleep 1s
46
done
47
​
48
# Wait for ssh to become available
49
echo "Waiting for sshd to be available"
50
for i in $(seq 1 30); do
51
if ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no [email protected]"$VM_IP" >/dev/null 2>/dev/null; then
52
break
53
fi
54
​
55
if [ "$i" == "30" ]; then
56
echo 'Waited 30 seconds for sshd to start, exiting...'
57
# Inform GitLab Runner that this is a system failure, so it
58
# should be retried.
59
exit "$SYSTEM_FAILURE_EXIT_CODE"
60
fi
61
​
62
sleep 1s
63
done
64
​
Copied!
1
#!/usr/bin/env bash
2
​
3
# /opt/libvirt-driver/cleanup.sh
4
​
5
currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
6
source ${currentDir}/base.sh # Get variables from base script.
7
​
8
set -eo pipefail
9
​
10
# Destroy VM.
11
virsh shutdown "$VM_ID"
12
​
13
# Undefine VM.
14
virsh undefine "$VM_ID"
15
​
16
# Delete VM disk.
17
if [ -f "$VM_IMAGE" ]; then
18
rm "$VM_IMAGE"
19
fi
Copied!
1
#!/usr/bin/env bash
2
​
3
# /opt/libvirt-driver/base.sh
4
​
5
VM_IMAGES_PATH="/var/lib/libvirt/images"
6
BASE_VM_IMAGE="$VM_IMAGES_PATH/gitlab-runner-centos7.qcow2"
7
VM_ID="runner-$CUSTOM_ENV_CI_RUNNER_ID-project-$CUSTOM_ENV_CI_PROJECT_ID-concurrent-$CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID-job-$CUSTOM_ENV_CI_JOB_ID"
8
VM_IMAGE="$VM_IMAGES_PATH/$VM_ID.qcow2"
9
​
10
_get_vm_ip() {
11
virsh -q domifaddr "$VM_ID" | awk '{print $4}' | sed -E 's|/([0-9]+)?$||'
12
}
Copied!
Then we will need to restart the service so the new configuration can be loaded
1
$ systemctl restart gitlab-runner
Copied!
How to use custom Gitlab runner
Now you should be able to see your custom runner registered under Gitlab Project --> Settings --> CI / CD
You can write a simple
gitlab-ci.yml and push it to test your build:.gitlab-ci.yml
1
stages:
2
- test
3
​
4
build:
5
stage: test
6
script:
7
- /usr/sbin/ip addr show
8
- /usr/bin/sudo yum install wget -y
9
- /usr/bin/cat /etc/*release
10
- /usr/bin/hostname
11
- /usr/bin/date
12
- /usr/bin/pwd
13
tags:
14
- custom
15
​
Copied!
​