How to install custom Gitlab runners with libvirt executor
Introduction
I have been recently discovering Gitlab CI and quite impressed by its capabilities. While going through Gitlab CI's, I wonder how can I run my pipelines at my lab. It would give me better build capabilities as some of my tests require VMs rather than containers. Then I started diving into on-premise Gitlab runners and managed to install a KVM based runner in my lab. Let me share my work:
Prerequisites
We need to have KVM installed on the server that we will be using as a Gitlab runner. That server will provision VMs based on your pipeline in gitlab-ci.yml
. If your server does not have KVM installed, you can use my Ansible role to install it quickly. Let me give you an example of how to use it:
---
- name: Install KVM
hosts: kvmserver
become: true
roles:
- role: yasarlaro.kvm
Once the KVM is installed, we need to create a base image to be consumed by Gitlab CI. I prefer to use CentOS 7 but you can choose a different one based on your project.
Now it is time to install the image so each new build request will use that base image:
virt-builder centos-7.8 \
--size 8G \
--output /var/lib/libvirt/images/gitlab-runner-centos7.qcow2 \
--format qcow2 \
--hostname gitlab-runner-centos7 \
--network \
--install curl \
--run-command 'curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | bash' \
--run-command 'curl -s "https://packagecloud.io/install/repositories/github/git-lfs/script.rpm.sh" | bash' \
--run-command 'useradd -m -p "" gitlab-runner -s /bin/bash' \
--install gitlab-runner,git,git-lfs,openssh-server \
--run-command "git lfs install --skip-repo" \
--ssh-inject gitlab-runner:file:/root/.ssh/id_rsa.pub \
--run-command "echo 'gitlab-runner ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers" \
--run-command "sed -E 's/GRUB_CMDLINE_LINUX=\"\"/GRUB_CMDLINE_LINUX=\"net.ifnames=0 biosdevname=0\"/' -i /etc/default/grub" \
--run-command "grub2-mkconfig -o /boot/grub2/grub.cfg" \
--run-command "sed 's/ONBOOT=no/ONBOOT=yes/g' -i /etc/sysconfig/network-scripts/ifcfg-eth0" \
--run-command "sed 's/SELINUX=enforcing/SELINUX=disabled/g' -i /etc/selinux/config"
Please note that the image disk to be used is gitlab-runner-centos7.qcow2.
Install gitlab-runner binary
To install GitLab Runner:
Add the official GitLab repository:
# For Debian/Ubuntu/Mint
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
# For RHEL/CentOS/Fedora
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash
2. Install Binary
# For Debian/Ubuntu/Mint
apt update && apt install gitlab-runner
# For RHEL/CentOS/Fedora
yum install gitlab-runner
Register Gitlab runner
Now it is time to register the runner and configure it.
[root@runner ~]# gitlab-runner register
Runtime platform arch=amd64 os=linux pid=24725 revision=943fc252 version=13.7.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab.com/
Enter the registration token:
xXxXxXxXxXxXxX
Enter a description for the runner:
[runner.onur.lab]:
Enter tags for the runner (comma-separated):
custom,libvirt,kvm
Registering runner... succeeded runner=p9njk8fx
Enter an executor: docker, parallels, shell, ssh, custom, docker-ssh, virtualbox, docker+machine, docker-ssh+machine, kubernetes:
custom
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
[root@runner ~]#
Configure Gitlab Runner
Now we need to tell Github runner how to spin up a VM once needed and how to terminate it once it is not needed anymore.
First thing is to edit config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "lab-runner1"
url = "https://gitlab.com/"
token = "xxxxx"
executor = "custom"
builds_dir = "/home/gitlab-runner/builds"
cache_dir = "/home/gitlab-runner/cache"
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.custom]
prepare_exec = "/opt/libvirt-driver/prepare.sh" # Path to a bash script to create VM.
run_exec = "/opt/libvirt-driver/run.sh" # Path to a bash script to run script inside of VM over ssh.
cleanup_exec = "/opt/libvirt-driver/cleanup.sh" # Path to a bash script to delete VM and disks.
As you can see here, we need some scripts to be placed under /opt/libvirt-driver path:
[root@runner libvirt-driver]# ls -l
total 16
-rwxr-xr-x 1 root root 437 Jan 15 14:42 base.sh
-rwxr-xr-x 1 root root 363 Jan 15 13:58 cleanup.sh
-rwxr-xr-x 1 root root 1561 Jan 15 15:20 prepare.sh
-rwxr-xr-x 1 root root 451 Jan 15 13:58 run.sh
Let me also provide you the content of those scripts below:
#!/usr/bin/env bash
# /opt/libvirt-driver/run.sh
currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base script.
VM_IP=$(_get_vm_ip)
ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no gitlab-runner@"$VM_IP" /bin/bash < "${1}"
if [ $? -ne 0 ]; then
# Exit using the variable, to make the build as failure in GitLab
# CI.
exit "$BUILD_FAILURE_EXIT_CODE"
fi
Then we will need to restart the service so the new configuration can be loaded
$ systemctl restart gitlab-runner
How to use custom Gitlab runner
Now you should be able to see your custom runner registered under Gitlab Project --> Settings --> CI / CD
You can write a simple gitlab-ci.yml
and push it to test your build:
stages:
- test
build:
stage: test
script:
- /usr/sbin/ip addr show
- /usr/bin/sudo yum install wget -y
- /usr/bin/cat /etc/*release
- /usr/bin/hostname
- /usr/bin/date
- /usr/bin/pwd
tags:
- custom
Last updated
Was this helpful?