Azure Infrastructure Automation with Gitlab CI

Introduction

Creating new resources in Azure is very easy thanks to Azure Portal, but what if you want to create an infrastructure with code. One of the options is to use ARM templates for infrastructure deployment. If you want to implement a proper CI/CD against your ARM templates, Gitlab CI can be a very strong candidate. Let's assume if you have a pipeline to create the infrastructure and delete it upon a successful deployment in order to save cost. Wouldn't it be perfect? If you say "yes", let's go deeper and see how I implemented the solution.

Setting up Gitlab Project

I am not going into the details of how you can create a Gitlab project as it is done by simply clicking a few buttons, but the most important part of the automation is to create variables which we will use in Gitlab shared runners to authenticate to Azure Cloud.

You can add variables by clicking "Settings" and then "CI/CD" in your project. The "Variables" blade should be visible and you can introduce such variables:

Do not forget to mark your variables as "Masked". Otherwise they will be visible in the Gitlab CI pipeline and they contain sensitive information about your Azure subscription

If you wish to mark your variables protected, please remember that they can be only accessed from protected branches

VARIABLE NAME	DESCRIPTION
SP_ID	Azure Service Principal account to run the deployment
SP_SECRET	Secret of the service principal account
SUBSCRIPTION_ID	Azure subscription where resources will be created
TENANT_ID	Tenant ID of your Azure subscription

Setting up .gitlab-ci.yaml

Now it is time to set up your pipeline to login to the Azure service principal account and deploy resources defined in your subscription. I am using Gitlab shared runners and "mcr.microsoft.com/azure-cli" container image to run the pipeline.

One of the important things is to use a dedicated resource group for the deployment tests so we can destroy whatever is inside that resource group to save costs.

variables:
  DEFAULT_RG:
    description: "Default resource group to deploy the resources for testing"
    value: "test-rg"
  DEFAULT_LOCATION:
    description: "Default location of the testing resource group"
    value: "northeurope"
  
default:
  image: mcr.microsoft.com/azure-cli
  before_script:
    - az login --service-principal --username $SP_ID --password $SP_SECRET --tenant $TENANT_ID
    - az account set --subscription $SUBSCRIPTION_ID
    - set -euo pipefail

set -euo pipefail will prevent pipeline to run further steps if one of the variables is empty

The remaining of the pipeline can change based on what you want to test but let me paste you the full content here so you can see what I am testing under that small scope. The respective Gitlab project can also be found here.

variables:
  DEFAULT_RG:
    description: "Default resource group to deploy the resources for testing"
    value: "test-rg"
  DEFAULT_LOCATION:
    description: "Default location of the testing resource group"
    value: "northeurope"
  
default:
  image: mcr.microsoft.com/azure-cli
  before_script:
    - az login --service-principal --username $SP_ID --password $SP_SECRET --tenant $TENANT_ID
    - az account set --subscription $SUBSCRIPTION_ID
    - set -euo pipefail

stages:
  - validate
  - deploy
  - destroy

validate virtual network:
  stage: validate
  script:
    - az deployment group validate -g ${DEFAULT_RG} -f virtual-network/virtual-network.json -p virtual-network/virtual-network.parameters.json
  
validate virtual machine:
  stage: validate
  script:
    - az deployment group validate -g ${DEFAULT_RG} -f virtual-machine/linux-vm.json -p virtual-machine/linux-vm.parameters.json -p adminPassword=$VM_INITIAL_PASSWORD

deploy virtual network:
  stage: deploy
  script:
    - az deployment group create -g ${DEFAULT_RG} -f virtual-network/virtual-network.json -p virtual-network/virtual-network.parameters.json

deploy virtual machine:
  stage: deploy
  script:
    - az deployment group create -g ${DEFAULT_RG} -f virtual-machine/linux-vm.json -p virtual-machine/linux-vm.parameters.json -p adminPassword=$VM_INITIAL_PASSWORD
  needs:
    - job: "deploy virtual network"

destroy test environment:
  stage: destroy
  script:
    - az deployment group create -g ${DEFAULT_RG} -f resource-group/empty-resource-group.json --mode "Complete"

This pipeline will run validation of the resource to be created under the test resource group and will deploy them. After the successful deployment, it will use a very good trick to delete all resources in the resource group by doing a complete deployment with an empty Azure resource.

Conclusion

The above approach can let you test your ARM templates with Gitlab CI pipeline at a minimal cost. There are various ways of testing Azure deployment including Azure DevOps pipeline but I wanted to test it under Gitlab CI.