Building Images using GitLab CI¶

Continuous Integration (CI) is a technique whereby software is tested frequently (i.e. continuously) in order to catch any potential issues early on. In CI, tests are run after every commit to the codebase.

GitLab at CERN supports CI out of the box and it is very easy to set up by just adding a single file to your repsository.

Here, we cover CI briefly in the context of RECAST. More information regarding CI can be found in the ATLAS Software tutorial talk on GitLab:

Your first CI Job¶

Add the following to a file named .gitlab-ci.yml:

variables:
  GIT_SUBMODULE_STRATEGY: recursive

stages:
  - build

build_job:
    stage: build
    image: gitlab-registry.cern.ch/atlas/athena/analysisbase:21.2.247
    script:
    - echo Hello World

The basic structure of a .gitlab-ci.yml is a set of stages and jobs, where each job belongs to a given stage.

In this simple example, a single job, named build_job, belongs to a single stage named build and every time the CI runs (i.e. every time you push a commit to GitLab via git push) the job will simply print Hello World.

Checking CI Jobs¶

The GitLab user interface allows you to check on the status of CI Jobs.

Building a Docker Image in CI¶

CERN supports building Docker images on CI. Let's take a minimal Dockerfile and build it in GitLab

Dockerfile:

FROM gitlab-registry.cern.ch/atlas/athena/analysisbase:21.2.247
COPY --chown=atlas . /code

This Dockerfile just adds the full code of the repo (denoted by the relative path . of the repository root) on top of the gitlab-registry.cern.ch/atlas/athena/analysisbase:21.2.247. Note that in this example we do not compile any code. You will need to add this as necessary.

We now prepare a .gitlab-ci.yml file which gives GitLab instructions on what to do each time someone pushes commits to GitLab. In this case we want it to build a Docker image based on the Dockerfile we just wrote.

.gitlab-ci.yml:

variables:
  GIT_SUBMODULE_STRATEGY: recursive

stages:
  - package

build_image:
  stage: package
  image:
    name: gitlab-registry.cern.ch/ci-tools/docker-image-builder
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context "${CI_PROJECT_DIR}"
                       --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
                       --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_NAME}-${CI_COMMIT_SHORT_SHA}"

Recommended Tag Structure¶

You will notice the --destination argument in the .gitlab-ci.yml script above. This controls the name of the Docker image that is produced in the CI step. Here, the image name will be <reponame>:<branchname>-<commit id>. This way images built from different branches or commits do not overwrite each other.

Container Registry¶

When the Docker image build succeeds the image is pushed to the GitLab container registry hosted at gitlab-registry.cern.ch.

Authentication¶

The images in the container registry inherit the access controls of your project. That is, if your project is private the images are as well and you will need to authenticate to the registry in order to be able to pull the images.

Docker¶

Using docker you can use docker login to authenticate to a registry.

docker login gitlab-registry.cern.ch
docker run --rm -it gitlab-registry.cern.ch/<your-project>/<your-image>:<your-tag> bash

Apptainer / Singularity¶

When using apptainer (formerly singularity) you will still need to authenticate against private container registries

apptainer remote login --username "${USER}" docker://gitlab-registry.cern.ch
apptainer exec -C docker://gitlab-registry.cern.ch/<your-project>/<your-image>:<your-tag> bash

To be able to use apptainer with recast-atlas (e.g. on lxplus9) you will additionally need to set the packtivity runtime information in your environment before running recast

export PACKTIVITY_CONTAINER_RUNTIME=singularity

Last update: November 9, 2023