GitLab Shared Runner

HPC PL12146
1

Hi there!

I was trying to setup a CI/CD Pipeline in GitLab but I keep getting the message “This GitLab instance does not provide any shared Runners yet. Instance administrators can register shared Runners in the admin area.”

I guess that since gitlab.unige.ch is a self-managed GitLab instance it doesn’t come with any pre-installed/configured runners.

Would it be possible to have a shared runner installed on it?

Thanks in advance!

Ashley

2 Likes
2

I also think that having some shared runner would be nice ( This was one of my suggestion for the new cluster), however in the meantime, it is possible to register yourself a runner for a given project or group. For this you only need to configure the ssh access to a machine ( for example the login node ). Note also that you should not launch heavy job on the login node and use the appropriate srun or sbatch command in your pipeline.

1 Like
3

Hi there,

FYI, do you know that there is a specific GitLab category in the Research Data Management forum (cf. https://rdm-forum.unige.ch/c/devops-gitlab/25 )?

In the past, we decided to not provide such functionality given that. as @Pablo.Strasser noticed (cf. GitLab Shared Runner - #2 by Pablo.Strasser ), it is still possible to register your own self-managed runner (on any platform, Windows included).

I started again the discussion within the DiSTIC about adding such functionality to our GitLab instance, I will come back ASAP.

Thx, bye,
Luca

4

You’re totally right. Frankly, I have never thought about configure the runner on the login node. I will try that out and let you know.

Thanks for the suggestion.

5

Hi Luca,

No, I didn’t. Thanks for the information.

It’s a pity we cannot benefit from such functionality, imho. Thanks for discussing the addition within the DiSTIC.

6

Dear @Ashley.Caselli, dear @Pablo.Strasser here is a clarification. You aren’t supposed to run a GitLab runner on the login node and we may kill it as this is not part of the normal usage of the login node.

You may use a compute node as runner if really needed, but this isn’t well appropriate, because we don’t want that you reserve unused resources for this purpose. Ideally you should have a dedicated machine (a server, a lab machine, a virtual machine) dedicated to this purpose.

We should as @Pablo.Strasser said maybe have a shared runner running on Yggdrasil if there is a reason to have that on an HPC facility.

Best

7

Hi

just to follow-up on the discussion about this we had at today’s HPC Lunch with @Yann.Sagon.

First, to clarify couple of things.
Indeed, a gitlab runner is not doing anything productive most of it’s life time, it just listens for jobs, so it makes little sense to run it as an HPC node, hoarding a process for nothing. And long-running process may be difficult to keep alive by a user.

However, once the runner receives a job, it executes it in more HPC-like fashion, with what they call an “executor”.
There are many executors, they usually just need to support capacity to run docker containers.
One could even imagine gitlab runner with an executor running jobs on slurm, but an HPC cluster generally does not support docker (for a good reason). It might be possible to do some conversion to singularity, but I do not think it exists yet, it’s not so trivial.
Interestingly, what @Pablo.Strasser suggests, running srun within gitlab CI, is also an option which results being similar to gitlab executor, but outside gitlab runner’s own approach of how to achieve this. If I understand correctly.

So, inevitably, gitlab runner needs to run as a service, and it usually needs to execute on something supporting docker.

For these reasons we use a kubernetes executor, on a dedicated cluster (currently old hardware). Gitlab runners are also used to deploy data analysis services (like this one) and various supporting services (like jupyterlab, monitoring platform, etc) on the same kubernetes cluster.

To upgrade and expand this cluster we have recently secured funding to build a kubernetes cluster this time managed by D-STIC (with @Yannael.Girerd).
At the time heavy load it will be used by us completely, but a lot of the time there will be no CI jobs and services will not be overloaded.

So a I had a thought that maybe we can share it with other users it in a similar fashion to how private HPC nodes are shared, with lower priority. Kubernetes, especially in the way it will be deployed in this case, has quite advanced resource management (more tuned to long-running tasks like gitlab runners), and it should be possible to setup such a design.

Whether this will be an interesting solution, is a matter of discussion.

Perhaps even, eventually, if other users also want dedicated kubernetes resources, the can contribute to expand it, with similar approach to how it is done with HPC, but for different kind of workloads, suitable for kubernetes.

Cheers

Volodymyr

8

Hi,

My suggestion on using sbatch/srun inside a runner was to use the basic shell executor which allows to run shell script from Gitlab. The use case I had in mind are simple preprocessing and postprocessing job like make or building singularity image. And use sbatch or srun for the rest. From a user point of view the script from the runner would behave as if run directly on baobab.

I was considering that the rule for using srun/sbatch or not should be the same than for a script run in the cluster ie long / heavy job should use srun/sbatch and only simple preprocessing job don’t need to.

Note however that this approach limit the capability of the runner but to me seem to integrate well into baobab.

It is clear that a Kubernetes runner has a lot more capability than my proposition and could interest some user for deploying analytics for websites. I personally run a Kubernetes cluster and runner on my “home” cluster.

I like the idea of a shared kubernetes cluster however note that having the power to deploy pods (ie write yaml file) allows to get full access to the node where the pods is deployed in. This is done similarly that in the docker case by mounting / into a container run as root.

Application run on kubernetes are safe of this flaw, but user having access to write their own manifest have way to access the whole node. I do hope that the user will be mature enough so that this do not create a road block.

Have a nice day.

Pablo Strasser

9

Hi

your suggestion of using calling slurm inside a runner is indeed an interesting one.

And it’s especially easy to debug, as you say. Though in a way, this makes runner rely on potentially non-repeatable environment which is established by the baobab user, which makes the project more difficult to migrate, scale, or ship, if necessary.

Also I am a bit concerned that it may be a bit much effort for many users. But perhaps it is possible to develop some common scripts to facilitate this.

In addition, it may be hard to maintain live runner, indeed you can not be sure it will be forever active on the login node.
It can be of course just systemd somewhere on a private node.
Or perhaps, since k8s is great just for that - keeping things live and well - your suggestion can be used together with k8s, shared gitlab runner would live in k8s and execute on baobab. It could be interesting to try, not sure.
Or you had in mind yet another option to reliably run long-running runner?

Concerning security implications, it indeed can become tricky, and also it evolved between different versions of k8s and gitlab, but I had in mind a pretty limited case, when kubernetes executor of gitlab runner does not give user permissions to create arbitrary objects in kubernetes. It can be restricted to just run CI from .gitlab-ci yaml, with very restrictive service account (or even none).
I can not say I have the complete knowledge of k8s, but I tried and it appears to be possible to make runner execute CI jobs with service account which has no permissions. It should be also possible to avoid mounting it altogether.

And in different cases, when the user is allowed to access the API, it can be still severely limited, preventing gaining much privileges (e.g. restricting hostPath volumes, analog of docker volume mounting).

Still, since this is indeed delicate, possibility to create own applications (with deployments, ingresses, pvs, etc) is to be reserved to separate, specialized runners, restricted to groups with more experienced and trusted users, with larger rights but still limited to their namespaces with suitable resource limits.

So it would seem that it should not mean that any user can do when they want with the whole node or cluster, unless they are given this access?..

By the way, the physical cluster will run actually vmware and virtual sub-clusters, so it’s possible to isolate cases further. Though this may make balancing sometimes difficult.

Cheers

Volodymyr

10

Hi,

I agree that gitlab runner on kubernetes should be safe. My security answer was more for the general idea of shared kubernetes access to make sure security is taken into account. I personally would completely support a kubernetes service offered by the university.

On the runner subject I mainly see two use cases for a runner for two differnts type of user:

  1. User needing the whole gitlab runner feature for creating images and others. For this case, a kubernetes or docker executor is better as it allows better repeatability and shipping.
  2. User mainly doing some simple scripting for baobab, compiling some file and other before running code natively without singularity. For this case an ssh or shell executor would be enough. This offer to the user the same environment than baobab at the cost of repeatability.

For the second approach an executor should be provided with systemd or other.

I hope this clarify a bit my view on that.

Pablo Strasser

11

Hi

I agree!
And as I said, the second option seems definitely interesting!
I am just wondering if it is something to provide to users (e.g. I provide to users of our group, or maybe D-STIC provides a shared runner) then a good way to maintain a common runner like this would be still in a kubernetes cluster.
Since managing long-running processes is basically what kubernetes is made for.
This would avoid making everybody maintain own systemd on some personal persistent server, not everybody has such a thing, many people just use laptops.

edit: but yeah, for individual somewhat advanced user without own kubernetes cluster and before shared or group runners are provided, your suggestion seems to be indeed the best!

Cheers

Volodymyr