2021-02-06 19:35:00
As part of my studying for the CDP course, I've expanded my homelab with a private instance of Gitlab. I've got to say: I like it! A lot. It's good software!
To accomodate my builds I expanded the RAM on my Docker host VM and set up three "gitlab-runners" to pick up jobs from Gitlab CICD pipelines. Microsoft's documentation is outstanding: the runners were installed and configured within minutes.
The only thing I really disliked was their instructions to "wget https://some-url | bash -". That always feels so fscking scary.
As part of my change management process the Docker host of course needed a reboot, to see if things some up correctly. They did and the "gitlab-runner" process was there as well. But it wasn't picking up any jobs! Only when I SSHd into the host and ran "sudo gitlab-runner run" would jobs start flowing.
At first I thought I just didn't understand the concept of the runner process well enough. Maybe I hadn't set them up correctly? Then I decided to do the logical thing: check the logs. I've been teaching my students to do so, so why didn't I? :D
"sudo systemctl status gitlab-runner -l" showed me the following:
$ sudo systemctl status gitlab-runner -l
â— gitlab-runner.service - GitLab Runner
Loaded: loaded
Active: active
...
Feb 06 19:24:37 gitlab-runner[20361]: WARNING: Checking for jobs... failed
runner=REDACTED status=couldn't execute POST against https://REDACTED/api/v4/jobs/request:
Post https://REDACTED/api/v4/jobs/request: x509: certificate signed by unknown authority
The self-signed cert isn't too surprising, since I still have a backlog item to get that fixed. I wanted to first get the basics right before getting a proper cert from my PKI. But I thought I had dealt with that by registering the runner with a CA cert override.
Checking "/etc/gitlab-runner/config.toml" showed me where I had gone wrong: the CA cert override path was relative, not exact.
[[runners]]
name = "REDACTED"
url = "https://REDACTED"
token = "REDACTED"
tls-ca-file = "./gitlab.pem"
executor = "docker"
I had assumed that the cert would be picked up by the runner config and stored elsewhere, instead of being referenced from the file system. Wrong! I made sure to copy the self-signed cert to "/etc/gitlab-runner/gitlab.pem" after which I corrected the "config.toml" file to use the correct path.
One quick restart of the runner service and now jobs are automatically picked up!
kilala.nl tags: work, studies, homelab,
View or add comments (curr. 3)
Posted by Cailin Coilleach
Just disocovered that we shouldn't forget to add the "gitlab-runner" user account on Linux into the "docker" group locally. Otherwise you can't use Docker commands through a "shell" executor.
Posted by Tess
Two very useful articles I found while working with the gitlab-runners and both the shell and docker executors:
All content, with exception of "borrowed" blogpost images, or unless otherwise indicated, is copyright of Tess Sluijter. The character Kilala the cat-demon is copyright of Rumiko Takahashi and used here without permission.
2021-02-06 22:40:00
Posted by Tess
I was also confused by the fact that I have >3 runners on that one host, but it looks like it was only running one job at a time. Very odd and a bit frustrating if you have a lot of smaller jobs to run.
Thanks to this Gitlab forum post I have learned about the concurrent and limit fields that can go into "config.toml". Basically: set each runner to limit 1 (or 2) and set concurrent to something larger than 1 (make assumptions on your capacity!). Turns out that my default config file has concurrent set to 1.