Gitlab-runner not picking up jobs after reboot

2021-02-06 19:35:00

As part of my studying for the CDP course, I've expanded my homelab with a private instance of Gitlab. I've got to say: I like it! A lot. It's good software! 

To accomodate my builds I expanded the RAM on my Docker host VM and set up three "gitlab-runners" to pick up jobs from Gitlab CICD pipelines. Microsoft's documentation is outstanding: the runners were installed and configured within minutes.

The only thing I really disliked was their instructions to "wget https://some-url | bash -". That always feels so fscking scary. 

As part of my change management process the Docker host of course needed a reboot, to see if things some up correctly. They did and the "gitlab-runner" process was there as well. But it wasn't picking up any jobs! Only when I SSHd into the host and ran "sudo gitlab-runner run" would jobs start flowing. 

At first I thought I just didn't understand the concept of the runner process well enough. Maybe I hadn't set them up correctly? Then I decided to do the logical thing: check the logs. I've been teaching my students to do so, so why didn't I? :D

"sudo systemctl status gitlab-runner -l" showed me the following:

$ sudo systemctl status gitlab-runner -l
‚óŹ gitlab-runner.service - GitLab Runner
   Loaded: loaded
   Active: active

...
Feb 06 19:24:37 gitlab-runner[20361]: WARNING: Checking for jobs... failed
runner=REDACTED status=couldn't execute POST against https://REDACTED/api/v4/jobs/request: 
Post https://REDACTED/api/v4/jobs/request: x509: certificate signed by unknown authority

The self-signed cert isn't too surprising, since I still have a backlog item to get that fixed. I wanted to first get the basics right before getting a proper cert from my PKI. But I thought I had dealt with that by registering the runner with a CA cert override. 

Checking "/etc/gitlab-runner/config.toml" showed me where I had gone wrong: the CA cert override path was relative, not exact.

[[runners]]
  name = "REDACTED"
  url = "https://REDACTED"
  token = "REDACTED"
  tls-ca-file = "./gitlab.pem"
  executor = "docker"

I had assumed that the cert would be picked up by the runner config and stored elsewhere, instead of being referenced from the file system. Wrong! I made sure to copy the self-signed cert to "/etc/gitlab-runner/gitlab.pem" after which I corrected the "config.toml" file to use the correct path. 

One quick restart of the runner service and now jobs are automatically picked up!


kilala.nl tags: , , ,

View or add comments (curr. 3)