Kilala.nl - Personal website of Tess Sluijter

Unimportant background
Login
  RSS feed

About me

Blog archives

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

> Weblog

> Sysadmin articles

> Maths teaching

Finished a lot of hard work: the CDP exam, Certified DevSecOps Professional

2021-03-04 10:10:00

I know, I know: the past weeks it's been nothing but Gitlab over here :D That's going to quiet down now. How did all of that get started though?

Back in January, I posted the following question on the BHIS Discord:

"When it comes to CICD, microservices and the whole modern API reality I'm quite out of my depth. I never was a developer, can't code my way out of a wet paper bag; was always on the sysadmin and secops side. 

Are you guys aware of any trainings or bootcamps that are squarely aimed at grabbing my demography (sysadmin, secops) by the scruff of their neck and dumping them through the whole process of building a sample API, automated building and testing and then ramming it onto something like Azure of CloudFoundry? 

I've been on the sidelines of plenty CICD, helping DevOps teams with their Linux and security troubles... but now I really need to know what they do all day.

Anything commercial, that lasts multiple days and is from a reputable vendor would be absolutely great. I don't care too much about which solutions are used in said training. Key words may include: Spring.boot, Maven, Git, Azure DevOps, Github Actions, Fortify. Just an all-in-one "journey" would be lovely."

I asked around with friends and colleagues. Most folks weren't aware of any such trainings, though one pointed me at Kode Kloud, another suggested Dev Champs and two of them suggested Practical DevSecOps.

PDSO's CDP course, Certified DevSecOps Professional, listed selling points that matched what I wanted:

Having now completed the whole course and having passed the exam, here's my impressions about PDSO's CDP course:

My overall verdict, was the CDP course worth it? Yes, it was. I learned a lot, I got to mess around with a lot of cool tools and the exam was challenging.

One tip that I'd give students is to also run a CI/CD environment of their own, with more projects than the one or two in the labs. I have gained so much extra knowledge from running Gitlab in my homelab, with 6-7 vulnerable apps! It's been awesome and educational. 

A few of my fellow students asked for pointers on the exam. I wouldn't want to give anything away that's covered by the NDAs, but I can tell you this much:

Basically, be ready to do high-paced learning and studying on-the-fly. In that regards, this exam isn't too different from the OSCP pen-testing exam: the concepts are the same, but you will need to do research on the job :)

Most importantly:

  1. As John Strand always says: "Document as you go!" Take notes all the way through your work, don't put that off until the end.
  2. Clone your exam repository to your local computer and pull updates regularly! I lost 11 hours of work on my exam, because my Gitlab got reprovisioned.

kilala.nl tags: , ,

View or add comments (curr. 0)

Practical DevSecOps CDP exam: heart attack moment

2021-02-28 20:44:00

An erase git repository

Let me tell you! When you're 11.5 hours into a 12 hour exam, this is NOT a screen you want to see on your main Gitlab that holds all your exam code. ( O_o)

Thank ${Deity} I cloned it all to my local system.

To clarify that a little bit: the CDP exam I took today is a practical exam where you spend twelve hours hacking, testing and building code that manages an application infrastructure. The whole exam, like the labs during class, are "in the cloud" run by Practical DevSecOps

Around 1700, while trying to deploy a Docker container or two, my Gitlab runner became unresponsive and my Docker daemon died. Then the app webserver died. And then other students started piping up in chat that their labs were stuck.

Finally, around 1730 my Gitlab server (which holds all my exam code) was reprovisioned. That is: erased, rebuilt, re-installed. My work for the past eleven hours was gone. 

So as I said: thank ${Deity} I had cloned my git repositores to my local machine. 


kilala.nl tags: , ,

View or add comments (curr. 0)

Quick notes: script to setup Gitlab runners and run as Ansible

2021-02-26 10:55:00

Just some quick notes I've been making on how to quickly get gitlab-runner up on a Linux box. I still feel very yucky about curl-in a file into sudo bash, so I'll probs grab the file locally instead and make sure it doesn't do anything nasty.

The following example was used on my Ansible host, to install gitlab-runner and to have it run as the local "ansible" user account instead of root. It registers and starts two runners.

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash
 
export GITLAB_RUNNER_DISABLE_SKEL=true; sudo -E yum install -y gitlab-runner
 
sudo gitlab-runner uninstall
 
sudo mkdir /etc/systemd/system/gitlab-runner.service.d/
cat > /tmp/exec_start.conf << EOF
 
[Service]
ExecStart=
ExecStart=/usr/bin/gitlab-runner "run" "--working-directory" "/home/ansible/gitlab" "--config" "/etc/gitlab-runner/config.toml" "--service" "gitlab-runner" "--user" "ansible"
EOF
 
sudo mv /tmp/exec_start.conf /etc/systemd/system/gitlab-runner.service.d/exec_start.conf
 
sudo systemctl daemon-reload
sudo systemctl enable gitlab-runner
sudo systemctl start gitlab-runner
 
sudo cp /tmp/broehaha-cachain.pem /etc/gitlab-runner/cachain.pem
 
read -p "gitlab reg token: " GITLAB_TOKEN
 
sudo gitlab-runner register --non-interactive
--tls-ca-file=/etc/gitlab-runner/cachain.pem
--tag-list ansible
--name ansible.corp.broehaha.nl
--registration-token ${GITLAB_TOKEN}
--url https://gitlab.corp.broehaha.nl
--executor shell
--locked=false
 
sudo gitlab-runner register --non-interactive
--tls-ca-file=/etc/gitlab-runner/cachain.pem
--tag-list ansible
--name ansible.corp.broehaha.nl
--registration-token ${GITLAB_TOKEN}
--url https://gitlab.corp.broehaha.nl
--executor shell
--locked=false

kilala.nl tags: , ,

View or add comments (curr. 0)

Over-doing it? Maybe... Almost time to chill a bit.

2021-02-20 17:13:00

Heh, it's a bit ironic, no? Six weeks ago I wondered whether I was over-doing it, with work and my studies. I'd just finished a few course and two exams and was about to start with a new client. 

Not two weeks later I've taken another two classes and I'm about to take another exam. A twelve hour, practical exam followed by documentation and reporting. 

I've promised myself that, once I'm done with the exam, I'll spend a few weeks on nothing but gaming! Genshin Impact here I come! :)

EDIT:

Ah. I just realized: I start teaching class again in 6-8 weeks. That'll require prep-time too :D


kilala.nl tags: , ,

View or add comments (curr. 0)

Security testing OWASP Juice Shop in Gitlab CI/CD

2021-02-20 16:10:00

Gitlab pipeline

After finishing the awesome BHIS "Modern Webapp Pen-testing" class (January), I immediately rolled into the "Certified DevSecOps Professional" course. I am lacking in experience with CI/CD, while having to support DevOps engineers every day.

The CDP labs by Practical SecDevOps are okay, but only testing Django.NV got stale.

What better way to learn about SAST, DAST, SCA and more than by running our beloved Juice Shop webapp through my own CI/CD pipeline?! :D 

Not only does this give me a private Juice Shop in a safe environment (my homelab), but it got me more familiar with Gitlab and all the things that come with DevSecOps / SecDevOps / Security in DevOps / however you wanna call it. 

The image above shows the Juice Shop project in my Gitlab, with its security testing and deployment stages. The last "Compliance" stage (with Inspec) didn't fit into the pic.

Running the pipeline builds a Docker image for Juice Shop, runs SAST, SCA, secret scanning and linters, then runs the Docker image on my testbox and runs Nikto, ZAP and SSLyze against it as DAST. All very much default/basic, but it's a start!


kilala.nl tags: , ,

View or add comments (curr. 0)

Practical DevSecOps and their CDP training

2021-02-19 07:42:00

I've been mentioning Gitlab for a while now and you might wonder why the sudden change. :D I'm working my way through the CDP training from Practical DevSecOps.

I needed a crash course that took me through a practical example of CI/CD pipelines, from A to Z, in a hurry. I'm in security and I need to advise DevOps engineers who work on those pipelines every day. I found it harder and harder to relate to them without having gone through their journey myself. Intellectually I understood most of the concepts, but everything stayed very vauge without me actually doing it hands-on.

So far the course is a resounding "okay". It's not wonderful, it's not bad, it's just that: pretty good. The slide decks are decent, the trainer narrating the videos has a nice voice, but the narration is quite literally reading from the text book. Some of the text on slides and in the labs was lifted directly from third party sources such as projects' Github pages or from articles like Annie Hedgpeth's series on running Inspec

They have a huge amount of online labs, which is good, even if they get repetitive. So what I've done is setup Gitlab in my homelab as well, and apply all the things the course teaches me to multiple intentionally-vulnerable web apps.

So I've got Git repos for Juice Shop (Node.JS and Angular), django.nv (Python and JS), Webgoat (Java), GoVWA (Go) and others, which I'm treating like they were projects for my simulated company. Each of these gets its own CI/CD pipeline to run code quality checks, SAST, DAST and automated build + deploy through Docker.

It's been one heck of a learning experience and I'm looking forward to the closing exam, which is another 24h practical exam. I love those!


kilala.nl tags: , ,

View or add comments (curr. 0)

Gitlab runner "shell" executor cannot upload artifacts

2021-02-17 20:30:00

When using a "shell" executor with gitlab-runner you may run into the following errors, when trying to upload artifacts to Gitlab.

ERROR: Uploading artifacts as "archive" to coordinator... error error=couldn't execute POST against https://gitlab.corp.broehaha.nl/api/v4/jobs/847/artifacts?artifact_format=zip&artifact_type=archive: Post https://gitlab.corp.broehaha.nl/api/v4/jobs/847/artifacts?artifact_format=zip&artifact_type=archive: proxyconnect tcp: tls: first record does not look like a TLS handshake

The issue here is that your "gitlab-runner" user account has picked up a http proxy configuration that's not sitting well with it.

In my homelab, the proxy settings are configured for all users using Ansible, through "/etc/profile". For the "gitlab-runner" user that apparently may be problematic when trying to talk to the internal Gitlab server. Quick and dirty work-around: unset the proxy settings from your environment.

echo "unset http_proxy; unset https_proxy" >> ~/.bashrc
echo "unset http_proxy; unset https_proxy" >> ~/.profile

kilala.nl tags: , ,

View or add comments (curr. 0)

Challenges running "owasp/zap2docker-stable" without docker:dind

2021-02-17 19:35:00

As part of the CDP course we're running unattended ZAP scans as part of integration testing, using the "owasp/zap2docker-stable" Docker container. The course materials tell you to run the CI/CD task using "docker:dind", a Docker-in-Docker solution. For some reason my Docker boxen aren't a fan of that; I'll have to debug that later.

Trying to run the ZAP container with a simple "shell" executor through gitlab-runner led to some fun challenges though! The course material suggests the following Docker run command:

docker run --user $(id -u):$(id -g) -w /zap -v $(pwd):/zap/wrk:rw --rm owasp/zap2docker-stable zap-baseline.py -t https://target:port -J zap-output.json

To sum it up: start the ZAP container, run the ZAP baseline script using your current UID and GID, mount your local directory as /zap/wrk and then write the results as a JSON file onto the mounted local directory.

This approach fails in two ways if you're not doing the fastest, dirty approach: running as the "root" user account.

Either you use it with "--user $(id -u):$(id -g)" and then you get the error message "Failed to start ZAP :(". Or you run it without that setting, then ZAP runs but it cannot save the output file, with a "permission denied: /zap/wrk/zap-output.json" message.

The issue here is that container has a very limited setup of users (as it should) and your uid+gid are most likely not in there. Under normal conditions, the ZAP scripts inside the container run as "zap:1000:1000" but that user doesn't have write access to your user's directory on the Docker host.

So... If you're running the ZAP container directly on your host and not as DinD, then you'll need to setup a temporary directory and setup write access for either uid:1000 or gid:1000 to it. The latter feels "better" to me. Then we'll end up with this (assuming Gitlab):

zap-baseline:
    stage: integration
    dependencies: []
    allow_failure: true
    tags:
        - shell
    before_script:
        - docker pull owasp/zap2docker-stable
        - mkdir output; chgrp -f 1000 output; chmod 770 output; cd output
    script: 
        - docker run --rm -v $(pwd):/zap/wrk owasp/zap2docker-stable zap-baseline.py -t http://target:port -J zap-output.json
    artifacts:
        paths: [output/zap-output.json]
        when: always 

kilala.nl tags: , ,

View or add comments (curr. 0)

Older blog posts