2021-03-04 10:10:00
I know, I know: the past weeks it's been nothing but Gitlab over here :D That's going to quiet down now. How did all of that get started though?
Back in January, I posted the following question on the BHIS Discord:
"When it comes to CICD, microservices and the whole modern API reality I'm quite out of my depth. I never was a developer, can't code my way out of a wet paper bag; was always on the sysadmin and secops side.
Are you guys aware of any trainings or bootcamps that are squarely aimed at grabbing my demography (sysadmin, secops) by the scruff of their neck and dumping them through the whole process of building a sample API, automated building and testing and then ramming it onto something like Azure of CloudFoundry?
I've been on the sidelines of plenty CICD, helping DevOps teams with their Linux and security troubles... but now I really need to know what they do all day.
Anything commercial, that lasts multiple days and is from a reputable vendor would be absolutely great. I don't care too much about which solutions are used in said training. Key words may include: Spring.boot, Maven, Git, Azure DevOps, Github Actions, Fortify. Just an all-in-one "journey" would be lovely."
I asked around with friends and colleagues. Most folks weren't aware of any such trainings, though one pointed me at Kode Kloud, another suggested Dev Champs and two of them suggested Practical DevSecOps.
PDSO's CDP course, Certified DevSecOps Professional, listed selling points that matched what I wanted:
Having now completed the whole course and having passed the exam, here's my impressions about PDSO's CDP course:
My overall verdict, was the CDP course worth it? Yes, it was. I learned a lot, I got to mess around with a lot of cool tools and the exam was challenging.
One tip that I'd give students is to also run a CI/CD environment of their own, with more projects than the one or two in the labs. I have gained so much extra knowledge from running Gitlab in my homelab, with 6-7 vulnerable apps! It's been awesome and educational.
A few of my fellow students asked for pointers on the exam. I wouldn't want to give anything away that's covered by the NDAs, but I can tell you this much:
Basically, be ready to do high-paced learning and studying on-the-fly. In that regards, this exam isn't too different from the OSCP pen-testing exam: the concepts are the same, but you will need to do research on the job :)
Most importantly:
kilala.nl tags: work, studies,
View or add comments (curr. 0)
2021-02-28 20:44:00
Let me tell you! When you're 11.5 hours into a 12 hour exam, this is NOT a screen you want to see on your main Gitlab that holds all your exam code. ( O_o)
Thank ${Deity} I cloned it all to my local system.
To clarify that a little bit: the CDP exam I took today is a practical exam where you spend twelve hours hacking, testing and building code that manages an application infrastructure. The whole exam, like the labs during class, are "in the cloud" run by Practical DevSecOps.
Around 1700, while trying to deploy a Docker container or two, my Gitlab runner became unresponsive and my Docker daemon died. Then the app webserver died. And then other students started piping up in chat that their labs were stuck.
Finally, around 1730 my Gitlab server (which holds all my exam code) was reprovisioned. That is: erased, rebuilt, re-installed. My work for the past eleven hours was gone.
So as I said: thank ${Deity} I had cloned my git repositores to my local machine.
View or add comments (curr. 0)
2021-02-26 10:55:00
Just some quick notes I've been making on how to quickly get gitlab-runner up on a Linux box. I still feel very yucky about curl-in a file into sudo bash, so I'll probs grab the file locally instead and make sure it doesn't do anything nasty.
The following example was used on my Ansible host, to install gitlab-runner and to have it run as the local "ansible" user account instead of root. It registers and starts two runners.
kilala.nl tags: work, studies,
View or add comments (curr. 0)
2021-02-20 17:13:00
Heh, it's a bit ironic, no? Six weeks ago I wondered whether I was over-doing it, with work and my studies. I'd just finished a few course and two exams and was about to start with a new client.
Not two weeks later I've taken another two classes and I'm about to take another exam. A twelve hour, practical exam followed by documentation and reporting.
I've promised myself that, once I'm done with the exam, I'll spend a few weeks on nothing but gaming! Genshin Impact here I come! :)
EDIT:
Ah. I just realized: I start teaching class again in 6-8 weeks. That'll require prep-time too :D
View or add comments (curr. 0)
2021-02-20 16:10:00
After finishing the awesome BHIS "Modern Webapp Pen-testing" class (January), I immediately rolled into the "Certified DevSecOps Professional" course. I am lacking in experience with CI/CD, while having to support DevOps engineers every day.
The CDP labs by Practical SecDevOps are okay, but only testing Django.NV got stale.
What better way to learn about SAST, DAST, SCA and more than by running our beloved Juice Shop webapp through my own CI/CD pipeline?! :D
Not only does this give me a private Juice Shop in a safe environment (my homelab), but it got me more familiar with Gitlab and all the things that come with DevSecOps / SecDevOps / Security in DevOps / however you wanna call it.
The image above shows the Juice Shop project in my Gitlab, with its security testing and deployment stages. The last "Compliance" stage (with Inspec) didn't fit into the pic.
Running the pipeline builds a Docker image for Juice Shop, runs SAST, SCA, secret scanning and linters, then runs the Docker image on my testbox and runs Nikto, ZAP and SSLyze against it as DAST. All very much default/basic, but it's a start!
kilala.nl tags: work, studies,
View or add comments (curr. 0)
2021-02-19 07:42:00
I've been mentioning Gitlab for a while now and you might wonder why the sudden change. :D I'm working my way through the CDP training from Practical DevSecOps.
I needed a crash course that took me through a practical example of CI/CD pipelines, from A to Z, in a hurry. I'm in security and I need to advise DevOps engineers who work on those pipelines every day. I found it harder and harder to relate to them without having gone through their journey myself. Intellectually I understood most of the concepts, but everything stayed very vauge without me actually doing it hands-on.
So far the course is a resounding "okay". It's not wonderful, it's not bad, it's just that: pretty good. The slide decks are decent, the trainer narrating the videos has a nice voice, but the narration is quite literally reading from the text book. Some of the text on slides and in the labs was lifted directly from third party sources such as projects' Github pages or from articles like Annie Hedgpeth's series on running Inspec.
They have a huge amount of online labs, which is good, even if they get repetitive. So what I've done is setup Gitlab in my homelab as well, and apply all the things the course teaches me to multiple intentionally-vulnerable web apps.
So I've got Git repos for Juice Shop (Node.JS and Angular), django.nv (Python and JS), Webgoat (Java), GoVWA (Go) and others, which I'm treating like they were projects for my simulated company. Each of these gets its own CI/CD pipeline to run code quality checks, SAST, DAST and automated build + deploy through Docker.
It's been one heck of a learning experience and I'm looking forward to the closing exam, which is another 24h practical exam. I love those!
kilala.nl tags: work, studies,
View or add comments (curr. 0)
2021-02-17 20:30:00
When using a "shell" executor with gitlab-runner you may run into the following errors, when trying to upload artifacts to Gitlab.
ERROR: Uploading artifacts as "archive" to coordinator... error error=couldn't execute POST against https://gitlab.corp.broehaha.nl/api/v4/jobs/847/artifacts?artifact_format=zip&artifact_type=archive: Post https://gitlab.corp.broehaha.nl/api/v4/jobs/847/artifacts?artifact_format=zip&artifact_type=archive: proxyconnect tcp: tls: first record does not look like a TLS handshake
The issue here is that your "gitlab-runner" user account has picked up a http proxy configuration that's not sitting well with it.
In my homelab, the proxy settings are configured for all users using Ansible, through "/etc/profile". For the "gitlab-runner" user that apparently may be problematic when trying to talk to the internal Gitlab server. Quick and dirty work-around: unset the proxy settings from your environment.
echo "unset http_proxy; unset https_proxy" >> ~/.bashrc echo "unset http_proxy; unset https_proxy" >> ~/.profile
kilala.nl tags: work, sysadmin,
View or add comments (curr. 0)
2021-02-17 19:35:00
As part of the CDP course we're running unattended ZAP scans as part of integration testing, using the "owasp/zap2docker-stable" Docker container. The course materials tell you to run the CI/CD task using "docker:dind", a Docker-in-Docker solution. For some reason my Docker boxen aren't a fan of that; I'll have to debug that later.
Trying to run the ZAP container with a simple "shell" executor through gitlab-runner led to some fun challenges though! The course material suggests the following Docker run command:
docker run --user $(id -u):$(id -g) -w /zap -v $(pwd):/zap/wrk:rw --rm owasp/zap2docker-stable zap-baseline.py -t https://target:port -J zap-output.json
To sum it up: start the ZAP container, run the ZAP baseline script using your current UID and GID, mount your local directory as /zap/wrk and then write the results as a JSON file onto the mounted local directory.
This approach fails in two ways if you're not doing the fastest, dirty approach: running as the "root" user account.
Either you use it with "--user $(id -u):$(id -g)" and then you get the error message "Failed to start ZAP :(". Or you run it without that setting, then ZAP runs but it cannot save the output file, with a "permission denied: /zap/wrk/zap-output.json" message.
The issue here is that container has a very limited setup of users (as it should) and your uid+gid are most likely not in there. Under normal conditions, the ZAP scripts inside the container run as "zap:1000:1000" but that user doesn't have write access to your user's directory on the Docker host.
So... If you're running the ZAP container directly on your host and not as DinD, then you'll need to setup a temporary directory and setup write access for either uid:1000 or gid:1000 to it. The latter feels "better" to me. Then we'll end up with this (assuming Gitlab):
zap-baseline:
stage: integration
dependencies: []
allow_failure: true
tags:
- shell
before_script:
- docker pull owasp/zap2docker-stable
- mkdir output; chgrp -f 1000 output; chmod 770 output; cd output
script:
- docker run --rm -v $(pwd):/zap/wrk owasp/zap2docker-stable zap-baseline.py -t http://target:port -J zap-output.json
artifacts:
paths: [output/zap-output.json]
when: always
kilala.nl tags: work, studies,
View or add comments (curr. 0)
All content, with exception of "borrowed" blogpost images, or unless otherwise indicated, is copyright of Tess Sluijter. The character Kilala the cat-demon is copyright of Rumiko Takahashi and used here without permission.