2025-10-16 12:41:00
In my last post I made a somewhat sloppy solution to do monthly rotation of my Lets Encrypt certificates for Mailcow. Turns out that acme.sh has a neat built-in solution for rotation!
If you make a weekly or monthly systemd timer, with the following service definition, it'll all work out nicely!
Contents of /usr/lib/systemd/system/mailcow-certificate.timer:
[Unit]
Description=Monthly Mailcow certificate renewal
[Timer]
OnCalendar=weekly
RandomizedDelaySec=60m
Persistent=true
[Install]
WantedBy=timers.target
Contents of /usr/lib/systemd/system/mailcow-certificate.service:
[Unit]
Description=Monthly Mailcow certificate renewal
After=network.target network-online.target systemd-networkd.service
[Service]
Type=oneshot
ExecStart=/bin/bash -c '/opt/acme.sh/acme.sh --cron --home "/root/.acme.sh"'
ExecStartPost=/bin/bash -c 'docker compose --file /opt/mailcow/docker-compose.yml restart'
KillMode=process
TimeoutStopSec=900
After that?
sudo systemctl enable mailcow-certificate.timer
sudo systemctl enable mailcow-certificate.service
sudo systemctl start mailcow-certificate.timer
kilala.nl tags: work, sysadmin,
View or add comments (curr. 0)
2025-10-07 19:48:00
A few weeks ago I set up a locally hosted Mailcow instance, with proper TLS certificates from Let's Encrypt. Good stuff.
At the time I'd put off automating certificate renewal, because I wasn't ready to deal with my DNS provider's API yet. Turns out that TransIP is dead simple to use with the official Acme.sh solution!
There's a ready-to-run implementation of Acme.sh for TransIP DNS. Here's the documentation for it.
All you need is to follow TransIP's documentation of opening up and securing the API for your account.
The script I'd showed before only needs a tiny adjustment to work:
#!/bin/bash
#
# Documentation:
# https://github.com/acmesh-official/acme.sh/wiki/dnsapi2#dns_transip
InstallDir="/opt/mailcow"
if [[ ($# -gt 0) && ($1 == "--force") ]]
then
Force="--force"
else
Force=""
fi
export TRANSIP_Username="myname"
export TRANSIP_Key_File="./api.pem"
acme.sh --set-default-ca --server https://acme-v02.api.letsencrypt.org/directory
acme.sh --issue --dns dns_transip --dnssleep 300 -d mail.${Domain} ${Force}
acme.sh --install-cert -d mail.broehaha.nl \
--cert-file ${InstallDir}/data/assets/ssl/nochain.pem \
--key-file ${InstallDir}/data/assets/ssl/key.pem \
--fullchain-file ${InstallDir}/data/assets/ssl/cert.pem
kilala.nl tags: work, sysadmin,
View or add comments (curr. 0)
2025-10-05 16:50:00
Last year I did my traditional comparison of incoming and outgoing exam objectives for the Linux+ exam. It's good for students to know the changes from XK0-005 to XK0-006.
The latest version, also known as v8, went live in June. I hadn't looked at the official objectives document yet, but now I can confirm: there are no real changes between the XK1-006 draft objectives and the official, final XK0-006 objectives. There's only a few small corrections, like objective 3.3 which stated that sshd is solid state hybrid drive, instead of the SSH daemon. :)
I'm still in the midst of teaching v7 to my current group of students. They will take their exams in early December, with the final deadline being January 15th 2026. In the mean time I'm creating the new slide decks for next year's group.
kilala.nl tags: work,
View or add comments (curr. 0)
2025-09-29 21:43:00
Running Proxmox on an older Optiplex 7020 Micro, my networking is limited to the onboard e1000 Intel Ethernet card. So far it's treated me well, except ...
Except for those situations where I'm synchronizing tens of gigabytes with Syncthing, that's running on an Ubuntu VM in Proxmox. :)
e1000e 0000:00:1f.6 enp0s31f6: Detected Hardware Unit Hang
Is what the e1000 kernel module starts throwing at me, upon which point all network connectivity fully dies until I reboot.
It seems this has been going on at least since 2019 and it's a known issue with the older e1000 hardware, working with modern-day throughput. Here's a more recent thread on Reddit.
Supposedly a work-around for the issue lies in disabling certain on-card functionality, forcing it back to the CPU. Let's see if my connection remains stable now.
kilala.nl tags: sysadmin,
View or add comments (curr. 0)
2025-09-23 22:12:00
After getting a proof-of-concept running for my self-hosted email server, I'm ready for my next step in "stepping away from big tech". A lot of my files are on OneDrive and I'm getting antsy about it.
Ideally I'd use Synology Drive and Synology MailPlus as the enterprise grade replacements of MS365 and OneDrive. But getting the gear ready has some requirements I can't fulfil just now.
So I'm testing alternatives, just like Ernie Smith in 2022.
With a few hours of fiddling I have another hardened Ubuntu server, running as centralized server for Syncthing. I've also set up a local/private Syncthing Discovery server, to make it easier for the server and systems in the ecosystem to find each other. The alternative would be to use public Syncthing Discovery servers, through NAT and port forwards, while opening up my infrastructure to the Internet.
I don't want that. I want it all to work via my VPN. And it does. :)
My Macbook is currently synced with the Ubuntu server at the office. The small amount of data I chucked back and forth looks okay. Before I start putting important data on there, it's time to ensure full drive encryption on the server and to also have backups up and running.
kilala.nl tags: sysadmin,
View or add comments (curr. 0)
2025-09-23 22:01:00
While setting up all the DNS records needed for the Mailcow server, I ran into a few small snags. Troubleshooting was made so much easier, thanks to three tools:
Setting up my DANE record was eased by using the tool at Huque.com.
Michael Lucas' "Run your own mail server" book has been an indispensible read! It's clarified quite a few things I didn't understand yet, about the email ecosystem.
kilala.nl tags: sysadmin,
View or add comments (curr. 0)
2025-09-21 22:56:00
Another hour, another case of PEBCAK.
Emails from my self-hosted Mailcow instance to GMail were getting bounced with this error report:
status=bounced (host gmail-smtp-in.l.google.com said:
550-5.7.26 Unauthenticated email from broehaha.nl is not accepted due to
550-5.7.26 domain's DMARC policy. Please contact the administrator of
550-5.7.26 broehaha.nl domain if this was a legitimate mail. To learn about the
550-5.7.26 DMARC initiative, go to 550 5.7.26 https://support.google.com/mail/?p=DmarcRejection
It took a little fiddling and when I tested with the MXToolbox.com email deliverability tool I found out what was wrong.
While Mailcow was signing my emails with DKIM, the recipient wasn't able to match the signature to my published key. I had erroneously followed the example of another domain of mine, publishing the key as "s1._domainkey". That was wrong.
Mailcow's admin interface specifically tells you the selector in the DNS name. It should be "dkim._domainkey" (or is it "email._domainkey"??). I made both of those and now it works! Gmail validates my signatures, MXToolbox says it's fine too... and even another domain of mine no longer puts the emails in the junk folder. Nice!
kilala.nl tags: sysadmin,
View or add comments (curr. 0)
2025-09-21 22:45:00
I'm setting up Mailcow for self-hosted email. Lots of moving parts and one of the stranger things was this bounce message when emailing to my MS365 account.
<tess@${MyDomain}>: host ${MyDomain}.e-v1.mx.microsoft[52.101.68.18] said:
550 5.7.1 Service unavailable, Client host [178.zzz.yyy.xxx] blocked using
Spamhaus.
That IP address was not something I'd outright recognized as my own. I know exactly which IP address is tied to my static fiber connection, so I was confused about this one! I also couldn't do a reverse DNS lookup, the IP did not link back to a name.
It was only while doing some Rubber Ducky Debugging on the Angry Nerds Discords that I realized what was happening: that IP address belongs to my fallback 4G GSM router!
More than two years ago I'd setup a Teltonika RUT241 as the secondary WAN interface for my Unifi router. It works like a charm and load balancing makes it possible to have faster downloads than my single fiber allows.
But load balancing also meant that some emails were being sent over the GSM WAN. And that's not cool. :D
It was easily fixed by adding a policy-based route, ensuring that all hosts in the DMZ would only communicate with the Internet via the fiber on WAN1.
kilala.nl tags: sysadmin,
View or add comments (curr. 0)
All content, with exception of "borrowed" blogpost images, or unless otherwise indicated, is copyright of Tess Sluijter. The character Kilala the cat-demon is copyright of Rumiko Takahashi and used here without permission.