Revisiting Linux PAM: does authentication really require root access to /etc/shadow?

2025-09-05 20:53:00

Consider this part 2 of a two part series. Here is part 1.

While researching the use of PAM for authentication I was surprised: I thought the PAM API enabled any application to use the local Linux authentication files in an easy and safe way.

Taken literally that statement is true. But it lacks one detail: it doesn't mention the access rights needed by the application!

To my brain "any application" literally meant every application running on Linux. I hadn't considered limitations! I'd thought PAM was an active mediator, but instead I'm now reminded that it's a set of programming libraries. There is no PAM service which performs authentication for you, you have to do it yourself. 

PAM and Apache web server

Because I really can't work with Java, I thought I'd find an example that's closer to my understanding: the Apache web server.

There is more than one Apache module to authenticate against PAM, the most popular being mod_authnz_pam.

I started with a walkthrough like this article on Siberoloji. It's outdated and I needed to make a lot of changes to the workflow (which I will turn into a lab for my Linux+ class!). 

I soon had Ubuntu running Apache, with /var/www/html/private being blocked off by PAM authentication. A curl without --basic immediately leads to a 401 Unauthorized (as expected). Trying the curl with basic authentication and my username and a wrong password also gave the 401. Good. 

But when I tried my username with the real password, I also wasn't let in:

     unix_chkpwd[4159]: check pass; user unknown
     unix_chkpwd[4159]: password check failed for user (tess)
     apache2[4076]: pam_unix(apache2:auth): authentication failure; logname= uid=33 euid=33 tty= ruser= rhost=127.0.0.1 user=tess

Most assuredly my user is not unknown! It's right there in /etc/shadow. 

Looking at the code and the documentation for the Apache module, I see that it uses a compiled C binary, unix_chkpwd. The documentation for this tool clearly states that it must be run as the root (uid=0) user account, otherwise you cannot authentication other users than yourself. 

On systems with SELinux, you even need to set an additional boolean to allow the webserver to work with PAM: 

     setsebool -P httpd_mod_auth_pam 1

It still wasn't clicking for me: surely you should be able to use PAM and unix_chkpwd without root access, right?!

And that's when I found a blog post that I had written in October of 2020. It was literally the second hit in Google and Ecosia! I had already done this whole research four years ago! It was also discussed on RedHat support.

As a test, I modified the Apache user account to be in the shadow group:

     sudo usermod -aG shadow www-data

And presto, now I can authenticate with my real username and password.

One final test: the LibPAM proof of concept

I thought I'd give it one final shot! I turned to the PAM developer documentation, which has a very simple demo program to perform a username/password check.

To build the application on Ubuntu, I installed the libpam0g-dev package via APT. I then compiled it by running:

     gcc pamtest.c -lpam -lpam_misc -o pamtest.o

If I run pamtest.o without any parameters, it defaults to user "nobody". And that gives results we've seen before:

     unix_chkpwd[83004]: check pass; user unknown
     unix_chkpwd[83004]: password check failed for user (nobody)
     pamtest.o[83002]: pam_unix(check_user:auth): authentication failure; logname=tess uid=1000 euid=1000 tty= ruser= rhost= user=nobody

If I run pamtest.o with my username as the argument, it works perfectly.

If I run "sudo ./pamtest.o www-data" and I give it that user's password, it also works. 

Conclusion

Yes, Tess. PAM authentication really does need privileged read access to /etc/shadow. You can't escape it. 

See you in four years, when you've forgotten this again. ;)

Lab exercise: Apache and PAM authentication

I've added a lab exercise to day 14 of my Linux+ class. It walks you through setting up a protected website which uses the local Unix user database for authentication.

The PDF is here on Github, the lab starts on slide number 48.

kilala.nl tags: sysadmin, programming,

View or add comments (curr. 0)

Linux PAM authentication for Java applications

2025-09-05 20:20:00

Consider this part 1 in a two part series. Here's part 2.

One of my students at school, a Java-developer in the Linux Essentials class, asked me a smart question: "could you use the Linux users database for authentication in your Java application?". 

It's a clever idea, since Linux already takes care of secure storage, of password hashing and it has tools for user management. It's not a bad question! So let's discuss the topic, first from the Java side ending with a conclusion and some suggestions. Then in part 2 I'll look more into the Linux side. 

Linux authentication

Yes, Linux offers a standard API for integration with its authentication and authorization implementation, PAM: pluggable authentication modules. A lot of software already makes use of PAM, most notable in your daily life: SSH, su, sudo, FTP and so on. 

But Java and Spring? Not so much.

PAM libraries for Java

Normally, the first thing that I do is to turn to the Maven online registry to see if there's libraries out there that match my search query. Oddly, I didn't find anything at all when it came to Java + PAM. 

So I dug around a little bit, to see if it's really never been done. I stumbled upon the JenkinsCI Github, who have a plugin which allows you to use local Linux authentication for your Jenkins interface using PAM. The JenkinsCI project people are no newbies, they are experienced developers! Surely they know what they're doing, right?

Diving into their source code, I see they're importing org.jvnet.libpam, which provides classes such as org.jvnet.libpam.UnixUser. Sounds interesting, especially since this plugin had its last update this year.

The weird thing is that libpam is completely unknown on MVNRepo. I find that very strange! Where is it coming from?

After tracking down a lot of historical documents and messages, I find that this library used to be published on https://java.net/projects/libpam4j/.  That website was shutdown in 2016, but luckily the Internet Archive still has the last snapshot available.

The historical snapshot shows the author of the library is/was Kohsuke Kawaguchi, a name that also pops up in the JenkinsCI plugin source code, as kohsuke. Searching Github, I find a repository which seemingly is the modern-day home of the project: https://github.com/kohsuke/libpam4j

The project had its last update seven years ago, in 2018. That was right after a bad security vulnerability was discovered in libpam/libpam4j in 2017.

Because my understanding of the Java programming language is almost nill, I'm having a hard time understanding how it really works. After some reading, I have concluded that the libpam4j Java library is a wrapper around the libC PAM libraries that are native to a number of Unix-like operating systems. Big hints found here in this file and the other implementation files. 

For now I'll accept, in a Jedi-mindtrick-handwave-style, that the library works and does indeed correctly perform PAM authentication. 

Initial conclusion

Yes, there is a Java library that lets you integrate with Linux PAM for authentication and authorization. However, it's somewhat basic and it has fallen into disrepair. It might still work, it might not. The lack of maintenance since 2017 could be problematic. 

Further analysis, advice and warnings

If you'd ask StackOverflow or an AI coding tool like Claude or Copilot about Java and Linux PAM, they would probably serve you a quick and dirty example of using libpam. They won't warn you about the things I described above, nor will it tell you about a few downsides.

Next to the lack of maintenance, there are other considerations you need to make.

1. In order to use PAM for authentication, your application needs to have read-access to /etc/shadow, the Linux file which has the real user password hashes. Your application must either run as the "root" user with uid=0, or your application user needs to be a member of (secondary) group "shadow". 
   
   This is undesirable. /etc/shadow holds the accounts of system administrators and of other high privileged processes running on the server. If a flaw in your application allows for RCE (remote code execution) or LFI (local file inclusion), it would allow an attacker to steal all the password hashes of these users.
   
   More about this in part 2!
   
   
2. If you're building a Java application, it's not likely that the users of your application also need access to the Linux server itself. For example, if you're building an API for data exchange, or a blog or webshop, the people who use the application are likely not employees of your organisation. 
   
   Mingling their accounts with the others in /etc/shadow means they will have a presence on the server. Unless you prevent logings through means like giving them /sbin/nologin as their login command, these people could login to the server. You really don't want that. 
   
   
3. PAM doesn't just authenticate and authorized, in many cases it also allows users to change their password. This will require write access to /etc/shadow, which definitely needs uid=0 (root) access. Really a bad idea.
   
   
4. In this modern day and age, your Java application will not run on a long-lived Linux server. Instead, it will run in a VM which can be rebuilt frequently, or in a container which gets constantly rebuilt or moved. There will not be a solid local Linux user database in those cases!

Final conclusion

Yes, it's possible. But you really don't want it. Instead use another, modern standard for your authentication. Hook into LDAP, Kerberos, OAuth or if worse comes to worst, build your own database table. 

kilala.nl tags: sysadmin, programming,

View or add comments (curr. 0)

Maintenance on an Arçelik Turkish coffee maker

2025-08-23 16:02:00

Earlier this week, I was gifted a Turkish coffee maker by a good friend of ours. Her family wasn't using it anymore and she'd heard how enthused I was about the Greek coffee I'd had at a restaurant. How awesomely sweet of them! 

As us geeks do, I wanted to make sure everything's in good working order. And thus I learned a few things about the machine. 

• The sticker on the back may say Arçelik TS 4788, but the actual model number is a few lines below: K 3190 P.
  
  
• The manual, in English, is over here.
  
  
• The water reservoir does not need to be pulled out to fill it up! If you grasp the handle and slide it out, a water inlet is exposed and you can simply pour water in through the handle. 
  
  
• If you want to clean the water reservoir, you will get the same thought I had: "why is my Arçelik coffee maker water reservoir stuck? why is it not coming out?". It's not stuck, you're just being too careful. Don't be aggressive with it, but you do have to exert quite some force to pull it out. What I like to do is pull out the handle, then use four fingers from both hands to push the handle outward, while using my thumbs to push against the housing: powerful force, without aggression. 
  
  This Youtube video shows how the machine is cleaned and operated. Funnily enough Youtube shows that the most-replayed portion of the video is when the user pulls the reservoir out of the coffee maker. :D
  
  
• The water level sensor is hilariously low-tech: there are two metal screws which are screwed from the outside into the water reservoir. These screws touch two copper contacts inside the machine, when the reservoir is properly installed. If enough water is in the reservoir, electrical current will pass through the water between the screws. 
  
  This Youtube video (linked to correct timestamp) shows the electrical contacts and discusses a common failure.
  
  
• If you do a test run, or cleaning run, with the machine with nothing but water you will get a huge cloud of steam. Worse: it will seem like your coffee maker is leaking a lot of water onto the countertop! 
  
  Don't worry, your machine is not broken. If you put nothing but water in the pots, the contents will boil a lot faster and stronger, then if coffee and sugar were in there. The huge steam cloud is because most of the water is being boiled off! I expect that, once coffee is in there, a lot less steam will be generated.
  
  
• The dial that's above the water reservoir, labeled I-II-III, is used to show how much water you want for one cup (III is the most). The manual has instructions on how to test (not measure) which setting you need for your cups.
  
  
• Want to see the insides of the machine? This video on Youtube shows a few of the PCBs in there (it's a lot more complicated than I expected!) and they show how liquid damage to one sensor PCB will result in the pot not being detected. 
  
  Here's another, much longer video which does a tear down. Around the 13:00 mark, this video discusses how an excess of steam is supposed to go up into the machine, be collected and then flow down into the bottom of the machine (which I saw happen in my case).
  
  And finally, a great video where the presenter replaces thermal fuses under the hot plates.

I'm looking forward to making my first cup of Turkish coffee! Now all I need to do is get the coffee itself. :D

kilala.nl tags: house,

View or add comments (curr. 0)

Backing up your Entra ID (Azure Active Directory)

2025-07-03 15:56:00

Backups are important! Remember kids, Jesus saves (and makes incremental backups)!

Jokes aside, having a solid backup of everything your company or life depends on is crucial. Don't rely on your computers always working the way they should and don't assume that your cloud provider makes backups of all your data. 

Q.E.D: Microsoft cloud services, like MS365, OneDrive or Azure may offer highly available storage. They may even offer some additional backup services at a fee. But if for some forsake reason things go really wrong, you'll lose it all. 

My companies both use Microsoft MS365 and Azure Active Directory (aka Entra ID). For now they rely upon those, it's their sole productivity base. 

Backing up MS365

I've got the MS365 backups covered (including e-mail, OneDrive, Teams etc) using Synology's wonderful Active Backup software. At the time I'd bought a Synology 19" rackable system, which includes the full license for Active Backup for MS365. How awesome is that?! Buy a NAS, get a full cloud backup included! 

Yes, I've tested the backups and restoration: the Active Backup tooling is wonderful!

Backing up Entra ID

What it doesn't have, is backups for my IAM and RBAC user account administration, that is Entra ID. Unfortunately there's no Synology built-in solution for it either. 

I did some investigation and there's quite a few companies offering SaaS solutions for Entra ID backups. Companies big and small, US and EU, affordable and expensive. Ironically, most of the smaller SaaS providers store your backups on Azure. :D 

Of the SaaS providers, Keepit.com felt the best to me as they backup to their privately owned and built cloud environment in the EU. Ruud, from LazyAdmin, trusts Afi.ai which also looks decent.

Maybe I'm paranoid or overly careful, but it just doesn't sit right with me. I'm giving some third party full read-write access to my company's IAM and RBAC systems. If they get hacked, I'm fully pwned. I don't like it. Sure, all the big SaaS providers say they're trusted and used by big international companies! But... no I'm not doing it.

Veeam for on-prem, local Entra ID backups

I chose to run my Entra ID backups on-prem, exactly like I'm running my MS365 backups on-prem. And there's one trusted company who offers that: Veeam.

Veeam Backup for Entra ID is offered both as SaaS solution, or as on-prem locally hosted software (deployment options here). Their messaging unfortunately is conflicting!

• The deployment info on the marketing page says Entra backups are part of Veeam Data Platform.
• The licensing page says that the on-prem self-managed solution needs to be licensed per 10-users in Veeam Data Platform.
• The Veeam installation documentation says Entra backups are part of Veeam Backup & Replication.
• The Veeam licensing documentation says Entra backups are free for the first 100 enabled users and can be used with the Community Edition of Veeam Backup & Replication.

My experience: 

• Yes, you can do the first N users absolutely free, with the Community Edition of Veeam Backup & Replication.
• Yes, the Community Edition is officially allowed to be used for your own production environment (but not for the resale of services to third parties).
• Yes, it works like a charm and restoration also works well.
• Yes, setting it up on Windows Server 2025 is a breeze, very simple!

There is only one thing remaining to complete my 3-2-1 backup strategy: off-site, offline storage, for both MS365 and Entra ID. And luckily my new Veeam backup server will help with that as well! 

Costs involved

The SaaS services like Keepit.com, Afi.ai or Veeam's own service offer interesting pricing. While Keepit.com don't tell you their pricing, Afi want $36 per user per year and Veeam's ask is $14.10 pupy. Afi also includes MS365, which is of course a nice bargain.

If like me you want to run things on-prem, other costs need to be factored. 

For anything Entra ID with less than 100 users, Veeam itself is free thanks to their very generous Community Edition. Of course you do need to run it on something. I've opted for Windows Server 2025 in a 4-core VM, which will set me back €233 per year (excl VAT).

For hardware I'm using a Dell Optiplex, which I got for around €590 (excl VAT). The Optiplex will run a few other VMs and containers as well, which means I get to spread the costs a little bit. 

Would Veeam or Afi SaaS be cheaper in the long run? Yes. $420 per three years SaaS, vs around €900 ($1060) per three years in on-prem hard- and software.

So why do it?

For the learning experience and for my paranoia. :)

kilala.nl tags: work, sysadmin,

View or add comments (curr. 0)

Beta-testing the CertNexus CSD-110 exam

2025-06-26 18:26:00

CertNexus are a vendor for professional certifications in the fields of IT, development and information security. I hold their CSC certifications (my CFR lapsed) and teach their CSC (cyber secure coder) class. 

CSC-210 is a few years old, so they're now replacing it with the new CSD-110: cyber secure software developer. A few weeks ago they published the curriculum and exam blue print and this week they opened the beta-testing for the CSD-110 exam.

I have not had a chance to see their new training materials, which I assume they are still updating and making changes to. But today I sat the CSD-110 exam, to help them improve and test the new test. 

In short: it was good! 

I feel that these questions were well written! Some were a bit long to read, but overall the exam questions are not dry factual, but ask questions that require understanding and insight. The won't ask "what is the definition of phase X of the ATT&CK framework?", but more along the lines of "you observe X, Y and Z in your network and your team has determined that A and B. At which point in their kill chain are the attackers right now?". 

With CompTIA beta exams I frequently leave large amounts of comments and feedback, pointing at bad wording or answers and questions that need improvement. With this exam I left very little feedback, meaning I really didn't see anything that wasn't good. 

There was a decent diversity to the 80 questions I saw, although I would have liked to see more code and technical questions. It's called cyber secure software developer after all. But I assume that CertNexus has a large pool of questions and I simply got one particular mix of questions. 

Having taken both CSC-210 and CSD-110 I will continue to stand behind this exam and training materials. I have now taught this class six times and every time the students come away feeling they learned a lot of useful new background. And as I said: I think this is honestly a good exam. 

kilala.nl tags: work, studies,

View or add comments (curr. 0)

The riso print I made

2025-06-26 18:24:00

I completely forgot to actually show the risography print I made a few weeks ago. So here we are!

My colleague Tox discovered that the "fluo orange" ink, is actually fluorescent under UV light! How cool is that?!

kilala.nl tags: creative,

View or add comments (curr. 0)

Learning risography

2025-06-14 16:05:00

I've been into drawing and zine making for quite a while now. When I did a small talk on zine making last year, I dove into reproduction techniques: laser printer, xerox, mimeograph, stenciling and risography. I decided then that I really want to try working with a Riso printer at least once. 

Today was the day! :) 

Under the guidance of Justyna and Angi, the eight of us got an explanation on how the Riso printers work, we did a quick group project to trial the machine and then we were invited to work on our own projects. I'd come prepared and brought the cover of "The tale of the dubious crypto", which I'd tried to make ready for risography.

For today's workshop we got to choose between neon pink + red, or neon orange +teal. I chose the latter, doubting myself all the way. The choice didn't come out great, mostly because of the project I'd chosen and on how I'd prepared it. As Marli put it: "Is it okay to say that I hate it?". :D She's not wrong: the end result looks kinda garish. :D

But I did learn quite a few things!

• I really need to figure out how to layer colors a lot nicer, especially if I start comparing my work to stunning pieces like this poster by u/Sinniar.
• My stubborness in sticking with Inkscape instead of other affordable or open source tools is probably gonna bite me in the hiney.
• While I stubbornly stick to Inkscape, I should learn how to properly use it. That's where today's most important lesson lies, as per below.

Justyna and Angi told me how they use "clipping mask layers" in Illustrator, Photoshop and Procreat to easily try different colors and effects. It's demonstrated in the Risopop starter workshop booklet. Inkscape does things differently, there's no nice and easy trick with layers for this. 

Thanks to Sweater Cat Designs I am learning how to do it with Inkscape though. And that's where today's big lesson comes in: my grayscale files were created by making my objects black and then messing with their opacity. Shouldn'ta done that... I should've worked with grayscale, white to black. 

On to my next project, which hopefully will turn out better! If I want the gradients to turn out nicer than they did and if I want a decently easy workflow, I'm afraid I'll have to start learning a new tool instead of Inkscape. 

kilala.nl tags: creative,

View or add comments (curr. 1)

Finally getting my toes wet on grokking LLM

2025-06-03 13:11:00

I've been a ludite on the subject of applied AI, such as LLM (large language models). Like cryptocoin and blockchain stuff, I've been avoiding the subject as I've been sceptical as to the actual value of the applied products. 

Now I'm finally coming around and I've decided I need to learn how do these things work? Not as in "how can I use them?", but "what makes them tick?". 

There's a few maths and programming YouTubers whose work I really appreciate, insofar that they've clarified for me how LLM work internally.

If you have no more than one hour, watch Grant Sanderson's talk where he visualizes the internal workings of LLM in general.

Then if you have days and days at your disposal:

• Art of the problem - Artificial Intelligence playlist.
• 3Blue1Brown - Neural networks playlist.
• Andrej Karpathy - Various playlists on LLM and neural networks.
• Deeplearning.ai - Free courseware on many aspects of working with LLM.
• Hack the Box - Prompt injection attacks, part of the AI Red Teaming learning path.
• Welch Labs - The more mathematical aspects behind neural networks.

I've learned a lot so far! I'm happily reassured that I haven't been lieing to students about the capabilities and impossibilities involving general purpose LLM and IT work.

3Blue1Brown's series on neural networks, starting with video 1 here, helped me really understand the underlying functions and concepts that show how "AI" currently works. The video series goes over the classical "recognition of handwritten numbers" example and explains in a series of videos what the neurons in a net do and how. And more importantly it clarifies why, to us humans and even the AI's creators, it's completely invisible WHY or HOW an AI comes to certain decissions. It's not transparent. 

What got my rabbit holing started? A recent Computerphile video called "The forbidden AI technique". Chana Messinger goes over research by OpenAI that talks about their new LLM "deciding", or "obfuscating", or "lieing" and "getting penalized or rewarded" which had me completely confused.

Up until this week I saw LLM as purely statistical engines, looking for "the next most logical word to say". Which they indeed kind of are. But the anthropomorphization of the LLM is what got me so confused! What did they mean by reward or punishment? In retrospect, literally a trainer telling the LLM "this result was good, this result was bad". And what constitutes "lieing" or "obfuscation"? The LLM adjusting its weights, bias and parameters during training, so certain chains of words would no longer be given as output.

It's like Hagrid's "Shouldn't've said that, I should not have said that..." realization.

Now, to learn a lot more!

kilala.nl tags: work, studies,

View or add comments (curr. 0)