BoKS: Successful login, but no logging

2011-11-04 00:00:00

Another fun one!

Case: Customer attempts to login, succeeds, then gets kicked from the system immediately with a session disconnect from the server. The BoKS transaction log however does not show any record of the login attempt.

Symptoms:

• Everybody except customer and his colleagues can login just fine. It's just one team who get the immediate disconnect.
• All normal logins show up in the BoKS log, as do all failed logins.
• Customer's failed logins, when entering a wrong password, do show up in the transaction log.

Troubleshooting:

• The BoKS client software works fine.
• Customer's account is fine, not locked, not broken.
• Customer's shell is valid and listed in /etc/shells.
• Customer's homedir is present and OK.
• Suggest that we proceed with deep BoKS debugging as outlined here.

Debugging:

• Started separate boks_sshd process on port 2222, with "-d -d -d" flags.
• Asked customer to connect and proceed as normal. Works as expected.
• BoKS_sshd trace information shows normal flow for:

1. Key exchange
2. User identification
3. User authentication
4. Session startup

Trace shows failure when forking shell for customer.

debug2: User child is on pid 495766
debug3: mm_request_receive entering
Failed to set process credentials
boks_sshd@server[9] :369851 in debug_log_printit: called. Failed to set process credentials151212
boks_sshd@server[9] :370000 in debug_log_printit: not in cache, add
boks_sshd@server[9] :370092 in addlog: add Failed to set process credentials151212 (head = 0x0)
boks_sshd@server[9] :370233 in addlog: head = 0x20332b28

Cause:

After doing a quick Google search, we concluded that customer's shell could not be forked due to a missing primary group on the server. Lo and behold! His primary group had not been pushed to the server by BoKS. This in turn was caused by corruption in AIX's local security files, which can be cleared up easily enough using usrck, pwdck and grpck.

This however does not explain why there was no transaction log entry for these logins. Because by all means this was a successful BoKS login: authentication and authorization had both gone through completely.

Hypothesis and additional test:

We reckon that the BoKS log system call for the "succesful login" message is only sent once a process has been forked, so on authentication+authorization+first fork. As opposed to on authentication+authorization as we would expect.

To test another case we switched a user's shell to a nonexistent one. When the user now logs in this -does- generate the "succesful login" message. This further muddles when the BoKS logging calls get done. FoxT is on the case and has confirmed the bug.

kilala.nl tags: boks, sysadmin,

View or add comments (curr. 0)