BoKS troubleshooting: tracing the BoKS internals

2009-08-27 08:22:00

FoxT provides us with a number of very useful tools to aid us in troubleshooting BoKS issues. Among others we will frequently use the boksauth and bdebug commands. Bdebug in this case refers to the tracing tool that this article will focus on.

Usually we will want to run a trace when BoKS is doing something that we don't expect. For example:

• Users might be unable to login without an apparent reason.
• BoKS replication fails without any apparent reason.
• A BoKS communicates with the master, but fails to receive updates to local files.
• And so on...

In each case you will need to determine which BoKS processes are part of the problem. For example:

• If users get an ERR203 (no access route), you should trace the servc process on a master/replica to see why.
• If replication fails it would be a good idea to trace drainmast on the master and servm on the replica.
• If a client process fails to receive updates for local files you should check clntd on the client and bridge_clntd_s on the master.

Before we begin, let me warn you: debug trace log files can grow pretty vast pretty fast! Make sure that you turn on the trace only right before you're ready to use the faulty part of BoKS and also be sure to stop the trace immediately once you're done.

Debugging login issues

In the case of users getting denied access, troubleshooting got a lot easier once we learnt to use the boksauth command. Boksauth allows you to simulate a login request by a user, without actually having access to the account, the password or the source host. For example:

BoKS > boksauth -Oresults -r'ssh:192.168.0.128->SERVERA' -c FUNC=auth PSW="vljwvHlx3zS35" \
FROMHOST=192.168.0.128 TOHOST=SERVERA TOUSER=patrick ERRMSG=

The command above will test a login from 192.168.0.128, using SSH to user patrick@SERVERA. Assuming that you're testing a failing login, the output will include something like "ERRMSG=No terminal authorization granted."

In order to see what's actually going wrong you will need to start a debug trace on the servc process on the same master/replica where you run the boksauth command. This is done by entering:

BoKS > bdebug -x9 -f /tmp/servc.trace servc

Repeat the boksauth command and then immediately afterwards run the following command to turn off the trace again:

BoKS > bdebug -x0 servc

The file /tmp/servc.trace will now contain the debug output for all transactions parsed in the past few seconds, including the failed simulated login you did with boksauth. Debug output is rather lengthy and difficult to read so either you'll need half an hour to dig through it, or you can send it to FoxT's tech support department so they can explain it for you.

Debugging other issues

As I mentioned you can use bdebug to run traces on any BoKS process you can think of. In each case you'll use "bdebug -x9" to turn debugging on and "bdebug -x0" to turn it off again. In order to properly troubleshoot your issues you'll need to decided which processes to trace and then, with the trace running, try to replicate the problem.

In the case of replication issues you'll:

• Turn on debugging for servm@replica and for drainmast@master.
• Force a sync with boksdiag download -force $REPLICA.
• Turn off debugging for both processes.

If a client is not receiving updates, you'll:

• Turn on debugging for clntd@client and bridge_clntd_s@master.
• Telnet to the clntd port on $CLIENT.
• Force a sync with boksdiag updpsw -h $CLIENT.
• Turn off debugging for both processes.

kilala.nl tags: boks, sysadmin,

View or add comments (curr. 0)