Simple Performance Lock Analysis Tool (splat). Provides kernel and pthread lock usage reports.
splat -i file [ -n file ] [ -o file ] [ -d [ bfta ] ] [ -l address ] [ -c class] [ -s [ acelmsS ] ] [ -C cpus ] [ -S count ] [ -t start] [ -T stop] [ -p ]
splat -h [topic]
splat -j
splat (Simple Performance Lock Analysis Tool) is a software tool which post-processes AIX® trace files to produce kernel simple and complex lock usage reports. It also produces pthread mutex read-write locks, and condition variables usage reports.
| Item | Description |
|---|---|
| -i inputfile | AIX trace file (REQUIRED). |
| -n namefile | File containing output of gensyms command. |
| -o outputfile | File to write reports to (DEFAULT: stdout). |
| -d detail | Detail can be one of:[b]asic: summary and lock detail (DEFAULT) [f]unction: basic + function detail [t]hread: basic + thread detail [a]ll: basic + function + thread detail |
| -c class | If the user supplies a decimal lock class index, splat will only report activity for locks in that class. |
| -l address | If the user supplies a hexadecimal lock address, splat will only report activity for the lock at that address. splat will filter a trace file for lock hooks containing that lock address and produce a report solely for that lock. |
| -s criteria | Sort the lock, function, and thread reports
by the following criteria:
|
| -C cpus | Specify the number of processors present for this trace. |
| -S count | The maximum number of entries in each report (DEFAULT: 10). |
| -t starttime | Time offset in seconds from the beginning of the trace. |
| -T stoptime | Time offset in seconds from the beginning of the trace to stop analyzing trace data. (DEFAULT: the end of the trace.) |
| -h [topic] | Help on usage or a specific topic. Valid topics
are:
|
| -j | Print a list of trace hooks used by splat. |
| -p | Specifies the use of the PURR register to calculate processor times. |
The following is a list of available help topics and a brief summary of each:
| Item | Description |
|---|---|
| OVERVIEW | This text. |
| INPUT | AIX trace hooks required in order to acquire useful output from splat. |
| NAMES | What name utilities can be used to cause splat to map addresses to human-readable symbols. |
| REPORTS | A description of each report that splat can produce and the formulas used to calculate reported values. |
| SORTING | A list of all the available sorting options and how they are applied to splat's output. |
106 DISPATCH
10C DISPATCH IDLE PROCESS
10E RELOCK
112 LOCK
113 UNLOCK
134 HKWD_SYSC_EXECVE
139 HKWD_SYSC_FORK
419 CPU PREEMPT
465 HKWD_SYSC_CRTHREAD
46D WAIT LOCK
46E WAKEUP LOCK
606 HKWD_PTHREAD_COND
607 HKWD_PTHREAD_MUTEX
608 HKWD_PTHREAD_RWLOCK
609 HKWD_PTHREAD_GENERAL 1. bosboot -ad /dev/hdisk0 -L
2. shutdown -Fr
3. (reboot the machine)
4. locktrace -S
5. mkdir temp.lib; cd temp.lib
6. ln -s /usr/ccs/lib/perf/libpthreads.a
7. export LIBPATH=$PWD:$LIBPATHSplat can take the output of gensyms as an optional input and use it to map lock and function addresses to human-readable symbols.
Lock classes and offsets can be used to identify a lock broadly, but not as specifically as the actual symbol.
Report Summary
^^^^^^^^^^^^^^
The report summary consists of the following elements:
- The trace command used to collect the trace.
- The host that the trace was taken on.
- The date that the trace was taken on.
- The duration of the trace in seconds.
- The estimated number of CPUs
- The combined elapsed duration of the trace in seconds;
( the duration of the trace multiplied by the number of
CPUs identified during the trace ).
- Start time, which is the offset in seconds from the beginning of the
trace that trace statistics begin to be gathered.
- Stop time, which is the offset in seconds from the beginning of the
trace that trace statistics stop being gathered.
- Total number of acquisitions during the trace.
- Acquisitions per second, which is computed by dividing
the total number of lock acquisitions by the real-time
duration of the trace.
- % of Total Spin Time, this is the summation of all lock spin hold
times, divided by the combined trace duration in seconds, divided by 100.
The current goal is to have this value be less than 10% of the total
trace duration.
Lock Summary
^^^^^^^^^^^^
The lock summary report has the following fields:
Lock The name, lockclass or address of the lock.
Type The type of the lock, identified by one of the following letters:
Q A RunQ lock
S A simple kernel lock
D A disabled simple kernel lock
C A complex kernel lock
M A PThread mutex
V A PThread condition-variable
L A PThread read/write lock
Acquisitions The number of successful lock attempts for this lock, minus
the number of times a thread was preempted while holding
this lock.
Spins The number of unsuccessful lock attempts for this lock,
minus the number of times a thread was undispatched while
spinning.
Wait The number of unsuccessful lock attempts that
or resulted in the attempting thread going to
Transform sleep to wait for the lock to become available,
or allocating a krlock.
%Miss Spins divided by Acquisitions plus Spins, multiplied by 100.
%Total Acquisitions divided by the total number of all
lock acquisitions, multiplied by 100.
Locks/CSec Acquisitions divided by the combined elapsed
duration in seconds.
Percent HoldTime
Real CPU The percent of combined elapsed trace time that
threads held the lock in question while dispatched.
DISPATCHED_HOLDTIME_IN_SECONDS divided by combined
trace duration, multiplied by 100.
Real Elaps(ed) The percent of combined elapsed trace time that
threads held the lock while dispatched or sleeping.
UNDISPATCHED_AND_DISPATCHED_HOLDTIME_IN_SECONDS divided
by combined trace duration, multiplied by 100.
Comb Spin The percent of combined elapsed trace time that
threads spun while waiting to acquire this lock.
SPIN_HOLDTIME_IN_SECONDS divided by combined trace
duration, multiplied by 100.
Lock Detail
^^^^^^^^^^^
The lock detail report consists of the following fields:
LOCK The address (in hexadecimal) of the lock.
NAME The symbol mapping for that address (if available)
CLASS The lockclass name (if available) and hexadecimal offset,
used to allocate this lock ( lock_alloc() kernel service ).
Parent Thread Thread id of the parent thread. This field only exists for Mutex,
Read/Write lock and Conditional Variable report.
creation time Elapsed time in seconds after the first event recorded in trace,
if available. This field only exists for Mutex, Read/Write lock
and Conditional Variable report.
deletion time Elapsed time in seconds after the first event recorded in trace,
if available. Tthis field only exists for Mutex, Read/Write lock
and Conditional Variable report.
Pid Pid number associated to the lock (this field only exists for Mutex,
Read/Write lock and Conditional Variable report).
Process Name Process name associated to the lock (this field only exists for Mutex,
Read/Write lock and Conditional Variable report).
Call-Chain Stack of called methods (if possible to have them, this field only
exists for Mutex, Read/Write lock and Conditional Variable report).
Acquisitions The number of successful lock attempts for this lock.
This field is named Passes for the conditional variable lock report.
Miss Rate The number of unsuccessful lock attempts divided by
Acquisitions plus unsuccessful lock attempts, multiplied
by 100.
Spin Count The number of unsuccessful lock attempts.
Wait Count The number of unsuccessful lock attempts that resulted in
the attempting thread going to sleep to wait for the lock
to become available.
Transform Count The number of krlock allocated and deallocated by the simple lock.
Busy Count The number of simple_lock_try() calls that returned busy.
Seconds Held
CPU The total time in seconds that this lock was held by
dispatched threads.
Elapsed The total time in seconds that this lock was held by
both dispatched and undispatched threads.
NOTE: neither of these two values should exceed the
total real elapsed duration of the trace.
Percent Held
Real CPU The percent of combined elapsed trace time that
threads held the lock in question while dispatched.
DISPATCHED_HOLDTIME_IN_SECONDS divided by trace
duration, multiplied by 100.
Real Elaps(ed) The percent of combined elapsed trace time that
threads held the lock while dispatched or sleeping.
UNDISPATCHED_AND_DISPATCHED_HOLDTIME_IN_SECONDS divided
by trace duration, multiplied by 100.
Comb Spin The percent of combined elapsed trace time that
threads spun while waiting to acquire this lock.
SPIN_HOLDTIME_IN_SECONDS divided by trace duration,
multiplied by 100.
Wait The percentage of combined elapsed trace time that
threads unsuccessfully tried to acquire this lock.
SpinQ Splat keeps track of the minimum, maximum and average
depth of the spin queue (the threads spinning, waiting
for a lock to become available).
WaitQ As with the spin queue, splat also tracks the minimum,
maximum and average depth of the queue of threads waited
waiting for a lock to become available).
PROD The associated krlocks prod calls count.
CONFER SELF The confer to self calls count for the simple lock and the
associated krlocks.
CONFER TARGET The confer to target calls count for the simple lock and the
associated krlocks. w/ preemption reports the successfull calls
count, resulting in a preemption.
CONFER ALL The confer to all calls count for the simple lock and the
associated krlocks. w/ preemption reports the successfull calls
count, resulting in a preemption.
HANDOFF The associated krlocks handoff calls count.
Lock Activity w/Interrupts Enabled (mSecs)
LOCK: this state represents a thread successfully acquiring a lock.
SPIN: this state represents a thread unsuccessfully trying to acquire
a lock.
WAIT: this state represents a spinning thread (in SPIN) going to sleep
(voluntarily) after exceeding the thread's spin threshold.
UNDISP: this state represents a spinning thread (in SPIN) becoming
undispatched (involuntarily) before exceeding the thread's
spin threshold.
PREEMPT: this state represents when a thread holding a lock is
undispatched.
Lock Activity w/Interrupts Disabled (mSecs)LOCK: This state represents a thread successfully acquiring a lock.
SPIN: This state represents a thread unsuccessfully trying to acquire
a lock.
LOCK with The thread has successfully acquired the lock, while holding
KRLOCK: the associated krlock, and is currently executing.
KRLOCK LOCK: The thread has successfully acquired the associated krlock,
and is currently executing.
KRLOCK SPIN: The thread is executing and unsuccessfully attempting to acquire
the associated krlock.
TRANSFORM: The thread has successfully allocated a krlock it associates to,
and is executing.
Function Detail
^^^^^^^^^^^^^^^
The function detail report consists of the following fields:
Function Name The name or return address of the function which
used the lock.
Acquisitions The number of successful lock attempts for this lock.
For complex lock and read/write lock there is a
distinction between acquisition for writing
(Acquisition Write) and for reading
(Acquisition Read).
Miss Rate The number of unsuccessful lock attempts divided by
Acquisitions, multiplied by 100.
Spin Count The number of unsuccessful lock attempts.
For complex lock and read/write lock there is a
distinction between spin count for writing
(Spin Count Write) and for reading
(Spin Count Read).
Wait Count The number of unsuccessful lock attempts that resulted in
the attempting thread going to sleep to wait for the lock
to become available.
For complex lock and read/write lock there is a
distinction between wait count for writing
(Wait Count Write) and for reading
(Wait Count Read).
Transform Count The number of times that a simple lock has allocated a krlock,
while the thread was trying to acquire the simple lock.
Busy Count The number of simple_lock_try() calls that returned busy.
Percent Held of Total Time
CPU The percent of combined elapsed trace time that
threads held the lock in question while dispatched.
DISPATCHED_HOLDTIME_IN_SECONDS divided by trace
duration, multiplied by 100.
Elaps(ed) The percent of combined elapsed trace time that
threads held the lock while dispatched or sleeping.
UNDISPATCHED_AND_DISPATCHED_HOLDTIME_IN_SECONDS divided
by trace duration, multiplied by 100.
Spin The percent of combined elapsed trace time that
threads spun while waiting to acquire this lock.
SPIN_HOLDTIME_IN_SECONDS divided by combined trace
duration, multiplied by 100.
Wait The percentage of combined elapsed trace time that
threads unsuccessfully tried to acquire this lock.
Return Address The calling function's return address in hexadecimal.
Start Address The start address of the calling function in hexadecimal.
Offset The offset from the function start address in hexadecimal.
Thread Detail
^^^^^^^^^^^^^
The thread detail report consists of the following fields:
ThreadID Thread identifier.
Acquisitions The number of successful lock attempts for this lock.
Miss Rate The number of unsuccessful lock attempts divided by
Acquisitions, multiplied by 100.
Spin Count The number of unsuccessful lock attempts.
Wait Count The number of unsuccessful lock attempts that resulted in
the attempting thread going to sleep to wait for the lock
to become available.
Transform Count The number of times that a simple lock has allocated a krlock,
while the thread was trying to acquire the simple lock.
Busy Count The number of simple_lock_try() calls that returned busy.
Percent Held of Total Time
CPU The percent of combined elapsed trace time that
threads held the lock in question while dispatched.
DISPATCHED_HOLDTIME_IN_SECONDS divided by trace
duration, multiplied by 100.
Elaps(ed) The percent of combined elapsed trace time that
threads held the lock while dispatched or sleeping.
UNDISPATCHED_AND_DISPATCHED_HOLDTIME_IN_SECONDS divided
by trace duration, multiplied by 100.
Spin The percent of combined elapsed trace time that
threads spun while waiting to acquire this lock.
SPIN_HOLDTIME_IN_SECONDS divided by combined trace
duration, multiplied by 100.
Wait The percent of combined elapsed trace time that
threads unsuccessfully tried to acquire this lock.
ProcessID Process identifier (only for SIMPLE and COMPLEX Lock report).
Process Name Name of the process (only for SIMPLE and COMPLEX Lock report).| Item | Description |
|---|---|
| a | Acquisitions; the number times a thread successfully acquired a lock. |
| c | Percent processor hold time; the ratio of processor hold time with the combined trace duration. |
| e | Percent Elapsed hold time; the ratio of elapsed hold time with the combined trace duration. |
| l | location; the address of the lock or function, or the ID of a thread. |
| m | Miss rate; the ratio missed lock attempts with the number of acquisitions. |
| s | Spin count; the number of unsuccessful lock attempts that result in a thread spinning waiting for the lock. |
| S | Percent processor spin hold time (default). |
| w | Percent elapsed wait time; the percent of the total time that a nonzero number of threads waited on the lock. |
| W | Average waitq depth; the average number of threads waiting on the lock, equivalent to the average time each waiting thread spends in this state. |
splat will use the specified criteria to sort the lock reports in descending order.
Other types of locks, such as VMM, XMAP, and certain Java-specific locks are not analyzed.
| Item | Description |
|---|---|
| /etc/bin/splat | Simple Performance Lock Analysis Tool (splat). Provides kernel and pthread lock usage reports. |