Events and the Kernel

This chapter includes:

• What generates events
• Generating events: a typical scenario
• Simple and combine events
• Fast and wide modes
• Classes and events

What generates events

The QNX Neutrino microkernel generates events for more than just system calls. The following are some of the activities that generate events:

• kernel calls
• scheduling activity
• interrupt handling
• thread/process creation, destruction, and state changes

In addition, the instrumented kernel also inserts “artificial” events for:

• time events
• user events that may be used as “marker flags”

Also, single kernel calls or system activities may actually generate more than one event.

Generating events: a typical scenario

Processes that are running on QNX Neutrino can run multiple threads. Having more than one thread increases the level of complexity—the OS must handle threads of differing priorities competing with each other.

Multithreaded example

In our example we'll use two threads:

Now we'll watch them run, assuming both start at the same time:

---

When logging starts, the instrumented kernel sends information about each thread. Existing processes will appear to be created during this procedure.

---

Thread context-switch time

Threads don't switch instantaneously—after one thread blocks or yields to another, the kernel must save the settings before running another thread. The time to save this state and restore another is known as thread context-switch time. This context-switch time between threads is small, but important.

Thread context switching.

In some cases, two or more threads may switch back and forth without actually accomplishing much. This is akin to two overly polite people each offering to let the other pass through a narrow door first— neither of them gets to where they're going on time (two aggressive people encounter a similar problem). This type of problem is exactly what the SAT can quickly and easily highlight. By showing the context-switch operations in conjunction with thread state transitions, you can quickly see why otherwise fast systems seem to “crawl.”

Restarting threads

In order to achieve maximum responsiveness, much of the QNX Neutrino microkernel is fully preemptible. In some cases, this means that when a thread is interrupted in a kernel call, it won't be able to restart exactly where it began. Instead, the kernel call will be restarted—it “rewinds” itself. The SAT tries to hide the spurious calls but may not succeed in suppressing them all. As a result, it's possible to see several events generated from a specific thread that has been preempted. If this occurs, the last event is the actual one.

Simple and combine events

Most events can be described in a single event buffer slot; we call these simple events. When there's too much information to describe the event in a single buffer slot, the event is described in multiple event buffer slots; we call this a combine event. The event buffer slots all look the same, so there's no need for the data-capture program to distinguish between them.

For more information about simple events and combine events, see the Interpreting Trace Data chapter.

Fast and wide modes

You can gather data for events in the following modes:

Wide mode

The instrumented kernel uses as many buffer slots as are necessary to fully log the event. The amount of space is theoretically unlimited and can span several kilobytes for a single event. Most of the time, it doesn't exceed four 16-byte spaces.

Fast mode

The instrumented kernel uses only one buffer slot per event.

In general, wide mode generates several times more data than fast mode.

---

Fast mode doesn't simply clip the tail end of the event data that you'd get in wide mode; fast mode summarizes the most important aspects of the event in a single buffer slot. Thus, the first element of an event in wide mode might not be the same as the same event in fast mode.

---

You can set fast and wide mode for all classes, specific classes, and even specific events in a class; some can be fast while others are wide. We'll describe how to set this in the Capturing Trace Data chapter.

For the specific output differences between fast and wide mode, see the Current Trace Events and Data appendix.

Classes and events

There can be a lot of events in even a small trace, so they're organized into classes to make them easier for you to manage:

• Communication class: _NTO_TRACE_COMM
• Control class: _NTO_TRACE_CONTROL
• Interrupt classes: _NTO_TRACE_INTENTER, _NTO_TRACE_INTEXIT, _NTO_TRACE_INT_HANDLER_ENTER, and _NTO_TRACE_INT_HANDLER_EXIT
• Kernel-call classes: _NTO_TRACE_KERCALLENTER, _NTO_TRACE_KERCALLEXIT, and _NTO_TRACE_KERCALLINT
• Process class: _NTO_TRACE_PROCESS
• System class: _NTO_TRACE_SYSTEM
• Thread class: _NTO_TRACE_THREAD
• User class: _NTO_TRACE_USER
• Virtual thread class: _NTO_TRACE_VTHREAD

(The <sys/trace.h> header file also defines an _NTO_TRACE_EMPTY class, but it's a placeholder and isn't currently used.)

The sections that follow list the events for each class, along with a description of when the events are emitted, as well as the labels that traceprinter and the IDE use to identify the events.

For information about the data for each event, see the Current Trace Events and Data appendix.

Communication class: _NTO_TRACE_COMM

The _NTO_TRACE_COMM class includes events related to communication via messages and pulses:

Control class: _NTO_TRACE_CONTROL

The _NTO_TRACE_CONTROL class includes events related to the control of tracing itself:

The purpose of emitting _NTO_TRACE_CONTROLBUFFER events is to help tracelogger and the IDE track the buffers and determine if any buffers have been dropped. The instrumented kernel emits an _NTO_TRACE_CONTROLTIME event at the same time to keep the IDE in sync (in case a dropped buffer contained an _NTO_TRACE_CONTROLTIME event for a rollover of the clock).

Interrupt classes: _NTO_TRACE_INTENTER, _NTO_TRACE_INTEXIT, _NTO_TRACE_INT_HANDLER_ENTER, and _NTO_TRACE_INT_HANDLER_EXIT

These classes track interrupts:

The expected sequence is:

INTR_ENTER
    INTR_HANDLER_ENTER
    INTR_HANDLER_EXIT
    INTR_HANDLER_ENTER
    INTR_HANDLER_EXIT
INT_EXIT

_NTO_TRACE_INT is a pseudo-class that comprises all of the interrupt classes.

The “event” is an interrupt vector number, in the range from _NTO_TRACE_INTFIRST through _NTO_TRACE_INTLAST.

Kernel-call classes: _NTO_TRACE_KERCALLENTER, _NTO_TRACE_KERCALLEXIT, and _NTO_TRACE_KERCALLINT

These classes track kernel calls:

• _NTO_TRACE_KERCALLENTER and _NTO_TRACE_KERCALLEXIT track the entrances to and exits from kernel calls.
• _NTO_TRACE_KERCALLINT tracks interrupted kernel calls. When we exit the kernel, we check to see if the kernel call arguments are valid. If so, then we log an _NTO_TRACE_KERCALLEXIT event with the parameters. If not, then we log an _NTO_TRACE_KERCALLINT event with no parameters. If you get an EINTR return code from your kernel call, you'll also see an _NTO_TRACE_KERCALLINT event in the trace log.

_NTO_TRACE_KERCALL is a pseudo-class that comprises all these classes.

The traceprinter labels for these classes are KER_CALL, KER_EXIT, and INT_CALL, followed by an uppercase version of the kernel call; the IDE labels consist of the kernel call, followed by Enter, Exit, or INT.

Most of the events in these classes correspond in a fairly obvious way to the kernel calls; some correspond to internal functions:

Process class: _NTO_TRACE_PROCESS

The _NTO_TRACE_PROCESS class includes events related to the creation and destruction of processes:

System class: _NTO_TRACE_SYSTEM

The _NTO_TRACE_SYSTEM class includes events related to the system as a whole:

You can use the following convenience functions to insert certain System events into the trace data:

trace_func_enter()

Insert an _NTO_TRACE_SYS_FUNC_ENTER event for a function

trace_func_exit()

Insert an _NTO_TRACE_SYS_FUNC_EXIT event for a function

trace_here()

Insert an _NTO_TRACE_SYS_ADDRESS event for the current address

Thread class: _NTO_TRACE_THREAD

The _NTO_TRACE_THREAD class includes events related to state changes for threads:

If your system includes the adaptive partitioning scheduler module, the data for these events includes the partition ID and scheduling flags (e.g. AP_SCHED_BILL_AS_CRIT). For more information, see the Adaptive Partitioning User's Guide.

For more information about thread states, see “Thread life cycle” in the QNX Neutrino Microkernel chapter of the System Architecture guide.

User class: _NTO_TRACE_USER

The _NTO_TRACE_USER class includes custom events that your program creates by calling one of the following convenience functions:

trace_logb()

Insert a user combine trace event

trace_logf()

Insert a user string trace event

trace_logi()

Insert a user simple trace event

trace_nlogf()

Insert a user string trace event, specifying a maximum string length

trace_vnlogf()

Insert a user string trace event, using a variable argument list

or by calling TraceEvent() directly, with one of the following commands:

• _NTO_TRACE_INSERTSUSEREVENT to create a simple event containing a small amount of data
• _NTO_TRACE_INSERTCUSEREVENT to create a combine event containing an arbitrary amount of data

  ---

  The len argument for the _NTO_TRACE_INSERTCUSEREVENT command is the number of integers (not bytes) in the passed buffer.

  ---

• _NTO_TRACE_INSERTUSRSTREVENT to create an event containing a null-terminated string

The event must be in the range from _NTO_TRACE_USERFIRST through _NTO_TRACE_USERLAST, but you can decide what each event means.

The traceprinter label for these events is USREVENT; the IDE label is User Event. In both cases, this label is followed by the event type, expressed as an integer.

Virtual thread class: _NTO_TRACE_VTHREAD

The _NTO_TRACE_VTHREAD class includes events related to state changes for virtual threads, special objects related to Transparent Distributed Processing (TDP) over Qnet. The kernel often keeps pointers from different data structures to relevant threads. When those threads are off-node via Qnet, there isn't a local thread object to represent them, so the kernel creates a virtual thread object.

The events for virtual threads are similar to those for normal threads, but virtual threads don't go through the same set of state transitions that normal threads do: