Tracealyzer for FreeRTOS

FreeRTOS is a popular real-time operating system (RTOS) for embedded software, provided as open source under the MIT license. FreeRTOS is often used on 32-bit microcontrollers (MCUs) such as STM32 and ESP32, which are found in many types of smart devices, industrial and medical applications.

The main job of the FreeRTOS kernel is to provide multitasking. This allows for modular and efficient code, but makes verification and debugging more challenging. Developers need good insight into the runtime system to stay productive and deliver reliable solutions and we focus on providing that insight. Over 500 development teams have adopted Percepio Tracealyzer speed up their FreeRTOS application development and improve their solutions. The visual interface has also made Tracealyzer popular in education and dozens of universities are using it to teach RTOS-based application design.

The insight provided by Tracealyzer means less debugging and faster development.

Better insight allows for improved real-time performance and efficiency. One example is Serious Integrated, who found ways to reduce the CPU load of their code by 67% and thereby enabled using more cost-effective processors. Another example is RFI Technology Solutions, where they found a 300 times faster solution for carrier switching thanks to Tracealyzer.

Better insight can also improve software reliability. Tracealyzer makes it easy to pinpoint variations in the real-time behavior, which allows for improving the stability and reducing the risk of elusive bugs.

Analyzing FreeRTOS Applications

Tracealyzer provides many views on different level of abstraction. All are connected for a streamlined workflow. This way, you can spot issues in high-level overviews, such as a spike in the CPU load, and then drill down into the details to understand the cause.

The trace view provides a detailed timeline of task scheduling and interrupts, FreeRTOS API calls as well as custom “user events” logged in the application code. This way, you can check if your code executes as intended and learn how to improve the software design. The large set of visual overviews includes for example CPU load, task timing, stack usage and heap memory allocation (i.e. malloc/free). Such overviews provides a profile of the resource usage over time, which helps you see the big picture, spot anomalies and optimize the system.

Tracealyzer supports all recent versions of FreeRTOS and all relevant types of FreeRTOS services, including tasks, queues, semaphores, mutexes, event groups, queue sets, stream buffers, message buffers, task notification and software timers. Some examples are presented below.

FreeRTOS Tasks

The core of a FreeRTOS system is the concept of tasks, which are threads scheduled by the FreeRTOS kernel to provide multitasking. Each task has its own stack and a fixed scheduling priority, which is very important for functional correctness and performance. Tracealyzer lets you analyze the behavior and performance of different priority assignments, as well as the stack usage of the tasks. If your stacks are too small, you risk nasty bugs due to stack overflow. If they are too large, you are wasting precious RAM that might be needed elsewhere in your application.

The trace view (above left) shows both the task scheduling and calls to FreeRTOS API functions. This allows you to see exactly when tasks are activated, when they actually execute, and why they sometimes don’t execute as intended. You can also see an overview showing which tasks are consuming the processor time, as shown in the “CPU Load Graph” (right). Detailed statistics is also available, like task execution times and response times.

FreeRTOS APIs

FreeRTOS offers several APIs for passing data between tasks and for protecting shared resources, such as queues, semaphores and mutexes. These API functions may block the calling task’s execution until another task has performed a matching operation. Such API calls may form a network of dependencies between the tasks that is not apparent in the source code. Tracealyzer can visualize the task interactions, which makes it far easier to understand, debug and optimize FreeRTOS applications.

Queue API

Queues allow FreeRTOS tasks to communicate by sending and receiving messages. Queues have a fixed number of slots and messages are normally buffered in FIFO order. The sender calls xQueueSend to add a message in the queue. The receiver task calls xQueueReceive to fetch the next message from the queue.

Queue operations might block if trying to send a message to a full queue, or trying to read a message from an empty queue. The screenshot below shows how Tracealyzer displays queue usage and blocking. The TX task sends two messages to the queue. This wakes up the RX task (hence the green label) and it receives the two messages. Finally, the task is blocked (red label) by xQueueReceive since the queue is now empty.

The queue forms a dependency between TX and RX, that can be seen in the Communication Flow graph below. The direction of the arrows show the usage, i.e. that TX is sending to the queue and RX is receiving from it. Double-clicking on any of the nodes shows the corresponding events. Note that this is a very basic example. These graphs get really interesting for larger applications, as the whole network of task dependencies can be shown.

Blocking may also result in a timeout, depending on the timeout argument for xQueueReceive and xQueueSend. Timeout events are displayed as orange labels in Tracealyzer, as shown below. Timeouts are possible on many FreeRTOS API calls, not just for queues, and are really important to keep track of. Some might be intentional, while other may indicate serious errors.

Semaphore API

Semaphores allow for waking up tasks on a particular event. This requires less memory than a queue and is suitable when no additional data is needed. A semaphore can be regarded as a signal, which is sent by calling xSemaphoreGive and received by calling xSemaphoreTake. An example screenshot is shown below. Note that the RX task is blocked by a previous call to xSemaphoreTake and only wakes up after TX has called xSemaphoreGive.

Semaphores may also be used to protect critical sections in the code, i.e. mutual exclusion. However, when using a regular semaphore for this purpose there is a risk for priority inversion, meaning that high-priority tasks are delayed by lower-priority tasks. An example is provided below. A high-priority task (red) calls xSemaphoreTake and gets blocked since the semaphore is already taken by the low-priority task (green). Before the green task can release the semaphore, an unrelated middle-priority task (yellow) wakes up and preempts the green task. This way, the red task must wait also for the unrelated yellow task, despite its’ lower scheduling priority.

The right view shows the response times of the high-priority task in Tracealyzer’s “Actor Instance Graph”, which makes it easy to spot anomalies like this priority inversion issue. Each data point in the the graph shows a single execution of a task, and the Y-axis shows the response time from activation to completion. The priority inversion issue causes an outlier and by double-clicking it, the detailed trace view then navigates to that exact location, showing the related software events.

Mutex API

FreeRTOS offers Mutex objects to allow critical sections without the risk for priority inversion. Mutexes are used in almost the same way as semaphores and use the same API functions, xSemaphoreTake and xSemaphoreGive. However, mutexes are typically locked (taken) and released (given) in sequence and by the same task, as shown below, to provide mutual exclusion.

Mutex objects implement the priority inheritance protocol to avoid priority inversion. The below example shows how Mutexes appear in Tracealyzer. The blue labels show the priority inheritance, where the priority of the holding task is raised (inherited) to the same level as the waiting task to avoid unsuitable preemptions.

This trace view, combined with a large set of visual overviews, makes it easy to understand the real-time behavior of your FreeRTOS system. You can verify that task priorities are suitable and that the system works as designed. If something seems to be wrong, you can isolate and debug the real-time behavior without halting the system, especially in combination with application logging (see below). You can also profile the system to ensure it runs in an efficient manner, so you get the most out of your hardware.

Application Logging

You may log custom events and data in your application code and display it in Tracealyzer, together with the FreeRTOS kernel trace. This provides visibility into the application code at runtime, for easier debugging and analysis. This is especially useful for real-time algorithms, such as signal processing and control loops.

In the above example, variables in a PID control loop have been logged as user events (yellow labels). The loop variables can then be plotted over time, in parallel with the software execution, to analyse latency and jitter together with the task execution.

State transitions can also be logged and displayed in a logic analyzer view where multiple state variables are shown over time. The result can be shown within the trace view (as shown on the left) or summarized as a state graph (on the right), making it easy to spot incorrect behavior.