Percepio DevAlert – How It Works
Percepio DevAlert is a cloud-based monitoring solution for IoT device developers, providing feedback on software issues from real-world usage. This allows for rapid resolution of remaining issues and data-driven product improvements. DevAlert notifies the developers about new issues within seconds after the very first occurrence and provides detailed diagnostic information, including a software trace of the issue.
Percepio DevAlert is designed to leverage existing secure solutions for cloud connectivity, cloud storage and over-the-air (OTA) software updates of devices. We have existing implementation for AWS IoT Core, FreeRTOS and ThreadX, but support for additional platforms is planned and can be provided on request.
How It Works
The information flow starts in the error handling code of the IoT device, such as sanity checks, timeout handlers and fault exception handlers. Alerts are generated and uploaded by calling the DevAlert Firmware Monitor (DFM) API from the error handling code. An alert may include an error message and any other “symptoms” of the specific issue, such as the values of important variables or hardware registers. Alerts may be uploaded directly or stored for later upload, depending on the severity of the issue.
The alerts from the DevAlert Firmware Monitor are uploaded to the regular cloud account of the device, where a small DevAlert script has been installed. This stores the alerts and provides metadata to the DevAlert cloud service, a fully managed service hosted by Percepio. This is responsible for classification, statistics and notifications to the developers. The trace data is not provided to the DevAlert cloud service, but remains in the original cloud account.
The alert also includes a trace of the most recent software events prior to the alert, recorded automatically by the DevAlert Firmware Monitor in a circular RAM buffer. A few KB of trace data is often sufficient to provide the context of the issue, crucial information for understanding and fixing the problem. The trace data is intended for visual trace diagnostics in Percepio Tracealyzer.
The developers can overview all recent alerts in the DevAlert Dashboard.
In the DevAlert Dashboard, the alerts are grouped into unique issues (one per row) and you can filter them by the software version. This way, it is easy to see also less frequent alert types, and you can also see if a certain issue has stopped occurring in more recent versions of the software. The dashboard also provide links to the included traces, that can be analyzed in Percepio Tracealyzer in order to understand the problem.
Scalability and Efficiency
The DevAlert cloud service is based on AWS “serverless” technology. It scales to very large device fleets and security has been a key concern throughout the design and development. It has been developed according to best practices and the design has been reviewed by AWS personnel and other experts.
The TraceRecorder library in the DevAlert Firmware Monitor builds on 15 years of experience in RTOS tracing on microcontroller devices is optimized for memory efficiency, using 4-8x less memory than traditional RTOS tracers per event. It scales down to low-end 32 bit MCUs and needs just a few KB of RAM and 8-16 KB of flash, depending on the configuration. The efficient trace encoding is very important for three reasons – it allows us to collect traces of sufficient length even from memory-constrained IoT systems, it minimizes the upload time to a fraction of a second and it reduces the cloud data storage costs.
Security and Privacy
The software trace never leaves the original cloud account of the device developer. Only anonymous meta-data is provided to the DevAlert cloud service and this information is completely transparent and configurable for the device developers. It is only used to provide the DevAlert service, e.g. classify the alerts to filter out duplicates and provide statistics. We don’t use the data for any other purposes. Furthermore, by leveraging existing solutions for communication and storage, you can ensure the data is protected using best practices for authentication and encryption. The DevAlert data is uploaded in the same way as your regular application data, and thereby just as secure.
Since DevAlert focuses on the behavior of the device software and does not identify the device users, it is not subject to privacy legislation such as GDPR.
Note that Percepio Tracealyzer is a desktop software, so the trace data stays private at all times.
Cloud-Side Operational Costs
DevAlert does not generate any data traffic unless an issue is reported, so normally there is no extra data transmitted. Alerts are only uploaded occasionally and the amount of data is typically quite small, a few KB, and you can configure this as needed.
The alerts need to be stored for some time, but due to the small amount of data and today’s cheap cloud storage, this is a negligible cost, especially when compared to the value of the provided information. Say that you have a large fleet of one million devices, with 1 alert per device and week on average, and each alert is 5 KB. This would produce about 260 GB per year. Storing this data for a year would cost about $72, assuming Amazon S3 standard storage. But since most data can be deleted after a short time, the storage costs can be much lower.
Sending out OTA updates after improving the software can be somewhat more costly, at least for large device fleets. An OTA update of 600 000 devices would cost $1,275. But this isn’t much compared to the alternative cost of letting bugs and other issues remain unfixed – damaged customer experience, reduced product sales, or even accidents and legal action. And you may reduce the OTA update costs by collecting less critical patches in planned periodic updates.