Select the version of your OS from the tabs below. If you don't know the version you are using, run the command
cat /etc/os-release or
cat /etc/issue on the board.
Remember that you can always refer to the Torizon Documentation, there you can find a lot of relevant articles that might help you in the application development.
Device monitoring is a feature available in the Torizon platform that makes it possible to check the health and monitor the performance of a device running TorizonCore.
The device runs a monitoring agent that collects data about the different parts of the system (CPU and memory usage, networking statistics, storage, critical errors, running processes, etc) and sends it to Torizon OTA, where the user can visualize information about the device or fleet of devices.
When investigating the best option for the monitoring agent in TorizonCore, we looked for an option that would be modular, event-driven, based on known and widely adopted standards, open-source, and with acceptable performance and resource usage on constraint devices. And we decided to integrate Fluent Bit in TorizonCore.
Fluent Bit is an open-source log processor and forwarder, which allows to collect any data like metrics and logs from different sources (hardware and software), enrich them with filters, and send them to multiple destinations.
In Fluent Bit, information is processed in a pipeline, with a very pluggable architecture. Data is collected with input plugins, filtered with filter plugins, and sent to remote servers with output plugins:
Fluent Bit is written in C and designed with performance in mind (high throughput with low CPU and memory usage).
For more information about the project, see the Fluent Bit official documentation.
This article complies to the Typographic Conventions for the Toradex Documentation.
Attention: In this article, you will need to execute the commands as root. You can log in as root or (better) use the
sudo command when logged in with a regular user to do it.
Fluent Bit is integrated and enabled in TorizonCore 5.4.0 and later versions. By default, it's configured to monitor CPU, memory, temperature, and the docker daemon, and send the information to Torizon OTA.
When you first boot TorizonCore, the Fluent Bit service will not start due to the absence of the
# systemctl status fluent-bit * fluent-bit.service - Fluent Bit Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Tue 2021-09-14 07:42:15 UTC; 5h 19min ago `- ConditionPathExists=/etc/fluent-bit/enabled was not met Sep 14 07:42:15 colibri-imx6-10492785 systemd: Condition check resulted in Fluent Bit being skipped.
As soon as the device is provisioned to Torizon OTA, the provisioning script will enable the Fluent Bit service by creating the
/etc/fluent-bit/enabled file. After that, Fluent Bit will start collecting data and sending it to a local TCP port in the device (8850 by default), where a data proxy in Aktualizr-Torizon is listening to. Aktualizr-Torizon will receive this monitoring data, format it in JSON, and send it to Torizon OTA.
Device monitoring is already enabled by default in TorizonCore 5.4.0 and later versions. The only thing you need to do is provision the device to Torizon OTA. No extra steps are required.
TorizonCore 5.3.0 has all the required infrastructure for device monitoring, but it is not enabled by default.
To enable device monitoring in TorizonCore 5.3.0, the first step is to provision the device to Torizon OTA.
Then you have to create a Fluent Bit configuration file:
[SERVICE] flush 1 daemon Off log_level info parsers_file parsers.conf plugins_file plugins.conf [INPUT] name cpu tag cpu interval_sec 300 [FILTER] Name nest Match cpu Operation nest Wildcard * Nest_under cpu [INPUT] name mem tag memory interval_sec 300 [FILTER] Name nest Match memory Operation nest Wildcard * Nest_under memory [INPUT] name thermal tag temperature name_regex thermal_zone0 interval_sec 300 [FILTER] Name nest Match temperature Operation nest Wildcard * Nest_under temperature [INPUT] name proc proc_name dockerd tag proc_docker fd false mem false interval_sec 300 [FILTER] Name nest Match proc_docker Operation nest Wildcard * Nest_under docker [OUTPUT] name tcp port 8850 format json_lines match *
Enable the Fluent Bit service:
# systemctl enable fluent-bit # systemctl restart fluent-bit
And configure the data proxy in Aktualizr-Torizon:
# mkdir -p /etc/systemd/system/aktualizr-torizon.service.d/ # sh -c 'echo "[Service]" > /etc/systemd/system/aktualizr-torizon.service.d/override.conf' # sh -c 'echo "Environment=\"AKTUALIZR_CMDLINE_PARAMETERS=--enable-data-proxy\"" >> /etc/systemd/system/aktualizr-torizon.service.d/override.conf' # systemctl daemon-reload # systemctl restart aktualizr-torizon
Now the device should be ready to send monitoring data to Torizon OTA.
Due to the missing infrastructure to support device monitoring, it's not possible to enable it in TorizonCore 5.2.0 and earlier versions.
Disabling device monitoring can be easily done by simply disabling the Fluent Bit service:
# systemctl stop fluent-bit # systemctl disable fluent-bit
Changing device monitoring to collect more information about other parts of the system is not a feature currently supported in Torizon OTA, but can be done if you are willing to set up your own servers to receive and visualize data coming from the device.
For that, you can just change Fluent Bit configuration in
/etc/fluent-bit/fluent-bit.conf. Fluent Bit is compatible with the most popular Cloud providers and protocols (AWS, Microsoft Azure, Google Cloud, Datadog, Elasticsearch, etc). For more information about how to configure Fluent Bit, see the official documentation.