Search by Tags

Device Monitoring in TorizonCore


Article updated at 24 Aug 2021
Subscribe for this article updates

Select the version of your OS from the tabs below. If you don't know the version you are using, run the command cat /etc/os-release or cat /etc/issue on the board.

Remember that you can always refer to the Torizon Documentation, there you can find a lot of relevant articles that might help you in the application development.

Torizon 5.4.0


Device monitoring is a feature available in the Torizon platform that makes it possible to check the health and monitor the performance of a device running TorizonCore.

The device runs a monitoring agent that collects data about the different parts of the system (CPU and memory usage, networking statistics, storage, critical errors, running processes, etc) and sends it to Torizon OTA, where the user can visualize information about the device or fleet of devices.

When investigating the best option for the monitoring agent in TorizonCore, we looked for an option that would be modular, event-driven, based on known and widely adopted standards, open-source, and with acceptable performance and resource usage on constraint devices. And we decided to integrate Fluent Bit in TorizonCore.

Fluent Bit

Fluent Bit is an open-source log processor and forwarder, which allows to collect any data like metrics and logs from different sources (hardware and software), enrich them with filters, and send them to multiple destinations.

In Fluent Bit, information is processed in a pipeline, with a very pluggable architecture. Data is collected with input plugins, filtered with filter plugins, and sent to remote servers with output plugins:

  • Input plugins: gather and parse information from different sources (CPU, disk, memory, network, temperature, processes, kernel, logs, etc).
  • Filter plugins: allow altering the data before delivering it to some destination (remove, add, change, nest, etc).
  • Output plugins: allow defining a destination for the data (Prometheus, Amazon, Azure, Google Cloud, Datadog, Elasticsearch, HTTP(S), Kafka, etc).

Fluent Bit is written in C and designed with performance in mind (high throughput with low CPU and memory usage).

For more information about the project, see the Fluent Bit official documentation.

This article complies to the Typographic Conventions for the Toradex Documentation.

Attention: In this article, you will need to execute the commands as root. You can log in as root or (better) use the sudo command when logged in with a regular user to do it.

Device monitoring implementation in TorizonCore

Fluent Bit is integrated and enabled in TorizonCore 5.4.0 and later versions. By default, it's configured to monitor CPU, memory, temperature, and the docker daemon, and send the information to Torizon OTA.

When you first boot TorizonCore, the Fluent Bit service will not start due to the absence of the /etc/fluent-bit/enabled file:

# systemctl status fluent-bit
* fluent-bit.service - Fluent Bit
     Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Tue 2021-09-14 07:42:15 UTC; 5h 19min ago
             `- ConditionPathExists=/etc/fluent-bit/enabled was not met

Sep 14 07:42:15 colibri-imx6-10492785 systemd[1]: Condition check resulted in Fluent Bit being skipped.

As soon as the device is provisioned to Torizon OTA, the provisioning script will enable the Fluent Bit service by creating the /etc/fluent-bit/enabled file. After that, Fluent Bit will start collecting data and sending it to a local TCP port in the device (8850 by default), where a data proxy in Aktualizr-Torizon is listening to. Aktualizr-Torizon will receive this monitoring data, format it in JSON, and send it to Torizon OTA.

Enabling device monitoring in TorizonCore 5.4.0 and later versions

Device monitoring is already enabled by default in TorizonCore 5.4.0 and later versions. The only thing you need to do is provision the device to Torizon OTA. No extra steps are required.

Enabling device monitoring in TorizonCore 5.3.0

TorizonCore 5.3.0 has all the required infrastructure for device monitoring, but it is not enabled by default.

To enable device monitoring in TorizonCore 5.3.0, the first step is to provision the device to Torizon OTA.

Then you have to create a Fluent Bit configuration file:

    flush        1
    daemon       Off
    log_level    info
    parsers_file parsers.conf
    plugins_file plugins.conf
    name         cpu
    tag          cpu
    interval_sec 300
    Name       nest
    Match      cpu
    Operation  nest
    Wildcard   *
    Nest_under cpu
    name         mem
    tag          memory
    interval_sec 300
    Name       nest
    Match      memory
    Operation  nest
    Wildcard   *
    Nest_under memory
    name         thermal
    tag          temperature
    name_regex   thermal_zone0
    interval_sec 300
    Name       nest
    Match      temperature
    Operation  nest
    Wildcard   *
    Nest_under temperature
    name         proc
    proc_name    dockerd
    tag          proc_docker
    fd           false
    mem          false
    interval_sec 300
    Name       nest
    Match      proc_docker
    Operation  nest
    Wildcard   *
    Nest_under docker
    name   tcp
    port   8850
    format json_lines
    match  *

Enable the Fluent Bit service:

# systemctl enable fluent-bit
# systemctl restart fluent-bit

And configure the data proxy in Aktualizr-Torizon:

# mkdir -p /etc/systemd/system/aktualizr-torizon.service.d/
# sh -c 'echo "[Service]" > /etc/systemd/system/aktualizr-torizon.service.d/override.conf'
# sh -c 'echo "Environment=\"AKTUALIZR_CMDLINE_PARAMETERS=--enable-data-proxy\"" >> /etc/systemd/system/aktualizr-torizon.service.d/override.conf'
# systemctl daemon-reload
# systemctl restart aktualizr-torizon

Now the device should be ready to send monitoring data to Torizon OTA.

Enabling device monitoring in TorizonCore 5.2.0 and earlier versions

Due to the missing infrastructure to support device monitoring, it's not possible to enable it in TorizonCore 5.2.0 and earlier versions.

Disabling device monitoring

Disabling device monitoring can be easily done by simply disabling the Fluent Bit service:

# systemctl stop fluent-bit
# systemctl disable fluent-bit

Changing device monitoring configuration

Changing device monitoring to collect more information about other parts of the system is not a feature currently supported in Torizon OTA, but can be done if you are willing to set up your own servers to receive and visualize data coming from the device.

For that, you can just change Fluent Bit configuration in /etc/fluent-bit/fluent-bit.conf. Fluent Bit is compatible with the most popular Cloud providers and protocols (AWS, Microsoft Azure, Google Cloud, Datadog, Elasticsearch, etc). For more information about how to configure Fluent Bit, see the official documentation.