Build TensorFlow Lite Applications With NPU Support on i.MX 93-Based Modules on Torizon OS

Introduction

This article covers the basic workflow to build containerized TensorFlow Lite applications with Neural Processing Unit (NPU) support on i.MX 93-based modules running Torizon OS.

Prerequisites

• Hardware Prerequisites:
  • An i.MX 93-based Toradex System-on-Module (SoM)
  • A compatible Carrier Board
• Software Prerequisites:
  • A Toradex System-on-Module with Torizon OS installed
  • A configured build environment, as described in the Configure Build Environment for Torizon Containers article
  • A quantized .tflite model supported by the Ethos-U delegate
    • CPU fallback remains available for models that are not compatible with the NPU

Build TensorFlow Lite Applications

On your host machine, create a Dockerfile to set up the necessary dependencies to run TensorFlow Lite with NPU support. The produced container includes the runtime provided by NXP, which is designed to run on any hardware accelerator available on the system.

tip

You do not need to use Toradex's packages to train your models. Toradex recommends using the upstream TensorFlow libraries for training.

Dockerfile

ARG IMAGE_ARCH=linux/arm64

FROM --platform=$IMAGE_ARCH torizon/debian-imx93:4

# Define build arguments for the application
ARG TF_LITE_MODEL="mobilenet_v1_1.0_224_quant.tflite"
ARG TF_LITE_MODEL_ETHOSU="mobilenet_v1_1.0_224_quant_vela.tflite"
ARG IMAGE_DATASET="grace_hopper.bmp"
ARG LABELMAP_FILE="labels.txt"
ARG INFERENCE_SCRIPT="label_image.py"

ARG WORKDIR_PATH="/app"

# Environment setup
ENV DEBIAN_FRONTEND=noninteractive \
    PYTHONDONTWRITEBYTECODE=1      \
    PYTHONUNBUFFERED=1

WORKDIR ${WORKDIR_PATH}

# Install dependencies
RUN apt-get update &&                          \
    apt-get install -y --no-install-recommends \
        \
        # Python dependencies
        python3      \
        python3-venv \
        python3-pip  \
        \
        # Python libraries
        python3-numpy \
        python3-pil   \
        \
        # TensorFlow Lite dependencies
        libtensorflow-lite2.16.2        \
        tensorflow-lite-ethosu-delegate \
        ethos-u-driver-stack            \
        python3-tflite-runtime          \
        \
    && rm -rf /var/lib/apt/lists/*

# Set up a Python virtual environment and install the TensorFlow Lite runtime
RUN python3 -m venv ${WORKDIR_PATH}/.venv --system-site-packages

# Install Python dependencies
RUN . ${WORKDIR_PATH}/.venv/bin/activate &&     \
    pip install --no-cache-dir --upgrade pip

# Copy the application code into the container
COPY ${TF_LITE_MODEL} ${WORKDIR_PATH}
COPY ${TF_LITE_MODEL_ETHOSU} ${WORKDIR_PATH}
COPY ${LABELMAP_FILE} ${WORKDIR_PATH}
COPY ${IMAGE_DATASET} ${WORKDIR_PATH}

COPY ${INFERENCE_SCRIPT} ${WORKDIR_PATH}

As an example, this article uses the NXP-provided object detection demo. Download the required resources using the commands below:

1. Download the input image:

   $ curl -L https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/lite/examples/label_image/testdata/grace_hopper.bmp -o grace_hopper.bmp

2. Download and extract the model:

   $ curl -L https://storage.googleapis.com/download.tensorflow.org/models/mobilenet_v1_2018_02_22/mobilenet_v1_1.0_224_quant.tgz | tar xz --wildcards --no-anchored '*.tflite'

3. Download the converted model to work with Ethos-U, or convert the preceding model using the Ethos-U Vela compiler provided by ARM:

   $ curl -L https://docs.toradex.com/119248-mobilenet_v1_1.0_224_quant_vela.tflite -o mobilenet_v1_1.0_224_quant_vela.tflite

4. Download and extract the labels file:

   $ curl -L https://storage.googleapis.com/download.tensorflow.org/models/mobilenet_v1_1.0_224_frozen.tgz | tar xz --strip-components=1 --wildcards --no-anchored '*/labels.txt'

5. Download the inference script:

   $ curl -L https://raw.githubusercontent.com/nxp-imx/tensorflow-imx/lf-6.12.49_2.2.0/tensorflow/lite/examples/python/label_image.py -o label_image.py

warning

Define all required build arguments in the Dockerfile before building the image. The TF_LITE_MODEL, TF_LITE_MODEL_ETHOSU, IMAGE_DATASET, LABELMAP_FILE, and INFERENCE_SCRIPT arguments must be set to valid file paths.

Finally, build the container image using the command below. Make sure to replace <username> and <image-name> with your Docker Hub username and the name for your image, respectively.

$ docker build -t <username>/<image-name> .

$ docker push <username>/<image-name>

Run the Application

Once you have built and pushed your Docker image to the container registry, you can run the container on your i.MX 93-based module.

info

Running TensorFlow Lite applications with NPU support on i.MX 93-based modules requires a module-specific NPU overlay:

• Lino iMX93: lino-imx93_npu_overlay.dtbo
• OSM iMX93: toradex-osm-imx93_npu_overlay.dtbo

Refer to the TorizonCore Builder documentation to enable the required overlay.

warning

The NXP downstream driver for the Ethos-U NPU requires the Cortex-M33 core to run properly. As a result, the Cortex-M33 core is not available for other purposes while the NPU is in use.

Use the following command to start and attach to the container with the necessary permissions and environment variables to access the NPU:

# docker run -it                        \
    -v /dev:/dev                        \
    --device-cgroup-rule "c 240:0 rmw"  \
    <username>/<image-name>

Finally, execute the TensorFlow Lite demo application inside the container:

## source .venv/bin/activate

## python3 label_image.py --model_file mobilenet_v1_1.0_224_quant_vela.tflite --image grace_hopper.bmp --label_file labels.txt --ext_delegate=/usr/lib/aarch64-linux-gnu/libethosu_delegate.so

Run the following commands to execute the application using the other hardware accelerator available on the system:

XNNPack (CPU)

## python3 label_image.py --model_file mobilenet_v1_1.0_224_quant.tflite --image grace_hopper.bmp --label_file labels.txt

Performance Comparison

The following table provides a performance comparison of the inference times for the CPU and NPU on the i.MX 93-based modules when running the TensorFlow Lite demo application:

HW accelerator	Inference Time	FPS (1/Inference Time)
XNNPack (CPU)	52.3 ms	19.1
NPU	4.6 ms	217.4

Additional Resources

• AI, Computer Vision, and Machine Learning on Toradex Computer on Modules
• Run the TensorFlow Lite Demo Application With NPU Support on Torizon OS
• Object Detection AI Training and Inference for Toradex Modules