Build TensorFlow Lite Applications With NPU Support on i.MX 95-Based Modules on Torizon OS

Introduction

This article covers the basic workflow to build containerized TensorFlow Lite applications with Neural Processing Unit (NPU) support on i.MX 95-based modules running Torizon OS.

Prerequisites

• Hardware Prerequisites:

  • An i.MX 95-based Toradex System-on-Module (SoM)
  • A compatible Carrier Board

• Software Prerequisites:

  • A Toradex System-on-Module with Torizon OS installed
  • A configured build environment, as described in the Configure Build Environment for Torizon Containers article
  • A quantized .tflite model supported by the Neutron delegate
    • CPU fallback remains available for models that are not compatible with the NPU

info

For i.MX 95 modules, Torizon OS must be built based on the Walnascar Yocto release. Refer to the AI, Computer Vision and Machine Learning on Toradex i.MX 95-based Modules article to set up a Walnascar Torizon Build.

Build TensorFlow Lite Applications

On your host machine, create a Dockerfile to set up the necessary dependencies to run TensorFlow Lite with NPU support. The produced container includes the runtime provided by NXP, which is designed to run on any hardware accelerator available on the system.

tip

You do not need to use Toradex's packages to train your models. Toradex recommends using the upstream TensorFlow libraries for training.

Dockerfile

ARG IMAGE_ARCH=linux/arm64

FROM --platform=$IMAGE_ARCH torizon/debian-imx95:4

# Define build arguments for the application
ARG TF_LITE_MODEL="mobilenet_v1_1.0_224_quant.tflite"
ARG TF_LITE_MODEL_NEUTRON="mobilenet_v1_1.0_224_quant_converted.tflite"
ARG IMAGE_DATASET="grace_hopper.bmp"
ARG LABELMAP_FILE="labels.txt"
ARG INFERENCE_SCRIPT="label_image.py"
ARG INFERENCE_BIN="label_image"

ARG WORKDIR_PATH="/app"

# Environment setup
ENV DEBIAN_FRONTEND=noninteractive \
    PYTHONDONTWRITEBYTECODE=1      \
    PYTHONUNBUFFERED=1

WORKDIR ${WORKDIR_PATH}

# Install dependencies
RUN apt-get update &&                          \
    apt-get install -y --no-install-recommends \
        \
        # Python dependencies
        python3      \
        python3-venv \
        python3-pip  \
        \
        # Python libraries
        python3-numpy \
        python3-pil   \
        \
        # TensorFlow Lite dependencies
        libtensorflow-lite2.19.0         \
        tensorflow-lite-neutron-delegate \
        neutron                          \
        python3-tflite-runtime           \
        \
    && rm -rf /var/lib/apt/lists/*

# Set up a Python virtual environment and install the TensorFlow Lite runtime
RUN python3 -m venv ${WORKDIR_PATH}/.venv --system-site-packages

# Install Python dependencies
RUN . ${WORKDIR_PATH}/.venv/bin/activate &&     \
    pip install --no-cache-dir --upgrade pip

# Copy the application code into the container
COPY ${TF_LITE_MODEL} ${WORKDIR_PATH}
COPY ${TF_LITE_MODEL_NEUTRON} ${WORKDIR_PATH}
COPY ${LABELMAP_FILE} ${WORKDIR_PATH}
COPY ${IMAGE_DATASET} ${WORKDIR_PATH}

COPY ${INFERENCE_SCRIPT} ${WORKDIR_PATH}
COPY ${INFERENCE_BIN} ${WORKDIR_PATH}

As an example, this article uses the NXP-provided object detection demo. Download the required resources using the commands below:

1. Download the input image:

   $ curl -L https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/lite/examples/label_image/testdata/grace_hopper.bmp -o grace_hopper.bmp

2. Download and extract the model:

   $ curl -L https://storage.googleapis.com/download.tensorflow.org/models/mobilenet_v1_2018_02_22/mobilenet_v1_1.0_224_quant.tgz | tar xz --wildcards --no-anchored '*.tflite'

3. Download the converted model to work with Neutron, or convert the preceding model using the eIQ Tool provided by NXP:

   $ curl -L https://docs.toradex.com/118984-mobilenet_v1_1.0_224_quant_converted.tflite -o mobilenet_v1_1.0_224_quant_converted.tflite

4. Download and extract the labels file:

   $ curl -L https://storage.googleapis.com/download.tensorflow.org/models/mobilenet_v1_1.0_224_frozen.tgz | tar xz --strip-components=1 --wildcards --no-anchored '*/labels.txt'

5. Download the inference script:

   $ curl -L https://raw.githubusercontent.com/nxp-imx/tensorflow-imx/lf-6.12.49_2.2.0/tensorflow/lite/examples/python/label_image.py -o label_image.py

6. Download the label_image binary file or compile it following the instructions available at TensorFlow C++ and Python Image Recognition Demo:

   $ curl -L https://docs.toradex.com/118985-label_image.bin -o label_image && chmod +x label_image

warning

Define all required build arguments in the Dockerfile before building the image. The TF_LITE_MODEL, TF_LITE_MODEL_NEUTRON, IMAGE_DATASET, LABELMAP_FILE, INFERENCE_SCRIPT and INFERENCE_BIN arguments must be set to valid file paths.

Finally, build the container image using the command below. Make sure to replace <username> and <image-name> with your Docker Hub username and the name for your image, respectively.

$ docker build -t <username>/<image-name> .

$ docker push <username>/<image-name>

Run the Application

Once you have built and pushed your Docker image to the container registry, you can run the container on your i.MX 95-based module. Use the following command to start and attach to the container with the necessary permissions and environment variables to access the NPU:

# docker run -it                        \
    -v /dev:/dev                        \
    --device-cgroup-rule "c 239:0 rmw"  \
    <username>/<image-name>

Finally, execute the TensorFlow Lite demo application inside the container:

## source .venv/bin/activate

## python3 label_image.py --model_file mobilenet_v1_1.0_224_quant_converted.tflite --image grace_hopper.bmp --label_file labels.txt --ext_delegate=/usr/lib/aarch64-linux-gnu/libneutron_delegate.so

Run the following commands to execute the application using the other hardware accelerators available on the system:

CPU

## python3 label_image.py --model_file mobilenet_v1_1.0_224_quant.tflite --image grace_hopper.bmp --label_file labels.txt

GPU

## ./label_image                                     \
    --tflite_model mobilenet_v1_1.0_224_quant.tflite \
    --labels labels.txt                              \
    --image grace_hopper.bmp                         \
    --use_gpu=true                                   \
    --gpu_backend=cl                                 \
    --gpu_precision_loss_allowed=true                \
    --gpu_experimental_enable_quant=true             \
    --gpu_inference_for_sustained_speed=false

Performance Comparison

The following table provides a performance comparison of the inference times for the CPU, GPU, and NPU on the SMARC iMX95 module when running the TensorFlow Lite demo application:

SoM	HW accelerator	Inference Time	FPS (1/Inference Time)
SMARC iMX95	XNNPack (CPU)	16.71 ms	59.84 FPS
SMARC iMX95	GPU	46.21 ms	21.64 FPS
SMARC iMX95	NPU	1.30 ms	769.23 FPS

info

The CPU outperforms the GPU due to the small model size. GPUs have higher overhead and are better suited for larger models.

Additional Resources

• AI, Computer Vision, and Machine Learning on Toradex Computer on Modules
• AI, Computer Vision and Machine Learning on Toradex i.MX 95-based Modules
• Run the TensorFlow Lite Demo Application With NPU Support on Torizon OS
• Object Detection AI Training and Inference for Toradex Modules