Building Machine Learning Software with Reference Images for Yocto Project
Introduction
In this article, we will show how to integrate to the Toradex Reference Images for Yocto Project Software the following AI runtimes:
Toradex BSP Version | meta-ml version | AI Runtimes |
---|---|---|
Quarterly: 6.0.0 | Based on NXP BSP L5.15.32_2.0.0 Download documentation (requires login) | TensorFlow Lite v2.8.0 ONNX Runtime 1.10.0 OpenCV 4.5.2 |
NXP eIQ
NXP eIQ software provides the basis for Machine Learning application optimized for i.MX SoCs, enabling Neural Network acceleration on NXP SoCs on the GPU or NPU through the OpenVX backend.
When executing inference on Cortex-A cores, NXP eIQ inference engines support multi-threaded execution.
eIQ based on NXP BSP L5.15.32_2.0.0
also supports DeepViewRT (install guide/documentation).
You can find more detailed information on the features of eIQ for each specific version on the i.MX Machine Learning User's Guide available on the NXP's Embedded Linux Documentation.
OpenVX provides NPU/GPU acceleration for all the eIQ AI runtimes, except OpenCV (as stated on the i.MX Machine Learning User's Guide).
TensorFlow Lite
As stated in the TensorFlow Lite Documentation:
TensorFlow Lite is a set of tools that enables on-device machine learning by helping developers run their models on mobile, embedded, and IoT devices.
In order to execute TensorFlow models with TensorFlow Lite, you need to use the TensorFlow Lite Converter. The TensorFlow Lite version needs to match the TensorFlow version used to design the model.
Not every TensorFlow model is directly convertible to TensorFlow Lite, because some TensorFlow operators (ops) do not have a TensorFlow Lite equivalent. However, in some situations, you can use a mix of TensorFlow and TensorFlow Lite ops by enabling the Select TensorFlow Ops feature. Please, see the TensorFlow Lite Documentation for more information about this feature and how to enable it.
TorizonCore
Currently, ONNX Runtime has not been validated to work on TorizonCore, to learn how to use Tensorflow-lite with Torizon, read the following article:
Pre-Requisites
- Verdin iMX8M Plus (CPU/GPU/NPU support).
- A compatible Carrier Board.
- Read the Build a Reference Image with Yocto Project article.
Adding eIQ recipes to Reference Images for Yocto Project
Cloning the Toradex BSP repository
In an empty directory, use git-repo to obtain the Toradex BSP on the version 6.0.0
, as explained in the section First-time Configuration of the Build a Reference Image with Yocto Project article:
To improve the comprehension of this article, we will create a directory named ~/yocto-ml-build
.
$ mkdir -p ~/yocto-ml-build/bsp-toradex
$ cd ~/yocto-ml-build/bsp-toradex
$ repo init -u git://git.toradex.com/toradex-manifest.git -b kirkstone-6.x.y -m tdxref/default.xml
$ repo sync
Source the file export
to setup the environment. On the first invocation, this also copies a sample configuration to build/conf/*.conf
$ . export
Getting eIQ
eIQ is provided on a Yocto layer called meta-imx/meta-ml.
The next steps expect the current directory to be <project-folder>/build
.
Git clone the meta-imx
repository to your project directory:
$ git clone --depth 1 -b kirkstone-5.15.32-2.0.0 https://github.com/nxp-imx/meta-imx.git ../meta-imx
Copying the Recipes to your environment
First, create a layer named meta-ml
, add it to your environment and remove the example recipe:
This step may fail due to missing packages in your computer:
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
lz4c
If that is the case, install the required packages and repeat this step.
$ bitbake-layers create-layer ../layers/meta-ml
$ bitbake-layers add-layer ../layers/meta-ml
$ rm -rf ../layers/meta-ml/recipes-example
Copy the recipes from meta-imx
to your layer:
$ cp -r ../meta-imx/meta-ml/recipes-* ../layers/meta-ml/
Adding the recipes to your distribution
Add the meta-ml
recipes to your image:
$ echo 'IMAGE_INSTALL:append += " tensorflow-lite tensorflow-lite-vx-delegate onnxruntime"' >> conf/local.conf
Add some image processing libraries to be able to execute additional image manipulations such as resize, crop, etc.:
$ echo 'IMAGE_INSTALL:append += " opencv python3-pillow adwaita-icon-theme "' >> conf/local.conf
In order to build the image a little bit faster, for now, we will remove the Qt packages. Keep it if you are planning to use Qt in your image.
$ echo 'IMAGE_INSTALL:remove = " packagegroup-tdx-qt5 wayland-qtdemo-launch-cinematicexperience "' >> conf/local.conf
Modify the TensorFlow Lite recipe as a workaround to incompatible build flags set by XNNPACK (TensorFlow Lite requirement):
$ sed -i "s/\(CXXFLAGS += \"-fPIC\"\)/&\\nTUNE_CCARGS:remove = \"-mcpu=cortex-a53\"\\nTUNE_CCARGS:append = \"-mtune=cortex-a53 -march=armv8-a+crc+simd+crypto\"\\n/" ../layers/meta-ml/recipes-libraries/tensorflow-lite/tensorflow-lite_2.8.0.bb
This command appends the following lines below CXXFLAGS += "-fPIC"
in the TensorFlow Lite recipe file:
TUNE_CCARGS:remove = "-mcpu=cortex-a53"
TUNE_CCARGS:append = "-mtune=cortex-a53 -march=armv8-a+crc+simd+crypto"
Which disables the forced optimizations for cortex-a53 in TensorFlow Lite, because XNNPACK builds for other arm architectures and selects the optimal functions at runtime. The command adds the A53 optimizations in a way that XNNPACK can override.
Configuring the Machine
If you want to build for a machine based on an NXP SoM, some downloads require you to read and accept the NXP/Freescale EULA available in <project-folder>/layers/meta-freescale/EULA
.
You have to state your acceptance by appending the following line to your <project-folder>/build/conf/local.conf
file:
ACCEPT_FSL_EULA = "1"
Select the SoM in <project-folder>/build/conf/local.conf
by uncommenting (removing the #
in) the line corresponding to your SoM:
MACHINE ?= "verdin-imx8mp"
Building
Build the tdx-reference-multimedia-image
image for your target SoM as explained on the Build a Reference Image with Yocto Project article:
$ bitbake tdx-reference-multimedia-image
In some situations of internet or server instability, trying to build may fail with:
do_fetch: Fetcher failure for URL:
In most cases, this issue is solved by re-trying to build.
Building with reduced RAM usage
Your computer may run out of RAM while compiling some packages (such as onnxruntime). To reduce the RAM usage, limit the number of threads used by BitBake and Make.
Flashing the image
To flash your image to the board, see the Quickstart Guide for your SoM.
Executing Demos
NXP provides an inference example, supporting CPU, GPU, and NPU.
To execute it, cd
to the example's directory:
# cd /usr/bin/tensorflow-lite-2.8.0/examples/
This demo will take an arbitrary picture (grace_hopper.bmp
) as an input of an image classification neural network based on
Mobilenet V1 (224x224 input size).
See more information about this demo on the NXP's i.MX Machine Learning User's Guide.
To run the demo:
# USE_GPU_INFERENCE=0 ./label_image -m mobilenet_v1_1.0_224_quant.tflite -i grace_hopper.bmp -l labels.txt --external_delegate_path=/usr/lib/libvx_delegate.so
# USE_GPU_INFERENCE=1 ./label_image -m mobilenet_v1_1.0_224_quant.tflite -i grace_hopper.bmp -l labels.txt --external_delegate_path=/usr/lib/libvx_delegate.so
# ./label_image -m mobilenet_v1_1.0_224_quant.tflite -i grace_hopper.bmp -l labels.txt
See below a comparison of Inference Time executing this demo:
Som | Inference Time | FPS (1/Inference Time) |
---|---|---|
Verdin iMX8M Plus - CPU only | 46.2 ms | 21.6 fps |
Verdin iMX8M Plus with GPU Support | 162 ms | 6.2 fps |
Verdin iMX8M Plus with NPU Support | 2.57 ms | 389.1 fps |
Alternatively, you can run the same example using a Python implementation:
# USE_GPU_INFERENCE=0 python3 label_image.py -e /usr/lib/libvx_delegate.so
# USE_GPU_INFERENCE=1 python3 label_image.py -e /usr/lib/libvx_delegate.so
# USE_GPU_INFERENCE=0 python3 label_image.py
As explained on the NXP's Application Note AN12964, the i.MX 8M Plus SoC requires an Warmup Time of about 7 seconds to initiate before delivering its expected high performance. You will observe this extra time when starting an application with NPU support.
Additional Resources
See the version-specific NXP's i.MX Machine Learning User's Guide for more information about eIQ.