Video Encoding and Playback With GStreamer (Linux)
Introduction
This article explains the software resources for multimedia playback and encoding on Toradex modules.
Video encoding and decoding (playback) are intensive tasks for the CPU, therefore various SoCs are equipped with dedicated hardware named Video Processing Units (VPU) or similar, and sometimes video processing operations can be done on the Graphics Processing Unit (GPU). Especially when using GStreamer, make sure that you are using plugins that support hardware acceleration whenever possible.
From BSP 5 Wayland/Weston is supported, as explained in the BSP Layers and Reference Images for Yocto Project Software. For GPU/VPU-enabled modules, GStreamer and Video4Linux2 (V4L2) are pre-installed in the Reference Multimedia Image.
If you don't want to do a custom OpenEmbedded build, consider using Torizon, where you can install libraries and packages inside Debian Containers. For information about how to build a container with GStreamer, refer to How to use Gstreamer on Torizon OS. If you are starting to work with cameras, refer to How to use Cameras on Torizon.
For more information about GStreamer, refer to the GStreamer Documentation. The GStreamer documentation - command line tools may be particularly useful to get started.
Prerequisites
- A Toradex SoM with the Reference Multimedia Image.
- A display or monitor connected to the board and enabled according to the Device Tree Overlays (Linux).
- A Qt demo must be automatically started on the enabled display. If you don't see anything, the display may not be correctly enabled.
- Disable the QT Demo application as described at Disable the Qt Demo Launched at Startup.
Disable the Qt Demo Launched at Startup
When using our Reference Multimedia Image, before you can play a video, you must stop and disable the Qt demo that is autorun at the startup:
# systemctl stop wayland-app-launch
# systemctl disable wayland-app-launch
Considerations about Multimedia on iMX Modules
NXP provides GStreamer plugins to access the multimedia libraries using the i.MX SoC's hardware acceleration units. You will find all of the details included on the page Embedded Linux for i.MX Applications Processors, especially on the Releases tab:
- i.MX Linux Release Notes: an overview of the current i.MX Gstreamer Plugins is available in the Multimedia section.
- i.MX Linux User's Guide: detailed information on how to use GStreamer, including several examples, is available in the Multimedia section.
Browse the documentation using the corresponding NXP BSP versions that match our BSP versions:
| Toradex BSP 5.x.y | NXP BSP |
|---|---|
| Up-to 5.1.0 | L5.4.24_2.1.0 |
| From 5.2.0 onward | L5.4.70_2.3.0 |
Command Line Tools
gst-launch-1.0
Quoting from GStreamer documentation:
gst-launch-1.0is a tool that builds and runs basic GStreamer pipelines.
In other words, the gst-launch-1.0 is a debugging tool that parses, builds and launches a GStreamer pipeline. It follows the basic structure below:
# gst-launch-1.0 <pipeline>
Pipelines
To summarize, a GStreamer pipeline is a way in which applications can interact with audio and video data. The two most basic elements of a pipeline are source, to generate and inject data into the pipeline, and sink, to get the data to an output. Intermediary components in the pipeline have both source and sink characteristics.
The ports through which GStreamer elements communicate with each other are called pads. There are two types of pads:
- Source pad: through which data exits an element.
- Sink pad: through which data enters an element.
For example, for we can take generic data from a source (appsrc) and connect directly to a sink:
# gst-launch-1.0 appsrc ! queue ! appsink
Following the same idea, we can also take data from a video or audio source, process, and take it to a sink element:
# gst-launch-1.0 videotestsrc ! video/x-raw ! autovideosink
# gst-launch-1.0 audiotestsrc ! audio/x-raw ! autoaudiosink
Or even convert the data before take it to a sink element:
# gst-launch-1.0 videotestsrc ! video/x-raw ! queue ! videoconvert ! autovideosink
# gst-launch-1.0 audiotestsrc ! audio/x-raw ! queue ! audioconvert ! audioresample ! autoaudiosink
gst-inspect-1.0
Quoting from GStreamer documentation:
gst-inspect-1.0is a tool that prints out information on available GStreamer plugins, information about a particular plugin, or information about a particular element.
# gst-inspect-1.0 <plugin/element>
gst-inspect-1.0 could be a helpful tool when developing a GStreamer pipeline. For example, you can use it to learn how to configure video source pads and capabilities:
# gst-inspect-1.0 videotestsrc
gst-inspect-1.0 videotestsrc output
Factory Details:
Rank none (0)
Long-name Video test source
Klass Source/Video
Description Creates a test video stream
Author David A. Schleef <ds@schleef.org>
Plugin Details:
Name videotestsrc
Description Creates a test video stream
Filename /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstvideotestsrc.so
Version 1.18.4
License LGPL
Source module gst-plugins-base
Source release date 2021-03-15
Binary package GStreamer Base Plugins (Debian)
Origin URL http://packages.qa.debian.org/gst-plugins-base1.0
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseSrc
+----GstPushSrc
+----GstVideoTestSrc
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-raw
format: { (string)AYUV64, (string)ARGB64, (string)GBRA_12LE, (string)GBRA_12BE, (string)Y412_LE, (string)Y412_BE, (string)A444_10LE, (string)GBRA_10LE, (string)A444_10BE, (string)GBRA_10BE, (string)A422_10LE, (string)A422_10BE, (string)A420_10LE, (string)A420_10BE, (string)RGB10A2_LE, (string)BGR10A2_LE, (string)Y410, (string)GBRA, (string)ABGR, (string)VUYA, (string)BGRA, (string)AYUV, (string)ARGB, (string)RGBA, (string)A420, (string)Y444_16LE, (string)Y444_16BE, (string)v216, (string)P016_LE, (string)P016_BE, (string)Y444_12LE, (string)GBR_12LE, (string)Y444_12BE, (string)GBR_12BE, (string)I422_12LE, (string)I422_12BE, (string)Y212_LE, (string)Y212_BE, (string)I420_12LE, (string)I420_12BE, (string)P012_LE, (string)P012_BE, (string)Y444_10LE, (string)GBR_10LE, (string)Y444_10BE, (string)GBR_10BE, (string)r210, (string)I422_10LE, (string)I422_10BE, (string)NV16_10LE32, (string)Y210, (string)v210, (string)UYVP, (string)I420_10LE, (string)I420_10BE, (string)P010_10LE, (string)NV12_10LE32, (string)NV12_10LE40, (string)P010_10BE, (string)Y444, (string)GBR, (string)NV24, (string)xBGR, (string)BGRx, (string)xRGB, (string)RGBx, (string)BGR, (string)IYU2, (string)v308, (string)RGB, (string)Y42B, (string)NV61, (string)NV16, (string)VYUY, (string)UYVY, (string)YVYU, (string)YUY2, (string)I420, (string)YV12, (string)NV21, (string)NV12, (string)NV12_64Z32, (string)NV12_4L4, (string)NV12_32L32, (string)Y41B, (string)IYU1, (string)YVU9, (string)YUV9, (string)RGB16, (string)BGR16, (string)RGB15, (string)BGR15, (string)RGB8P, (string)GRAY16_LE, (string)GRAY16_BE, (string)GRAY10_LE32, (string)GRAY8 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
multiview-mode: { (string)mono, (string)left, (string)right }
video/x-bayer
format: { (string)bggr, (string)rggb, (string)grbg, (string)gbrg }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
multiview-mode: { (string)mono, (string)left, (string)right }
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SRC: 'src'
Pad Template: 'src'
Element Properties:
animation-mode : For pattern=ball, which counter defines the position of the ball.
flags: readable, writable
Enum "GstVideoTestSrcAnimationMode" Default: 0, "frames"
(0): frames - frame count
(1): wall-time - wall clock time
(2): running-time - running time
background-color : Background color to use (big-endian ARGB)
flags: readable, writable, controllable
Unsigned Integer. Range: 0 - 4294967295 Default: 4278190080
blocksize : Size in bytes to read per buffer (-1 = default)
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 4096
do-timestamp : Apply current stream time to buffers
flags: readable, writable
Boolean. Default: false
flip : For pattern=ball, invert colors every second.
flags: readable, writable
Boolean. Default: false
foreground-color : Foreground color to use (big-endian ARGB)
flags: readable, writable, controllable
Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295
horizontal-speed : Scroll image number of pixels per frame (positive is scroll to the left)
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0
is-live : Whether to act as a live source
flags: readable, writable
Boolean. Default: false
k0 : Zoneplate zero order phase, for generating plain fields or phase offsets
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0
kt : Zoneplate 1st order t phase, for generating phase rotation as a function of time
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0
kt2 : Zoneplate 2nd order t phase, t*t/256 cycles per picture
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0
kx : Zoneplate 1st order x phase, for generating constant horizontal frequencies
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0
kx2 : Zoneplate 2nd order x phase, normalised to kx2/256 cycles per horizontal pixel at width/2 from origin
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0
kxt : Zoneplate x*t product phase, normalised to kxy/256 cycles per vertical pixel at width/2 from origin
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0
kxy : Zoneplate x*y product phase
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0
ky : Zoneplate 1st order y phase, for generating content vertical frequencies
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0
ky2 : Zoneplate 2nd order y phase, normailsed to ky2/256 cycles per vertical pixel at height/2 from origin
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0
kyt : Zoneplate y*t product phase
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0
motion : For pattern=ball, what motion the ball does
flags: readable, writable
Enum "GstVideoTestSrcMotionType" Default: 0, "wavy"
(0): wavy - Ball waves back and forth, up and down
(1): sweep - 1 revolution per second
(2): hsweep - 1/2 revolution per second, then reset to top
name : The name of the object
flags: readable, writable, 0x2000
String. Default: "videotestsrc0"
num-buffers : Number of buffers to output before sending EOS (-1 = unlimited)
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
parent : The parent of the object
flags: readable, writable, 0x2000
Object of type "GstObject"
pattern : Type of test pattern to generate
flags: readable, writable
Enum "GstVideoTestSrcPattern" Default: 0, "smpte"
(0): smpte - SMPTE 100% color bars
(1): snow - Random (television snow)
(2): black - 100% Black
(3): white - 100% White
(4): red - Red
(5): green - Green
(6): blue - Blue
(7): checkers-1 - Checkers 1px
(8): checkers-2 - Checkers 2px
(9): checkers-4 - Checkers 4px
(10): checkers-8 - Checkers 8px
(11): circular - Circular
(12): blink - Blink
(13): smpte75 - SMPTE 75% color bars
(14): zone-plate - Zone plate
(15): gamut - Gamut checkers
(16): chroma-zone-plate - Chroma zone plate
(17): solid-color - Solid color
(18): ball - Moving ball
(19): smpte100 - SMPTE 100% color bars
(20): bar - Bar
(21): pinwheel - Pinwheel
(22): spokes - Spokes
(23): gradient - Gradient
(24): colors - Colors
timestamp-offset : An offset added to timestamps set on buffers (in ns)
flags: readable, writable
Integer64. Range: 0 - 9223372036854775807 Default: 0
typefind : Run typefind before negotiating (deprecated, non-functional)
flags: readable, writable, deprecated
Boolean. Default: false
xoffset : Zoneplate 2nd order products x offset
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0
yoffset : Zoneplate 2nd order products y offset
flags: readable, writable
Integer. Range: -2147483648 - 2147483647 Default: 0
For example, you can configure video/x-raw or video/x-bayer in terms of format,height, width, framerate, multiview-mode. The pipeline would look like the following example:
# gst-launch videotestsrc ! "video/x-raw, format=<video-format>, width=<supported-width>, height=<supported-height>, framerate=<framerate>, multiview-mode=<supported-mode>" ! autovideosink
You can also use gst-inspect-1.0 without an argument and grep to filter patterns. For example, you can serch for video4linux plugins, audio sources, and others:
# gst-inspect-1.0 | v4l2
video4linux2: v4l2src: Video (video4linux2) Source
video4linux2: v4l2sink: Video (video4linux2) Sink
video4linux2: v4l2radio: Radio (video4linux2) Tuner
video4linux2: v4l2deviceprovider (GstDeviceProviderFactory)
# gst-inspect-1.0 | grep audiosrc
jack: jackaudiosrc: Audio Source (Jack)
autodetect: autoaudiosrc: Auto audio source
Video Playback
Some videos are included in the /home/root/video directory of the Reference Multimedia Image, for easy evaluation:
# cd /home/root/video
# ls
You can play them with gplay-1.0, gst-play-1.0 or gst-launch-1.0, depending on your SoM:
Keep in mind that, depending on the SoM capabilities, a video with a specific encoding may not play. Try different formats.
# gplay-1.0 <video file>
# gst-play-1.0 <video file>
# gst-launch-1.0 playbin uri=file://<video file>
For more advanced use cases, consult the GStreamer documentation, as suggested in the previous section.
If you want to learn how to configure your screen, take a look at Working with Weston.
Video Encoding and Decoding
Encoding refers to the process of converting raw video/image to a more compact format. As well, decoding refers to the process of uncompressing and converting the encoded media to a raw format or, in other words, a format that can be displayed on a screen.
To encode a video, for example, it can be done as follows:
# gst-launch-1.0 videotestsrc ! video/x-raw ! x264enc ! video/x-h264 ! fakesink
To decode a video file and send it to a sink, it can be done using decodebin as follow:
# gst-launch-1.0 filesrc location=<encoded_file_location> ! queue ! decodebin ! videoconvert ! autovideosink
For more information about encoding and decoding plugins (encodebin, x264enc, decodebin, and others) and other pipeline elements, you can use the gst-inspect-1.0 tool.
Limitations
When using Gstreamer for video decoding, the pipelines it creates by using tools as gst-play-1.0 and gplay-1.0 tend to prioritize software-based video decoding and colorspace conversion, which can lead to a performance limitation of approximately 1 frame per second. It can be seen in a situation such as the following:
# gst-play-1.0 video/testvideo_h264.mp4
...
0:00:10.0 / 0:00:10.0
Reached end of play list.
Total showed frames (8), playing for (0:00:10.051291666), fps (0.796).
However, by creating a dedicated pipeline, this problem can be overcome, as in the following examples:
# gst-launch-1.0 filesrc location=video/testvideo_h264.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! imxvideoconvert_g2d ! queue ! waylandsink
...
Total showed frames (301), playing for (0:00:10.033499117), fps (30.000).
Freeing pipeline ...
# gst-play-1.0 video/testvideo_h264.mp4 --videosink "imxvideoconvert_g2d ! waylandsink"
0:00:10.0 / 0:00:10.0
Reached end of play list.
Total showed frames (300), playing for (0:00:10.051500000), fps (29.846).
Additional Resources
If you are looking for information about the GStreamer debbuging tools, please refer to Debugging tools article on GStreamer documentation.
If you are looking for information on how to use cameras with containers in Torizon OS, refer to the article How to use Cameras on Torizon.
If you are looking for information on how to use the CSI Camera Set 5MP AR0521 Color, refer to First Steps with CSI Camera Set 5MP AR0521 Color (Linux) or First Steps with CSI Camera Set 5MP AR0521 Color (Torizon).
If you want to use a webcam, read the article Webcam (Linux) in addition to the current article.
If you want to stream video over the network, read the article Audio/Video over RTP With GStreamer (Linux).