Search by Tags

Linux - Booting

 

Compare with Revision




Subscribe for this article updates

Boot Sequence

The boot sequence is something which isn't standardised in the ARM world. Each SoC vendor uses its own custom boot ROM implementation. However, most ARM SoC boot sequence execute roughly similar steps after reset:

  • The CPU internal boot ROM reads and latches the values present at GPIOs with strapping functionality.
  • The state of the strapping pins and SoC internal fuse settings are used to determine the boot device used for booting.
  • Configuration data is read from the start of the boot device. The data is used to configure the DDR RAM and locate the boot loader.
  • The DDR RAM gets initialized by the boot ROM.
  • The boot loader is copied from the boot device to RAM and control is passed to the boot loader. This is the earliest where customised code (in the boot loader) can be executed.

The Colibri Vybrid based modules have a slightly different boot sequence: The boot loader is directly copied to the SoC internal SRAM before any DDR RAM initialization. The boot loader then initializes the DDR RAM.

The Tegras use the term 'boot configuration table (BCT)' for the configuration data. The NXP/Freescale SoCs i.MX 6 and Vybrid use 'image vector table (IVT)', 'device configuration data (DCD)', and if booting from NAND additionally a 'boot control blocks (BCB)' data structure.

We use a slightly customized U-Boot as boot loader. (U-Boot)

  • U-Boot reads its environment variable from the boot device. If these data is corrupted or inexistent (e.g. has never been explicitly written) it prints '*** Warning - bad CRC, using default environment' and loads initial settings.
  • U-Boot analyses the environment variables to find out where the kernel and the rootfs are stored and what kernel command line is requested.
  • U-Boot auto detects the available RAM size and eMMC/NAND flash size and properties.
  • U-Boot sets the Ethernet MAC address and configures the hardware in preparation for the Linux kernel.
  • U-Boot loads the kernel into RAM and passes control to the kernel.
  • The kernel initializes Linux, mounts the rootfs and starts 'init' to initialize the Linux user space.

Boot Scenarios

  • Boot device is the eMMC or NAND flash provided on the module. U-Boot, kernel and rootfs are stored on the boot medium. The U-Boot environment in our images is set for this scenarios.
  • Boot device is a SD card. U-Boot, kernel and rootfs are stored on the boot medium. The strapping pins have to be set accordingly (Colibri T20/T30 only).
  • By changing the U-Boot environment the storage location of the kernel and rootfs can be changed to something different than the boot device.
  • The kernel can be loaded from eMMC/NAND flash, SD card, USB memory stick or a TFTP server.
  • The rootfs can be on eMMC/NAND flash, SD card, USB memory stick or a NFS server.

Common Combinations

  • Everything on eMMC/NAND flash –> Regular production and testing use.
  • Everything on SD card -> When you want to change between different OS', i.e. WinCE on eMMC/NAND flash, Linux on SD card (Colibri T20/T30 only).
  • U-Boot on eMMC/NAND flash, kernel and rootfs on SD card -> When you want to change between different kernels / rootfs'.
  • U-Boot on eMMC/NAND flash, kernel from TFTP and rootfs from NFS -> During kernel and application development and debugging.

See the release notes of our Linux image for instructions on how to activate eMMC/NAND flash or SD card boot on a Colibri T20/T30. For the other scenarios the U-Boot environment of a module that either does eMMC/NAND flash or SD card boot is modified as described below.

Initramfs

The rootfs can also live in RAM. Such a rootfs is loaded into RAM by the bootloader. Any changes to the rootfs content will be reverted by rebooting the module. The bootloader can load the file containing the initramfs from the same sources as it can load the kernel.

For more info see Initramfs and tmpfs.

U-Boot

Console

U-Boot's console is available on the primary aka Full Function (FF) UART aka UART1 on Apalis resp. UART_A on Colibris.

While U-Boot's console output is also visible on the parallel RGB display (and with the carrier boards RAMDAC on VGA, not supported on Apalis/Colibri T30 so far) a USB keyboard directly connected to the module does not work as of yet.

Environment Variables

The following listing shows the content of the U-Boot environment as present in our T20_LinuxImageV2.0 for NAND flash boot.

Tegra2 # printenv
baudrate=115200
bootcmd=run flashboot; run nfsboot
bootdelay=5
defargs=video=tegrafb vmalloc=128M usb_high_speed=1
ethaddr=00:14:2d:48:8a:58
fdtaddr=182ac8
flashargs=ip=off root=/dev/mtdblock0 rw rootfstype=yaffs2
flashboot=run setup; setenv bootargs ${defargs} ${flashargs} ${mtdparts} ${setupargs};\
 echo Booting from NAND...; nboot ${loadaddr} 0 ${lnxoffset} && bootm
ipaddr=192.168.10.2
lnxoffset=0xc80000
loadaddr=0x408000
memargs=mem=372M@0M fbmem=12M@372M nvmem=128M@384M
mmcboot=echo Loading RAM disk and kernel from MMC/SD card...; mmc init && fatload\
 mmc 0:1 0xC08000 rootfs-ext2.img.gz && fatload mmc 0:1 ${loadaddr} uImage;run ramboot
mtdparts=mtdparts=tegra_nand:1023744K@23808K(USR),3072K@0K(BCT),256K@4096K(PT),\
2048K@5376K(EBT),256K@8448K(BMP),2048K@9728K(ENV),8192K@12800K(LNX),256K@22272K(ARG)
netmask=255.255.255.0
nfsargs=ip=:::::eth0:on root=/dev/nfs rw netdevwait
nfsboot=run setup; setenv bootargs ${defargs} ${nfsargs} ${mtdparts} ${setupargs};\
 echo Booting from NFS...; usb start; dhcp; bootm
ramargs=initrd=0xA1800000,32M ramdisk_size=32768 root=/dev/ram0 rw
ramboot=run setup; setenv bootargs ${defargs} ${ramargs} ${mtdparts} ${setupargs};\
 echo Booting from RAM...; bootm
sdargs=root=/dev/mmcblk0p1 ip=:::::eth0:off rw,noatime rootfstype=ext3 rootwait gpt gpt_sector=18945
sdboot=run setup; setenv bootargs ${defargs} ${sdargs} ${mtdparts} ${setupargs};\
 echo Booting from MMC/SD card...; mmc read 0 ${loadaddr} 0x2a00 0x4000; bootm
serverip=192.168.10.1
setup=setenv setupargs asix_mac=${ethaddr} no_console_suspend=1 console=tty1 console=ttyS0,${baudrate}n8 debug_uartport=lsport,0 ${memargs}
stderr=serial,lcd
stdin=serial,usbkbd
stdout=serial,lcd
ubiargs=ubi.mtd=0 root=ubi0:rootfs rootfstype=ubifs
ubiboot=run setup; setenv bootargs ${defargs} ${ubiargs} ${mtdparts} ${setupargs};\
 echo Booting from NAND...; ubi part kernel-ubi && ubi read ${loadaddr} kernel; bootm
usbboot=echo Loading RAM disk and kernel from USB stick...; usb start && fatload\
 usb 0:1 0xC08000 rootfs-ext2.img.gz && fatload usb 0:1 ${loadaddr} uImage;run ramboot

Environment size: 2122/4092 bytes
Tegra2 #

Resetting the Environment to the Defaults

Tegra2 # env default -f -a
## Resetting to default environment
Tegra2 # saveenv

Creating a New or Changing an Existing Variable

Creating a new variable or changing an existing one is done using the setenv command. You must enclose the argument with ' ' if you don't want to expand variables or multiple statements are given. To make changes permanent they need to be stored in flash with saveenv:

Tegra2 # setenv defargs 'video=tegrafb vmalloc=248M usb_high_speed=1'
Tegra2 # saveenv
Saving Environment to NAND...
Erasing Nand...
Erasing at 0xe00000 -- 100% complete.
Writing to Nand... done

Booting Methods

U-Boot on eMMC/NAND Flash, Kernel and Rootfs on SD Card

Note: For Colibri T30 only U-Boots from June 2013 or newer are able to read from SD card. Note2: With our latest BSPs we recommend using regular MBR partitioned SD cards rather than GPT ones.

#create on your Linux PC a SD card with a GPT partition formatted for ext3
$ sudo parted /dev/sdX mklabel gpt
$ sudo parted /dev/sdX mkpart "colibri ext3 1 -1"
$ sudo mkfs.ext3 /dev/sdX1

#copy the rootfs & kernel to the SD card
$ sudo mkdir /mnt/sd
$ sudo mount /dev/sdX1 /mnt/sd
$ sudo cp -Ppr /path to image/rootfs/* /mnt/sd/
$ sudo cp /path to kernel/uImage /mnt/sd/boot/uImage
$ sudo umount /mnt/sd

#put the card into your Colibri T20 system, connect a terminal and boot to the U-Boot prompt
#change the U-Boot environment so that it loads kernel and rootfs from SD card.
#check the blockdevice name by booting from internal memory first. It may change between image versions. 
Tegra2 # setenv mmcargs 'root=/dev/mmcblk0p1 ip=:::::eth0:off rw,noatime rootfstype=ext3 rootwait'
Tegra2 # setenv mmcboot 'run setup; setenv bootargs ${defargs} ${mmcargs} ${mtdparts} ${setupargs} ${vidargs};\
 echo Loading kernel and rootfs from MMC/SD card...; mmc dev 0 && mmc part && ext2load mmc 0:1 ${loadaddr} /boot/uImage; bootm'
Tegra2 # saveenv
Tegra2 # run mmcboot

#put the card into your Apalis/Colibri T30 system, connect a terminal and boot to the U-Boot prompt (Below example uses 8-bit
#slot on Apalis T30, use 'mmc dev 2 && ext2load mmc 2:1 ${loadaddr} /boot/uImage' for 4-bit slot e.g. on Ixora)
#change the U-Boot environment so that it loads kernel and rootfs from MMC/SD card.
#check the blockdevice name by booting from internal memory first. It may change between image versions.
Apalis T30 # setenv sdargs 'ip=off root=/dev/mmcblk1p1 rw,noatime rootfstype=ext3 rootwait'
Apalis T30 # setenv sdboot 'run setup; setenv bootargs ${defargs} ${sdargs} ${setupargs} ${vidargs};\
 echo Loading kernel and rootfs from MMC/SD Card...; mmc dev 1 && ext2load mmc 1:1 ${loadaddr} /boot/uImage; bootm ${loadaddr}'
Apalis T30 # saveenv
Apalis T30 # run sdboot

Note: For versions prior to V2.1Beta3_20140318 one has to replace above 'mmc dev [dev]' command by 'mmc part [dev]' command as U-Boot changed in that respect. Note2: Our latest BSPs are prepared for SD card booting out-of-the-box and the environment modifications as outlined above should not be required (e.g. just enter 'run sdboot').

U-Boot on eMMC/NAND Flash, Kernel from TFTP and Rootfs from NFS

Install and configure a DHCP, TFTP and NFS server on your development workstation as explained here.

  • U-Boot obtains the IP address and the name of the file containing the kernel from the DHCP server.
  • U-Boot loads the kernel from the TFTP server with 'serverip' address.
  • U-Boot sets the kernel's commandline so that the kernel mounts the rootfs from a NFS server.

  • Linux uses DHCP again to get an IP address and the root-path arguments.
  • Linux tries to mount the NFS root file system using the root-path.

Prepare Kernel and rootfs

TFTP

Copy the kernel and if applicable the device-tree to the tftp directory. Make sure that the kernel file name matches with the filename field you specified in the DHCP configuration and the device tree name with the name configured in U-Boot.

sudo cp -Ppr /path to kernel/uImage* /srv/tftp/

NFS

Copy the rootfs to /srv/nfs/rootfs (while preserving symlinks, timestamps..)

sudo cp -Ppr /path to image/rootfs/* /srv/nfs/rootfs

Restart the Servers

After configuration changes and when changing the served NFS files the servers need to be restarted:
e.g. on Ubuntu: mk@ubuntu: ~ $ sudo service isc-dhcp-server restart; sudo service tftpd-hpa restart; sudo service nfs-kernel-server restart
e.g. on Fedora: [root@vm_one ~]# systemctl restart dhcpd.service; systemctl restart nfs-server.service

U-Boot Variables for TFTP/NFS Boot

See note about the MAC address (ethaddr) at the end of this article.

The module's IP address used for TFTP client, the IP address of the TFTP server, the MAC address:

ipaddr=192.168.10.2
serverip=192.168.10.1
ethaddr=00:14:2d:48:8a:58

The nfsrootdebug cmdline option enables debugging messages to debug mounting root file system via NFS:

defargs=$defargs nfsrootdebug

Booting

stop in U-Boot and then:

Tegra2 # run nfsboot

to make this the default boot option:

Tegra2 # setenv bootcmd 'run nfsboot'
Tegra2 # saveenv

NFS and connman

In our images we use connman to manage network connections.
When using an NFS mounted rootfs connman is not started during boot. This is because connman takes an already configured NIC down before bringing it up again. In the case of a NFS boot this unmounts the rootfs and makes the boot fail.
Should you require connman when using NFS, e.g. to do tests with wireless connectivity, you could change the connman.service file. Make sure to change eth0 to the name of the used NIC on Apalis T30. This is likely enp7s0.

--- lib/systemd/system/connman.service~ 2014-05-02 17:01:49.000000000 +0200
+++ lib/systemd/system/connman.service  2014-06-16 09:48:08.454798634 +0200
@@ -3,14 +3,15 @@
 After=syslog.target
 Before=remote-fs.target
 # only if not NFS mount, connman will disconnect your rootfs otherwise!
-ConditionKernelCommandLine=!root=/dev/nfs
+#ConditionKernelCommandLine=!root=/dev/nfs
 
 [Service]
 Type=dbus
 BusName=net.connman
 Restart=on-failure
 ExecStartPre=-/usr/lib/connman/wired-setup
-ExecStart=/usr/sbin/connmand -n
+#ExecStart=/usr/sbin/connmand -n
+ExecStart=/usr/sbin/connmand -n -I eth0
 StandardOutput=null
 
 [Install]

Various Variables

Default Boot Scenario

After a timeout U-Boot automatically executes the statements stored in the environment variable bootcmd. By default this runs flashboot (resp. emmcboot or ubiboot), should that fail a fall back to nfsboot is programmed. To make booting from SD card the default boot scenario do the following:

Tegra2 # printenv bootcmd
bootcmd=run flashboot; run nfsboot
Tegra2 # setenv bootcmd 'run mmcboot'
Tegra2 # saveenv

USB High Speed

To enable USB high speed have a look here: USB 2.0 High Speed (480Mbps)

Ethernet MAC Address

If no environment variable 'ethaddr' exists the Ethernet MAC address is read from the 'config block'. If you want to force an Ethernet MAC address add or change the ethaddr environment variable. Config block is a data structure shared with WinCE. It contains information written during manufacturing and configuration for the WinCE boot loader. See Config Block

Tegra2 # setenv ethaddr 00:01:02:03:04:05
Tegra2 # saveenv

eMMC Fast Boot Mode (Apalis/Colibri iMX6)

Not only does the eMMC fast boot mode allow for faster booting but it also proves more robust across various operating conditions.

Our Embedded Linux BSP V2.6 beta 1 release automatically migrates to eMMC fast boot mode when updating. Since U-Boot needs an updated U-Boot, images can not directly be downgraded anymore! Otherwise no further manual actions are needed. This section provides detailed about the eMMC fast boot mode in case custom migration/scripts are necessary.

In a not yet migrated module the fuse 0 5 has the value 00005062:

Apalis iMX6 # fuse sense 0 5
Sensing bank 0:

Word 0x00000005: 00005062

The eMMC fast boot mode can be enabled as a one-time-only fuse setting on the i.MX 6 SoC side and this mode requires an updated U-Boot to avoid a boot hang:

Apalis iMX6 # fuse prog 0 5 0x00005072
Programming bank 0 word 0x00000005 to 0x00005072...
Warning: Programming fuses is an irreversible operation!
This may brick your system.
Use this command only if you are sure of what you are doing!

Really perform this fuse programming? 

Please note that for an unattended update procedure one could avoid the confirmation question by adding an additional -y argument (e.g. fuse prog -y 0 5 0x00005072).

A few notes:

  • Programming the i.MX 6 SoC fuses is an irreversible operation!
  • Both the i.MX 6 SoC as well as the eMMC have to be configured to the exact same matching boot settings, otherwise booting will fail!
  • On the eMMC side one has to configure the hardware boot area partition settings as outlined below. So far none of our BSPs configures those settings!
  • The boot loader has to be programmed into the hardware boot area partition as automatically done since BSP V2.5 beta 2 and as outlined below. This implies that one can not easily downgrade to a BSP before V2.5 beta 2. But remember a yet unpublished U-Boot boot loader is anyway required!
  • To allow for seamless use of hardware/software system resets together with the eMMC fast boot mode one has to enable the eMMC hardware reset as well like outlined further below.

eMMC Hardware Boot Area Partitions (Apalis/Colibri iMX6 and Apalis/Colibri T30)

As part of our Embedded Linux BSPs V2.5 beta 2 (released on November 6, 2015) we unified the way all our modules with eMMC flashes are booting:

  • U-Boot now supports eMMC boot support commands (e.g. mmc bootbus and partconf).
  • U-Boot is now booted off the primary eMMC hardware boot area partition (e.g. /dev/mmcblk0boot0 in Linux).
  • The U-Boot environment is located at the end of the primary eMMC hardware boot area partition just before the config block (see below).
  • The Toradex factory configuration block (e.g. containing the serial number used for the MAC address) is located in the very last block thereof.
  • When updating from an older BSP version the config block is migrated to its new location and the relevant changes to the eMMC's configuration are done.

Boot Area Partitions Technical Background

Some pointers to further reading and how one can examine and alter the involved eMMC configuration from Linux or U-Boot.

A good overview about the thematic can be found in Micron’s TN-FC-06: Booting from Embedded MMC (e.MMC) - JEDEC v. 4.41.

Which supersedes TN-29-18.

Which in term supersedes TN-52-06.

As follows the relevant EXT_CSD registers:

BOOT_BUS_WIDTH Register (EXT_CSD [177]) = 0x16 (needed to set DDR mode and ACK at boot)
PARTITION_CONFIG Register (EXT_CSD [179]) = 0x48 (needed to enable booting from boot partition)

Boot Area Partitions Linux

Booting from boot partition disabled:

root@colibri-imx6:~# cat /sys/kernel/debug/mmc0/ios 
clock:          52000000 Hz
actual clock:   49500000 Hz
vdd:            21 (3.3 ~ 3.4 V)
bus mode:       2 (push-pull)
chip select:    0 (don't care)
power mode:     2 (on)
bus width:      3 (8 bits)
timing spec:    7 (sd uhs DDR50)
signal voltage: 0 (3.30 V)

root@colibri-imx6:~# cat /sys/devices/soc0/soc.0/2100000.aips-bus/2198000.usdhc/mmc_host/mmc0/mmc0\:0001/boot_info
[ 1502.828197] mmc0: BKOPS_EN bit is not set
boot_info:0x07;
  ALT_BOOT_MODE:1 - Supports alternate boot method
  DDR_BOOT_MODE:1 - Supports alternate dual data rate during boot
  HS_BOOTMODE:1 - Supports high speed timing during boot
boot_size:4096KB
boot_partition:0x00;
  BOOT_ACK:0 - No boot acknowledge sent
  BOOT_PARTITION-ENABLE: 0 - Device not boot enabled
boot_bus:0x00
  BOOT_MODE:0 - Use single data rate + backward compatible timings in boot operation
  RESET_BOOT_BUS_WIDTH:0 - Reset bus width to x1, single data rate and backwardcompatible timings after boot operation
  BOOT_BUS_WIDTH:0 - x1 (sdr) or x4 (ddr) bus width in boot operation mode

For i.MX6 based modules, migrating to booting off the primary hardware boot area partition incl. flashing a new U-Boot there:

root@colibri-imx6:~# echo 0 > /sys/block/mmcblk0boot0/force_ro
root@colibri-imx6:~# dd if=u-boot.imx of=/dev/mmcblk0boot0 bs=512 seek=2
[  222.394014]  mmcblk0boot0: unknown partition table
598+0 records in
598+0 records out
root@colibri-imx6:~# echo 1 > /sys/block/mmcblk0boot0/force_ro
root@colibri-imx6:~# echo 10 > /sys/devices/soc0/soc.0/2100000.aips-bus/2198000.usdhc/mmc_host/mmc0/mmc0\:0001/boot_bus_config
root@colibri-imx6:~# echo 8 > /sys/devices/soc0/soc.0/2100000.aips-bus/2198000.usdhc/mmc_host/mmc0/mmc0\:0001/boot_config

Booting from boot partition enabled:

root@colibri-imx6:~# cat /sys/devices/soc0/soc.0/2100000.aips-bus/2198000.usdhc/mmc_host/mmc0/mmc0\:0001/boot_info
[   76.145225] mmc0: BKOPS_EN bit is not set
boot_info:0x07;
  ALT_BOOT_MODE:1 - Supports alternate boot method
  DDR_BOOT_MODE:1 - Supports alternate dual data rate during boot
  HS_BOOTMODE:1 - Supports high speed timing during boot
boot_size:4096KB
boot_partition:0x48;
  BOOT_ACK:1 - Boot acknowledge sent during boot operation
  BOOT_PARTITION-ENABLE: 1 - Boot partition 1 enabled
boot_bus:0x0a
  BOOT_MODE:1 - Use single data rate + high speed timings in boot operation mode
  RESET_BOOT_BUS_WIDTH:0 - Reset bus width to x1, single data rate and backwardcompatible timings after boot operation
  BOOT_BUS_WIDTH:2 - x8 (sdr/ddr) bus width in boot operation mode

A few notes:

  • Changing boot_config takes effect immediately and does not require any power-cycle or the like.
  • Setting boot_partition back to 0x00 actually allows switching back to our current way of booting.
  • By default eMMC boot partitions are read-only and disabling via force_ro is only temporary and does not persist.

Boot Area Partitions U-Boot

Enable boot partition from within U-Boot (requires enabling CONFIG_SUPPORT_EMMC_BOOT first):

mmc bootbus dev boot_bus_width reset_boot_bus_width boot_mode
- Set the BOOT_BUS_WIDTH field of the specified device

Configure the eMMC to 8-bit, reset bus width settings to default after boot operation, single data rate and high-speed timings in boot operation mode:

Colibri iMX6 # mmc bootbus 0 2 0 1

mmc partconf dev boot_ack boot_partition partition_access
- Change the bits of the PARTITION_CONFIG field of the specified device

Enable booting from the (hardware) boot area partition 1 and the boot acknowledge to be sent during boot operation:

Colibri iMX6 # mmc partconf 0 1 1 0

Resetting to old behaviour:

Colibri iMX6 # mmc bootbus 0 0 0 0
Colibri iMX6 # mmc partconf 0 0 0 0

eMMC Hardware Reset (Apalis/Colibri iMX6)

By default the eMMC hardware reset signal RST_n is not enabled. Whether or not it should be permanently disabled or enabled can be configured one-time-only from within U-Boot:

mmc rst-function dev value
 - Change the RST_n_FUNCTION field of the specified device
   WARNING: This is a write-once field and 0 / 1 / 2 are the only valid values.

The following one-time-only setting permanently enables the eMMC hardware reset signal:

Colibri iMX6 # mmc rst-function 0 1