Decode · ARM

ARMv8 exception levels: EL3, EL2, EL1, EL0

[    0.013647] CPU: All CPU(s) started at EL1

ARMv8 defines four exception levels: privilege rings for the AArch64 execution state. From most-privileged to least:

  • EL3: Secure Monitor. The only level allowed to switch between Secure and Non-Secure worlds. Home of TF-A's BL31 (or Qualcomm's QSEE, or another vendor EL3 monitor). On boot, everything starts at EL3 and drops down through the levels; the EL3 monitor stays resident for the life of the system.
  • EL2: Hypervisor. Where KVM, Xen, and other hypervisors live. On a system without virtualization, EL2 is often bypassed but a stub of EL2 code is typically loaded so KVM can be enabled dynamically at runtime.
  • EL1: OS kernel. Where Linux (and most other kernels) run. If a hypervisor is present at EL2, Linux runs as a guest at "EL1 under EL2." Otherwise it runs at "bare EL1."
  • EL0: User applications. No privileged instructions. This is where every process runs after fork+exec.

The kernel line CPU: All CPU(s) started at EL1 (or at EL2) tells you where U-Boot handed off to Linux: which is a decision that gets made by U-Boot's armv8_switch_to_el1 function based on the board's boot flow.

Booting at EL2 is the modern default because it leaves the option for KVM (dynamically enabled later) or for a hypervisor to be loaded before the kernel takes over. Most current SBCs, dev boards, and desktop-class ARM64 boot at EL2.

Booting at EL1 happens when the vendor decided the platform doesn't need virtualization. Common on networking Snapdragon / IPQ (QSEE owns EL3 and doesn't ship an EL2 hypervisor stub) and on some tightly-locked-down consumer devices where the vendor didn't want to expose the KVM attack surface. Consequence: no in-guest virtualization is possible without replacing the firmware stack.

Debugging tip: if you're trying to enable KVM on an ARM64 board and kvm_arch_init refuses to load, check the boot log for the "CPU started at" line first. If it says EL1, no amount of Linux config will help: the platform firmware didn't give the kernel a hypervisor entry point.

References

Full walkthrough: /blog/qualcomm-sbl-secure-boot

Source log: /samples/bootintel-4.txt

More decoders: /decode · /faq

Have a boot log with something you can't decode? Paste it into the free fingerprinter. It identifies bootloader / kernel / SoC / autoboot exposure client-side, nothing uploaded.