Diffing Boot Logs to Catch Vendor-Introduced Regressions

Firmware updates ship security improvements roughly half the time. The other half, they ship security regressions — usually unintentional, occasionally explicit, almost always undocumented. The only way to know which one happened on your device is to compare what booted before against what boots now. Doing that well is harder than it looks.

The naive approach (and why it doesn't work)

First-pass instinct: capture before, capture after, run diff:

$ diff -u old.log new.log | head -100
- [    0.001234] init: starting v1.2.3
+ [    0.001891] init: starting v1.2.4
- [    1.456789] eth0: link up 1Gbps
+ [    1.512345] eth0: link up 1Gbps
- [    2.789012] dhcp: leased 192.168.1.42 from 192.168.1.1
+ [    2.812345] dhcp: leased 192.168.1.99 from 192.168.1.1
- [    3.123456] sshd: listening on 192.168.1.42:22
+ [    3.187654] sshd: listening on 192.168.1.99:22
+ [    3.234567] telnetd: listening on 0.0.0.0:23
[... 1400 more lines of timestamp + DHCP + MAC noise ...]

The signal in this hypothetical (telnetd re-enabled on every interface) is real and serious. It's also buried in noise that diff treats as equally important. Three classes of noise dominate:

• Boot-time timestamps — [ 1.456789] jitters by a few milliseconds every reboot.
• DHCP-assigned values — IP addresses, lease times, DNS server lists.
• Randomized identifiers — kernel ASLR base addresses, ephemeral port numbers, MAC randomization for Wi-Fi station mode.

Normalizing before diffing

The fix is to pass the log through a normalizer that scrubs the volatile bits before comparison. A reasonable starting set:

sed -E '
  s/^\[ *[0-9]+\.[0-9]+\]/[T]/;            # timestamps
  s/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/{IP}/g;  # IPv4
  s/[0-9a-fA-F:]{17}/{MAC}/g;                # MACs
  s/0x[0-9a-fA-F]{8,}/{ADDR}/g;              # kernel addrs
  s/\b[0-9]{5,}\b/{NUM}/g;                  # long numerics
' boot.log > boot.normalized

Apply this to both before and after logs and the resulting diff has 90%+ less noise. The free Boot Log Scrubber does this client-side with a maintained pattern set.

The regressions worth watching for

After normalizing, the diffs that actually matter fall into a handful of recurring patterns:

• New listening port appearing. Any service binding to0.0.0.0:<new-port> in the after-log that wasn't there before. Common culprits: vendor "remote support" daemons, debug shells re-enabled, UPnP toggled on, IPMI defaults reverted.
• Package version downgrade. Linux BusyBox v1.34 becomes BusyBox v1.30. Usually due to a botched build using a stale buildroot; the result is reopening every CVE fixed between those versions.
• Kernel command-line change. New init=, changed quiet/loglevel, or — worst case —init=/bin/sh reintroduced for "recovery convenience." The bootloader echoes the cmdline; the change shows up in the first 30 lines.
• Service silently re-enabled. telnetd, vsftpd, an old web admin port. Often re-added during "restoring legacy device compatibility" updates.
• U-Boot environment expanded. New variables that enable network boot, debug commands, or autoboot interruption. Each adds to the local attack surface.
• Module list change. A driver that wasn't loaded before is now loaded. Worth investigating; kernel modules expand syscall and ioctl surfaces.

What BootIntel does with this

Capturing two logs and diffing them by hand works exactly once. Doing it across a fleet of 50 devices on a monthly cadence does not. BootIntel stores every scan per device and presents diffs as a finding-shaped delta — not raw text:

• New findings introduced by the update (severity, evidence line, affected component).
• Resolved findings the update closed (so you can credit the vendor when they actually fix something).
• Unchanged findings still present, with provenance ("present since first scan" vs "reintroduced after firmware X.Y").

The diff view lives on each device's page in the dashboard. A single-device walkthrough of what surfaces: OpenWrt coverage.

The case for keeping every boot log

Even if you don't diff actively, retaining boot logs solves three problems on the day you need them:

1. When a new CVE drops, you can re-run detection against old logs to know which historical builds were vulnerable — critical for incident-response timelines.
2. When a vendor claims a fix, you have the receipts (before/after evidence) to validate the claim.
3. When auditors ask, "show me the security posture of this device on date X" — you have the artifact.

Default retention is 30 days on free, 90 days on Starter, and a year on Pro. For regulated industries the 1-year window is usually the constraint that makes Pro the right tier; pricing.

Related reading

• Reading OpenWrt Boot Logs for Security Issues — what the individual lines mean.
• U-Boot Security Misconfigurations — the bootloader half of the diff, where regressions hide most often.
• Capture UART Boot Logs in a Browser — reproducibly capture both the before and the after.
• Boot Log Scrubber — the normalizer to run before diffing.