Roadmaps / Embedded Linux / The milestone project
Read first
The milestone project
Scenario. fleet-gw is a gateway image on 5,000 remote sites. It runs a small service, gwd, that reads one-line sensor readings from a TCP feed (a script on your host plays the sensor) and stores them. Customers sit in the EU, so the product is covered by the Cyber Resilience Act. You promise ten years of security fixes (the CRA minimum is five years from placing on the market, unless the expected use is shorter; Article 13(8)). Each site takes an update about once a quarter and has an unreliable supply: assume (state it as an assumption) that 1 update in 200 sees a power cut during the update window, so one quarterly wave of 5,000 sites meets about 25 cuts. The slot budget comes from your Week 1 measurement: each root slot is twice your release image, rounded up to 64 MiB, and the debug image must fit too; ADR-3 records what the second slot costs. Targets you measure and report: image size, boot to gwd ready, update time, and bundle size.
| Phase | Checkpoint | Done when |
|---|---|---|
| 1 | Reproducible image | One kas file builds the image; two builds, each with its own empty sstate and download cache (a shared cache proves little), give the same package list and versions, and the image hash too, or every difference is listed with its cause (for example wic UUIDs and timestamps; use buildhistory and diffoscope); a script boots the image in QEMU and a smoke test passes; a seeded unpinned layer (a branch name instead of a commit) is caught by a check script |
| 2 | Kernel and device tree | A kernel config fragment (not a full config) is built and you list which options you turned off and the size and boot-time saving; your out-of-tree driver for knaxus,gw-demo binds to a node you added to the device tree and logs its property; a seeded wrong compatible string leaves it unbound and your check script flags it |
| 3 | Safe updates | A signed RAUC bundle installs to the inactive slot and boots; 100 power cuts at random points leave 0 unbootable systems, and each trial ends on a working slot; a tampered bundle and a bundle signed by the wrong key are both rejected; a new image that boots but fails its health check rolls back by itself (see the health check below) |
| 4 | Release and compliance | The image ships an SBOM (SPDX 3) and a CVE report; every open CVE in gwd's own closure (its libraries and the packages you added) has a status and a reason, and the kernel CVEs are triaged by a written policy with a count by status; a seeded vulnerable library triggers a rebuild, a signed bundle and a staged rollout to five simulated units that halts when one unit fails its health check; a ten-year support timeline shows the dates your promise outlasts; on paper, with timestamps, the early-warning text is ready within 24 hours of the seeded CVE, the notification text within 72 hours, and the final report text within 14 days after the signed bundle exists (the CRA periods) |
Starter pack (Week 1). An unverified skeleton: check each field against the kas documentation for the release you install, and run it before you trust it. Build with kas build fleetgw.yml, then boot with the runqemu line in the meta-rauc-community README for qemuarm64. Expect several hours for the first build and about 60 to 100 GB of disk for one build.
header:
version: 14 # use the header version your kas release documents
machine: qemuarm64
distro: poky
target: fleetgw-image # your image recipe in meta-fleetgw
repos:
meta-fleetgw: { path: ../meta-fleetgw }
poky:
url: https://git.yoctoproject.org/poky
commit: <40-character hash from git ls-remote, branch wrynose>
layers: { meta, meta-poky }
meta-openembedded:
url: https://github.com/openembedded/meta-openembedded
commit: <hash>
layers: { meta-oe, meta-python, meta-networking }
meta-rauc: { url: https://github.com/rauc/meta-rauc, commit: <hash> }
meta-rauc-community:
url: https://github.com/rauc/meta-rauc-community
commit: <hash> # no wrynose branch (master declares Wrynose compatibility): pin a commit
layers: { meta-rauc-qemuarm }
local_conf_header:
fleetgw: |
INIT_MANAGER = "systemd"
DISTRO_FEATURES:append = " rauc"
PREFERRED_PROVIDER_virtual/bootloader = "u-boot"
# set the U-Boot environment provider as the meta-rauc-community README for qemuarm64 says
# add IMAGE_FEATURES += "read-only-rootfs" in Week 4, together with the data partition
gwd is any program of about 100 lines that reads lines from a TCP port and appends them to a file under /data. Health check. Mark good with the rauc-mark-good service the RAUC project ships (check your pinned layer), gated by a gw-health.service of your own: it checks that gwd answers within 30 seconds and calls reboot if not. The meta-rauc-community U-Boot script keeps BOOT_ORDER and BOOT_x_LEFT (boot attempts left); when the attempts run out, U-Boot boots the other slot. WatchdogSec= restarts a hung service but does not reboot the machine; to reboot use FailureAction=reboot on the health unit. QEMU has no hardware watchdog, so a hung kernel or init is untested: list it in ADR-7. Commands in this roadmap were not run end to end by its author.
Fault table. The faults you inject across the checkpoints.
| # | Fault | How to inject | What you should see |
|---|---|---|---|
| 1 | Unpinned layer | Change a kas branch to a name instead of a commit | The check script fails the build |
| 2 | Wrong device tree match | Change the compatible string in your node | The driver does not bind; the check flags it |
| 3 | Power cut during an update | Kill the QEMU process: half the trials at uniformly random times, half on a trigger line in the serial log (a script watches for it and kills at once for a short window, or after 0 to 2 seconds for a long stage). Read the log to count where each cut really landed; a window of tens of milliseconds may be missed, so report the trials that landed, and use fault 3b for it | Old or new slot boots; never neither |
| 3b | Storage-level cut (Senior) | Put the disk image behind dm-flakey or dm-log-writes on a Linux host (root needed: use a disposable VM) and drop or reorder writes after a cut | The same outcomes, or a failure the kill never showed |
| 4 | Tampered bundle | Flip one byte in the bundle | rauc install rejects it |
| 5 | Boots but broken | A build whose gwd exits at start | The health check fails; the system reverts to the old slot |
| 6 | Vulnerable library | Vendor an old release of a small library (for example an old zlib source tarball with a published CVE; record its ID) into the gwd recipe, so the SBOM names it | The CVE report lists it; the drill starts |
Evidence to hand in. A repository with the kas file, your layer, the check scripts and the trial logs; a README that opens with the scenario and its numbers; ADRs, each naming a rejected option: 1 Yocto against Buildroot, 2 kernel config approach, 3 slot layout and boot order, 4 update tool, 5 CVE and triage policy; Senior adds 6 key rotation and 7 the limits of the power-cut method; capture tables and raw files; a one-page design doc and one diagram of the boot and update path (bootloader, slots, data partition).
Level variants.
| Level | Adds to the core scope |
|---|---|
| Fresher | Core: faults 1 to 6 and one CVE drill |
| Senior | After Week 12, a second update tool (SWUpdate or Mender; about 8 hours: integrate it into the same A/B layout, build its bundle, repeat the same trials with the same trigger method) measured against RAUC on bundle size, install time, rollback behaviour and trial results; fault 3b (about 2 hours); ADRs 6 and 7 |
| Staff | A platform-policy memo (8 hours): one page for three product lines: the kernel and Yocto release policy, the update tool, the key and certificate lifecycle and the support window, each with a number, a cost and the check that would show it failing. Example line: "Kernel: follow the Yocto LTS kernel; cost: one rebuild and test cycle a month; check: days from a kernel CVE to a signed bundle" |
| Principal | A compliance review checklist (6 hours): 15 yes-or-no checks a team lead can run in 30 minutes on any product (SBOM per release, update signed, support period stated, reporting contact, key storage). Done when two engineers other than you run it on a product you built to fail five of the checks, both find at least 4 of the 5, and they differ on at most 3 of 15 answers (agreement alone is weak: two coin flips agree on about 7 of 15). A build-or-buy memo (8 hours): Yocto in-house, a vendor distribution or a managed update service, with costs for ten years and the exit route. Done when two readers can state the choice, its ten-year cost and the exit route after one read |
Do this at work (Checkpoint W, 24 hours). Pick one real product and run Checkpoints 3 and 4 on it with its real update path, on bench units or an emulator: randomised power cuts at the update stages, and a CVE drill from its real SBOM. Then apply the checklist (on the Staff path, a 10-check list you write in 2 hours) to at least three products. Senior: optional, as promotion evidence. Evidence a promotion committee can accept: the real decision (for example "ship SBOMs and a signed A/B update on the two products still on a single slot"), who made it, the named stakeholder (product security or legal) who approved it, its scope (products and sites), adoption (teams using the checklist) and the measured outcome (unrecoverable updates, time to ship a CVE fix), re-checked at 90 days with the date recorded. Pass bars: at least 100 cuts on real units with 0 unrecoverable, and ADR-7's untested classes named; the checklist adopted by at least three teams after 90 days; time from awareness of a seeded CVE to a signed bundle at or below the target you set and the committee accepted. Illustrative hand-in: "Two single-slot products moved to A/B; 120 cuts on 6 bench units, 0 unrecoverable; checklist used by 4 teams at day 90; CVE-to-bundle time 9 days against a 14-day target; approved by product security." A lab hand-in alone is not promotion evidence.
Solo version. Nothing needs a team or a lab. A script plays the sensor and picks the cut times, one peer or a forum thread scores the hand-in, and five QEMU instances stand in for the fleet. A product manager or legal contact is replaced by the regulation text and the Commission's guidance pages.
Laptop-scale note. Five instances stand in for 5,000 sites, so every fleet number is an extrapolation from your trials, labelled as one. A cloud build host helps with Week 1 only if you set a spend limit that really stops (a budget alert does not) and delete it afterwards.
Worked reference: what a "3 Proven" hand-in for Checkpoint 3 looks like
Illustrative numbers, not a recorded run; yours will differ, but the shape of the table and the arithmetic should match. A script cuts power with the trigger method (fault 3); the "landed" counts come from the log; 100 cuts in total, plus 5 no-cut runs of fault 5. A single cut during the first boot of the new slot costs one boot attempt, so the new slot boots again; the old slot returns only when the attempts run out (here 3).
| Stage where the power is cut | Trials | Old slot boots | New slot boots | Unbootable |
|---|---|---|---|---|
| Writing the inactive slot | 45 | 45 | 0 | 0 |
| Switching the boot order (the environment write; 20 landed) | 20 | 12 | 8 | 0 |
| First boot of the new slot, before "mark good" (attempts left 3 to 2) | 25 | 0 | 25 (boots again) | 0 |
| After "mark good" | 10 | 0 | 10 | 0 |
No cut: a build whose gwd exits (fault 5), three failed attempts (not counted in the 100) | 5 | 5 (fallback) | 0 | 0 |
Reading the table: 0 unbootable in 100 trials bounds the failure rate below about 3% at 95% confidence (the "rule of three": 3 divided by 100) for this stage mix only. Per stage the bound is looser: 0 of 20 in the boot-order window allows up to 15%. In the field the cut lands in proportion to time: if the environment write takes 20 ms of a 90 second update, about 1 cut in 4,500 lands in it, so one wave of about 25 cuts meets it about 0.006 times (illustrative). "Boot order last" is only atomic if the environment write is: use redundant environment copies (CONFIG_ENV_REDUND) and say how fw_setenv writes them. A run of 1,000 trials in the risky windows bounds them near 0.3%; measure one trial's time first (at 3 minutes a trial that is 50 hours unattended, or 10 hours on 5 parallel instances). ADR-3 excerpt: "Decision: A/B root slots, boot order changed last, redundant environment, one shared data partition. Rejected: an in-place update of one slot with a recovery partition. A cut during the write leaves no good root and recovery needs a second download over a bad link. Cost: the second slot takes flash equal to the image. Not exercised: torn writes, eMMC cache loss, wear (QEMU kill only). Check: the cut table above, repeated at 1,000 trials in the risky windows before release."
Shapes of a good hand-in for the other checkpoints (illustrative). Checkpoint 1: "Package lists identical (412 of 412 packages); image hash differs; cause: wic UUID and file timestamps; fixed with a fixed disk UUID and SOURCE_DATE_EPOCH; hashes now match." Checkpoint 2: "Option CONFIG_X off: image 3.1 MiB smaller, boot 0.4 s faster; seeded compatible = "knaxus,gw-demo1": the check prints no driver matched and exits 1." Checkpoint 4: "gwd closure: 9 packages, 2 open CVEs (the vendored zlib: affected, in use by gwd). Kernel: 640 reported; 410 not affected (option off), 190 patched, 40 open with a 14-day triage date. Clock: seeded CVE at T0; early-warning text at T0+3 h; notification text at T0+20 h; bundle signed at T0+26 h; final report text due 14 days after T0+26 h."