Ongoing · Working paper No. 1 · v0.3 · Updated 2026-11-15
CI mesh. Two Airs and a laptop gate.
Karthik Kamalakannan · Started Oct 2026
Our CI runs on two 8 GB M1 MacBook Airs, CASE and TARS, plus team laptops that take jobs only when a gate says it is safe. Identical labels balance the load. A case label keeps deploys off laptops. p90 queue wait fell from 27 s to 4 s in a small early sample. We report the design, five failures and what is still open.[1][2]
Keywords: self-hosted runners, GitHub Actions, label routing, job slots, laptop mesh
1Introduction
Our CI started as one spare MacBook Air called CASE: a Linux VM and GitHub's self-hosted runners.[1] It worked. It was also a single point of failure with a keyboard attached.
So CASE got a twin, TARS. Then the team's laptops started helping when they are plugged in and idle.[2]
2Design
2.1 Topology
Each Air runs a Colima Linux VM with 6 GiB of 8 GB and Rosetta for amd64 images, two job slots, a watchdog that restarts a hung VM, and an hourly canary. Runner names differ. Labels are identical, so GitHub hands each job to whichever runner is idle. Three canaries dispatched at once: two on TARS, one on CASE.
case.2.2 Label scheme
self-hosted, linux, airlock matches any runner, laptops included. An org or lane label narrows it. case is carried only by the two office Airs. Anything that deploys, migrates or touches prod secrets requires case.
2.3 Slots
Each machine has two slots, claimed by a runner job hook. Light jobs take one. Heavy jobs, like image builds and rspec, take both. Waiting jobs go in priority order: FeatureOS main, FeatureOS PRs, SupportWire, internal repos.
2.4 Laptop gate
A MacBook Pro runs an OrbStack VM with 20 of 24 GB and five runners for the two monorepos. A gate runs every 60 seconds. Every check must pass for five stable minutes: kill switch and approved machines, AC power, VPN, network quality, GitHub reachable, load, free disk, and demand. A failed check stops idle runners only. A job that started finishes.
- okKill switch + approved
- failAC power
- okVPN
- okNetwork quality
- okGitHub reachable
- okLoad
- okFree disk
- okDemand
Gate: idle · new jobs: none · running job: finishes
3Results
p90 queue wait 27 s → 4 s
| Before | After | |
|---|---|---|
| Median wait | 3 s | 3 s |
| p90 wait | 27 s | 4 s |
| Worst wait | 9.5 min | 46 s |
The median was always fine. The tail moved. Load split after the change: 9 jobs on CASE, 9 on TARS, 7 on the laptop.
4Failure analysis
- 1Artifacts
Cause. 8.7 GB of image tarballs, 90% of Actions storage.
Fix. 1-day retention, no tarball when the image is published, org default 90 to 7 days.
- 2Disk guard
Cause. Cleanup measured the 20 GB system disk, not the 100 GB Docker disk.
Fix. Caches moved to the data disk, guard points at it. System disk 81% to 60%.
- 3FileVault
Cause. A remote reboot parked TARS at the unlock screen.
Fix. Remote reboots only via fdesetup authrestart.
- 4QEMU
Cause. One cache miss: a 59 minute build, other jobs stuck up to 53 minutes.
Fix. Rosetta via the Colima daemon, driver: docker. QEMU dropped everywhere.
- 5Slot waits
Cause. Our slot hook makes jobs wait where GitHub cannot see it.
Fix. Rule-based stuck-job detector every 5 minutes, Jev judges flagged jobs.
5Open problems
- Open Memory
Two 8 GB Airs are still memory bound. Heavy jobs run one at a time per machine.
- Open Power cuts
After a power cut, TARS still waits for a human.
- Next Routing around a full machine
Runners on a full machine should look busy to GitHub, so the job goes to the free one.
References
- [1]We turned a spare MacBook Air into our CI. Skcript blog, 2026-10-28.
- [2]We now have a CI mesh with our laptops. Skcript blog, 2026-11-12.